Display control device, display control method and display control program

Abstract

To provide technologies which display such that a positional relationship between an obstacle and a vehicle can be more intuitively grasped.SOLUTION: A display control device (6) is installed in a vehicle (V) in which an imaging unit (4) having a capturing area determined around the vehicle and a display device (5) displaying an image are mounted. The display control device displays a display image on the display unit, the display image being obtained by converting a captured image captured by the imaging unit into a bird' eye image viewed in a virtual viewpoint (PV) set above the vehicle. The display control device sets the virtual viewpoint in an orthogonal plane (Mb) that is a virtual plane orthogonal to a virtual line (Lb) connecting between an object (B) located in the capturing area and the vehicle, and sets a direction of line of view (LV) from the virtual viewpoint toward a wall plane at a projection plane (Mp) that is a virtual plane having a bottom plane (Mp1) corresponding to a road surface (Rs) below the vehicle and a wall plane (Mp2) raised to extend from the bottom plane around the bottom plane.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a display control device, a display control method, and a display control program. [Background technology]

[0002] Patent Document 1 discloses an overhead image generating device that aims to display the positional relationship between an obstacle and a vehicle so that the positional relationship can be intuitively understood. This device includes an image data acquisition unit, an obstacle information acquisition unit, an overhead image generating unit, and a display control unit. The image data acquisition unit acquires peripheral image data obtained by photographing the periphery of the vehicle. The obstacle information acquisition unit acquires position information of an obstacle detected in the periphery of the vehicle. The overhead image generating unit generates an overhead image by performing a viewpoint conversion process on the peripheral image acquired by the image data acquisition unit based on the position information of the obstacle acquired by the obstacle information acquisition unit so that the vehicle and the obstacle can be viewed from above. Specifically, the overhead image generating unit performs a viewpoint conversion process so that a virtual viewpoint is located on an extension line from the vehicle to the obstacle. The display control unit displays the overhead image generated by the overhead image generating unit on a display panel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6724821 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in the case of a tall obstacle such as a pole, it may be difficult to grasp the sense of distance between the vehicle and the obstacle in an overhead image in which a virtual viewpoint is positioned on an extension line from the vehicle to the obstacle. In this way, in the device disclosed in Patent Document 1, a situation may arise in which it is difficult to grasp the sense of distance between the vehicle and obstacles around it in the image.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, etc. That is, the present invention provides a technique for displaying, for example, a positional relationship between an obstacle and a vehicle so that the positional relationship can be more intuitively understood. [Means for solving the problem]

[0006] The display control device (6) is applied to a vehicle (V) equipped with an imaging unit (4) that captures the surrounding area and a display device (5) that displays the image, and is configured to cause the display device to display a display image obtained by converting the image captured by the imaging unit into a bird's-eye view image from a virtual viewpoint (PV) set above the vehicle. The display control device according to claim 1 comprises: a processor (61) capable of executing computer program instructions; a memory (62) connected to said processor and storing said computer program instructions; Equipped with The processor executes the computer program instructions to: The virtual viewpoint is set within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; The direction of the line of sight (LV) from the virtual viewpoint is set so as to face the wall surface on a projection plane (Mp), which is a virtual surface having a bottom surface (Mp1) corresponding to the road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the periphery of the bottom surface. The display control method according to claim 8 is a method executed by the display control device, The virtual viewpoint is set within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; The direction of the line of sight (LV) from the virtual viewpoint is set so as to face the wall surface on a projection plane (Mp), which is a virtual surface having a bottom surface (Mp1) corresponding to the road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the periphery of the bottom surface. The display control program according to claim 9 is a computer program executed by the display control device, The process executed by the display control device includes: A process of setting the virtual viewpoint within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; A process of setting a direction of a line of sight (LV) from the virtual viewpoint so as to face a wall surface (Mp2) in a projection plane (Mp), which is a virtual plane having a bottom surface (Mp1) corresponding to a road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the bottom surface; Includes.

[0007] In addition, in each section of the application documents, each element may be given a reference number in parentheses. However, such a reference number merely indicates an example of the correspondence between the element and the specific means described in the embodiment described later. Therefore, the present invention is not limited in any way by the description of the reference numbers above. [Brief description of the drawings]

[0008] [Figure 1] 1 is a plan view showing a schematic configuration of a vehicle equipped with an in-vehicle system including a display control device according to an embodiment of the present invention. [Diagram 2] 2 is a block diagram showing a schematic device configuration of the in-vehicle system shown in FIG. 1. [Diagram 3] 3 is a block diagram showing a schematic functional configuration of the display control device shown in FIG. 2. [Figure 4] 4 is a side view showing an example of an operation scene of the display control device shown in FIG. 3. [Diagram 5] 4 is a conceptual diagram showing an outline of the operation of the display control device shown in FIG. 3. [Figure 6] 4 is a conceptual diagram showing an outline of the operation of the display control device shown in FIG. 3. [Figure 7] 4 is a conceptual diagram showing an outline of the operation of the display control device shown in FIG. 3. [Figure 8]4 is a conceptual diagram showing an outline of the operation of the display control device shown in FIG. 3. [Figure 9] 4 is a flowchart showing an outline of the operation of the display control device shown in FIG. 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (Embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that various modified examples applicable to one embodiment may be hindered from understanding the embodiment if they are inserted in the middle of a series of explanations related to the embodiment. For this reason, the modified examples are described together after the series of explanations related to the embodiment.

[0010] (Overall system configuration) First, the overall configuration of the in-vehicle system 1 will be described with reference to FIG. 1 and FIG. 2. The in-vehicle system 1 is a so-called vehicle surroundings monitoring system, and is configured to be mounted on a vehicle V, such as an automobile traveling on a road, so that obstacles existing around the vehicle V can be monitored from inside a cabin V0 inside the vehicle V. The vehicle body V1 is formed in a box shape that covers the cabin V0 from the outside. In FIG. 1, the "front", "rear", "left", "right" and "up" of the vehicle V are defined as shown in the figure. These directional concepts are set based on the driver who is riding in the vehicle V traveling forward on a straight road on a horizontal ground. The vehicle width center line LC1 is an imaginary straight line that passes through the vehicle center point Pc and is parallel to the front-rear direction, i.e., the vehicle overall length direction. The vehicle length center line LC2 is an imaginary straight line that passes through the vehicle center point Pc and is parallel to the left-right direction, i.e., the vehicle width direction.

[0011] As shown in Fig. 1, a front bumper V2 and headlights V3 are attached to the front end of a vehicle body V1. A rear bumper V4 and brake lights V5 are attached to the rear end of the vehicle body V1. A left door mirror V6 is attached to the left end of the vehicle body V1. A right door mirror V7 is attached to the right end of the vehicle body V1. A dashboard V9 and a steering wheel V10 are provided in front of a driver's seat V8 inside the vehicle compartment V0.

[0012] 2, the in-vehicle system 1 includes a vehicle state sensor 2, an obstacle detection sensor 3, an imaging unit 4, a display device 5, and a display control device 6. Below, specific device configurations or functional configurations of each of these components constituting the in-vehicle system 1 will be described in order.

[0013] The vehicle state sensor 2 is provided to obtain information or signals corresponding to the driving state of the vehicle V and output the information or signals to the display control device 6. The "driving state" includes physical quantities related to the behavior of the vehicle, such as vehicle speed and angular velocity. The "driving state" also includes quantities related to the driving operation state, such as accelerator operation amount, brake operation amount, shift position, steering angle, etc. In other words, the vehicle state sensor 2 is a collective term for well-known sensors necessary for vehicle driving control, such as a shift position sensor, a vehicle speed sensor, a steering angle sensor, a yaw rate sensor, etc., for the sake of simplicity of illustration and explanation.

[0014] The obstacle detection sensor 3 is provided to detect obstacles around the vehicle V and output the detection result to the display control device 6. Specifically, the vehicle V is equipped with at least one type of well-known sensor capable of detecting obstacles around the vehicle V, such as an ultrasonic sensor, a radar sensor, a laser radar sensor, etc., as the obstacle detection sensor 3.

[0015] The imaging unit 4 is mounted on the vehicle V so as to capture an image of an obstacle or the like present in the capture area, with the periphery, i.e., a predetermined range in the vicinity of the vehicle V, being the capture area. Specifically, the imaging unit 4 is provided with a front camera 41, a rear camera 42, a left side camera 43, and a right side camera 44. The front camera 41 is an imaging device whose capture area is the front of the vehicle V, and is attached to the front end of the vehicle body V1. The rear camera 42 is an imaging device whose capture area is the rear of the vehicle V, and is attached to the rear end of the vehicle body V1. The left side camera 43 is an imaging device whose capture area is the left side of the vehicle V, and is attached to the left door mirror V6. The right side camera 44 is an imaging device whose capture area is the right side of the vehicle V, and is attached to the right door mirror V7.

[0016] The display device 5 is provided in the vehicle interior V0 so as to display various images visible to at least the occupants including the driver. Specifically, the display device 5 is mounted on the dashboard V9. Images displayed by the display device 5 include, for example, navigation images, second task content images such as a television screen during automatic driving, various information display images such as traffic congestion information, images captured by the imaging unit 4, images obtained by subjecting such captured images to various image processing, and the like. The display control device 6 is configured to execute a display control operation, i.e., a control operation for image display on the display device 5.

[0017] Specifically, the display control device 6 includes a processor 61 and a memory 62. The processor 61, which can execute computer program instructions, has a configuration as a CPU or MPU. CPU stands for Central Processing Unit. MPU stands for Micro Processing Unit. The memory 62 is connected to the processor 61. The memory 62 has a configuration as a non-volatile non-transient substantial storage medium that holds computer program instructions executed by the processor 61 and various data such as look-up tables required when executing the computer program instructions. Specifically, the memory 62 includes at least one of a ROM, a flash memory, a magnetic disk, and the like. ROM stands for Read Only Memory. The display control device 6 is configured so that the processor 61 reads out and executes computer program instructions from the memory 62, thereby executing various processes for display control operations.

[0018] (Display Control Device) FIG. 3 shows an example of a functional configuration realized by the display control device 6 shown in FIG. 2, when the processor 61 reads out and executes a display control program from the memory 62. FIG. 4 shows a state in which an object B, which is an obstacle, is present around, i.e., in the vicinity of, the vehicle V. The directional concepts of "front", "rear", "left", "up", and "down" shown in FIG. 4 are consistent with those in FIG. 1. FIGS. 5 to 8 are diagrams for explaining an example of a display control operation when a display image based on a captured image including the object B is displayed on the display device 5 in such a state. Hereinafter, a specific configuration of the display control device 6 according to this embodiment will be described with reference to FIGS. 1 to 8.

[0019] The display control device 6 is configured to cause the display device 5 to display a display image obtained by converting the image captured by the imaging unit 4 into a bird's-eye image from a line of sight LV from a virtual viewpoint PV set above the vehicle V. Specifically, as shown in Fig. 3, the display control device 6 has, as its functional configuration, an image data acquisition unit 601, a vehicle information acquisition unit 602, an obstacle information acquisition unit 603, and a display information generation unit 604.

[0020] The image data acquisition unit 601 acquires, i.e. receives, image data captured by the imaging unit 4 from the imaging unit 4 and holds the data for a predetermined period of time. The vehicle information acquisition unit 602 acquires the traveling state of the vehicle V from the vehicle state sensor 2. The obstacle information acquisition unit 603 acquires the detection state of an object B within the area captured by the imaging unit 4 from the obstacle detection sensor 3. The display information generation unit 604 generates a display image based on the information acquired by the image data acquisition unit 601, the vehicle information acquisition unit 602, and the obstacle information acquisition unit 603, and outputs the display image to the display device 5.

[0021] Specifically, the display information generating unit 604 has a line of sight setting unit 641, a viewpoint conversion processing unit 642, and a synthesis processing unit 643. The line of sight setting unit 641 is configured to set a line of sight LV, that is, to set a virtual viewpoint PV and a direction of the line of sight LV from the virtual viewpoint PV. The viewpoint conversion processing unit 642 is configured to convert an image captured by the imaging unit 4 into a bird's-eye image by the line of sight LV from the virtual viewpoint PV. The synthesis processing unit 643 is configured to generate display image data to be displayed on the display device 5 by superimposing, that is, synthesizing, various image display contents, etc. on the image converted by the viewpoint conversion processing unit 642.

[0022] (Operation overview) Below, an overview of the operation of the display control device 6 according to this embodiment, and an overview of the method and program executed by the display control device 6 will be described with reference to the drawings, along with the effects achieved by these. In the following description, the display control device 6 having the above configuration, the display control method which is a method executed by it, and the display control program which is a computer program executed by it may be collectively referred to simply as "this embodiment."

[0023] 4 to 8 show an example of the operation when an object B, which is a pole-shaped obstacle, is present to the right rear of the vehicle V. The object B has a predetermined height (for example, about 65 to 80 cm) from the road surface Rs. In FIGS. 5 to 8, the projection plane Mp is a virtual plane having a planar bottom surface Mp1 corresponding to the road surface Rs below the vehicle V and a wall surface Mp2 erected from the bottom surface Mp1 around the bottom surface Mp1. In this embodiment, the bottom surface Mp1 is formed in a rectangular shape having sides parallel to the vehicle width center line LC1 and the vehicle length center line LC2. The corners of the rectangular shape may be formed in an R shape having a predetermined radius of curvature. The wall surface Mp2 has a shape in which four flat surfaces surrounding the bottom surface Mp1 are connected in a ring shape, and is erected so as to expand obliquely outward from the outer edge of the bottom surface Mp1. The angle between the bottom surface Mp1 and the wall surface Mp2 is referred to as the erection angle θ1 of the wall surface Mp2. The erection angle θ1 is, for example, about 30 to 60°, and typically 45°. The connection portion between the bottom surface Mp1 and the wall surface Mp2 and the corner portion of the wall surface Mp2 may be formed in a curved shape. The wall surface Mp2 may be defined as a portion of the bottom surface Mp1 whose height from the horizontal surface portion is equal to or greater than a predetermined height, for example, a portion of the vehicle V that is equal to or greater than 0.2 m in actual size. In this manner, in this embodiment, the projection surface Mp is formed in a bathtub shape that opens upward.

[0024] First, as shown in FIGS. 5 and 6, the line-of-sight setting unit 641 calculates an object azimuth line segment Lb, which is a virtual line segment connecting the object B and the vehicle V. In the present embodiment, the object azimuth line segment Lb is a portion passing through the space between the object B and the vehicle V on a virtual straight line connecting the object center point Pb and the vehicle center point Pc, that is, a portion between the outer edge of the object B and the outer edge of the vehicle V. The object center point Pb is the center point of the object B existing in the imaging region in a plan view. "Plan view" means looking at an object with a line of sight parallel to the vertical direction and directed from top to bottom. That is, the object azimuth line segment Lb is a horizontal line segment orthogonal to the vertical direction.

[0025] Next, the line-of-sight setting unit 641 calculates an orthogonal plane Mb, which is a virtual plane orthogonal to the object azimuth line segment Lb. The orthogonal plane Mb is set to pass through an intermediate point Px, which is an internal division point of the object azimuth line segment Lb. The intermediate point Px is typically the midpoint. Therefore, the orthogonal plane Mb is typically a plane passing through the central position on the object azimuth line segment Lb. In the present embodiment, the orthogonal plane Mb is formed as a vertical plane orthogonal to the horizontal plane.

[0026] Subsequently, the line-of-sight setting unit 641 sets a virtual viewpoint PV within the orthogonal plane Mb. Also, as shown in FIGS. 6 and 7, the line-of-sight setting unit 641 sets the direction of the line of sight LV from the virtual viewpoint PV so as to face the first wall surface Mp21. The first wall surface Mp21 is the portion of the wall surface Mp2 that is closest to the object B. A portion adjacent to the first wall surface Mp21 and close to the object B in the circumferential direction of the wall surface Mp2 is referred to as a second wall surface Mp22. That is, as shown in FIG. 6, in the first wall surface Mp21 and the second wall surface Mp22 that are close to the object B and adjacent to each other, if the distance between the object B and the first wall surface Mp21 is D1 and the distance between the object B and the second wall surface Mp22 is D2, then D1 < D2. In the present embodiment, the line-of-sight setting unit 641 sets the line of sight LV in the direction where the first wall surface Mp21 close to the object B is located.

[0027] Further, the line of sight setting unit 641 sets a depression angle θ2 from the virtual viewpoint PV in the line of sight LV. The depression angle θ2 is an angle between the line of sight LV and a horizontal line LH passing through the virtual viewpoint PV. Here, FIG. 7 corresponds to a side cross-sectional view of the orthogonal plane Mb in FIG. 6. In this embodiment, the line of sight setting unit 641 sets the depression angle θ2 so that the line of sight LV is perpendicular to the first wall surface Mp21 in the side cross-sectional view of the orthogonal plane Mb as shown in FIG. 7. That is, θ2[°]≒90-θ1. Specifically, for example, when θ1=45°, θ2≒45°.

[0028] Furthermore, the line of sight setting unit 641 sets the line of sight LV so that a characteristic shape portion, which is at least a part of the vehicle V, is included in the display image, i.e., the field of view. The "characteristic shape portion" is a portion in the display image where it is easy to visually recognize which part of the vehicle V is displayed. Specifically, the characteristic shape portion includes the vehicle center point Pc, the outer edge of the vehicle body V1 close to the object B, the front bumper V2, the headlight V3, the rear bumper V4, the brake lamp V5, the left door mirror V6, the right door mirror V7, or the driver's seat V8. FIG. 8 conceptually illustrates an example of a display image in this specific example. In the display example of FIG. 8, an image display content corresponding to the brake lamp V5 is displayed as the first shape characteristic portion GV1 of the vehicle V. Also, an image display content corresponding to the rear bumper V4 is displayed as the second shape characteristic portion GV2.

[0029] The viewpoint conversion processing unit 642 generates bird's-eye image data by projecting the captured image data by the imaging unit 4 onto the projection plane Mp by viewpoint conversion processing using the virtual viewpoint PV and line of sight LV set by the line of sight setting unit 641. Then, the synthesis processing unit 643 generates display image data by superimposing, i.e., synthesizing, the host vehicle polygon, which is an image display content corresponding to the vehicle V, on the bird's-eye image data. Note that the contents of these viewpoint conversion processes and synthesis processes are similar to the publicly known or well-known technologies described in JP 2011-182059 A, JP 2019-153138 A, and JP 2021-118435 A, except for the position of the virtual viewpoint PV, the direction of the line of sight LV, and the shape of the projection plane Mp. Therefore, further details of these processes will not be described in this specification.

[0030] Fig. 9 is a flowchart showing an outline of the operation according to this embodiment. In Fig. 9, "S" is an abbreviation of "step." The processor 61 performs the display control process shown in Fig. 9 by reading out a display control program according to this embodiment from the memory 62 and executing it. In this display control process, the processor 61 executes the processes of steps 101 to 106 in order.

[0031] In step 101, the processor 61 acquires the detection state of the object B in the shooting area by the imaging unit 4, which may be an obstacle. In step 102, the processor 61 calculates the orthogonal plane Mb corresponding to the detected object B. In step 103, the processor 61 determines the direction of the line of sight LV. That is, as shown in FIG. 6, the processor 61 identifies the first wall surface Mp21 closest to the object B, and determines the direction of the virtual viewpoint PV and the line of sight LV in the planar view so as to face the first wall surface Mp21. In step 104, the processor 61 determines the depression angle θ2. In step 105, the processor 61 executes a viewpoint conversion process using the virtual viewpoint PV and the line of sight LV determined, i.e., set, in the above steps. In step 106, the processor 61 generates display image data by a synthesis process, and outputs the generated display image data to the display device 5.

[0032] As described above in detail, in this embodiment, the virtual viewpoint PV and the line of sight LV are set so that the sense of distance between the object B, which may be an obstacle, and the vehicle V can be easily grasped. Specifically, in this embodiment, the virtual viewpoint PV is set in an orthogonal plane Mb perpendicular to the object orientation line segment Lb, which is a virtual line segment connecting the object B and the vehicle V. In addition, in this embodiment, the direction of the line of sight LV is set so as to look diagonally down from the virtual viewpoint PV toward the wall surface Mp2. As a result, the tall object B, which may be an obstacle, appears three-dimensionally, making it easier to grasp the sense of distance from the vehicle V on the image. Therefore, according to this embodiment, it is possible to display the positional relationship between the obstacle and the vehicle V on the display device 5 so that the positional relationship can be more intuitively grasped.

[0033] In this embodiment, the line of sight LV is set in the direction in which the first wall surface Mp21 close to the object B is located. This makes it easier to grasp the sense of distance between the vehicle V and the object B in the image, as shown in Fig. 8. Note that, even if the line of sight LV is directed toward the second wall surface Mp22, it is possible to obtain a displayed image that makes it easier to grasp the sense of distance, but by directing the line of sight LV toward the first wall surface Mp21, it becomes even easier to intuitively grasp the sense of distance.

[0034] In this embodiment, the depression angle θ2 of the line of sight LV is set so that the line of sight LV is perpendicular to the first wall surface Mp21 in a side cross-sectional view by the orthogonal plane Mb. Specifically, for example, when θ1=45°, the line of sight LV is set to look down at an angle of about 45° with respect to the vertical line. This makes it possible to display a tall object B such as a pole well while suppressing the sense of incongruity felt by occupants such as drivers as much as possible. In the above specific example, if the depression angle θ2 is too small, the bending of the pole-shaped object B at the rising part of the first wall surface Mp21, i.e., the boundary part between the first wall surface Mp21 and the bottom surface Mp1, becomes noticeable. In addition, the object B appears thicker and shorter than it actually is. On the other hand, if the depression angle θ2 is too large, the object B appears thinner and longer than it actually is. In addition, the bending of the grid lines of the projection surface Mp becomes noticeable.

[0035] In this embodiment, the line of sight LV is set so that at least a portion of the characteristic shape of the vehicle V is included in the displayed image. This makes it easier for a driver or other occupant to visually recognize which part of the vehicle V or the host vehicle polygon is displayed together with the object B. Therefore, according to this embodiment, it is possible to display on the display device 5 so that the positional relationship between the obstacle and the vehicle V can be more intuitively grasped.

[0036] (Modification) The present invention is not limited to the above-mentioned embodiment and examples. Therefore, the above-mentioned embodiment and the like can be modified as appropriate. Representative modified examples will be described below. In the following description of the modified examples, differences from the above-mentioned embodiment and the like will be mainly described. In addition, the same reference numerals are used for parts that are the same or equivalent in the above-mentioned embodiment and the following modified examples. Therefore, in the following description of the modified examples, the description in the above-mentioned embodiment and the like may be used as appropriate for components having the same reference numerals as the above-mentioned embodiment and the like, unless there is a technical contradiction or special additional explanation.

[0037] The present invention is not limited to the specific device configuration shown in the above embodiment. For example, the application of the present invention is not limited to automobiles running on roads, but may also be applied to off-road vehicles, military vehicles, manned drones, etc. The obstacle detection sensor 3 may be omitted. That is, the obstacle information acquisition unit 603 may acquire detection information of the object B based on image data captured by the imaging unit 4. There is also no particular limitation on the type, number, mounting position, etc. of the cameras in the imaging unit 4. That is, for example, the front camera 41 may be disposed in the vehicle cabin V0. Furthermore, the display control device 6 may be disposed in an external computer provided outside the vehicle V and connected to the vehicle V by wireless communication.

[0038] The functional components shown in Fig. 3 are merely for the sake of convenience in order to simplify the explanation of one embodiment of the present invention. Therefore, the illustration in Fig. 3 does not mean that these functional components must be realized as some kind of module or hardware in the display control device 6.

[0039] The program according to the present invention, which enables the execution of various operations, procedures, or processes described in the above embodiments, can be downloaded or upgraded via V2X communication. V2X is an abbreviation for Vehicle to X. Alternatively, such a program can be downloaded or upgraded via a terminal device provided in a manufacturing plant, a maintenance plant, a dealer, or the like of the vehicle. Such a program may be stored on a memory card, an optical disk, a magnetic disk, or the like.

[0040] In this way, each of the above functional configurations and processes may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, each of the above functional configurations and processes may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, each of the above functional configurations and processes may be realized by one or more dedicated computers configured by combining a processor and a memory programmed to execute one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transient tangible storage medium as instructions executed by the computer. In other words, each of the above functional configurations and processes may be expressed as a computer program including a procedure for realizing it, or as a non-transient tangible storage medium storing the program.

[0041] The present invention is not limited to the specific operation mode shown in the above embodiment. That is, for example, in the above embodiment, the shape of the projection plane Mp was formed into a substantially rectangular bathtub shape in a plan view. For this reason, although the line of sight LV is perpendicular to the first wall surface Mp21 in a side cross-sectional view by the orthogonal plane Mb, it is not necessarily perpendicular to the first wall surface Mp21 in a three-dimensional manner. That is, the line of sight LV is not necessarily parallel to the normal direction of the first wall surface Mp21. However, the present invention is not limited to such an aspect. That is, the shape of the projection plane Mp may be a bowl-shaped or inverted dome-shaped shape as described in JP 2011-182059 A and JP 2021-118435 A. In this case, the line of sight LV is parallel to the normal direction of the first wall surface Mp21. Therefore, in this case, the extension line of the line of sight LV is the normal line of the first wall surface Mp21. In addition, there is no particular limitation on the relationship between the projection plane Mp and the vehicle center point Pc. That is, in the techniques described in the above publications, the center of the projection plane Mp and the vehicle center point Pc are coincident, but the present invention is not limited to such an embodiment. Therefore, the projection plane Mp may be set so that the vehicle center point Pc is offset from the center of the projection plane Mp. Specifically, for example, the vehicle center point Pc may be offset from the center of the projection plane Mp in a direction in which the target object, which is the most noteworthy object B, is present. In this case, the target object may be selected according to the traveling direction of the vehicle V, as described later. Specifically, for example, when the vehicle V is traveling forward, the target object is selected in the front or front side of the vehicle V, which is the shooting range of the front camera 41.

[0042] In the specific example shown in the above embodiment, the object orientation line segment Lb was a horizontal line segment overlapping with a virtual line connecting the object center point Pb and the vehicle center point Pc. Therefore, the orthogonal plane Mb was a vertical plane perpendicular to the horizontal plane. However, the present invention is not limited to such an aspect. That is, for example, the object orientation line segment Lb may be a line segment connecting the shortest distance between the vehicle V and the object B. Alternatively, the object orientation line segment Lb may be a line segment overlapping with a virtual line connecting the three-dimensional center of gravity of the vehicle V and the three-dimensional center of gravity of the object B. In this case, the object orientation line segment Lb may be inclined with respect to the horizontal plane depending on the height of the object B. Therefore, in this case, if the orthogonal plane Mb is to be a vertical plane, the orthogonal plane Mb may be inclined with respect to the object orientation line segment Lb. Alternatively, in this case, the orthogonal plane Mb may be set to be slightly inclined with respect to the vertical plane.

[0043] For example, if an object B present behind the vehicle V is displayed on the display device 5 from a viewpoint looking backward while the vehicle V is traveling forward, it may be difficult for a driver or other occupant to grasp the situation. In this way, if the line of sight LV faces a direction different from the traveling direction of the vehicle V, the displayed image may be unnatural to the driver or other occupant. Therefore, in the process of setting the line of sight LV and the virtual viewpoint PV, the traveling direction of the vehicle V may be taken into consideration. In other words, the line of sight LV, i.e., the virtual viewpoint PV may be set based on the traveling direction of the vehicle V. Specifically, for example, when the vehicle V is traveling forward, the virtual viewpoint PV is set forward of the vehicle center point Pc. On the other hand, when the vehicle V is traveling backward, the virtual viewpoint PV is set behind the vehicle center point Pc. This makes it possible to display a display image that satisfactorily suppresses the unnatural feeling of the driver or other occupant.

[0044] Needless to say, the elements constituting the above embodiment are not necessarily essential, except when expressly stated as essential or when clearly considered essential in principle. In addition, when the numbers, amounts, ranges, etc. of the components are mentioned, the present invention is not limited to the specific numerical values, except when expressly stated as essential or when clearly limited to a specific numerical value in principle. Similarly, when the shapes, directions, positional relationships, etc. of the components are mentioned, the present invention is not limited to the shapes, directions, positional relationships, etc., except when expressly stated as essential or when clearly limited to a specific shape, direction, positional relationship, etc. in principle. In addition, the modified examples are not limited to the above examples. In addition, multiple modified examples may be combined with each other. Furthermore, all or part of the above embodiment and all or part of the modified examples may be combined with each other.

[0045] (Perspective) As is apparent from the above description of the embodiments and modifications, the disclosure of this specification includes at least the following aspects. [Point 1-1] A display control device (6) is applied to a vehicle (V) equipped with an imaging unit (4) that captures the surroundings as an imaging region and a display device (5) that displays the image, and causes the display device to display a display image obtained by converting an image captured by the imaging unit into a bird's-eye view image from a virtual viewpoint (PV) set above the vehicle, a processor (61) capable of executing computer program instructions; a memory (62) connected to said processor and storing said computer program instructions; Equipped with The processor executes the computer program instructions to: The virtual viewpoint is set within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; A direction of a line of sight (LV) from the virtual viewpoint is set so as to face a wall surface (Mp2) in a projection plane (Mp), which is a virtual plane having a bottom surface (Mp1) corresponding to a road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the bottom surface. Display control device. [Point 1-2] The line of sight is set in a direction in which the wall surface close to the object is located. A display control device according to aspect 1-1. [Points 1-3] The line of sight is set based on a traveling direction of the vehicle. A display control device according to aspect 1-1 or 1-2. [Points 1-4] A depression angle (θ2) from the virtual viewpoint in the line of sight is set so as to be perpendicular to the wall surface. The display control device according to any one of Aspects 1-1 to 1-3. [Points 1-5] The line of sight is set so that at least a portion of the vehicle is included in the display image. The display control device according to any one of Aspects 1-1 to 1-4. [Points 1-6] At least a portion of the vehicle includes a vehicle center point (Pc), an outer edge of a vehicle body (V1) close to the object, a front bumper (V2), a headlight (V3), a rear bumper (V4), a brake lamp (V5), a door mirror (V6, V7), or a driver's seat (V8). A display control device according to aspects 1-5. [Point 1-7] The orthogonal plane is set so as to pass through a center position of the virtual line segment. The display control device according to any one of Aspects 1-1 to 1-6. [Point 2-1] A display control method is applied to a vehicle (V) equipped with an imaging unit (4) that captures the surroundings as an imaging region and a display device (5) that displays the image, and is executed by a display control device (6) that causes the display device to display a display image obtained by converting an image captured by the imaging unit into a bird's-eye view image from a virtual viewpoint (PV) set above the vehicle, The virtual viewpoint is set within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; A direction of a line of sight (LV) from the virtual viewpoint is set so as to face a wall surface (Mp2) in a projection plane (Mp), which is a virtual plane having a bottom surface (Mp1) corresponding to a road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the bottom surface. Display control method. [Point 2-2] The line of sight is set in a direction in which the wall surface close to the object is located. A display control method according to aspect 2-1. [Point 2-3] The line of sight is set based on a traveling direction of the vehicle. A display control method according to aspect 2-1 or 2-2. [Point 2-4] A depression angle (θ2) from the virtual viewpoint in the line of sight is set so as to be perpendicular to the wall surface. The display control method according to any one of Aspects 2-1 to 2-3. [Point 2-5] The line of sight is set so that at least a portion of the vehicle is included in the display image. The display control method according to any one of Aspects 2-1 to 2-4. [Point 2-6] At least a portion of the vehicle includes a vehicle center point (Pc), an outer edge of a vehicle body (V1) close to the object, a front bumper (V2), a headlight (V3), a rear bumper (V4), a brake lamp (V5), a door mirror (V6, V7), or a driver's seat (V8). A display control method according to aspect 2-5. [Point 2-7] The orthogonal plane is set so as to pass through a center position of the virtual line segment. The display control method according to any one of Aspects 2-1 to 2-6. [Point 3-1] A display control program is applied to a vehicle (V) equipped with an imaging unit (4) that captures the surroundings as an imaging region and a display device (5) that displays the image, the display device displaying a display image obtained by converting an image captured by the imaging unit into a bird's-eye view image from a virtual viewpoint (PV) set above the vehicle, the display control program being executed by a display control device (6), The process executed by a processor (61) provided in the display control device is A process of setting the virtual viewpoint within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; A process of setting a direction of a line of sight (LV) from the virtual viewpoint so as to face a wall surface (Mp2) in a projection plane (Mp), which is a virtual plane having a bottom surface (Mp1) corresponding to a road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the bottom surface; Including, Display control program. [Point 3-2] The line of sight is set in a direction in which the wall surface close to the object is located. A display control program according to aspect 3-1. [Point 3-3] The line of sight is set based on a traveling direction of the vehicle. A display control program according to aspect 3-1 or 3-2. [Point 3-4] A depression angle (θ2) from the virtual viewpoint in the line of sight is set so as to be perpendicular to the wall surface. The display control program according to any one of Aspects 3-1 to 3-3. [Point 3-5] The line of sight is set so that at least a portion of the vehicle is included in the display image. The display control program according to any one of Aspects 3-1 to 3-4. [Point 3-6] At least a portion of the vehicle includes a vehicle center point (Pc), an outer edge of a vehicle body (V1) close to the object, a front bumper (V2), a headlight (V3), a rear bumper (V4), a brake lamp (V5), a door mirror (V6, V7), or a driver's seat (V8). A display control program according to aspect 3-5. [Point 3-7] The orthogonal plane is set so as to pass through a center position of the virtual line segment. The display control program according to any one of Aspects 3-1 to 3-6. [Explanation of symbols]

[0046] 4. Imaging unit 5 Display device 6 Display Control Device 61 processors 62 Memory 603 Obstacle Information Acquisition Unit 604 Display information generation section 641 Line of sight setting section 642 Viewpoint conversion processing unit 643 Synthesis Processing Unit

Claims

1. A display control device (6) is applied to a vehicle (V) equipped with an imaging unit (4) that captures the surroundings as an imaging region and a display device (5) that displays the image, and causes the display device to display a display image obtained by converting an image captured by the imaging unit into a bird's-eye view image from a virtual viewpoint (PV) set above the vehicle, A processor (61) capable of executing computer program instructions; a memory (62) connected to said processor and storing said computer program instructions; Equipped with The processor executes the computer program instructions to: The virtual viewpoint is set within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; A direction of a line of sight (LV) from the virtual viewpoint is set so as to face a wall surface (Mp2) in a projection plane (Mp), which is a virtual plane having a bottom surface (Mp1) corresponding to a road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the bottom surface. Display control device.

2. The line of sight is set in a direction in which the wall surface close to the object is located. The display control device according to claim 1 .

3. The line of sight is set based on a traveling direction of the vehicle. The display control device according to claim 1 .

4. A depression angle (θ2) from the virtual viewpoint in the line of sight is set so as to be perpendicular to the wall surface. The display control device according to claim 1 .

5. The line of sight is set so that at least a portion of the vehicle is included in the display image. The display control device according to claim 1 .

6. At least a portion of the vehicle includes a vehicle center point (Pc), an outer edge of a vehicle body (V1) close to the object, a front bumper (V2), a headlight (V3), a rear bumper (V4), a brake lamp (V5), a door mirror (V6, V7), or a driver's seat (V8). The display control device according to claim 5 .

7. The orthogonal plane is set so as to pass through a center position of the virtual line segment. The display control device according to claim 1 .

8. A display control method is applied to a vehicle (V) equipped with an imaging unit (4) that captures the surroundings as an imaging region and a display device (5) that displays the image, and is executed by a display control device (6) that displays on the display device a display image obtained by converting an image captured by the imaging unit into a bird's-eye view image from a virtual viewpoint (PV) set above the vehicle, The virtual viewpoint is set within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; A direction of a line of sight (LV) from the virtual viewpoint is set so as to face a wall surface (Mp2) in a projection plane (Mp), which is a virtual plane having a bottom surface (Mp1) corresponding to a road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the bottom surface. Display control method.

9. A display control program is applied to a vehicle (V) equipped with an imaging unit (4) that captures the surroundings as an imaging region and a display device (5) that displays the image, and is executed by a display control device (6) that displays, on the display device, a display image obtained by converting an image captured by the imaging unit into a bird's-eye view image from a virtual viewpoint (VP) set above the vehicle, The process executed by the display control device includes: A process of setting the virtual viewpoint within an orthogonal plane (Mb) that is a virtual plane perpendicular to a virtual line segment (Lb) connecting an object (B) present in the shooting area and the vehicle; A process of setting a direction of a line of sight (VL) from the virtual viewpoint so as to face a wall surface (Mp2) in a projection plane (Mp), which is a virtual plane having a bottom surface (Mp1) corresponding to a road surface (Rs) below the vehicle and a wall surface (Mp2) erected from the bottom surface around the bottom surface; Including, Display control program.

Images