Vehicle surrounding environment display device, control method for vehicle surrounding environment display device, and program

Abstract

To prevent a user from losing a positional sense of a virtual viewpoint based on a vehicle icon, while securing a high degree of freedom for the user to operate the virtual viewpoint.SOLUTION: A vehicle surrounding environment display device generates a virtual space corresponding to a surrounding environment of a vehicle on the basis of detection information of an external sensor of the vehicle, and displays, on a display, an image in a virtual space viewed from a virtual viewpoint operated by a user of the vehicle. The virtual space includes a vehicle icon corresponding to the vehicle. The vehicle surrounding environment display device includes an image display unit for controlling the virtual viewpoint such that, as the operating distance of the virtual viewpoint operated by the user becomes longer, the virtual viewpoint is more likely to be close to the vehicle icon, or, as the total operation time of the virtual viewpoint operated by the user becomes longer, the virtual viewpoint is more likely to be close to the vehicle icon.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a vehicle surrounding environment display device, a control method for the vehicle surrounding environment display device, and a program.

Background Art

[0002] Conventionally, as a technical document related to a vehicle surrounding environment display device, Japanese Unexamined Patent Application Publication No. 2020-088697 is known. This publication discloses a peripheral monitoring device that generates a virtual space including an image of the host vehicle and projects the surrounding environment of the vehicle as a three-dimensional video into the virtual space. A user can freely view the environment around the vehicle by operating a virtual viewpoint in the virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the above-described conventional device, the image of the host vehicle may disappear while the user is freely operating the virtual viewpoint in the virtual space. If the user continues to operate the virtual viewpoint in a state where the image of the host vehicle has disappeared, there is a risk of losing the sense of position of the virtual viewpoint based on the image of the host vehicle.

Means for Solving the Problems

[0005] One aspect of the present invention is a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the host vehicle based on detection information of an external sensor of the host vehicle and displays an image in the virtual space as seen from a virtual viewpoint operated by a user of the host vehicle on a display. In the virtual space, a host vehicle icon corresponding to the host vehicle is arranged, and the virtual viewpoint is controlled so that the virtual viewpoint is more likely to approach the host vehicle icon as the operation distance of the virtual viewpoint by the user becomes longer, or the virtual viewpoint is controlled so that the virtual viewpoint is more likely to approach the host vehicle icon as the total operation time of the virtual viewpoint by the user becomes longer. The image display unit is provided.

[0006] According to the vehicle surrounding environment display device according to one aspect of the present invention, by controlling the virtual viewpoint so that the virtual viewpoint is more likely to approach the host vehicle icon as the operation distance or the total operation time of the virtual viewpoint by the user becomes longer, even if the host vehicle icon is out of sight in the image seen from the virtual viewpoint, the user can easily find the host vehicle icon by the operation. Therefore, it is possible to suppress the user from losing the sense of position of the virtual viewpoint based on the host vehicle icon while ensuring a high degree of freedom in operating the virtual viewpoint by the user.

[0007] In the vehicle surrounding environment display device according to one aspect of the present invention, the image display unit may bring the fixation point, which is the center of the rotational movement of the virtual viewpoint in the virtual space, closer to the host vehicle icon as the operation distance of the virtual viewpoint by the user becomes longer. Further, the image display unit may bring the fixation point of the virtual viewpoint closer to the host vehicle icon as the total operation time of the virtual viewpoint by the user becomes longer. According to this vehicle surrounding environment display device, by bringing the fixation point of the virtual viewpoint closer to the host vehicle icon as the operation distance or the total operation time of the virtual viewpoint becomes longer, the possibility that the host vehicle icon appears in the image seen from the virtual viewpoint by the user's operation increases. Therefore, it is possible to suppress the user from losing the sense of position of the virtual viewpoint based on the host vehicle icon while ensuring a high degree of freedom in operating the virtual viewpoint by the user.

[0008] In a vehicle surrounding environment display device according to an aspect of the present invention, an image display unit sets a non-gaze point control area that surrounds the host vehicle icon in a virtual space, and the closer the operation distance of the virtual viewpoint by the user while the gaze point is located outside the non-gaze point control area is, or the longer the total operation time of the virtual viewpoint by the user while the gaze point is located outside the non-gaze point control area is, the closer the gaze point of the virtual viewpoint is to the host vehicle icon, or the closer the total operation time of the virtual viewpoint by the user while the gaze point is located outside the non-gaze point control area is, the closer the gaze point of the virtual viewpoint is to the host vehicle icon. Gaze point control is performed, and when the gaze point is located within the non-gaze point control area, gaze point control may not be performed. According to this vehicle surrounding environment display device, since gaze point control is not executed while the gaze point is located within the non-gaze point control area set surrounding the host vehicle icon, it is possible to suppress the deterioration of the user's operability by performing gaze point control until the gaze point is located near the host vehicle icon.

[0009] In a vehicle surrounding environment display device according to an aspect of the present invention, an image display unit shortens the distance between the gaze point and the virtual viewpoint as the operation distance of the virtual viewpoint by the user increases, or shortens the distance between the gaze point and the virtual viewpoint as the total operation time of the virtual viewpoint by the user increases. According to this vehicle surrounding environment display device, by shortening the distance between the gaze point and the virtual viewpoint as the operation distance or the total operation time of the virtual viewpoint increases, the virtual viewpoint also approaches the host vehicle icon as the gaze point approaches the host vehicle icon of the vehicle. Therefore, it is possible to suppress the user from losing the positional sense of the virtual viewpoint based on the host vehicle icon while ensuring a high degree of freedom in operating the virtual viewpoint by the user.

[0010] In a vehicle surrounding environment display device according to an aspect of the present invention, an image display unit decreases the depression angle of the virtual viewpoint as the operation distance of the virtual viewpoint by the user increases, or decreases the depression angle of the virtual viewpoint as the total operation time of the virtual viewpoint by the user increases. According to this vehicle surrounding environment display device, by reducing the depression angle of the virtual viewpoint with respect to the fixation point as the operation distance or the total operation time of the virtual viewpoint increases, it is possible to suppress the user from losing the positional sense of the virtual viewpoint based on the own vehicle icon while ensuring a high degree of freedom in operating the virtual viewpoint by the user.

[0011] Another aspect of the present invention is a control method for a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the own vehicle based on detection information of an external sensor of the own vehicle and displays an image in the virtual space viewed from a virtual viewpoint operated by a user of the own vehicle on a display. In the virtual space, an own vehicle icon corresponding to the own vehicle is arranged, and the virtual viewpoint is controlled so that the virtual viewpoint is more likely to approach the own vehicle icon as the operation distance of the virtual viewpoint by the user increases, or the virtual viewpoint is controlled so that the virtual viewpoint is more likely to approach the own vehicle icon as the total operation time of the virtual viewpoint by the user increases.

[0012] According to the control method for a vehicle surrounding environment display device according to another aspect of the present invention, by controlling the virtual viewpoint so that the virtual viewpoint is more likely to approach the own vehicle icon as the operation distance or the total operation time of the virtual viewpoint by the user increases, even if the own vehicle icon disappears from the image viewed from the virtual viewpoint, the user can easily find the own vehicle icon by the operation. Therefore, it is possible to suppress the user from losing the positional sense of the virtual viewpoint based on the own vehicle icon while ensuring a high degree of freedom in operating the virtual viewpoint by the user.

[0013] Still another aspect of the present invention is a program for operating the ECU of a host vehicle as a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the host vehicle based on detection information of external sensors of the host vehicle and displays an image in the virtual space as viewed from a virtual viewpoint operated by a user of the host vehicle on a display. The program arranges a host vehicle icon corresponding to the host vehicle in the virtual space and controls the virtual viewpoint so that the virtual viewpoint is more likely to approach the host vehicle icon as the operation distance of the virtual viewpoint by the user becomes longer, or controls the virtual viewpoint so that the virtual viewpoint is more likely to approach the host vehicle icon as the total operation time of the virtual viewpoint by the user becomes longer.

[0014] According to the program according to still another aspect of the present invention, by controlling the virtual viewpoint so that the virtual viewpoint is more likely to approach the host vehicle icon as the operation distance or the total operation time of the virtual viewpoint by the user becomes longer, even if the host vehicle icon is out of sight in the image viewed from the virtual viewpoint, the user can easily find the host vehicle icon by the operation, so that while ensuring a high degree of freedom of operation of the virtual viewpoint by the user, it is possible to suppress the user from losing the sense of position of the virtual viewpoint based on the host vehicle icon.

Advantages of the Invention

[0015] According to each aspect of the present invention, it is possible to suppress the user from losing the sense of position of the virtual viewpoint based on the host vehicle icon while ensuring a high degree of freedom of operation of the virtual viewpoint by the user.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Mode for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0018] FIG. 1 is a block diagram showing a vehicle surrounding environment display device 100 according to an embodiment. The vehicle surrounding environment display device 100 shown in FIG. 1 is mounted on a vehicle such as a passenger car or a freight car (hereinafter referred to as the own vehicle), and is a device for supporting a user's recognition of the surrounding environment of the vehicle. The vehicle surrounding environment display device 100 generates a virtual space reflecting the surrounding environment of the own vehicle, and displays an image in the virtual space seen from a virtual viewpoint operated by the user on a display. The vehicle surrounding environment display device 100 displays the surrounding environment of the own vehicle on the display as a so-called 3D view.

[0019] The user may be a driver of the own vehicle, a passenger of the own vehicle, or an owner of the own vehicle. The user may be an operator who performs remote support for the own vehicle by a remote support system. In the remote support system, an operator can make a determination of the running of the own vehicle (determination of progress, turning right or left, stopping, etc.) or perform a driving operation of the own vehicle through remote support equipment provided outside the vehicle and communicable with the own vehicle. The own vehicle is not limited to a vehicle that can be remotely supported by a remote support system. The own vehicle may be a vehicle having an automatic driving function or a vehicle not having an automatic driving function.

[0020] [Configuration of Vehicle Surroundings Display Device] As shown in FIG. 1, the vehicle surroundings display device 100 includes an ECU [Electronic Control Unit] 10 that comprehensively manages the device. The ECU 10 is an electronic control unit having a CPU [Central Processing Unit] and a storage unit. The storage unit is composed of, for example, ROM [ReadOnly Memory], RAM [Random Access Memory], EEPROM [Electrically Erasable Programmable Read-Only Memory], etc. In the ECU 10, various functions are realized, for example, by the CPU executing a program stored in the storage unit. The ECU 10 may be composed of a plurality of electronic units. The ECU 10 is connected to an external camera 1 (external sensor), a radar sensor 2 (external sensor), a user operation reception unit 3, and a display 4.

[0021] The external camera 1 is an imaging device that images the external situation of the host vehicle. The external camera 1 includes, for example, a front camera that images the front of the host vehicle, a back camera that images the rear of the host vehicle, and a side camera that images the left and right sides of the host vehicle. The number of cameras of the external camera 1 is not particularly limited and may be one. The external camera 1 transmits imaging image information to the ECU 10.

[0022] The radar sensor 2 is a detection device that detects objects around the host vehicle using radio waves (e.g., millimeter waves) or light. The radar sensor 2 can be configured to include a millimeter wave radar or a lidar [LIDAR: Light Detection and Ranging]. The radar sensor 2 transmits object detection information regarding the detected object to the ECU 10. Note that the radar sensor 2 and the external camera 1 constitute external sensors for detecting the surrounding environment of the host vehicle. The object detection information of the radar sensor 2 or the imaging image information of the external camera 1 corresponds to the detection information of the external sensor.

[0023] The user operation reception unit 3 is a device that receives operations of the virtual viewpoint by the user. The user operation reception unit 3 can be, for example, an input unit of an HMI [Human Machine Interface] provided in the host vehicle. The input unit includes, for example, a touch panel display, buttons, levers, switches, etc. Further, the user operation reception unit 3 may be able to receive operations by voice recognition or gestures.

[0024] As the user operation reception unit 3, input devices of a mobile terminal or a computer communicatively connected to the host vehicle may be used. Further, an operator terminal of a remote support system may be used as the user operation reception unit 3.

[0025] The display 4 is, for example, a center display mounted on the dashboard of the host vehicle. The display 4 may be a display of a tablet-type computer installable in the host vehicle, or may be a HUD [Head Up Display]. The display 4 is not limited to being mounted on the host vehicle. The display 4 may be an operator display of a remote support system provided in a facility away from the host vehicle. The display 4 may be a display of a mobile terminal carried by the user, or may be a display of the user's tablet-type computer or desktop computer.

[0026] Next, the functional configuration of the ECU 10 will be described. As shown in FIG. 1, the ECU 10 has a virtual space generation unit 11 and an image display unit 12. Some of the functions of the ECU 10 described below may be implemented in a mode executed by a server communicable with the host vehicle (for example, a server of a remote support system), a mobile terminal, or a computer (for example, a tablet-type computer or a desktop computer).

[0027] The virtual space generation unit 11 generates a virtual space corresponding to the surrounding environment of the host vehicle based on, for example, the imaging image information of the external camera 1. The surrounding environment of the host vehicle includes, for example, the position of the white line of the lane in which the host vehicle travels. The surrounding environment of the host vehicle may include the situation (position, traveling direction, etc.) of other vehicles such as the preceding vehicle and the adjacent parallel vehicles.

[0028] The virtual space is generated, for example, as a 3D video synthesized from a plurality of images. The image synthesis method is not particularly limited. The virtual space generation unit 11 generates a virtual space as a 3D video by, for example, projecting each image onto a global coordinate system that serves as a reference for the virtual space and associating the overlapping pixels with each other.

[0029] The virtual space generation unit 11 arranges a host vehicle icon corresponding to the host vehicle in the virtual space. The host vehicle icon is arranged as a three-dimensional icon. The host vehicle icon can be formed by polygon, voxel, or other CG processing. The virtual space generation unit 11 may generate a host vehicle icon that reflects the state of the host vehicle. The virtual space generation unit 11 may reflect the lighting state of the lighting devices of the host vehicle (lighting states of headlights, turn signals, brake lights, etc.) in the lighting state of the lighting devices in the host vehicle icon, and may also reflect the steering angle of the tires of the host vehicle in the tires of the host vehicle icon.

[0030] When the virtual space generation unit 11 recognizes other vehicles based on the imaging image information of the external camera 1, it may arrange a three-dimensional other vehicle icon corresponding to the other vehicle in the virtual space. Similarly, the virtual space generation unit 11 may arrange a three-dimensional pedestrian icon in the virtual space. The virtual space generation unit 11 may recognize other vehicles, etc. based on the object detection information of the radar sensor 2 instead of the imaging image information of the external camera 1, or may recognize other vehicles, etc. using both the external camera 1 and the radar sensor 2.

[0031] Note that the virtual space generation unit 11 may recognize other vehicles and the like around the host vehicle by using the information on the surrounding environment recognized by other vehicles through vehicle-to-vehicle communication. The virtual space generation unit 11 may, for example, communicate with a traffic information management server managed by a country to obtain image information of cameras installed on the road and various traffic information, and recognize other vehicles and the like by using these.

[0032] In addition, the virtual space generation unit 11 may predict the behavior of other vehicles based on the imaging image information of the external camera 1 or the object detection information of the radar sensor 2, and may display the prediction result of the behavior of other vehicles in association with the other vehicle icon. The virtual space generation unit 11 may, for example, display the predicted travel route of other vehicles by an arrow icon or the like, and may display the predicted stop position of other vehicles during deceleration by a block-shaped icon extending in the lane width direction or the like. Similarly, the virtual space generation unit 11 may display the prediction result of the behavior of pedestrians in association with the pedestrian icon.

[0033] The method for generating the virtual space is not limited to the method of synthesizing a plurality of images of the external camera 1. The virtual space generation unit 11 does not need to generate the virtual space as a 3D video as long as the surrounding environment of the host vehicle can be recognized by the user. The virtual space generation unit 11 may generate the virtual space by arranging the host vehicle icon, the white line, and the other vehicle icon so that the positional relationship between the white line and other vehicles with respect to the host vehicle can be understood.

[0034] The image display unit 12 displays an image in the virtual space viewed from the virtual viewpoint operated by the user in the virtual space generated by the virtual space generation unit 11 on the display 4. The image display unit 12 moves the virtual viewpoint 50 in response to the user operation input to the user operation reception unit 3.

[0035] FIG. 2 is a diagram for explaining the own-vehicle icon and the virtual viewpoint. FIG. 2 shows the own-vehicle icon M and the virtual viewpoint 50. The plane on which the own-vehicle icon M is arranged corresponds to the horizontal plane of the global coordinate system. FIG. 2 shows the line of sight D of the virtual viewpoint 50, the fixation point C of the virtual viewpoint 50, and the depression angle α of the virtual viewpoint 50. In FIG. 2, the virtual viewpoint 50 is illustrated as the icon of a camera for easy understanding. The depression angle α is the angle formed between the plane on which the own-vehicle icon M is arranged in the vertical plane or the horizontal plane of the global coordinate system and the line of sight D. Note that it is not necessary to use the virtual viewpoint 50 as an icon.

[0036] The fixation point C of the virtual viewpoint 50 is the point that becomes the center of the rotational movement of the virtual viewpoint 50 in the virtual space. When the user performs an operation of horizontally rotating the virtual viewpoint 50, the virtual viewpoint 50 moves so as to draw an arc centered on the fixation point C as viewed from the up-and-down direction (the vertical direction of the global coordinate system) of the own-vehicle icon M. Here, FIG. 3(a) is a diagram showing an example of the rotational movement of the virtual viewpoint. FIG. 3(a) is a view seen from above the own-vehicle icon M. FIG. 3(a) shows the locus Ra of the virtual viewpoint 50 that moves so as to draw a circle with the fixation point C as the center of rotation. Also, the straight-line distance between the fixation point C and the virtual viewpoint 50 is shown as the viewpoint distance L. In FIG. 3(a), the viewpoint distance L corresponds to the radius of rotation of the circular locus Ra.

[0037] Similarly, the virtual viewpoint 50 can also perform a vertical rotation centered on the fixation point C. When the user performs an operation of vertically rotating the virtual viewpoint 50, the virtual viewpoint 50 moves so as to draw an arc centered on the fixation point C as viewed from the front-and-back direction (the horizontal direction of the global coordinate system) of the own-vehicle icon M in FIG. 2. Note that, in addition to the rotational movement, the virtual viewpoint 50 may be capable of moving back and forth along the line of sight D (zooming in and out of the image) and parallel movement. When no intervention control is being performed, the virtual viewpoint 50 can be moved to an arbitrary position within the virtual space and can face an arbitrary direction by the operation of the user.

[0038] The virtual viewpoint 50 is set to an initial position. The initial position is, for example, a position from above the rear of the own-vehicle icon M looking down at the own-vehicle icon M. The initial position is not particularly limited as long as the own-vehicle icon M is displayed at the position on the display 4. The initial position may be changeable by the user.

[0039] The image display unit 12 controls the virtual viewpoint 50 so that the virtual viewpoint 50 is more likely to approach the own-vehicle icon M as the operation distance of the virtual viewpoint 50 by the user becomes longer. The operation distance is the distance that the virtual viewpoint 50 has moved within the virtual space by the user's operation. The image display unit 12 may reset the count of the operation distance when the state where no user operation is input continues for a certain period of time.

[0040] Specifically, the image display unit 12 indirectly controls the virtual viewpoint 50 to be more likely to approach the own-vehicle icon M by performing a fixation point control (intervention control) that brings the fixation point C of the virtual viewpoint 50 closer to the own-vehicle icon M as the operation distance of the virtual viewpoint 50 becomes longer. The fixation point control is not performed while the user has stopped the operation.

[0041] In the fixation point control, the degree of approach of the fixation point C according to the operation distance (the ease of approaching the own-vehicle icon M) may be a fixed value or a variable value. Specifically, the image display unit 12 may be configured such that while the operation distance increases by a certain distance, the fixation point C approaches the own-vehicle icon M by a predetermined distance. The image display unit 12 may be configured such that while the operation distance increases by a certain distance, the fixation point C approaches the own-vehicle icon M by a predetermined ratio in the entire separation distance between the fixation point C and the own-vehicle icon M.

[0042] Alternatively, the image display unit 12 may be configured such that the distance by which the fixation point C approaches the host vehicle icon M increases accelerationally as the operation distance increases. That is, the image display unit 12 makes it easier for the virtual viewpoint 50 to approach the host vehicle icon M accelerationally as the operation distance increases. The image display unit 12 may vary the distance by which the fixation point C approaches the host vehicle icon M according to the operation distance by using table data or arithmetic expressions in which the operation distance and the distance by which the fixation point C approaches the host vehicle icon M are associated in advance. Instead of the distance, the ratio in the entire separation distance between the fixation point C and the host vehicle icon M may be used. The degree of approach of the fixation point C can be set in a manner that causes less discomfort to the user. The degree of approach of the fixation point C may be adjustable by the user.

[0043] Further, the image display unit 12 may control the viewpoint distance L (radius of rotation) between the virtual viewpoint 50 and the fixation point C in fixation point control. Specifically, the image display unit 12 shortens the viewpoint distance L as the operation distance of the virtual viewpoint 50 increases in fixation point control. As a result, as the operation distance of the virtual viewpoint 50 increases, the virtual viewpoint 50 and the fixation point C approach each other, and the host vehicle icon M is more likely to be reflected in the image viewed from the virtual viewpoint 50 in combination with the approach of the fixation point C to the host vehicle icon M.

[0044] Furthermore, the image display unit 12 may control the depression angle α of the virtual viewpoint 50 in fixation point control. Specifically, the image display unit 12 decreases the depression angle α of the virtual viewpoint 50 as the operation distance of the virtual viewpoint 50 by the user increases. As a result, as the operation distance of the virtual viewpoint 50 increases, the position of the virtual viewpoint 50 becomes lower, so that the host vehicle icon M is more likely to be reflected in the image viewed from the virtual viewpoint 50 compared to the case where the virtual viewpoint 50 looks down from a high position.

[0045] Here, FIG. 3(b) is a diagram showing an example of the rotational movement of the virtual viewpoint 50 when gaze point control is performed. In FIG. 3(b), the trajectory Rb of the rotational movement of the virtual viewpoint 50 when gaze point control is performed is shown. Also, the position 50a of the virtual viewpoint 50 after the rotational movement and the position Ca of the gaze point C after the movement are indicated by dashed lines. In FIG. 3(b), it is assumed that the user inputs only the operation of rotational movement in the clockwise direction.

[0046] In FIG. 3(b), the image display unit 12 controls so that the gaze point C approaches the own vehicle icon M in conjunction with the rotational movement of the virtual viewpoint 50. Also, the image display unit 12 controls so that the viewing point distance L between the virtual viewpoint 50 and the gaze point C becomes shorter in conjunction with the rotational movement of the virtual viewpoint 50. Further, the image display unit 12 controls so that the depression angle α of the virtual viewpoint 50 becomes smaller in conjunction with the rotational movement of the virtual viewpoint 50. As a result, the virtual viewpoint 50 makes a rotational movement while gradually approaching the own vehicle icon M as shown by the trajectory Rb in FIG. 3(b), and the depression angle α also becomes smaller, so that the own vehicle icon M is reflected in the image viewed from the virtual viewpoint 50.

[0047] The user can input an operation to freely move the virtual viewpoint 50 even during gaze point control. On the other hand, since the gaze point C, which is the center of rotation of the virtual viewpoint 50, can be brought closer to the own vehicle icon M, the own vehicle icon M is more likely to be reflected in the image in the virtual space viewed from the virtual viewpoint 50. In other words, it becomes less likely that the own vehicle icon M is cut off from the image displayed on the display 4.

[0048] Note that the image display unit 12 may enable the user to move the virtual viewpoint 50 together with the gaze point C during gaze point control. In this case, both the user's operation and the gaze point control are reflected in the movement of the gaze point C. When the user continues an operation to move the gaze point C away from the own vehicle icon M for a certain period of time or more, the gaze point C may be moved giving priority to the user's operation. Also in this case, since the gaze point C is located on the own vehicle icon M side when viewed from the virtual viewpoint 50, the own vehicle icon M is reflected in the image viewed from the virtual viewpoint 50.

[0049] The image display unit 12 may use the total operation time of the virtual viewpoint 50 by the user instead of the operation distance of the virtual viewpoint 50 by the user. The total operation time is the total value of the time during which the virtual viewpoint 50 moves within the virtual space due to the user's operation. The total operation time is not counted while the user stops operating the virtual viewpoint 50. When the state where no user operation is input continues for a certain period of time, the image display unit 12 may reset the counting of the total operation time.

[0050] In this case, the image display unit 12 controls the virtual viewpoint 50 so that the virtual viewpoint 50 is more likely to approach the own vehicle icon M as the total operation time of the virtual viewpoint 50 by the user becomes longer. The image display unit 12 performs fixation point control (intervention control) to bring the fixation point C closer to the own vehicle icon M as the total operation time of the virtual viewpoint 50 becomes longer. The image display unit 12 may shorten the viewpoint distance L as the total operation time of the virtual viewpoint 50 becomes longer in the fixation point control. The image display unit 12 may decrease the depression angle α as the total operation time of the virtual viewpoint 50 becomes longer in the fixation point control.

[0051] Next, the determination of the unnecessary fixation point control will be described. The image display unit 12 may be in a mode where the fixation point control is not executed when a predetermined condition is satisfied. Specifically, the image display unit 12 may set a fixation point control unnecessary area where the fixation point control is not executed.

[0052] The fixation point control unnecessary area is an area where the fixation point control is not performed when the fixation point C is included in the area. The fixation point control unnecessary area is set to include the own vehicle icon M within the virtual space. The fixation point control unnecessary area may be set as a rectangular area when viewed from directly above the own vehicle icon M centered on the own vehicle icon M, or may be set as an area within a certain distance from the own vehicle icon M. The position of the own vehicle icon M within the fixation point control unnecessary area is not particularly limited.

[0053] When the image display unit 12 sets a non-gaze point control area, it does not perform gaze point control while the gaze point C is located within the non-gaze point control area. While the image display unit 12 does not perform gaze point control (while the gaze point C is located within the non-gaze point control area), it is not necessary to count the operation distance of the virtual viewpoint 50 by the user.

[0054] While the gaze point C is located outside the non-gaze point control area, the image display unit 12 starts counting the operation distance of the virtual viewpoint 50 by the user. When the gaze point C is located outside the non-gaze point control area, the image display unit 12 executes gaze point control to bring the gaze point C closer to the own vehicle icon M as the operation distance of the virtual viewpoint 50 by the user becomes longer. Note that when the gaze point C enters the non-gaze point control area during the execution of gaze point control, the image display unit 12 resets or temporarily stops counting the operation distance. The same applies when using the total operation time instead of the operation distance for gaze point control.

[0055] Here, FIG. 4 is a diagram showing an example of the rotational movement of the virtual viewpoint when the gaze point C is located within the non-gaze point control area. In FIG. 4, the user inputs only a clockwise rotational movement operation. FIG. 4 shows the non-gaze point control area W, the trajectory Rc of the virtual viewpoint 50, the first position 50b and the second position 50c of the rotationally moved virtual viewpoint 50. As shown in FIG. 4, when the gaze point C is located within the non-gaze point control area W, gaze point control is not performed, so the virtual viewpoint 50 rotates along the trajectory Rc that draws a circle around the non-moving gaze point C.

[0056] FIG. 5 is a diagram showing an example of gaze point control when the gaze point C is located outside the non-gaze point control area W. Also in FIG. 5, the user inputs only a clockwise rotational movement operation. FIG. 5 shows the trajectory Rd of the virtual viewpoint 50, the third position 50d and the fourth position 50e of the rotationally moved virtual viewpoint 50. Also shown are the position Cd of the gaze point C corresponding to the third position 50d of the virtual viewpoint 50 and the position Ce of the gaze point C corresponding to the fourth position 50e of the virtual viewpoint 50.

[0057] As shown in FIG. 5, when the fixation point C of the virtual viewpoint 50 deviates from the fixation point control unnecessary area W, fixation point control is executed. The fixation point C is controlled to gradually approach the own vehicle icon M in conjunction with the rotational movement of the virtual viewpoint 50. The fixation point control shown in FIG. 5 also includes control of the viewpoint distance L. The fixation point control may include control of the depression angle α.

[0058] Note that the image display unit 12 may determine whether or not to execute fixation point control according to the position of the virtual viewpoint 50 instead of the position of the fixation point C. The image display unit 12 may execute fixation point control when the virtual viewpoint 50 is located outside the fixation point control unnecessary area W, and may not execute fixation point control when the virtual viewpoint 50 is located inside the fixation point control unnecessary area W.

[0059] Alternatively, the image display unit 12 may determine whether or not to execute fixation point control using both the position of the virtual viewpoint 50 and the position of the fixation point C. The image display unit 12 may execute fixation point control of the virtual viewpoint 50 when either the virtual viewpoint 50 or the fixation point C is located outside the fixation point control unnecessary area W.

[0060] The image display unit 12 does not necessarily need to use the fixation point control unnecessary area W as a determination of the necessity of fixation point control. The image display unit 12 may, for example, not execute fixation point control while the distance between the fixation point C and the own vehicle icon M is less than a first distance threshold. The image display unit 12 may not execute fixation point control while the distance between the virtual viewpoint 50 and the own vehicle icon M is less than a second distance threshold. Each threshold is a threshold with a preset value.

[0061] Subsequently, the automatic control of the virtual viewpoint 50 will be described. The image display unit 12 may automatically control the position and orientation of the virtual viewpoint 50 so that the own vehicle icon M is reflected on the screen when a predetermined condition is satisfied.

[0062] Here, FIG. 6 is a diagram for explaining the automatic control intervention area. FIG. 6 shows the automatic control intervention area H, the fifth position 50f, the sixth position 50g, and the seventh position 50h of the virtual viewpoint 50. The automatic control intervention area H is, for example, an area within a certain distance from the own vehicle icon M. The automatic control intervention area H may be set as a rectangular area similar to the non-gaze point control area W.

[0063] In FIG. 6, it is assumed that the virtual viewpoint 50 has moved from the initial position to the fifth position 50f by the user's operation. The fifth position 50f is a position outside the automatic control intervention area H. In this case, the image display unit 12 executes automatic control of the virtual viewpoint 50. Specifically, the image display unit 12 automatically moves the virtual viewpoint 50 to the sixth position 50g. At the sixth position 50g, the virtual viewpoint 50 is moved so that the virtual viewpoint 50 faces the own vehicle icon M. Thereafter, the image display unit 12 moves the virtual viewpoint 50 to the seventh position 50h so that the own vehicle icon M seen from the virtual viewpoint 50 expands. The seventh position 50h is a position where the virtual viewpoint 50 enters the automatic control intervention area H.

[0064] As shown in FIG. 6, when the virtual viewpoint 50 is separated from the own vehicle icon M by a certain distance or more, the image display unit 12 can make the user notice the position of the own vehicle icon M by automatically controlling the position and orientation of the virtual viewpoint 50. The image display unit 12 may be in a mode of automatically moving the virtual viewpoint 50 to the initial position.

[0065] FIG. 7 is a diagram for explaining the automatic control intervention area H as seen from the back of the own vehicle icon M. As shown in FIG. 7, the automatic control intervention area H and the non-gaze point control area W can be set as three-dimensional areas with determined heights. Note that the automatic control intervention area H and the non-gaze point control area W may be areas set with respect to the horizontal plane (areas having no height). In this case, it is determined whether or not to execute automatic control based on whether or not the position of the virtual viewpoint 50 projected onto the horizontal plane is included in the automatic control intervention area H.

[0066] Instead of the position of the virtual viewpoint 50, the image display unit 12 may determine whether to execute automatic control according to the position of the fixation point C. Even if the virtual viewpoint 50 is located outside the automatic control intervention area H, when the fixation point C of the virtual viewpoint 50 is located within the automatic control intervention area H, the automatic control of the virtual viewpoint 50 is not executed. Even if the virtual viewpoint 50 is located within the automatic control intervention area H, when the fixation point C is located outside the automatic control intervention area H, the image display unit 12 executes the automatic control of the virtual viewpoint 50 because there is a possibility that the user has lost sight of the own vehicle icon M.

[0067] Alternatively, the image display unit 12 may determine whether to execute automatic control using both the position of the virtual viewpoint 50 and the position of the fixation point C. The image display unit 12 may execute the automatic control of the virtual viewpoint 50 when either one of the virtual viewpoint 50 or the fixation point C is located outside the automatic control intervention area H.

[0068] In addition to the distances between the virtual viewpoint 50 and the fixation point C and the own vehicle icon M, the image display unit 12 may use the fact that the own vehicle icon M is missing from the image viewed from the virtual viewpoint 50 as a condition for executing automatic control. Even if the virtual viewpoint 50 is far away from the own vehicle icon M but the own vehicle icon M is shown in the image, it can be considered that the sense of position of the virtual viewpoint 50 with respect to the own vehicle icon M is not lost. Therefore, even if the virtual viewpoint 50 or the fixation point C is located outside the automatic control intervention area H, when the own vehicle icon M is shown in the image, the image display unit 12 does not necessarily execute the automatic control of the virtual viewpoint 50, giving priority to the user's operation.

[0069] [Program] The program causes the ECU 10 to function as the virtual space generation unit 11 and the image display unit 12 described above. The program is provided by a non-temporary recording medium such as a ROM or a semiconductor memory, for example. The program may also be provided via communication such as a network.

[0070] [Control Method of Vehicle Surrounding Environment Display Device] Next, a control method for the vehicle surrounding environment display device 100 according to the present embodiment will be described with reference to the drawings. FIG. 8 is a flowchart showing an example of the control method for the vehicle surrounding environment display device 100 according to the present embodiment.

[0071] As shown in FIG. 8, the ECU 10 of the vehicle surrounding environment display device 100 determines, as S10, whether or not the virtual viewpoint 50 is located within the automatic control intervention area H by the image display unit 12. When the ECU 10 determines that the virtual viewpoint 50 is not located within the automatic control intervention area H (S10: NO), it proceeds to S11. When the ECU 10 determines that the virtual viewpoint 50 is located within the automatic control intervention area H (S10: YES), it proceeds to S12.

[0072] In S11, the ECU 10 executes automatic control of the virtual viewpoint 50 by the image display unit 12. The image display unit 12 automatically moves the virtual viewpoint 50, for example, to a position where the own vehicle icon M is reflected on the screen of the display 4. Thereafter, the ECU 10 ends the current control.

[0073] In S12, the ECU 10 determines whether or not the fixation point C is located outside the fixation point control unnecessary area W by the image display unit 12. When the ECU 10 determines that the fixation point C is not located outside the fixation point control unnecessary area W (S12: NO), it ends the current control. When the ECU 10 determines that the fixation point C is located outside the fixation point control unnecessary area W (S12: YES), it proceeds to S13.

[0074] In S13, the ECU 10 executes fixation point control according to the operation distance or the total operation time of the virtual viewpoint 50 by the user by the image display unit 12. The image display unit 12 executes fixation point control to bring the fixation point C of the virtual viewpoint 50 closer to the own vehicle icon M as the operation distance of the virtual viewpoint 50 becomes longer, for example. The image display unit 12 may control the viewpoint distance L and the depression angle α of the virtual viewpoint 50 in the fixation point control. Thereafter, the ECU 10 ends the current control.

[0075] Note that the vehicle surrounding environment display device 100 does not necessarily need to perform the determination in S10. In this case, the process in S11 is unnecessary. Also, the vehicle surrounding environment display device 100 does not necessarily need to perform the determination in S12, and may always execute the fixation point control.

[0076] According to the vehicle surrounding environment display device 100 according to the present embodiment described above, by controlling the virtual viewpoint 50 so that the virtual viewpoint 50 is more likely to approach the host vehicle icon M as the operation distance or the total operation time of the virtual viewpoint 50 by the user becomes longer, even if the host vehicle icon M disappears from the image viewed from the virtual viewpoint 50, the user can easily find the host vehicle icon M by the operation. Therefore, it is possible to suppress the user from losing the sense of position of the virtual viewpoint 50 while ensuring a high degree of freedom of operation of the virtual viewpoint 50 by the user.

[0077] Also, according to the vehicle surrounding environment display device 100, by bringing the fixation point C of the virtual viewpoint 50 closer to the host vehicle icon M as the operation distance or the total operation time of the virtual viewpoint 50 becomes longer, the possibility that the host vehicle icon M appears in the image viewed from the virtual viewpoint 50 by the user's operation increases. Therefore, it is possible to suppress the user from losing the sense of position of the virtual viewpoint 50 while ensuring a high degree of freedom of operation of the virtual viewpoint 50 by the user.

[0078] Furthermore, according to the vehicle surrounding environment display device 100, since the fixation point control is not executed while the fixation point C is located within the fixation point control unnecessary area W set surrounding the host vehicle icon M, it is possible to suppress the deterioration of the user's operability by performing the fixation point control until the fixation point C is located in the vicinity of the host vehicle icon M.

[0079] Further, according to the vehicle surrounding environment display device 100, as the operation distance or the total operation time of the virtual viewpoint 50 becomes longer, the viewpoint distance L, which is the distance between the fixation point C and the virtual viewpoint 50, is shortened, so that the virtual viewpoint 50 also approaches the own-vehicle icon M as the fixation point C approaches the own-vehicle icon M of the vehicle. Therefore, it is possible to suppress the user from losing the sense of position of the virtual viewpoint 50 while ensuring a high degree of freedom of operation of the virtual viewpoint 50 by the user.

[0080] Further, according to the vehicle surrounding environment display device 100, by reducing the depression angle α of the virtual viewpoint 50 with respect to the fixation point C as the operation distance or the total operation time of the virtual viewpoint 50 becomes longer, it is possible to suppress the user from losing the sense of position of the virtual viewpoint 50 while ensuring a high degree of freedom of operation of the virtual viewpoint 50 by the user.

[0081] As described above, the embodiments of the present invention have been described. However, the present invention is not limited to the above-described embodiments. The present invention can be implemented in various forms in which various changes and improvements are made based on the knowledge of those skilled in the art, including the above-described embodiments.

[0082] The vehicle surrounding environment display device 100 may be configured to directly control the virtual viewpoint 50 so that the virtual viewpoint 50 is more likely to approach the own-vehicle icon M, instead of the above-described fixation point control. The vehicle surrounding environment display device 100 may perform intervention control so that the position of the virtual viewpoint 50 approaches the own-vehicle icon M as the operation distance or the total operation time of the virtual viewpoint 50 by the user becomes longer, without using the fixation point C.

[0083] The vehicle surrounding environment display device 100 may control the virtual viewpoint 50 to be more likely to approach the own-vehicle icon M by increasing the resistance of the operation to move the virtual viewpoint 50 in the direction away from the own-vehicle icon M and decreasing or setting the resistance of the operation to move the virtual viewpoint 50 in the direction approaching the own-vehicle icon M to zero as the operation distance or the total operation time of the virtual viewpoint 50 by the user becomes longer.

Description of Reference Numerals

[0084] 1... External camera, 2... Radar sensor, 3... User operation reception unit, 4... Display, 10... ECU, 11... Virtual space generation unit, 12... Image display unit, 50... Virtual viewpoint, 100... Vehicle surrounding environment display device.

Claims

1. A vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the host vehicle based on detection information from external sensors of the host vehicle and displays an image in the virtual space as viewed from a virtual viewpoint operated by a user of the host vehicle on a display, in which a host vehicle icon corresponding to the host vehicle is arranged in the virtual space, and an image display unit that controls the virtual viewpoint so that the virtual viewpoint is more likely to approach the host vehicle icon as the operation distance of the virtual viewpoint by the user becomes longer, or controls the virtual viewpoint so that the virtual viewpoint is more likely to approach the host vehicle icon as the total operation time of the virtual viewpoint by the user becomes longer. Vehicle surrounding environment display device.

2. The image display unit according to claim 1, wherein the fixation point, which is the rotation center of the virtual viewpoint in the virtual space, approaches the host vehicle icon as the operation distance of the virtual viewpoint by the user becomes longer, or the fixation point of the virtual viewpoint approaches the host vehicle icon as the total operation time of the virtual viewpoint by the user becomes longer. Vehicle surrounding environment display device.

3. The image display unit sets a fixation point control unnecessary area that surrounds the host vehicle icon in the virtual space, and performs fixation point control to bring the fixation point of the virtual viewpoint closer to the host vehicle icon as the operation distance of the virtual viewpoint by the user becomes longer while the fixation point is located outside the fixation point control unnecessary area, or to bring the fixation point of the virtual viewpoint closer to the host vehicle icon as the total operation time of the virtual viewpoint by the user becomes longer while the fixation point is located outside the fixation point control unnecessary area, and does not perform the fixation point control while the fixation point is located within the fixation point control unnecessary area. Vehicle surrounding environment display device according to claim 2.

4. The image display unit shortens the distance between the fixation point and the virtual viewpoint as the operation distance of the virtual viewpoint by the user becomes longer, or shortens the distance between the fixation point and the virtual viewpoint as the total operation time of the virtual viewpoint by the user becomes longer. The vehicle surrounding environment display device according to claim 2.

5. The image display unit reduces the depression angle of the virtual viewpoint as the operation distance of the virtual viewpoint by the user becomes longer, or reduces the depression angle of the virtual viewpoint as the total operation time of the virtual viewpoint by the user becomes longer. The vehicle surrounding environment display device according to claim 2.

6. A control method for a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the host vehicle based on detection information of an external sensor of the host vehicle and displays an image in the virtual space as seen from a virtual viewpoint operated by a user of the host vehicle, A host vehicle icon corresponding to the host vehicle is arranged in the virtual space, The virtual viewpoint is controlled so that the virtual viewpoint is more likely to approach the host vehicle icon as the operation distance of the virtual viewpoint by the user becomes longer, or the virtual viewpoint is controlled so that the virtual viewpoint is more likely to approach the host vehicle icon as the total operation time of the virtual viewpoint by the user becomes longer. A control method for a vehicle surrounding environment display device.

7. A program for operating the ECU of the host vehicle as a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the host vehicle based on detection information of an external sensor of the host vehicle and displays an image in the virtual space as seen from a virtual viewpoint operated by a user of the host vehicle, A host vehicle icon corresponding to the host vehicle is arranged in the virtual space, A program that controls the virtual viewpoint so that the virtual viewpoint tends to approach the own vehicle icon as the operation distance of the virtual viewpoint by the user becomes longer, or controls the virtual viewpoint so that the virtual viewpoint tends to approach the own vehicle icon as the total operation time of the virtual viewpoint by the user becomes longer.

Images