Vehicle surrounding environment display device, control method for vehicle surrounding environment display device, and program

Abstract

This vehicle surrounding environment display device generates a virtual space corresponding to a surrounding environment of a host vehicle on the basis of detection information of an external sensor of the host vehicle, and displays, on a display, an image in a virtual space viewed from a virtual viewpoint operated by a user of the host vehicle. The virtual space includes a host vehicle icon corresponding to the host vehicle, and the vehicle surrounding environment display device includes an image display unit for controlling the virtual viewpoint such that, as controlled by the user, the greater the operating distance of the virtual viewpoint becomes, the easier it is to bring the virtual viewpoint closer to the host vehicle icon, or, as controlled by the user, the longer the total time of operating the virtual viewpoint becomes, the easier it is to bring the virtual viewpoint closer to the host vehicle icon.

Description

Vehicle surrounding environment display device, vehicle surrounding environment display device control method, and program

[0001] The present invention relates to a vehicle surrounding environment display device, a control method for a vehicle surrounding environment display device, and a program.

[0002] Japanese Patent Application Laid-Open Publication No. 2020-088697 is a known technical document relating to a vehicle surroundings environment display device. This publication describes a surroundings monitoring device that generates a virtual space including an image of the vehicle itself and projects the surroundings of the vehicle into the virtual space as a three-dimensional image, allowing the user to freely view the surroundings of the vehicle by manipulating the virtual viewpoint within the virtual space.

[0003] JP 2020-088697 A

[0004] In the conventional device described above, the image of the vehicle may become cut off while the user is freely manipulating the virtual viewpoint in the virtual space. If the user continues to manipulate the virtual viewpoint while the image of the vehicle is cut off, the user may lose track of the position of the virtual viewpoint relative to the image of the vehicle.

[0005] One aspect of the present disclosure is a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the vehicle based on detection information from an external sensor of the vehicle, and displays on a display an image of the virtual space as seen from a virtual viewpoint operated by a user of the vehicle, wherein a vehicle icon corresponding to the vehicle is placed in the virtual space, and the device is equipped with an image display unit that controls the virtual viewpoint so that the longer the distance the user operates the virtual viewpoint, the closer the virtual viewpoint becomes to the vehicle icon, or controls the virtual viewpoint so that the longer the total time the user operates the virtual viewpoint, the closer the virtual viewpoint becomes to the vehicle icon.

[0006] According to one aspect of the vehicle surrounding environment display device of the present disclosure, the virtual viewpoint is controlled so that the longer the user operates the virtual viewpoint or the longer the total operation time, the closer the virtual viewpoint becomes to the vehicle icon. This makes it possible for the user to easily find the vehicle icon by operating the virtual viewpoint even if it is out of view in the image seen from the virtual viewpoint. This ensures a high degree of freedom for the user to operate the virtual viewpoint while preventing the user from losing their sense of position of the virtual viewpoint relative to the vehicle icon.

[0007] In a vehicle surrounding environment display device according to an aspect of the present disclosure, the image display unit may bring a fixation point, which is the center of rotational movement of the virtual viewpoint in the virtual space, closer to the host vehicle icon as the user operates the virtual viewpoint. Furthermore, the image display unit may bring the fixation point of the virtual viewpoint closer to the host vehicle icon as the user operates the virtual viewpoint for a longer total time. According to this vehicle surrounding environment display device, by bringing the fixation point of the virtual viewpoint closer to the host vehicle icon as the user operates the virtual viewpoint for a longer total time, the likelihood that the host vehicle icon will appear in an image viewed from the virtual viewpoint due to the user's operation increases. Therefore, it is possible to prevent the user from losing a sense of position of the virtual viewpoint relative to the host vehicle icon while ensuring a high degree of freedom in operating the virtual viewpoint.

[0008] In a vehicle surrounding environment display device according to one aspect of the present disclosure, the image display unit sets a gaze point control-free area surrounding a host vehicle icon in a virtual space, and performs gaze point control such that the gaze point of the virtual viewpoint is brought closer to the host vehicle icon as the user operates the virtual viewpoint while the gaze point is located outside the gaze point control-free area, or the gaze point of the virtual viewpoint is brought closer to the host vehicle icon as the user operates the virtual viewpoint while the gaze point is located outside the gaze point control-free area, and does not need to be performed while the gaze point is located within the gaze point control-free area. With this vehicle surrounding environment display device, gaze point control is not performed while the gaze point is located within the gaze point control-free area set surrounding the host vehicle icon, thereby preventing impairment of user operability due to performing gaze point control even when the gaze point is located near the host vehicle icon.

[0009] In a vehicle surrounding environment display device according to one aspect of the present disclosure, the image display unit shortens the distance between the point of gaze and the virtual viewpoint as the user's operation distance of the virtual viewpoint increases, or shortens the distance between the point of gaze and the virtual viewpoint as the user's total operation time of the virtual viewpoint increases. With this vehicle surrounding environment display device, by shortening the distance between the point of gaze and the virtual viewpoint as the user's operation distance or total operation time of the virtual viewpoint increases, the virtual viewpoint also approaches the host vehicle icon as the point of gaze approaches the host vehicle icon, thereby ensuring a high degree of freedom in the user's operation of the virtual viewpoint while preventing the user from losing their sense of position of the virtual viewpoint relative to the host vehicle icon.

[0010] In a vehicle surrounding environment display device according to one aspect of the present disclosure, the image display unit decreases the depression angle of the virtual viewpoint as the user operates the virtual viewpoint longer, or decreases the depression angle of the virtual viewpoint as the user operates the virtual viewpoint longer. This vehicle surrounding environment display device reduces the depression angle of the virtual viewpoint relative to the gaze point as the user operates the virtual viewpoint longer. This ensures a high degree of freedom in operating the virtual viewpoint while preventing the user from losing track of the position of the virtual viewpoint relative to the host vehicle icon.

[0011] Another aspect of the present disclosure is a control method for a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of a vehicle based on detection information from an external sensor of the vehicle, and displays on a display an image of the virtual space as seen from a virtual viewpoint operated by a user of the vehicle, wherein a vehicle icon corresponding to the vehicle is placed in the virtual space, and the virtual viewpoint is controlled so that the longer the distance the user operates the virtual viewpoint, the more likely the virtual viewpoint is to approach the vehicle icon, or the longer the total time the user operates the virtual viewpoint, the more likely the virtual viewpoint is to approach the vehicle icon.

[0012] According to another aspect of the present disclosure, a method for controlling a vehicle surrounding environment display device controls the virtual viewpoint so that the longer the user operates the virtual viewpoint or the longer the total operation time, the closer the virtual viewpoint becomes to the vehicle icon. This makes it possible for the user to easily find the vehicle icon by operating the virtual viewpoint even if it is cut off from the image viewed from the virtual viewpoint. This prevents the user from losing their sense of position of the virtual viewpoint relative to the vehicle icon while ensuring a high degree of freedom for the user to operate the virtual viewpoint.

[0013] Yet another aspect of the present disclosure is a program that operates an ECU of a vehicle as a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the vehicle based on detection information from an external sensor of the vehicle, and displays on a display an image of the virtual space as seen from a virtual viewpoint operated by a user of the vehicle, and places a vehicle icon corresponding to the vehicle in the virtual space, and controls the virtual viewpoint so that the longer the distance the user operates the virtual viewpoint, the more likely the virtual viewpoint is to approach the vehicle icon, or controls the virtual viewpoint so that the longer the total time the user operates the virtual viewpoint, the more likely the virtual viewpoint is to approach the vehicle icon.

[0014] According to a program relating to yet another aspect of the present disclosure, the virtual viewpoint is controlled so that the longer the user operates the virtual viewpoint or the longer the total operation time, the closer the virtual viewpoint becomes to the vehicle icon.This makes it possible for the user to easily find the vehicle icon by operating the virtual viewpoint even if it is cut off from the image seen from the virtual viewpoint.This ensures a high degree of freedom for the user to operate the virtual viewpoint while preventing the user from losing their sense of the position of the virtual viewpoint relative to the vehicle icon.

[0015] According to each aspect of the present disclosure, it is possible to prevent the user from losing track of the position of the virtual viewpoint relative to the host vehicle icon while ensuring a high degree of freedom for the user to operate the virtual viewpoint.

[0016] 1 is a block diagram showing a vehicle surrounding environment display device according to an embodiment; FIG. 2 is a diagram for explaining a host vehicle icon and a virtual viewpoint; (a) a diagram showing an example of rotational movement of the virtual viewpoint; (b) a diagram showing an example of rotational movement of the virtual viewpoint when gaze point control is performed; (c) a diagram showing an example of rotational movement of the virtual viewpoint when the gaze point is located within an area where gaze point control is not required; (d) a diagram showing an example of gaze point control when the gaze point is located outside the area where gaze point control is not required; (e) a diagram for explaining an automatic control intervention area; (f) a diagram for explaining the automatic control intervention area as seen from behind the host vehicle icon; and (g) a flowchart showing an example of a control method for the vehicle surrounding environment display device according to the present embodiment.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0018] FIG. 1 is a block diagram showing a vehicle surroundings environment display device 100 according to one embodiment. The vehicle surroundings environment display device 100 shown in FIG. 1 is mounted on a vehicle such as a passenger car or a freight vehicle (hereinafter referred to as the host vehicle) to support the user in recognizing the vehicle's surroundings environment. The vehicle surroundings environment display device 100 generates a virtual space that reflects the host vehicle's surroundings environment and displays an image of the virtual space as seen from a virtual viewpoint operated by the user on a display. The vehicle surroundings environment display device 100 displays the host vehicle's surroundings environment on a display as a so-called 3D view.

[0019] The user may be the driver of the vehicle, a passenger of the vehicle, or the owner of the vehicle. The user may also be an operator who remotely supports the vehicle using a remote assistance system. In the remote assistance system, the operator can make decisions about the vehicle's driving (decisions about proceeding, turning right or left, stopping, etc.) or operate the vehicle through remote assistance equipment installed outside the vehicle and capable of communicating with the vehicle. The vehicle is not limited to vehicles that can be remotely supported by the remote assistance system. The vehicle may be a vehicle with an autonomous driving function or a vehicle without an autonomous driving function.

[0020] [Configuration of Vehicle Surrounding Environment Display Device] As shown in FIG. 1 , the vehicle surrounding environment display device 100 includes an ECU (Electronic Control Unit) 10 that manages the device. The ECU 10 is an electronic control unit having a CPU (Central Processing Unit) and a storage unit. The storage unit is composed of, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and an EEPROM (Electrically Erasable Programmable Read-Only Memory). The ECU 10 realizes various functions by, for example, executing programs stored in the storage unit in the CPU. The ECU 10 may be composed of multiple electronic units. The ECU 10 is connected to an external camera 1 (external sensor), a radar sensor 2 (external sensor), a user operation reception unit 3, and a display 4.

[0021] The external camera 1 is an imaging device that captures images of the external situation of the vehicle. The external camera 1 includes, for example, a front camera that captures images in front of the vehicle, a back camera that captures images behind the vehicle, and side cameras that capture images on the left and right sides of the vehicle. The number of cameras in the external camera 1 is not particularly limited, and may be one. The external camera 1 transmits captured image information to the ECU 10.

[0022] The radar sensor 2 is a detection device that detects objects around the vehicle using radio waves (e.g., millimeter waves) or light. The radar sensor 2 may include a millimeter-wave radar or a LIDAR (Light Detection and Ranging). The radar sensor 2 transmits object detection information relating to the detected objects to the ECU 10. The radar sensor 2 and the external camera 1 constitute an external sensor for detecting the environment around the vehicle. The object detection information of the radar sensor 2 or the image information captured by the external camera 1 corresponds to the detection information of the external sensor.

[0023] The user operation reception unit 3 is a device that receives operations of the virtual viewpoint by the user. The user operation reception unit 3 can be, for example, an input unit of an HMI (Human Machine Interface) provided in the vehicle. The input unit includes, for example, a touch panel display, buttons, levers, switches, etc. The user operation reception unit 3 may also be capable of receiving operations by voice recognition or gestures.

[0024] A mobile terminal or a computer input device connected to the vehicle may be used as the user operation reception unit 3. Alternatively, an operator terminal of the remote assistance system may be used as the user operation reception unit 3.

[0025] The display 4 is, for example, a center display mounted on the dashboard of the vehicle. The display 4 may be a display of a tablet computer that can be installed in the vehicle, or may be a HUD (Head Up Display). The display 4 is not limited to being mounted on the vehicle. The display 4 may be a display for an operator of a remote assistance system installed in a facility away from the vehicle. The display 4 may be a display of a mobile terminal carried by the user, or may be a display of the user's tablet computer or desktop computer.

[0026] Next, a description will be given of the functional configuration of the ECU 10. As shown in Fig. 1, the ECU 10 has a virtual space generation unit 11 and an image display unit 12. Some of the functions of the ECU 10 described below may be executed by a server (e.g., a server of a remote assistance system), a mobile terminal, or a computer (e.g., a tablet computer or a desktop computer) that can communicate with the vehicle.

[0027] The virtual space generation unit 11 generates a virtual space corresponding to the surrounding environment of the vehicle, for example, based on image information captured by the external camera 1. The surrounding environment of the vehicle includes, for example, the position of the white lines of the lane in which the vehicle is traveling. The surrounding environment of the vehicle may also include the status (position, direction of travel, etc.) of other vehicles, such as a preceding vehicle or an adjacent vehicle traveling alongside the vehicle.

[0028] The virtual space is generated as, for example, a 3D image obtained by combining multiple images. The method of combining the images is not particularly limited. For example, the virtual space generation unit 11 generates the virtual space as a 3D image by projecting each image onto a global coordinate system that serves as the reference for the virtual space and associating overlapping pixels with each other.

[0029] The virtual space generation unit 11 places a host vehicle icon corresponding to the host vehicle in the virtual space. The host vehicle icon is placed as a three-dimensional icon. The host vehicle icon can be formed using polygon, voxel, or other CG processing. The virtual space generation unit 11 may generate a host vehicle icon that reflects the state of the host vehicle. The virtual space generation unit 11 may reflect the lighting status of the host vehicle's lights (such as the lighting status of headlights, turn signals, and brake lights) in the lighting status of the host vehicle icon, and may reflect the steering angle of the host vehicle's tires in the tires of the host vehicle icon.

[0030] When the virtual space generation unit 11 recognizes another vehicle based on the image information captured by the external camera 1, it may place a three-dimensional other vehicle icon corresponding to the other vehicle in the virtual space. Similarly, the virtual space generation unit 11 may place a three-dimensional pedestrian icon in the virtual space. The virtual space generation unit 11 may recognize another vehicle, etc. based on object detection information from the radar sensor 2 instead of the image information captured by the external camera 1, or may recognize another vehicle, etc. using both the external camera 1 and the radar sensor 2.

[0031] The virtual space generation unit 11 may recognize other vehicles around the vehicle using information about the surrounding environment recognized by other vehicles through vehicle-to-vehicle communication. The virtual space generation unit 11 may acquire image information from cameras installed on the road and various types of traffic information by communicating with a traffic information management server managed by the government, for example, and use this information to recognize other vehicles.

[0032] Furthermore, the virtual space generation unit 11 may predict the behavior of other vehicles based on image information captured by the external camera 1 or object detection information from the radar sensor 2, and may display the predicted results of the behavior of other vehicles in association with other vehicle icons. For example, the virtual space generation unit 11 may display the predicted path of other vehicles using an arrow icon or the like, or may display the predicted stopping position of other decelerating vehicles using a block icon extending in the lane width direction or the like. Similarly, the virtual space generation unit 11 may display the predicted results of pedestrian behavior in association with pedestrian icons.

[0033] The method for generating the virtual space is not limited to the method of combining multiple images from the external camera 1. The virtual space generation unit 11 does not need to generate the virtual space as a 3D image as long as the user can recognize the surrounding environment of the host vehicle. The virtual space generation unit 11 may generate the virtual space by arranging a host vehicle icon, white lines, and other vehicle icons so that the positional relationship of the host vehicle to the white lines and other vehicles can be seen.

[0034] The image display unit 12 displays on the display 4 an image of the virtual space as seen from a virtual viewpoint operated by the user in the virtual space generated by the virtual space generation unit 11. The image display unit 12 moves the virtual viewpoint 50 in response to a user operation input to the user operation reception unit 3.

[0035] FIG. 2 is a diagram for explaining a host vehicle icon and a virtual viewpoint. FIG. 2 shows a host vehicle icon M and a virtual viewpoint 50. The plane on which the host vehicle icon M is arranged corresponds to the horizontal plane of the global coordinate system. FIG. 2 also shows the line of sight D of the virtual viewpoint 50, the gaze point C of the virtual viewpoint 50, and the depression angle α of the virtual viewpoint 50. For ease of understanding, FIG. 2 illustrates the virtual viewpoint 50 as a camera icon. The depression angle α is the angle between the line of sight D and the plane on which the host vehicle icon M is arranged in a vertical plane or the horizontal plane of the global coordinate system. Note that the virtual viewpoint 50 does not need to be an icon.

[0036] The gaze point C of the virtual viewpoint 50 is the center of rotational movement of the virtual viewpoint 50 in the virtual space. When the user performs an operation to rotate the virtual viewpoint 50 horizontally, the virtual viewpoint 50 moves in an arc centered on the gaze point C when viewed from the up-down direction of the host vehicle icon M (the vertical direction of the global coordinate system). Here, FIG. 3A is a diagram showing an example of rotational movement of the virtual viewpoint. FIG. 3A is a diagram seen from above the host vehicle icon M. FIG. 3A shows a trajectory Ra of the virtual viewpoint 50 moving in a circle centered on the gaze point C. The linear distance between the gaze point C and the virtual viewpoint 50 is also shown as a viewpoint distance L. In FIG. 3A, the viewpoint distance L corresponds to the radius of rotation of the circular trajectory Ra.

[0037] Similarly, the virtual viewpoint 50 can also be rotated vertically around the gaze point C. When the user performs an operation to rotate the virtual viewpoint 50 vertically, the virtual viewpoint 50 moves in an arc centered on the gaze point C when viewed from the fore-and-aft direction of the host vehicle icon M in FIG. 2 (the horizontal direction of the global coordinate system). In addition to rotational movement, the virtual viewpoint 50 may also be capable of forward and backward movement (enlarging and reducing the image) along the line of sight D and translational movement. When intervention control is not being performed, the virtual viewpoint 50 can be moved to any position in the virtual space by a user operation and can face any direction.

[0038] An initial position is set for the virtual viewpoint 50. The initial position is, for example, a position looking down on the host vehicle icon M from above and behind the host vehicle icon M. The initial position is not particularly limited as long as it is a position where the host vehicle icon M is displayed on the display 4. The initial position may be changeable by the user.

[0039] The image display unit 12 controls the virtual viewpoint 50 so that the longer the operation distance of the virtual viewpoint 50 by the user, the closer the virtual viewpoint 50 becomes to the host vehicle icon M. The operation distance is the distance the virtual viewpoint 50 has moved in the virtual space due to the user's operation. The image display unit 12 may reset the count of the operation distance if a state in which the user has not input any operation continues for a certain period of time.

[0040] Specifically, the image display unit 12 performs gaze point control (intervention control) to bring the gaze point C of the virtual viewpoint 50 closer to the host vehicle icon M as the operation distance of the virtual viewpoint 50 becomes longer, thereby indirectly controlling the virtual viewpoint 50 so that it becomes easier to approach the host vehicle icon M. The gaze point control is not performed while the user stops operating the virtual viewpoint 50.

[0041] In the gaze point control, the degree of approach of the gaze point C according to the operation distance (ease of approach to the host vehicle icon M) may be a fixed value or a variable value. Specifically, the image display unit 12 may be configured so that the gaze point C approaches the host vehicle icon M by a predetermined distance while the operation distance increases by a certain distance. The image display unit 12 may be configured so that the gaze point C approaches the host vehicle icon M by a predetermined percentage of the total separation distance between the gaze point C and the host vehicle icon M while the operation distance increases by the certain distance.

[0042] Alternatively, the image display unit 12 may be configured to accelerate the distance by which the gaze point C approaches the host vehicle icon M as the operation distance increases. In other words, the image display unit 12 makes it easier for the virtual viewpoint 50 to approach the host vehicle icon M as the operation distance increases. The image display unit 12 may vary the distance by which the gaze point C approaches the host vehicle icon M according to the operation distance by using table data or an arithmetic expression that previously associates the operation distance with the distance by which the gaze point C approaches the host vehicle icon M. Instead of distance, a proportion of the total separation distance between the gaze point C and the host vehicle icon M may be used. The degree of approach of the gaze point C can be configured to minimize discomfort for the user. The degree of approach of the gaze point C may be adjustable by the user.

[0043] Furthermore, in the gaze point control, the image display unit 12 may control the viewpoint distance L (radius of rotation) between the virtual viewpoint 50 and the gaze point C. Specifically, in the gaze point control, the image display unit 12 shortens the viewpoint distance L as the operation distance of the virtual viewpoint 50 becomes longer. As a result, the virtual viewpoint 50 and the gaze point C become closer as the operation distance of the virtual viewpoint 50 becomes longer, and together with the gaze point C becoming closer to the host vehicle icon M, the host vehicle icon M becomes more likely to be reflected in the image viewed from the virtual viewpoint 50.

[0044] Furthermore, the image display unit 12 may control the depression angle α of the virtual viewpoint 50 in the gaze point control. Specifically, the image display unit 12 reduces the depression angle α of the virtual viewpoint 50 as the operation distance of the virtual viewpoint 50 by the user increases. As a result, the position of the virtual viewpoint 50 decreases as the operation distance of the virtual viewpoint 50 increases, and therefore the host vehicle icon M is more likely to be reflected in the image viewed from the virtual viewpoint 50 compared to when the virtual viewpoint 50 looks down from a higher position.

[0045] Here, Fig. 3(b) is a diagram showing an example of rotational movement of the virtual viewpoint 50 when the point of interest control is performed. Fig. 3(b) shows a trajectory Rb of the rotational movement of the virtual viewpoint 50 when the point of interest control is performed. In addition, a position 50a of the virtual viewpoint 50 after the rotational movement and a position Ca of the point of interest C after the movement are shown by dashed lines. In Fig. 3(b), it is assumed that the user has input only a clockwise rotational movement operation.

[0046] 3B, the image display unit 12 controls the point of gaze C to approach the host vehicle icon M in conjunction with the rotational movement of the virtual viewpoint 50. Furthermore, the image display unit 12 controls the viewpoint distance L between the virtual viewpoint 50 and the point of gaze C to decrease in conjunction with the rotational movement of the virtual viewpoint 50. Furthermore, the image display unit 12 controls the depression angle α of the virtual viewpoint 50 to decrease in conjunction with the rotational movement of the virtual viewpoint 50. As a result, the virtual viewpoint 50 rotates and moves gradually closer to the host vehicle icon M as shown by the trajectory Rb in FIG. 3B, and the depression angle α also decreases, so that the host vehicle icon M appears in the image viewed from the virtual viewpoint 50.

[0047] Even during the point of gaze control, the user can freely input an operation to move the virtual viewpoint 50. On the other hand, the point of gaze C, which is the rotation center of the virtual viewpoint 50, can be moved closer to the host vehicle icon M, so that the host vehicle icon M is more likely to be reflected in the image in the virtual space seen from the virtual viewpoint 50. In other words, the host vehicle icon M is less likely to be cut off from the image displayed on the display 4.

[0048] The image display unit 12 may allow the user to perform an operation to move the virtual viewpoint 50 together with the point of interest C during the point of interest control. In this case, both the user's operation and the point of interest control are reflected in the movement of the point of interest C. If the user continues to perform an operation to move the point of interest C away from the host vehicle icon M for a certain period of time or more, the user's operation may be given priority and the point of interest C may be moved. In this case, too, the point of interest C is positioned on the host vehicle icon M side as viewed from the virtual viewpoint 50, so that the host vehicle icon M is reflected in the image viewed from the virtual viewpoint 50.

[0049] The image display unit 12 may use the total time for which the user operates the virtual viewpoint 50 instead of the operation distance of the virtual viewpoint 50 by the user. The total operation time is the total value of the time for which the virtual viewpoint 50 moves in the virtual space by the user's operation. The time while the user stops operating the virtual viewpoint 50 is not counted toward the total operation time. The image display unit 12 may reset the count of the total operation time if a certain period of time continues in which the user does not input any operation.

[0050] In this case, the image display unit 12 controls the virtual viewpoint 50 so that the longer the total time the user operates the virtual viewpoint 50, the more likely the virtual viewpoint 50 is to approach the host vehicle icon M. The image display unit 12 performs gaze point control (intervention control) to bring the gaze point C closer to the host vehicle icon M as the total time the virtual viewpoint 50 is operated becomes longer. In the gaze point control, the image display unit 12 may shorten the viewpoint distance L as the total time the virtual viewpoint 50 is operated becomes longer. In the gaze point control, the image display unit 12 may reduce the depression angle α as the total time the virtual viewpoint 50 is operated becomes longer.

[0051] Next, a description will be given of a determination as to whether or not the gaze point control is necessary. The image display unit 12 may be configured not to execute the gaze point control when a predetermined condition is satisfied. Specifically, the image display unit 12 may set a gaze point control unnecessary area where the gaze point control is not executed.

[0052] The area where gaze point control is not required is an area where gaze point control is not performed when the area includes the gaze point C. The area where gaze point control is not required is set in the virtual space so as to include the host vehicle icon M. The area where gaze point control is not required may be set as a rectangular area centered on the host vehicle icon M when viewed from directly above the host vehicle icon M, or may be set as an area within a certain distance from the host vehicle icon M. The position of the host vehicle icon M within the area where gaze point control is not required is not particularly limited.

[0053] When the image display unit 12 sets the area where the point of interest control is not required, the image display unit 12 does not perform the point of interest control while the point of interest C is located within the area where the point of interest control is not required. The image display unit 12 does not need to count the operation distance of the virtual viewpoint 50 by the user while the point of interest control is not being performed (while the point of interest C is located within the area where the point of interest control is not required).

[0054] The image display unit 12 starts counting the operation distance of the virtual viewpoint 50 by the user while the gaze point C is located outside the area where gaze point control is not required. When the gaze point C is located outside the area where gaze point control is not required, the image display unit 12 executes gaze point control such that the longer the operation distance of the virtual viewpoint 50 by the user, the closer the gaze point C is to the host vehicle icon M. Note that if the gaze point C enters the area where gaze point control is not required during execution of gaze point control, the image display unit 12 resets or temporarily suspends counting the operation distance. The same applies when the total operation time is used for gaze point control instead of the operation distance.

[0055] Here, Fig. 4 is a diagram showing an example of rotational movement of the virtual viewpoint when the point of interest C is located within an area where point of interest control is not required. In Fig. 4, the user has input only a clockwise rotational movement operation. Fig. 4 shows the area W where point of interest control is not required, the trajectory Rc of the virtual viewpoint 50, and the first position 50b and second position 50c of the rotationally moved virtual viewpoint 50. As shown in Fig. 4, when the point of interest C is located within the area W where point of interest control is not required, point of interest control is not performed, so the virtual viewpoint 50 rotates along the trajectory Rc that describes a circle centered on the fixed point of interest C.

[0056] Fig. 5 is a diagram showing an example of gaze point control when the gaze point C is located outside the gaze point control unnecessary area W. In Fig. 5, the user also inputs only a clockwise rotational movement operation. Fig. 5 shows the trajectory Rd of the virtual viewpoint 50, and the third position 50d and fourth position 50e of the rotationally moved virtual viewpoint 50. Also shown is the position Cd of the gaze point C corresponding to the third position 50d of the virtual viewpoint 50, and the position Ce of the gaze point C corresponding to the fourth position 50e of the virtual viewpoint 50.

[0057] As shown in Fig. 5, when the gaze point C of the virtual viewpoint 50 moves out of the gaze point control unnecessary area W, gaze point control is executed. The gaze point C is controlled so as to gradually approach the host vehicle icon M in conjunction with the rotational movement of the virtual viewpoint 50. The gaze point control shown in Fig. 5 also includes control of the viewpoint distance L. The gaze point control may also include control of the depression angle α.

[0058] The image display unit 12 may determine whether or not to perform the gaze point control based on the position of the virtual viewpoint 50, rather than the position of the gaze point C. The image display unit 12 may be configured to perform the gaze point control when the virtual viewpoint 50 is located outside the gaze point control unnecessary area W, and not to perform the gaze point control when the virtual viewpoint 50 is located within the gaze point control unnecessary area W.

[0059] Alternatively, the image display unit 12 may determine whether or not it is necessary to perform the gaze point control using both the position of the virtual viewpoint 50 and the position of the gaze point C. The image display unit 12 may perform the gaze point control of the virtual viewpoint 50 when either the virtual viewpoint 50 or the gaze point C is located outside the area W where gaze point control is not required.

[0060] The image display unit 12 does not necessarily need to use the gaze point control unnecessary area W to determine whether gaze point control is unnecessary. The image display unit 12 may be configured not to perform gaze point control while the distance between the gaze point C and the host vehicle icon M is less than a first distance threshold, for example. The image display unit 12 may be configured not to perform gaze point control while the distance between the virtual viewpoint 50 and the host vehicle icon M is less than a second distance threshold. Each threshold is a threshold value that is set in advance.

[0061] Next, a description will be given of automatic control of the virtual viewpoint 50. When a predetermined condition is satisfied, the image display unit 12 may automatically control the position and orientation of the virtual viewpoint 50 so that the host vehicle icon M is reflected on the screen.

[0062] 6 is a diagram illustrating the automatic control intervention area. Fig. 6 shows the automatic control intervention area H, a fifth position 50f, a sixth position 50g, and a seventh position 50h of the virtual viewpoint 50. The automatic control intervention area H is, for example, an area within a certain distance from the host vehicle icon M. The automatic control intervention area H may be set as a rectangular area, similar to the gaze point control-unnecessary area W.

[0063] In FIG. 6 , it is assumed that the virtual viewpoint 50 has been moved from its initial position to a fifth position 50f by a user operation. The fifth position 50f is a position outside the automatic control intervention area H. In this case, the image display unit 12 executes automatic control of the virtual viewpoint 50. Specifically, the image display unit 12 automatically moves the virtual viewpoint 50 to a sixth position 50g. At the sixth position 50g, the virtual viewpoint 50 is moved so that it faces the host vehicle icon M. Thereafter, the image display unit 12 moves the virtual viewpoint 50 to a seventh position 50h so that the host vehicle icon M is enlarged as viewed from the virtual viewpoint 50. The seventh position 50h is a position where the virtual viewpoint 50 enters the automatic control intervention area H.

[0064] 6 , when the virtual viewpoint 50 is away from the host vehicle icon M by a certain distance or more, the image display unit 12 automatically controls the position and orientation of the virtual viewpoint 50, thereby making it possible to make the user aware of the position of the host vehicle icon M when the user loses sight of the host vehicle icon M. The image display unit 12 may be configured to automatically move the virtual viewpoint 50 to an initial position.

[0065] 7 is a diagram illustrating the automatic control intervention area H as viewed from behind the host vehicle icon M. As shown in FIG. 7, the automatic control intervention area H and the point of view control-free area W can be set as three-dimensional areas with a fixed height. Note that the automatic control intervention area H and the point of view control-free area W may also be areas set relative to a horizontal plane (areas with no height). In this case, whether or not automatic control will be performed is determined depending on whether or not the position of the virtual viewpoint 50 projected onto the horizontal plane is included in the automatic control intervention area H.

[0066] The image display unit 12 may determine whether or not to perform automatic control based on the position of the point of gaze C, instead of the position of the virtual viewpoint 50. Even if the virtual viewpoint 50 is located outside the automatic control intervention area H, the image display unit 12 does not perform automatic control of the virtual viewpoint 50 if the point of gaze C of the virtual viewpoint 50 is located within the automatic control intervention area H. Even if the virtual viewpoint 50 is located within the automatic control intervention area H, if the point of gaze C is located outside the automatic control intervention area H, the image display unit 12 performs automatic control of the virtual viewpoint 50 because there is a possibility that the user has lost sight of the host vehicle icon M.

[0067] Alternatively, the image display unit 12 may determine whether to perform automatic control using both the position of the virtual viewpoint 50 and the position of the point of gaze C. The image display unit 12 may perform automatic control of the virtual viewpoint 50 when either the virtual viewpoint 50 or the point of gaze C is located outside the automatic control intervention area H.

[0068] Note that the image display unit 12 may use, as a condition for executing automatic control, not only the distance between the virtual viewpoint 50 or the point of gaze C and the host vehicle icon M, but also the fact that the host vehicle icon M is out of view from the image viewed from the virtual viewpoint 50. If the host vehicle icon M is displayed in the image even when the virtual viewpoint 50 is significantly away from the host vehicle icon M, the image display unit 12 can consider that the sense of position of the virtual viewpoint 50 relative to the host vehicle icon M has not been lost. For this reason, even if the virtual viewpoint 50 or the point of gaze C is located outside the automatic control intervention area H, the image display unit 12 may prioritize the user's operation and not execute automatic control of the virtual viewpoint 50 if the host vehicle icon M is displayed in the image.

[0069] [Program] The program causes the ECU 10 to function (operate) as the virtual space generation unit 11 and the image display unit 12. The program is provided by a non-transitory recording medium such as a ROM or a semiconductor memory. Alternatively, the program may be provided via communication such as a network.

[0070] [Method of Controlling the Vehicle Surrounding Environment Display Device] Next, a method of controlling the vehicle surrounding environment display device 100 according to this embodiment will be described with reference to the drawings. Fig. 8 is a flowchart showing an example of the method of controlling the vehicle surrounding environment display device 100 according to this embodiment.

[0071] 8, in S10, the ECU 10 of the vehicle surrounding environment display device 100 determines, using the image display unit 12, whether the virtual viewpoint 50 is located within the automatic control intervention area H. If the ECU 10 does not determine that the virtual viewpoint 50 is located within the automatic control intervention area H (S10: NO), the ECU 10 proceeds to S11. If the ECU 10 determines that the virtual viewpoint 50 is located within the automatic control intervention area H (S10: YES), the ECU 10 proceeds to S12.

[0072] In S11, the ECU 10 executes automatic control of the virtual viewpoint 50 using the image display unit 12. The image display unit 12 automatically moves the virtual viewpoint 50 to a position where, for example, the host vehicle icon M appears on the screen of the display 4. Thereafter, the ECU 10 ends the current control.

[0073] In S12, the ECU 10 determines whether the gaze point C is located outside the gaze point control unnecessary area W using the image display unit 12. If the ECU 10 does not determine that the gaze point C is located outside the gaze point control unnecessary area W (S12: NO), the ECU 10 ends the current control. If the ECU 10 determines that the gaze point C is located outside the gaze point control unnecessary area W (S12: YES), the ECU 10 proceeds to S13.

[0074] In S13, the ECU 10 executes gaze point control using the image display unit 12 in accordance with the operation distance or total operation time of the virtual viewpoint 50 by the user. For example, the image display unit 12 executes gaze point control such that the longer the operation distance of the virtual viewpoint 50, the closer the gaze point C of the virtual viewpoint 50 to the host vehicle icon M. The image display unit 12 may control the viewpoint distance L and depression angle α of the virtual viewpoint 50 in the gaze point control. Thereafter, the ECU 10 ends the current control.

[0075] The vehicle surrounding environment display device 100 does not necessarily have to perform the determination in S10. In this case, the process of S11 is not necessary. Furthermore, the vehicle surrounding environment display device 100 does not necessarily have to perform the determination in S12, and may always perform the gaze point control.

[0076] According to the vehicle surrounding environment display device 100 of this embodiment described above, the virtual viewpoint 50 is controlled so that the longer the user's operating distance or total operating time of the virtual viewpoint 50, the closer the virtual viewpoint 50 becomes to the vehicle icon M. This makes it possible for the user to easily find the vehicle icon M by operating the virtual viewpoint 50 even if the vehicle icon M is cut off from the image seen from the virtual viewpoint 50. This ensures a high degree of freedom for the user to operate the virtual viewpoint 50 while preventing the user from losing their sense of position of the virtual viewpoint 50.

[0077] Furthermore, according to the vehicle surrounding environment display device 100, the longer the operating distance or total operating time of the virtual viewpoint 50, the closer the gaze point C of the virtual viewpoint 50 is to the vehicle icon M, which increases the possibility that the vehicle icon M will be reflected in the image viewed from the virtual viewpoint 50 due to the user's operation.This therefore ensures a high degree of freedom for the user to operate the virtual viewpoint 50 while preventing the user from losing their sense of position of the virtual viewpoint 50.

[0078] Furthermore, according to the vehicle surrounding environment display device 100, gaze point control is not performed while the gaze point C is located within the gaze point control-free area W set around the vehicle icon M, thereby preventing the impairment of user operability caused by performing gaze point control even when the gaze point C is located near the vehicle icon M.

[0079] Furthermore, according to the vehicle surrounding environment display device 100, the viewpoint distance L, which is the distance between the gaze point C and the virtual viewpoint 50, is shortened as the operation distance or total operation time of the virtual viewpoint 50 becomes longer, so that as the gaze point C approaches the host vehicle icon M, the virtual viewpoint 50 also approaches the host vehicle icon M, thereby ensuring a high degree of freedom for the user to operate the virtual viewpoint 50 while preventing the user from losing their sense of position of the virtual viewpoint 50.

[0080] Furthermore, according to the vehicle surrounding environment display device 100, the depression angle α of the virtual viewpoint 50 relative to the gaze point C is reduced as the operating distance or total operating time of the virtual viewpoint 50 becomes longer, thereby ensuring a high degree of freedom for the user to operate the virtual viewpoint 50 while preventing the user from losing their sense of position of the virtual viewpoint 50.

[0081] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. The present invention can be embodied in various forms, including the above-described embodiments, with various modifications and improvements made based on the knowledge of those skilled in the art.

[0082] Instead of the gaze point control described above, the vehicle surrounding environment display device 100 may be configured to directly control the virtual viewpoint 50 so that the virtual viewpoint 50 tends to approach the host vehicle icon M. The vehicle surrounding environment display device 100 may also perform intervention control, without using the gaze point C, so that the position of the virtual viewpoint 50 approaches the host vehicle icon M as the operation distance or total operation time of the virtual viewpoint 50 by the user increases.

[0083] The vehicle surrounding environment display device 100 may control the virtual viewpoint 50 so that it becomes easier to move closer to the vehicle icon M by increasing the resistance to operations that move the virtual viewpoint 50 away from the vehicle icon M, and by decreasing or setting the resistance to zero to operations that move the virtual viewpoint 50 closer to the vehicle icon M, as the user's operating distance or total operating time of the virtual viewpoint 50 becomes longer.

[0084] 1...external camera, 2...radar sensor, 3...user operation reception unit, 4...display, 10...ECU, 11...virtual space generation unit, 12...image display unit, 50...virtual viewpoint, 100...vehicle surrounding environment display device

Claims

1. A vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of a vehicle based on detection information from an external sensor of the vehicle, and displays on a display an image of the virtual space as seen from a virtual viewpoint operated by a user of the vehicle, wherein a vehicle icon corresponding to the vehicle is placed in the virtual space, and the vehicle surrounding environment display device is equipped with an image display unit that controls the virtual viewpoint so that the longer the operation distance of the virtual viewpoint by the user, the closer the virtual viewpoint becomes to the vehicle icon, or controls the virtual viewpoint so that the longer the total time for which the user operates the virtual viewpoint, the closer the virtual viewpoint becomes to the vehicle icon.

2. The vehicle surrounding environment display device as described in claim 1, wherein the image display unit brings a gaze point, which is the center of rotation of the virtual viewpoint in the virtual space, closer to the vehicle icon as the user operates the virtual viewpoint over a longer distance, or brings the gaze point of the virtual viewpoint closer to the vehicle icon as the user operates the virtual viewpoint over a longer total time.

3. The vehicle surrounding environment display device of claim 2, wherein the image display unit sets a gaze point control-free area surrounding the vehicle icon within the virtual space, and performs gaze point control such that the gaze point of the virtual viewpoint is brought closer to the vehicle icon as the operation distance of the virtual viewpoint by the user while the gaze point is located outside the gaze point control-free area increases, or the gaze point of the virtual viewpoint is brought closer to the vehicle icon as the total operation time of the virtual viewpoint by the user while the gaze point is located outside the gaze point control-free area increases, and does not perform the gaze point control while the gaze point is located within the gaze point control-free area.

4. A vehicle surrounding environment display device as described in claim 2, wherein the image display unit shortens the distance between the gaze point and the virtual viewpoint as the user operates the virtual viewpoint over a longer distance, or shortens the distance between the gaze point and the virtual viewpoint as the user operates the virtual viewpoint for a longer total time.

5. A vehicle surrounding environment display device as described in claim 2, wherein the image display unit reduces the depression angle of the virtual viewpoint as the operation distance of the virtual viewpoint by the user increases, or reduces the depression angle of the virtual viewpoint as the total time for which the user operates the virtual viewpoint increases.

6. A control method for a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of a vehicle based on detection information from an external sensor of the vehicle, and displays on a display an image of the virtual space as seen from a virtual viewpoint operated by a user of the vehicle, wherein a vehicle icon corresponding to the vehicle is placed in the virtual space, and the virtual viewpoint is controlled so that the longer the operation distance of the virtual viewpoint by the user, the closer the virtual viewpoint becomes to the vehicle icon, or the longer the total time for which the user operates the virtual viewpoint, the closer the virtual viewpoint becomes to the vehicle icon.

7. A program that operates an ECU of a host vehicle as a vehicle surrounding environment display device that generates a virtual space corresponding to the surrounding environment of the host vehicle based on detection information from an external sensor of the host vehicle, and displays on a display an image of the virtual space as seen from a virtual viewpoint operated by a user of the host vehicle, the program placing a host vehicle icon corresponding to the host vehicle in the virtual space, and controlling the virtual viewpoint so that the longer the distance the user operates the virtual viewpoint, the closer the virtual viewpoint becomes to the host vehicle icon, or controlling the virtual viewpoint so that the longer the total time the user operates the virtual viewpoint, the closer the virtual viewpoint becomes to the host vehicle icon.

Images