Display system and display method

Abstract

To convey the presence of a traffic participant in the periphery of an own vehicle to a driver realistically in a recognizable manner.SOLUTION: A display system includes: a location acquiring section that acquires a current location of an own vehicle; an environment image generating section that generates a virtual environment image showing a surrounding environment of the own vehicle on the basis of the current location of the own vehicle and map information; a partial video image extracting section that extracts a participant video image, which is a video image portion of a traffic participant from the real environment video image; a turn determining section that determines the presence of an intention to turn of a driver of the own vehicle; and a display control section that generates and displays on a display device a composite image by inlaying the participant video image into the virtual environment image at a corresponding position. The display control section, if the own vehicle is not trying to turn, displays the composite image based on a front virtual image including the virtual environment image corresponding to a front line of sight of the driver, and if the own vehicle is trying to turn, displays the composite image based on a lateral virtual image including the virtual environment image corresponding to a line of sight in a direction of turn of the driver on the display device.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a display system and a display method for displaying the surrounding environment of a vehicle. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transport systems that take into consideration vulnerable transport participants have become more active. To achieve this, we are focusing on research and development into preventive safety technologies to further improve road safety and convenience.

[0003] Patent Document 1 discloses an image receiving and displaying device that performs geometric transformation on images captured using a camera installed outside the vehicle, converting the images into images as seen from above the vehicle or as seen from a virtual mirror in front of the vehicle, and displays the images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-200819 Summary of the Invention [Problem to be solved by the invention]

[0005] In preventive safety technology, a challenge is to provide information using a display device that complements the driver's perception to ensure safe driving of the vehicle, and to convey to the driver the traffic environment around the vehicle in an easily recognizable manner. In this regard, the image displayed by the technology described in Patent Document 1 is an image viewed from a fixed point such as above the vehicle or from a virtual mirror, and for a driver who needs to dynamically change his or her line of sight in accordance with changes in the driving situation, such as going straight or turning left or right, the image may be significantly different from the scene that is actually perceived by the eyes. For this reason, the above-mentioned conventional technology has limitations in terms of easily conveying the traffic environment to the driver. In order to solve the above-mentioned problems, the present application aims to achieve preventive safety for vehicle driving by conveying the traffic environment around the vehicle to the driver in an easily recognizable manner in accordance with the actual driving scene, thereby contributing to the development of a sustainable transportation system. [Means for solving the problem]

[0006] One aspect of the present invention includes a position acquisition unit that acquires the current position of the vehicle, an environment image generation unit that generates a virtual environment image that is a virtual image showing the environment around the vehicle based on the current position of the vehicle and map information, a partial image extraction unit that acquires a real environment image around the vehicle and extracts a participant image that is an image portion of traffic participants from the real environment image, a turn determination unit that determines whether the driver of the vehicle is about to turn, and a composite image that is generated by fitting each of the extracted participant images into a corresponding position on the virtual environment image and displaying the composite image on a display device. and a display control unit, wherein the environmental image generation unit generates a forward virtual image including the virtual environment image corresponding to the driver's line of sight ahead when the vehicle is not about to turn, and generates a side virtual image including the virtual environment image corresponding to the driver's line of sight in the turning direction when the vehicle is about to turn, and the display control unit displays the composite image based on the forward virtual image on the display device when the vehicle is not about to turn, and displays the composite image based on the side virtual image on the display device when the vehicle is about to turn. According to another aspect of the present invention, the display control unit displays the composite image based on the side virtual image in a display range on the display screen of the display device that is shifted toward the turning direction from the display range of the composite image based on the front virtual image. According to another aspect of the present invention, the display control unit displays the composite image on the display screen of the display device so that the position on the composite image corresponding to the direction in which the vehicle is traveling is the center of the display range of the composite image. According to another aspect of the present invention, the display screen of the display device is a display panel provided on an instrument panel of the vehicle and extending laterally in the vehicle width direction. According to another aspect of the present invention, the display screen of the display device is configured by a plurality of display panels arranged in the vehicle width direction on an instrument panel of the vehicle. According to another aspect of the invention, the display device includes an additional display panel disposed in front of a pillar of the host vehicle. According to another aspect of the present invention, when the vehicle is about to turn, the environmental image generation unit generates the lateral virtual image including the virtual environment image corresponding to the driver's line of sight in the turning direction, with the field of view of the virtual environment image corresponding to the driver's forward line of sight widened in the turning direction. According to another aspect of the present invention, when the vehicle is about to turn, the display control unit displays the composite image based on the side virtual image on the display screen of the display device in a range that is expanded toward the turning direction from the display range for the composite image based on the front virtual image. According to another aspect of the present invention, the display control unit generates a composite image based on the virtual environment image at the current time and the participant image at a predetermined time interval, and displays the composite image at the current time on the display device in real time. According to another aspect of the present invention, the virtual environment image is an image that provides an overhead view of the surrounding environment including the current position of the vehicle, and a virtual vehicle representation, which is a graphic representation of the vehicle, is superimposed at a position corresponding to the vehicle on the virtual environment image. Another aspect of the present invention is a display method executed by a computer provided in a display system, the display method comprising the steps of: acquiring the current position of the vehicle; generating a virtual environment image, which is a virtual image showing the environment surrounding the vehicle, based on the current position of the vehicle and map information; acquiring a real environment image around the vehicle and extracting a participant image, which is an image portion of traffic participants, from the real environment image; determining whether the driver of the vehicle is attempting to turn; and generating a composite image by fitting each of the extracted participant images into a corresponding position on the virtual environment image and displaying the composite image on a display device; wherein in the generating step, when the vehicle is not attempting to turn, a forward virtual image is generated that includes the virtual environment image corresponding to the driver's line of sight ahead, and when the vehicle is attempting to turn, a side virtual image is generated that includes the virtual environment image that corresponds to the driver's line of sight in the turning direction; and in the displaying step, when the vehicle is not attempting to turn, the composite image based on the forward virtual image is displayed on the display device, and when the vehicle is attempting to turn, the composite image based on the side virtual image is displayed on the display device. [Effects of the Invention]

[0007] According to the present invention, the traffic environment around the vehicle can be conveyed to the driver in a manner that is easily recognizable, in accordance with the actual driving scene. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a vehicle equipped with a display system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of the interior of the vehicle. [Figure 3] FIG. 3 is a diagram showing the configuration of a display system according to one embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of the display positions of a composite image based on a front virtual image and a composite image based on a side virtual image on a display device. [Figure 5] FIG. 5 is a diagram showing an example of a composite image based on a side virtual image when the vehicle is turning right. [Figure 6] FIG. 6 is a flow chart showing the steps of the display method executed by the processor of the display system. [Figure 7] FIG. 7 is a diagram showing another example of a display device on which the display system displays a composite image. [Figure 8] FIG. 8 is a diagram showing yet another example of a display device on which the display system displays a composite image. [Figure 9] FIG. 9 is a diagram showing another example of the display positions on the display device of the composite image based on the front virtual image and the composite image based on the side virtual image. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] Fig. 1 is a diagram showing an example of the configuration of a host vehicle 2, which is a vehicle equipped with a display system 1 according to one embodiment of the present invention, and Fig. 2 is a diagram showing an example of the configuration of the interior of the host vehicle 2. The display system 1 is installed in the host vehicle 2, and displays a virtual environment image, which is a virtual image of the surrounding environment of the host vehicle 2 (hereinafter also simply referred to as the surrounding environment), on a display device 12, to notify the driver D of the presence of traffic participants in the surrounding environment.

[0011] The host vehicle 2 is equipped with a front camera 3a that captures images of the surrounding environment in front of the host vehicle 2, and a left side camera 3b and a right side camera 3c that capture images of the left and right sides of the host vehicle 2. Hereinafter, the front camera 3a, left side camera 3b, and right side camera 3c will be collectively referred to as cameras 3. The front camera 3a is disposed, for example, near the front bumper, and the left side camera 3b and right side camera 3c are disposed, for example, on the left and right door mirrors. The host vehicle 2 may further be equipped with a rear camera (not shown) that captures images of the surrounding environment behind the vehicle. The host vehicle 2 is also equipped with an object detection device 4 that detects objects present in the surrounding environment. The object detection device 4 may be, for example, a radar, a sonar, and / or a lidar.

[0012] The vehicle 2 is further equipped with a vehicle monitoring device 5 that collects at least information on the vehicle's 2 traveling speed and information on the operation of its turn signals (not shown), a GNSS receiver 6 that receives location information on the vehicle's 2 current position from a GNSS satellite, and a navigation device 7 that provides route guidance using map information.

[0013] A display device 12 is provided in the cabin of the vehicle 2 on an instrument panel 13. The display device 12 has a display panel extending laterally in the vehicle width direction as a display screen 15. The display device 12 is, for example, a touch panel. The display screen 15 of the display device 12 extends, for example, from the bottom of the pillar 11a on the driver's seat 10 side to the bottom of the pillar 11b on the opposite side along the vehicle width direction. Hereinafter, the pillars 11a and 11b will also be collectively referred to as pillars 11.

[0014] FIG. 3 is a diagram showing the configuration of the display system 1. As shown in FIG. The display system 1 includes a processor 20 and a memory 21. The memory 21 is, for example, configured with a volatile and / or non-volatile semiconductor memory and / or a hard disk drive. The processor 20 is, for example, a computer including a CPU. The processor 20 may be configured to include a ROM in which a program is written, a RAM for temporarily storing data, and the like. The processor 20 includes, as functional elements or functional units, a position acquisition unit 23, a turning determination unit 24, an environmental image generation unit 25, a partial video extraction unit 26, a vehicle detection unit 27, and a display control unit 28.

[0015] These functional elements of the processor 20 are realized, for example, by the processor 20, which is a computer, executing a display program 22 stored in the memory 21. The display program 22 can be stored in any computer-readable storage medium. Alternatively, all or part of the functional elements of the processor 20 can be configured by hardware each including one or more electronic circuit components.

[0016] The position acquisition unit 23 receives position information from the GNSS receiver 6 and acquires the current position of the host vehicle 2 .

[0017] The turn determination unit 24 determines whether the driver D of the host vehicle 2 is attempting to turn (i.e., whether the driver D intends to turn). For example, the turn determination unit 24 determines whether the driver of the host vehicle 2 is attempting to turn based on information related to the operation of a turn indicator (not shown) of the host vehicle 2 and / or information about the planned travel route of the host vehicle 2, according to conventional technology. That is, for example, when the turn indicator of the host vehicle 2 is operated, the turn determination unit 24 determines that the driver D is attempting to turn in the direction indicated by the turned-on turn indicator. Furthermore, for example, when the host vehicle 2 is traveling before an intersection and the planned travel route is to turn right or left at the intersection, the turn determination unit 24 determines that the driver D is attempting to turn to turn right or left. Here, information related to the operation of the turn indicator of the host vehicle 2 can be acquired from the vehicle monitoring device 5. Furthermore, information about the planned travel route of the host vehicle 2 can be acquired from the navigation device 7.

[0018] Note that the vehicle 2 may be controlled by an autonomously driving computer instead of a human driver D. For this reason, hereinafter, "the driver of the vehicle 2 is attempting to turn" will be simply referred to as "the vehicle 2 is attempting to turn," as a broad concept that includes cases where the vehicle 2 is controlled by a computer. In this sense, "the driver D's intention to turn" can also be rephrased as "the decision to turn the vehicle 2."

[0019] The environment image generation unit 25 generates a virtual environment image, which is a virtual image showing the surrounding environment of the vehicle 2, based on the current position of the vehicle 2 and map information. The map information can be acquired, for example, from the navigation device 7. In this embodiment, the virtual environment image generated by the environment image generation unit 25 is, for example, a 3D image (stereoscopic display image) that provides a bird's-eye view of the surrounding environment including the current position of the vehicle.

[0020] In this embodiment, in particular, based on the determination by the turn determination unit 24 regarding the driver D's intention to turn, when the host vehicle 2 is not about to turn, the environmental image generation unit 25 generates a front virtual image including a virtual environment image corresponding to the driver D's line of sight ahead (i.e., viewed from the front line of sight). Furthermore, based on the determination by the turn determination unit 24 regarding the driver D's intention to turn, when the host vehicle 2 is about to turn, the environmental image generation unit 25 generates a side virtual image including a virtual environment image corresponding to the driver D's line of sight in the turning direction (i.e., viewed from the line of sight ahead).

[0021] The partial image extraction unit 26 acquires an image of the real environment around the vehicle 2 by the camera 3, and extracts a participant image, which is an image portion of a traffic participant, from the acquired real environment image. The vehicle detection unit 27 detects the positions of surrounding vehicles in the surrounding environment from the real environment video, and vehicle attributes including vehicle type, size, and / or color. The size of the surrounding vehicles can be calculated based on the angle of view of the surrounding vehicles in the real environment video and the distance to the surrounding vehicles detected by the object detection device 4, for example, according to conventional technology. Furthermore, the vehicle types of the surrounding vehicles can be identified by image matching with template images that indicate the sizes and shapes of various vehicle types, such as trucks, buses, cars, and motorcycles, pre-stored in the memory 21, for example, according to conventional technology.

[0022] Furthermore, the vehicle detection unit 27 determines whether or not there is a possibility that the detected surrounding vehicle will come into contact with the host vehicle 2. For example, the vehicle detection unit 27 determines whether or not there is a possibility of contact, in accordance with conventional technology, based on information about the speed of the surrounding vehicles, information about the lighting status of their turn signals, information about the traveling speed of the host vehicle 2, and / or the result of the determination by the turn determination unit 24 as to whether or not the driver D intends to turn. Here, the information about the speed of the surrounding vehicles and information about the lighting status of their turn signals can be acquired from real-world video. Furthermore, the information about the traveling speed of the host vehicle 2 can be acquired from the vehicle monitoring device 5.

[0023] The display control unit 28 generates a composite image by fitting each of the participant videos extracted by the partial video extraction unit 26 into the virtual environment image generated by the environment image generation unit 25 at the corresponding positions on the virtual environment image, and displays the composite image on the display device 12. For example, the display control unit 28 generates a composite image based on the virtual environment image and the participant video at the current time at predetermined time intervals, and displays the composite image at the current time on the display device 12 in real time.

[0024] The size of the participant image when it is embedded in the virtual environment image can be, for example, a size calculated for the traffic participant according to conventional technology and reduced to match the scale of the virtual environment image at the position where the participant image is embedded. The actual size of the traffic participant can be calculated based on the angle of view of the traffic participant in the real environment image and the distance to the traffic participant detected by the object detection device 4, similar to the size of the surrounding vehicles described above.

[0025] The display control unit 28 may further generate a composite image by superimposing a virtual vehicle representation, which is a graphic representation of the host vehicle 2 (or showing the host vehicle 2), at a corresponding position on the virtual environment composite image. For example, the virtual vehicle representation is a graphic display that simulates the movement of the host vehicle as seen from behind, and the composite image may be a so-called chase view from a perspective that follows the host vehicle from behind.

[0026] In particular, in this embodiment, when the vehicle 2 is not about to turn, based on the determination by the turn determination unit 24 regarding the driver D's intention to turn, the display control unit 28 displays a composite image based on the front virtual image on the display device 12. Furthermore, when the vehicle 2 is about to turn, based on the determination by the turn determination unit 24 regarding the driver D's intention to turn, the display control unit 28 displays a composite image based on the side virtual image on the display device 12.

[0027] At this time, the display control unit 28 displays the composite image based on the side virtual image at a position on the display screen 15 of the display device 12 that is shifted toward the turning direction from the position where the composite image based on the front virtual image is displayed.

[0028] Fig. 4 is a diagram showing an example of the display positions of a composite image based on a forward virtual image and a composite image based on a side virtual image on the display device 12. Fig. 4 is an enlarged view of the front portion of the driver's seat 10 in the interior view of the vehicle shown in Fig. 2, with the steering wheel omitted. The dashed dotted line CL in the figure is a line projected onto the instrument panel 13 along the longitudinal direction of the vehicle 2 from the center line of the body of the driver D seated in the driver's seat 10.

[0029] 4, a front image range 17a, which is the display range of a composite image based on a front virtual image, is set to a rectangular range indicated by a dotted line in the figure and centered on line CL. Furthermore, when turning left, a left image range 17b, which is the display range of a composite image based on a side virtual image from a leftward viewpoint, is set to a position shifted to the left, which is the turning direction, relative to the front image range 17a. Furthermore, when turning right, a right image range 17c, which is the display range of a composite image based on a side virtual image from a rightward viewpoint, is set to a position shifted to the right, which is the turning direction, relative to the front image range 17a, near the bottom of the pillar 11a.

[0030] For example, the display control unit 28 displays the composite image on the display screen 15 of the display device 12 so that the position on the composite image corresponding to the direction of travel of the host vehicle is at the center of the display range of the composite image. This makes it easier for the driver D to intuitively recognize the traffic environment from the displayed composite image.

[0031] In addition, the composite image based on the forward virtual image, the composite image based on the lateral virtual image from the left viewpoint, and the composite image based on the lateral virtual image from the right viewpoint are each selectively displayed (i.e., when one composite image is displayed, the other composite images are not displayed), so the forward image range 17a, the left image range 17b, and the right image range 17c do not necessarily need to be separated from each other as shown in the figure, and some areas may overlap each other.

[0032] The display control unit 28 can, for example, switch between displaying the composite image when going straight, turning left, and turning right, in such a manner that the composite image moves between the forward image range 17a, the left image range 17b, and the right image range 17c.

[0033] In the display system 1 having the above configuration, the surrounding environment of the vehicle 2 is displayed as a composite image on the display device 12, so that the driver D can, for example, when turning at an intersection where there are many traffic situation check items, know the presence of pedestrians and the like in blind spots such as the pillar 11 on the screen of the display device 12, thereby reducing the driving burden. Also, the composite image displayed on the display device 12 shows the images of traffic participants embedded and synthesized as participant images, so that the presence of traffic participants such as pedestrians can be conveyed to the driver D in a realistic manner (i.e., in a realistic manner).

[0034] Furthermore, in the display system 1, the composite image is based on a three-dimensional virtual environment image that provides a bird's-eye view of the area around the current position of the vehicle 2. This makes it easier for the driver D to grasp the positional relationship between the vehicle and traffic participants in the surrounding environment, as well as the positional relationships between traffic participants, compared to a bird's-eye view in which the image synthesized from multiple camera images is prone to distortion. In other words, the display system 1 has the excellent effect of using a three-dimensional virtual environment image as a base to easily grasp the three-dimensional positional relationship of the traffic environment including the vehicle and traffic participants, and by embedding and synthesizing images of the traffic participants into the virtual environment image, it is possible to realistically convey to the driver D the presence of the traffic participants and details of their movements.

[0035] In addition, when the vehicle 2 is not about to turn, the display system 1 displays a composite image based on a forward virtual image corresponding to the driver D's line of sight ahead, and when the vehicle 2 is about to turn, it displays a composite image based on a side virtual image corresponding to the driver D's line of sight in the turning direction, so that the traffic conditions in the turning direction can be conveyed to the driver D using an image that is easy to understand intuitively and in line with the driver D's line of sight.

[0036] Furthermore, in display system 1, when host vehicle 2 is about to turn, the display position of the composite image on display device 12 is shifted toward the turning direction compared to the display position when host vehicle 2 is not about to turn, so that the position of the composite image on display device 12 moves in the direction in which driver D's head turns when turning. Therefore, display system 1 reduces the amount of line-of-sight movement required when driver D views the composite image, making it easier for driver D to intuitively recognize the traffic situation from the composite image.

[0037] Moreover, by presenting traffic participants as participant images, the driver D can easily correlate the participant images with the traffic participants existing in the real environment, and can easily recognize the traffic participants in the real space. Furthermore, the virtual environment image can be displayed without unnecessary information other than, for example, the position and dimensions of the intersection, lanes, and sidewalks, so the driver D can concentrate on the necessary information without being distracted by unnecessary information.

[0038] In the display system 1, the display control unit 28 may highlight the participant video of a traffic participant who is a pedestrian or a cyclist when displaying the composite image. The highlighting can be performed, for example, by displaying at least a part of the border of the participant video (i.e., the boundary with the virtual environment image) in a warm color, by increasing or changing (e.g., blinking) the brightness of the participant video compared to the surroundings, by strengthening the warm color tone of the participant video, or the like. As a result, the display system 1 can more reliably and realistically notify the driver D of the presence of pedestrians and bicycles that the driver D may easily overlook.

[0039] Furthermore, for traffic participants that are surrounding vehicles, the display control unit 28 may generate a composite image by fitting and compositing a virtual vehicle representation, which is a graphic representation according to the vehicle attributes of the surrounding vehicles detected by the vehicle detection unit 27, into a corresponding position on the virtual environment image as a surrounding vehicle display. For example, when the vehicle type indicated by the vehicle attributes is a truck, the display control unit 28 can generate a composite image by fitting and compositing a virtual vehicle representation of a truck stored in advance in the memory 21 into the virtual environment image using a color and size according to the color and size indicated by the vehicle attributes. As a result, in the display system 1, for vehicles among traffic participants whose detailed information (e.g., sense of speed, color, size, model) can be easily expressed using graphic representations, they are displayed using virtual vehicle representations, thereby reducing the processing load required to generate a composite image and output it to a display device.

[0040] FIG. 5 is a diagram showing an example of a composite image based on a side virtual image that the display control unit 28 displays on the display device 12 when the host vehicle 2 turns right at an intersection. The composite image 30 based on the side virtual image displays a virtual host vehicle representation 32 of the host vehicle 2, a participant video 33 of a pedestrian, and a surrounding vehicle display 34 of a surrounding vehicle approaching the host vehicle 2 as an oncoming vehicle, all in a three-dimensional virtual environment image 31 that provides a bird's-eye view of the surrounding environment at the current position of the host vehicle 2. In the example of FIG. 5, a default virtual vehicle representation is used for the surrounding vehicle display 34. In the composite image 30 shown in FIG. 5, for example, when the driver D of the host vehicle 2 touches the surrounding vehicle display 34, the display control unit 28 switches the surrounding vehicle display 34 from the virtual vehicle representation of the surrounding vehicle to the participant video, or vice versa, with each touch.

[0041] The display control unit 28 may also switch the surrounding vehicle display from the virtual vehicle representation to a participant video of a traffic participant corresponding to the virtual vehicle representation in response to a touch on a virtual vehicle representation, which is a surrounding vehicle display, on the composite image displayed on the touch panel display device 12. That is, the virtual vehicle representation may be set as the default surrounding vehicle display, and the surrounding vehicle display may be switched between the virtual vehicle representation and the participant video by a touch operation on the composite image displayed on the display device 12. For example, in the composite image 30 shown in FIG. 5 , when the driver D of the host vehicle 2 touches the surrounding vehicle display 34, the display control unit 28 switches the surrounding vehicle display 34 from the virtual vehicle representation of the surrounding vehicle to the participant video, or vice versa, with each touch. This allows the display system 1 to allow the driver D to switch the display of the surrounding vehicles to the participant video as needed, thereby obtaining more detailed visual information about the surrounding vehicles.

[0042] The display control unit 28 may also highlight a surrounding vehicle display (a virtual vehicle representation or participant image of the surrounding vehicle) corresponding to the surrounding vehicle in the composite image when there is a possibility that the surrounding vehicle may come into contact with the host vehicle, i.e., when the vehicle detection unit 27 determines that there is a possibility that the surrounding vehicle may come into contact with the host vehicle 2. As a result, the display system 1 can more reliably notify the driver D of the presence of a nearby vehicle with which there is a possibility of contact or collision.

[0043] The above-mentioned highlighting can be performed, similar to the highlighting of the participant images of pedestrian traffic participants described above, by, for example, displaying a warm-colored border on the surrounding vehicle display, increasing the brightness of the surrounding vehicle display compared to the surroundings or changing it over time, strengthening the warm-colored tone of the surrounding vehicle display, etc.

[0044] Next, the procedure of the operation of the display system 1 will be described. 6 is a flowchart showing the procedure of a display method executed by the processor 20, which is a computer of the display system 1, for displaying the surrounding environment of the vehicle 2. This process is executed repeatedly.

[0045] When the process starts, first, the position acquisition unit 23 acquires the current position of the host vehicle 2 (S100). Then, the turn determination unit 24 determines whether the driver D of the host vehicle 2 intends to turn (S102). Next, the environmental image generation unit 25 generates a virtual environmental image, which is a virtual image showing the surrounding environment of the host vehicle 2, depending on whether there is an intention to turn, based on the current position and map information of the host vehicle 2 and the result of the determination of whether there is an intention to turn (S104). Here, "generating a virtual environmental image depending on whether there is an intention to turn" means generating a front virtual image when the host vehicle 2 is not attempting to turn, and generating a side virtual image when the host vehicle 2 is attempting to turn.

[0046] Next, the partial image extraction unit 26 acquires a real-world image of the surroundings of the vehicle 2 from the on-board camera 3, and extracts a participant image, which is an image portion of traffic participants, from the real-world image (S106). Furthermore, the vehicle detection unit 27 detects the positions of surrounding vehicles, which are vehicles within the surrounding environment, and vehicle attributes, including the vehicle type, size, and / or color, from the real-world image (S108). At this time, the vehicle detection unit 27 may determine whether or not there is a possibility that the detected surrounding vehicles will come into contact with the vehicle.

[0047] Next, the display control unit 28 generates a composite image by fitting a surrounding vehicle display showing the detected surrounding vehicle and the extracted participant video of at least the pedestrian at corresponding positions on the virtual environment image (S110). Here, the surrounding vehicle display may be a virtual vehicle representation showing the surrounding vehicle by default.

[0048] Then, the display control unit 28 displays the generated composite image at a position on the display device 12 that corresponds to whether or not there is an intention to turn (S112). In particular, in this embodiment, when the display control unit 28 displays the composite image based on the side virtual image on the display device 12, it displays the composite image at a position shifted toward the turning direction from the position where the composite image based on the front virtual image is displayed.

[0049] In step S112, the display control unit 28 may highlight the video of the pedestrian or bicycle participant as described above. Also, when the vehicle detection unit 27 determines that the detected surrounding vehicle may come into contact with the host vehicle, the display control device may highlight the corresponding surrounding vehicle display.

[0050] Next, the display control unit 28 determines whether any of the surrounding vehicle displays on the composite image displayed on the touch panel display device 12 has been touched (S114). If any of the surrounding vehicle displays has been touched (S114, YES), the display control unit 28 switches the surrounding vehicle display between a virtual vehicle representation and a participant image of the surrounding vehicle (S116), and ends this process. On the other hand, if none of the surrounding vehicle displays has been touched (S114, NO), the display control device ends this process.

[0051] After this process is completed, the processor 20 returns to step S100 and repeats this process, displaying the composite image at the current time on the display device 12 in real time.

[0052] 5. Other Embodiments In the above-described embodiment, the display screen 15 of the display device 12 extends horizontally in the vehicle width direction, but this is not limited thereto. As shown in FIG. 7 , the display screen 15 of the display device 12 may be configured to include any number of display panels, for example, six display panels 151a, 151b, 151c, 151d, 151e, and 151f, arranged in the vehicle width direction on the instrument panel 13. Hereinafter, the display panels 151a, 151b, 151c, 151d, 151e, and 151f will be collectively referred to as display panels 151. When the host vehicle 2 is about to turn, during a right turn and / or a left turn, the display control unit 28 can display a composite image based on a side virtual image on another display panel 151 that is located closer to the turning direction than one display panel 151 that displays a composite image based on a front virtual image. This makes it possible to easily configure the display device 12 that can display a composite image based on a front virtual image and a composite image based on a side virtual image at positions shifted from each other.

[0053] 8, the display device 12 may include an additional display panel 152 disposed in front of at least one pillar 11 of the vehicle 2. When the vehicle 2 is about to turn, the display control unit 28 can display a composite image based on the side virtual image on the additional display panel 152 on the side of the turning direction when turning right and / or turning left. This can further reduce the amount of line-of-sight movement required for the driver D to view the composite image. The same applies to the configuration shown in FIG. 7.

[0054] Furthermore, when the host vehicle 2 is about to turn, the environmental image generation unit 25 may generate, as the side virtual image, a side virtual image including a virtual environment image corresponding to the line of sight of the driver D in the turning direction, which is obtained by widening the angle of view of the virtual environment image corresponding to the driver D's forward line of sight in the turning direction. This makes it possible to display a seamless virtual image that connects the forward line of sight to the side line of sight, similar to the view that the driver D can see as his or her line of sight moves, and therefore makes it possible to provide the driver D with a virtual image that is easier to intuitively recognize.

[0055] In the case of the side virtual image described above, the display control unit 28 displays a composite image based on the side virtual image on the display screen 15 of the display device 12 in a left image area 18b (a rectangular area indicated by a dashed line in the figure) or a right image area 18c (a rectangular area indicated by a dashed line in the figure), which is a display area obtained by expanding a front image area 17a (a dotted rectangle in the figure), which is a display area for a composite image based on a front virtual image, toward the turning direction, as shown in Fig. 9. This allows the side virtual image to be displayed in a display area expanded in the direction of the line of sight movement from the front to the side that the driver D makes when turning, making it possible to provide the driver D with a virtual image in a manner that is easier to intuitively recognize.

[0056] In addition, in the above-described embodiment, the real environment image is acquired from the camera 3 mounted on the vehicle 2, but it may also be acquired from a street light camera present in the surrounding environment via road-to-vehicle communication, etc. The real environment video may also be acquired from an on-board camera provided on a vehicle around the vehicle 2 by communication via a communication network or by vehicle-to-vehicle communication.

[0057] In the above-described embodiment, the display control unit 28 highlights the participant images of traffic participants such as pedestrians and cyclists, but it may also highlight only pedestrians who require special attention, such as children and elderly people. The highlighting may be in the form described above, or may be a blinking display or an enlarged display.

[0058] The display control unit 28 can cause the display device 12 to display a composite image based on a clear virtual environment image, regardless of environmental conditions, even when direct visibility is poor, such as at night or in rainy weather.

[0059] The camera 3 may be an infrared camera, which can notify the driver D of the presence of a pedestrian that cannot be seen with the naked eye in the dark by displaying the composite image.

[0060] In response to a touch at any position on the composite image displayed on the display device 12, the display control unit 28 may further embed and composite a partial image of the real environment image at the touched position onto the virtual environment image to generate a composite image.

[0061] The present invention is not limited to the configurations of the above-described embodiments, and can be implemented in various forms without departing from the spirit of the invention.

[0062] 6. Configurations Supported by the Above Embodiments The above-described embodiment supports the following configurations.

[0063] (Configuration 1) A display system comprising: a position acquisition unit that acquires the current position of the vehicle; an environmental image generation unit that generates a virtual environment image, which is a virtual image showing the environment surrounding the vehicle, based on the current position of the vehicle and map information; a partial image extraction unit that acquires real environment video around the vehicle and extracts participant video, which is a video portion of traffic participants, from the real environment video; a turn determination unit that determines whether the driver of the vehicle is attempting to turn; and a display control unit that generates a composite image by fitting each of the extracted participant videos into a corresponding position on the virtual environment image and displays the composite image on a display device, wherein the environmental image generation unit generates a front virtual image including the virtual environment image corresponding to the driver's line of sight toward the front when the vehicle is not attempting to turn, and generates a side virtual image including the virtual environment image corresponding to the driver's line of sight in the turning direction when the vehicle is attempting to turn, and the display control unit displays the composite image based on the front virtual image on the display device when the vehicle is not attempting to turn, and displays the composite image based on the side virtual image on the display device when the vehicle is attempting to turn. According to the display system of configuration 1, the viewpoint of the composite image displayed on the display device is changed according to the driving scene (whether or not the vehicle is about to turn), making it easier for the driver to intuitively recognize the traffic environment in the direction the vehicle is about to travel. In other words, according to the display system of configuration 1, the traffic environment around the vehicle can be conveyed to the driver in a manner that is easily recognized according to the actual driving scene.

[0064] (Configuration 2) The display control unit displays the composite image based on the side virtual image at a position on the display screen of the display device that is shifted toward the turning direction from the position at which the composite image based on the front virtual image is displayed. According to the display system of configuration 2, the display position of the composite image displayed on the display device is changed according to the driving scene, so the driver has to move their eyes less when viewing the composite image, making it easier for them to intuitively recognize the traffic situation in the direction they are heading.

[0065] (Configuration 3) A display system according to configuration 1 or 2, wherein the display control unit displays the composite image on the display screen of the display device so that the position on the composite image corresponding to the direction of travel of the vehicle is the center of the display range of the composite image. According to the display system of configuration 3, it becomes easier for the driver to intuitively recognize the traffic environment from the displayed synthetic image.

[0066] (Configuration 4) The display system according to configuration 2 or 3, wherein the display screen of the display device is a display panel provided on an instrument panel of the vehicle and extending horizontally in the vehicle width direction. According to the display system of configuration 4, the display position of the composite image based on the side virtual image can be easily shifted in the turning direction from the display position of the composite image based on the front virtual image.

[0067] (Configuration 5) The display system according to configuration 2 or 3, wherein the display screen of the display device is composed of a plurality of display panels arranged in the vehicle width direction on an instrument panel of the vehicle. According to the display system of configuration 5, it is possible to easily configure a display device that can display a composite image based on a front virtual image and a composite image based on a side virtual image at mutually shifted positions.

[0068] (Configuration 6) The display system according to configuration 4 or 5, wherein the display device includes an additional display panel disposed in front of a pillar of the vehicle. According to the display system of configuration 6, the composite image is displayed near the line of sight of the driver when turning, which further reduces the amount of line of sight movement required for the driver to view the composite image.

[0069] (Configuration 7) The display system described in Configuration 1, wherein the environmental image generation unit generates the lateral virtual image including the virtual environment image corresponding to the driver's line of sight in the turning direction, by widening the field of view of the virtual environment image corresponding to the driver's forward line of sight in the turning direction when the vehicle is about to turn. According to the display system of configuration 7, it is possible to display a seamless virtual image that connects from the forward line of sight to the side line of sight, similar to the view that the driver can see as the driver moves his or her line of sight, and therefore it is possible to provide the driver with a virtual image that is easier to intuitively recognize.

[0070] (Configuration 8) A display system as described in Configuration 7, wherein, when the vehicle is about to turn, the display control unit displays the composite image based on the side virtual image on the display screen of the display device in a range that is expanded toward the turning direction from the display range for the composite image based on the front virtual image. According to the display system of configuration 8, the side virtual image is displayed in a display range expanded in the direction of the driver's line of sight movement from the front to the side when turning, so that the virtual image can be provided to the driver in a manner that makes it easier for them to intuitively recognize.

[0071] (Configuration 9) A display system described in any of configurations 1 to 8, wherein the display control unit generates a composite image based on the virtual environment image at the current time and the participant image at a predetermined time interval, and displays the composite image at the current time on the display device in real time. According to the display system of configuration 9, a synthetic image is displayed on the display device at a position that follows the movement of the driver's line of sight in an ever-changing traffic environment, and the appearance and movement of traffic participants can be communicated to the driver in a manner that allows for easy spatial recognition and gives the driver a realistic sense of presence of the traffic participants.

[0072] (Configuration 10) A display system described in any of configurations 1 to 9, wherein the virtual environment image is an overhead image of the surrounding environment including the current position of the vehicle, and a virtual vehicle representation, which is a graphic representation of the vehicle, is superimposed at a position corresponding to the vehicle on the virtual environment image. According to the display system of configuration 10, the display system is based on a virtual environment image that provides a bird's-eye view of the area around the current position of the vehicle, including a virtual vehicle representation of the vehicle itself. This makes it easier for the driver to grasp the positional relationship between traffic participants and the vehicle itself, and the positional relationship between traffic participants, compared to a bird's-eye view synthesized from camera images, which is prone to image distortion.

[0073] (Configuration 11) A display method executed by a computer provided in a display system, comprising the steps of: acquiring the current position of the vehicle; generating a virtual environment image, which is a virtual image showing the environment surrounding the vehicle, based on the current position of the vehicle and map information; acquiring a real environment image around the vehicle and extracting participant images, which are image portions of traffic participants, from the real environment image; determining whether the driver of the vehicle is attempting to turn; and generating a composite image by fitting each of the extracted participant images into the corresponding positions on the virtual environment image and displaying the composite image on a display device; wherein in the generating step, when the vehicle is not attempting to turn, a forward virtual image is generated including the virtual environment image corresponding to the driver's forward line of sight, and when the vehicle is attempting to turn, a side virtual image is generated including the virtual environment image corresponding to the driver's line of sight in the turning direction; and in the displaying step, when the vehicle is not attempting to turn, the composite image based on the forward virtual image is displayed on the display device, and when the vehicle is attempting to turn, the composite image based on the side virtual image is displayed on the display device. According to the display method of configuration 11, the traffic environment can be easily grasped based on a virtual environment image, and the presence of traffic participants can be realistically conveyed to the driver through participant images. In addition, the traffic situation in the turning direction can be conveyed to the driver D through an intuitively easy-to-understand image that corresponds to the driver D's line of sight when turning. [Explanation of symbols]

[0074] 1...display system, 2...host vehicle, 3...camera, 3a...front camera, 3b...left side camera, 3c...right side camera, 4...object detection device, 5...vehicle monitoring device, 6...GNSS receiver, 7...navigation device, 10...driver's seat, 11, 11a, 11b...pillar, 12, 14...display device, 13...instrument panel, 15...display screen, 151, 151a, 151b, 151c, 151d, 151e, 151f...display panel, 152...additional display panel, 17a...forward image range, 17b, 18b...left image range, 17c, 18c...right image range, 20...processor, 21...memory, 22...display program, 23...position acquisition unit, 24...turn determination unit, 25...environment image generation unit, 26...partial image extraction unit, 27...vehicle detection unit, 28...display control unit, 30...synthetic image, 31...virtual environment image, 32...virtual vehicle representation, 33...participant image, 34...surrounding vehicle display, D...driver.

Claims

1. a position acquisition unit that acquires the current position of the vehicle; an environmental image generating unit that generates a virtual environmental image, which is a virtual image showing the surrounding environment of the vehicle, based on the current position of the vehicle and map information; a partial image extraction unit that acquires a real environment image around the vehicle and extracts a participant image, which is an image portion of a traffic participant, from the real environment image; a turn determination unit that determines whether the driver of the vehicle is attempting to turn; a display control unit that generates a composite image by fitting each of the extracted participant videos into the virtual environment image at a corresponding position on the virtual environment image and displays the composite image on a display device; Equipped with the environmental image generation unit generates a forward virtual image including the virtual environment image corresponding to the driver's line of sight ahead when the host vehicle is not about to turn, and generates a side virtual image including the virtual environment image corresponding to the driver's line of sight in the turning direction when the host vehicle is about to turn; the display control unit displays the composite image based on the front virtual image on the display device when the host vehicle is not about to turn, and displays the composite image based on the side virtual image on the display device when the host vehicle is about to turn. Display system.

2. the display control unit displays the composite image based on the side virtual image in a display range on a display screen of the display device that is shifted toward the turning direction from a display range of the composite image based on the front virtual image. The display system of claim 1 .

3. the display control unit displays the composite image on the display screen of the display device so that a position on the composite image corresponding to a destination in the traveling direction of the host vehicle is at the center of a display range of the composite image. The display system of claim 1 .

4. The display screen of the display device is a display panel provided on an instrument panel of the vehicle and extending horizontally in the vehicle width direction. The display system of claim 2 .

5. The display screen of the display device is composed of a plurality of display panels arranged in the vehicle width direction on an instrument panel of the host vehicle. The display system of claim 2 .

6. The display device includes an additional display panel disposed in front of a pillar of the host vehicle. The display system of claim 4 .

7. When the host vehicle is about to turn, the environmental image generation unit generates the side virtual image including the virtual environment image corresponding to the driver's line of sight in the turning direction, by widening the angle of view of the virtual environment image corresponding to the driver's line of sight in the turning direction. The display system of claim 1 .

8. and when the host vehicle is about to turn, the display control unit displays the composite image based on the side virtual image on the display screen of the display device in a range that is larger than a display range for the composite image based on the front virtual image toward the turning direction. The display system of claim 7.

9. the display control unit generates a composite image based on the virtual environment image and the participant image at a current time at a predetermined time interval, and displays the composite image at the current time on the display device in real time. The display system of claim 1 .

10. The virtual environment image is an image overlooking the surrounding environment including the current position of the vehicle, and a virtual vehicle representation, which is a graphic representation of the vehicle, is superimposed and displayed at a position on the virtual environment image corresponding to the vehicle. A display system according to any one of claims 1 to 9.

11. A display method executed by a computer included in a display system, acquiring a current position of the vehicle; generating a virtual environment image that is a virtual image showing an environment surrounding the vehicle based on a current position of the vehicle and map information; acquiring a real-world image of the surroundings of the vehicle and extracting a participant image, which is an image portion of a traffic participant, from the real-world image; determining whether a driver of the host vehicle is intending to turn; a step of generating a composite image by fitting each of the extracted participant images into the virtual environment image at a corresponding position on the virtual environment image, and displaying the composite image on a display device; and In the generating step, when the host vehicle is not about to turn, a forward virtual image including the virtual environment image corresponding to the driver's line of sight is generated, and when the host vehicle is about to turn, a side virtual image including the virtual environment image corresponding to the driver's line of sight in the turning direction is generated; In the displaying step, when the host vehicle is not about to turn, the composite image based on the front virtual image is displayed on the display device, and when the host vehicle is about to turn, the composite image based on the side virtual image is displayed on the display device. Display method.

Images