Display system and display method

Abstract

To convey the presence of a traffic participant in the periphery of an own vehicle to a driver realistically in a recognizable manner.SOLUTION: A display system includes: a location acquiring section that acquires a current location of an own vehicle; an environment image generating section that generates a virtual environment image, which is a virtual image showing a surrounding environment of the own vehicle, on the basis of the current location of the own vehicle and map information; a partial video image extracting section that acquires a real environment video image of a surrounding of the own vehicle and extracts a participant video image, which is a video image portion of a traffic participant from the real environment video image; and a display control section that generates and displays on a display device a composite image by inlaying each of the extracted participant video images into the virtual environment image at a corresponding position on the virtual environment image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a display system and a display method for displaying the surrounding environment of a vehicle. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transport systems that take into consideration vulnerable transport participants have become more active. To achieve this, we are focusing on research and development into preventive safety technologies to further improve road safety and convenience.

[0003] Patent Document 1 discloses an image receiving and displaying device that performs geometric transformation on an image captured by a camera installed outside the vehicle to convert it into an image as seen from a predetermined position outside the vehicle and displays it. In this image receiving and displaying device, an image portion of a predetermined object extracted from the image is replaced with an icon in the converted image, or the icon is displayed by combining it with a map image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-200819 Summary of the Invention [Problem to be solved by the invention]

[0005] In preventive safety technology, when providing information using a display device that complements the driver's perception in order to ensure the safe driving of the vehicle, a challenge is to inform the driver in an easily recognizable manner of the presence of traffic participants around the vehicle. In this regard, the technology described in Patent Document 1 only displays icons of traffic participants within a video or map image showing the surrounding environment, and is limited in conveying the real presence of traffic participants to the driver. In order to solve the above-mentioned problems, the present application aims to achieve preventive safety for the driving of a vehicle by eliminating information unnecessary for driving and simply displaying necessary information when conveying information about the surroundings of the vehicle to the driver using a display device, while conveying the presence of traffic participants in an easily recognizable and realistic manner, thereby contributing to the development of a sustainable transportation system. [Means for solving the problem]

[0006] One aspect of the present invention is a display system comprising: a position acquisition unit that acquires the current position of the vehicle; an environmental image generation unit that generates a virtual environment image, which is a virtual image showing the environment surrounding the vehicle, based on the current position of the vehicle and map information; a partial image extraction unit that acquires a real environment image around the vehicle and extracts a participant image, which is a video portion of traffic participants, from the real environment image; and a display control unit that generates a composite image by fitting each of the extracted participant images into the virtual environment image at a corresponding position on the virtual environment image and displaying the composite image on a display device. According to another aspect of the present invention, the display control unit highlights, in the composite image, participant images of the traffic participants who are pedestrians. According to another aspect of the present invention, a vehicle detection unit is provided that detects the positions of surrounding vehicles, which are vehicles within the surrounding environment, and vehicle attributes including vehicle type, size, and / or color from the real environment image, and the display control unit generates the composite image by fitting and synthesizing a virtual vehicle representation, which is a graphic representation corresponding to the vehicle attributes of the surrounding vehicles, into the corresponding position on the virtual environment image as a surrounding vehicle display for the traffic participants, thereby generating the composite image. According to another aspect of the present invention, the display device is a touch panel, and the display control unit, in response to a user's operation on the display device, moves the viewpoint of the composite image so that the position indicated by the operation is the center and displays it on the display device, and / or enlarges the composite image by a predetermined magnification and displays it on the display device. According to another aspect of the present invention, the vehicle detection unit determines whether or not there is a possibility that the surrounding vehicle will come into contact with the host vehicle, and when there is a possibility that the surrounding vehicle will come into contact with the host vehicle, the display control unit highlights the surrounding vehicle display corresponding to the surrounding vehicle in the composite image. According to another aspect of the present invention, the display control unit generates a composite image based on the virtual environment image at the current time and the participant image at a predetermined time interval, and displays the composite image at the current time on the display device in real time. According to another aspect of the present invention, the display device is disposed in front of a pillar on the driver's seat side of the vehicle. According to another aspect of the present invention, the virtual environment image is an image that provides an overhead view of the surrounding environment including the current position of the vehicle, and a virtual vehicle representation, which is a graphic representation of the vehicle, is superimposed at a position corresponding to the vehicle on the virtual environment image. Another aspect of the present invention is a display method executed by a computer provided in a display system, the display method comprising the steps of: acquiring the current position of the vehicle; generating a virtual environment image, which is a virtual image showing the environment surrounding the vehicle, based on the current position of the vehicle and map information; acquiring a real environment image around the vehicle and extracting participant images, which are image portions of traffic participants, from the real environment image; and generating a composite image by inserting each of the extracted participant images into the virtual environment image at a corresponding position on the virtual environment image and displaying the composite image on a display device. [Effects of the Invention]

[0007] According to the present invention, in a display system that displays the surrounding environment of a vehicle, information unnecessary for driving is eliminated, necessary information is displayed simply, and the presence of traffic participants, etc., can be communicated to the driver in an easily recognizable and realistic manner. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a vehicle equipped with a display system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of the interior of the vehicle. [Figure 3] FIG. 3 is a diagram showing the configuration of a display system according to one embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of a composite image displayed on a display device by the display system. [Figure 5] FIG. 5 is a diagram showing an example of a composite image before the viewpoint is moved, for explaining the movement of the viewpoint center of a composite image by a touch operation. [Figure 6] FIG. 6 is a diagram showing an example of a composite image after the viewpoint has been moved, for explaining the movement of the center of the viewpoint of the composite image by a touch operation. [Figure 7] FIG. 7 is a diagram showing an example of a composite image before viewpoint movement and enlargement, for explaining enlarged display of a composite image by a touch operation. [Figure 8] FIG. 8 is a diagram showing an example of a composite image after viewpoint movement and enlargement, for explaining enlarged display of a composite image by a touch operation. [Figure 9] FIG. 9 is a flowchart showing the steps of a display method executed by a processor of the display system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] Fig. 1 is a diagram showing an example of the configuration of a host vehicle 2, which is a vehicle equipped with a display system 1 according to one embodiment of the present invention, and Fig. 2 is a diagram showing an example of the configuration of the interior of the host vehicle 2. The display system 1 is installed in the host vehicle 2, and displays a virtual environment image, which is a virtual image of the surrounding environment of the host vehicle 2 (hereinafter also simply referred to as the surrounding environment), on a display device 12, to notify the driver D of the presence of traffic participants in the surrounding environment.

[0011] The host vehicle 2 is equipped with a front camera 3a that captures images of the surrounding environment in front of the host vehicle 2, and a left side camera 3b and a right side camera 3c that capture images of the left and right sides of the host vehicle 2. Hereinafter, the front camera 3a, left side camera 3b, and right side camera 3c will be collectively referred to as cameras 3. The front camera 3a is disposed, for example, near the front bumper, and the left side camera 3b and right side camera 3c are disposed, for example, on the left and right door mirrors. The host vehicle 2 may further be equipped with a rear camera (not shown) that captures images of the surrounding environment behind the vehicle. The host vehicle 2 is also equipped with an object detection device 4 that detects objects present in the surrounding environment. The object detection device 4 may be, for example, a radar, a sonar, and / or a lidar.

[0012] The vehicle 2 is further equipped with a vehicle monitoring device 5 that collects at least information on the vehicle's 2 traveling speed and information on the operation of its turn signals (not shown), a GNSS receiver 6 that receives location information on the vehicle's 2 current position from a GNSS satellite, and a navigation device 7 that provides route guidance using map information.

[0013] A display device 12 is disposed in the cabin of the vehicle 2 in front of a pillar 11a on the driver's seat 10 side, which is located on the right side in the vehicle width direction. The display device 12 is, for example, a touch panel. When the driver's seat 10 is located on the left side in the vehicle width direction, the display device 12 may be disposed in front of a pillar 11b on the left side (i.e., on the driver's seat side). Hereinafter, the pillars 11a and 11b will also be collectively referred to as pillars 11. Another display device 14 used by the navigation device 7 to display map information is provided at a central position in the vehicle width direction of an instrument panel 13 in front of the driver's seat 10.

[0014] FIG. 3 is a diagram showing the configuration of the display system 1. As shown in FIG. The display system 1 includes a processor 20 and a memory 21. The memory 21 is, for example, configured with a volatile and / or non-volatile semiconductor memory and / or a hard disk drive. The processor 20 is, for example, a computer including a CPU. The processor 20 may be configured to include a ROM in which a program is written, a RAM for temporarily storing data, and the like. The processor 20 includes, as functional elements or functional units, a position acquisition unit 23, an environmental image generation unit 25, a partial video extraction unit 26, a vehicle detection unit 27, and a display control unit 28.

[0015] These functional elements of the processor 20 are realized, for example, by the processor 20, which is a computer, executing a display program 22 stored in the memory 21. The display program 22 can be stored in any computer-readable storage medium. Alternatively, all or part of the functional elements of the processor 20 can be configured by hardware each including one or more electronic circuit components.

[0016] The position acquisition unit 23 receives position information from the GNSS receiver 6 and acquires the current position of the host vehicle 2 . The environment image generation unit 25 generates a virtual environment image, which is a virtual image showing the surrounding environment of the vehicle 2, based on the current position of the vehicle 2 and map information. The map information can be acquired, for example, from the navigation device 7. In this embodiment, the virtual environment image generated by the environment image generation unit 25 is, for example, a 3D image (stereoscopic display image) that provides a bird's-eye view of the surrounding environment including the current position of the vehicle.

[0017] The partial image extraction unit 26 acquires an image of the real environment around the vehicle 2 using the camera 3, and extracts a participant image, which is an image portion of a traffic participant, from the acquired real environment image. The vehicle detection unit 27 detects the positions of surrounding vehicles, which are vehicles within the surrounding environment, and vehicle attributes including vehicle type, size, and / or color from the real environment image. The size of the surrounding vehicles can be calculated, for example, according to conventional technology, based on the angle of view of the surrounding vehicles in the real environment image and the distance to the surrounding vehicles detected by the object detection device 4. Furthermore, the vehicle types of the surrounding vehicles can be identified, for example, according to conventional technology, by image matching with template images that indicate the sizes and shapes of various vehicle types, such as trucks, buses, cars, and motorcycles, pre-stored in the memory 21.

[0018] Furthermore, the vehicle detection unit 27 determines whether or not there is a possibility that the detected surrounding vehicle will come into contact with the host vehicle 2. For example, the vehicle detection unit 27 determines whether or not there is a possibility of contact, according to conventional technology, based on information about the speed of the surrounding vehicles, information about the lighting status of their turn signals, information about the traveling speed of the host vehicle 2, information about the operation of the turn signals, and / or information about the planned traveling route of the host vehicle 2. Here, the information about the speed of the surrounding vehicles and the information about the lighting status of their turn signals can be acquired from real-world video. The information about the traveling speed of the host vehicle 2 and information about the operation of the turn signals can be acquired from the vehicle monitoring device 5. Furthermore, information about the planned traveling route of the host vehicle 2 can be acquired from the navigation device 7.

[0019] The display control unit 28 generates a composite image by fitting each of the participant videos extracted by the partial video extraction unit 26 into the virtual environment image generated by the environment image generation unit 25 at the corresponding positions on the virtual environment image, and displays the composite image on the display device 12. For example, the display control unit 28 generates a composite image based on the virtual environment image and the participant video at the current time at predetermined time intervals, and displays the composite image at the current time on the display device 12 in real time.

[0020] The size of the participant image when it is embedded in the virtual environment image can be, for example, a size calculated for the traffic participant according to conventional technology and reduced to match the scale of the virtual environment image at the position where the participant image is embedded. The actual size of the traffic participant can be calculated based on the angle of view of the traffic participant in the real environment image and the distance to the traffic participant detected by the object detection device 4, similar to the size of the surrounding vehicles described above.

[0021] The display control unit 28 may further generate a composite image by superimposing a virtual vehicle representation, which is a graphic representation of the host vehicle 2 (or a graphic representation indicating the host vehicle 2), at a corresponding position on the virtual environment composite image. For example, the virtual vehicle representation is a graphic display that simulates the movement of the host vehicle as seen from behind, and the composite image may be a so-called chase view from a perspective that follows the host vehicle from behind.

[0022] In the display system 1 having the above configuration, the surrounding environment of the vehicle 2 is displayed as a composite image on the display device 12, so that the driver D can, for example, when turning at an intersection where there are many traffic situation check items, know the presence of pedestrians and the like in blind spots such as the pillar 11 on the screen of the display device 12, thereby reducing the driving burden. Also, the composite image displayed on the display device 12 shows the images of traffic participants embedded and synthesized as participant images, so that the presence of traffic participants such as pedestrians can be conveyed to the driver D in a realistic manner (i.e., in a realistic manner).

[0023] Furthermore, in the display system 1, the composite image is based on a three-dimensional virtual environment image that provides a bird's-eye view of the area around the current position of the vehicle 2. This makes it easier for the driver D to grasp the positional relationship between the vehicle and traffic participants in the surrounding environment, as well as the positional relationships between traffic participants, compared to a bird's-eye view in which the image synthesized from multiple camera images is prone to distortion. Furthermore, by using a virtual environment image, it is possible to eliminate information that is unnecessary for driving in the real world and simply display necessary information about the surrounding environment. Furthermore, in the display system 1, by embedding and synthesizing participant images into the virtual environment image, the presence of traffic participants that require consideration when driving can be communicated to the driver in an easily recognizable and realistic manner.

[0024] Moreover, by presenting traffic participants as participant images, the driver D can easily correlate the participant images with the traffic participants existing in the real environment, and can easily recognize the traffic participants in the real space. Furthermore, the virtual environment image can be displayed without unnecessary information other than, for example, the position and dimensions of the intersection, lanes, and sidewalks, so the driver D can concentrate on the necessary information without being distracted by unnecessary information.

[0025] In addition, in the display system 1, the display device 12 is arranged, for example, at the position of the pillar 11 on the side of the driver's seat 10, so that the driver D can obtain information from the composite image displayed on the display device 12 with minimal eye movement.

[0026] In the display system 1, the display control unit 28 may highlight the participant video of a traffic participant who is a pedestrian or a cyclist when displaying the composite image. The highlighting can be performed, for example, by displaying at least a part of the border of the participant video (i.e., the boundary with the virtual environment image) in a warm color, by increasing or changing (e.g., blinking) the brightness of the participant video compared to the surroundings, by strengthening the warm color tone of the participant video, or the like. As a result, the display system 1 can more reliably and realistically notify the driver D of the presence of pedestrians and bicycles that the driver D may easily overlook.

[0027] For surrounding vehicles that are traffic participants, a composite image may be generated by embedding and compositing participant images of the surrounding vehicles into the virtual environment image in the same manner as described above. However, in this embodiment, for traffic participants that are surrounding vehicles, the display control unit 28 embeds and composites a virtual vehicle representation, which is a graphic representation corresponding to the vehicle attributes of the surrounding vehicles detected by the vehicle detection unit 27, into a corresponding position on the virtual environment image as a surrounding vehicle display, to generate a composite image. For example, when the vehicle attribute indicates a truck, the display control unit 28 can embed and composite a virtual vehicle representation of a truck stored in advance in the memory 21 into the virtual environment image using a color and size corresponding to the color and size indicated by the vehicle attribute, to generate a composite image. As a result, in the display system 1, for vehicles among traffic participants whose detailed information (e.g., sense of speed, color, size, model) can be easily expressed using graphic representations, they are displayed using virtual vehicle representations, thereby reducing the processing load required to generate a composite image and output it to a display device. Whether the surrounding vehicles are displayed as virtual vehicle representations or participant images can be switched by, for example, a setting button or the like (not shown) displayed on the display device 14 by the display control unit .

[0028] Fig. 4 is a diagram showing an example of a composite image displayed by the display control unit 28 on the display device 12. Fig. 4 shows a composite image when the host vehicle 2 turns right at an intersection. The composite image 30 displayed on the display device 12 displays a three-dimensional virtual environment image 31 that provides an overhead view of the surrounding environment at the current position of the host vehicle 2, a virtual host vehicle representation 32 showing the host vehicle 2, a participant video 33 of a traffic participant who is a pedestrian, and a surrounding vehicle display 34 of a surrounding vehicle approaching the host vehicle 2 as an oncoming vehicle.

[0029] The display control unit 28 may also highlight a surrounding vehicle display (a virtual vehicle representation or participant image of the surrounding vehicle) corresponding to the surrounding vehicle in the composite image when there is a possibility that the surrounding vehicle may come into contact with the host vehicle, i.e., when the vehicle detection unit 27 determines that there is a possibility that the surrounding vehicle may come into contact with the host vehicle 2. As a result, the display system 1 can more reliably notify the driver D of the presence of a nearby vehicle with which there is a possibility of contact or collision.

[0030] The above-mentioned highlighting can be performed, similar to the highlighting of the participant images of pedestrian traffic participants described above, by, for example, displaying a warm-colored border on the surrounding vehicle display, increasing the brightness of the surrounding vehicle display compared to the surroundings or changing it over time, strengthening the warm-colored tone of the surrounding vehicle display, etc.

[0031] In response to a user's operation on display device 12, which is a touch panel, display control unit 28 also moves the viewpoint of the composite image so that the position indicated by the operation becomes the center, and displays the composite image on display device 12, and / or enlarges the composite image by a predetermined magnification and displays it on display device 12. The user's operation is, for example, a touch operation on display device 12, which is a touch panel. In response to a touch on a part of the displayed composite image, display control unit 28 moves the viewpoint of the composite image so that the touched position becomes the center, and displays it on display device 12, and / or enlarges the composite image by a predetermined magnification and displays it on display device 12.

[0032] As a result, in the display system 1, the driver D can freely change the center position and / or display magnification of the composite image as needed, thereby making it easier to grasp the surrounding environment.

[0033] 5 and 6 are diagrams showing an example of moving the viewpoint of a composite image display by touching the composite image. When position P1 of the star shown is tapped on composite image 30a shown in Fig. 5, composite image 30b is displayed with the viewpoint center position moved to position P1, as shown in Fig. 6. Note that returning to the original viewpoint center position can be performed, for example, by tapping a BACK button (not shown) superimposed on the composite image by display control unit 28.

[0034] 7 and 8 are diagrams showing an example of enlarging a composite image by touching the composite image. When position P2 of the star shown in the figure is double-tapped in composite image 30c shown in FIG. 7, the center of viewpoint is moved to position P2 and composite image 30d with an enlarged display magnification is displayed, as shown in FIG. 8. For example, display control unit 28 can repeat the above-mentioned movement of the center of viewpoint and magnification enlargement each time the displayed composite image is double-tapped. Returning to the original center of viewpoint and original display magnification can be performed, as described above, by, for example, tapping a BACK button that display control unit 28 displays superimposed on the composite image.

[0035] The user's operation on the display device 12 described above is not limited to a touch operation, but may be any operation. For example, the operation may be performed by operating a switch button (not shown) displayed on the display device 14.

[0036] Next, the procedure of the operation of the display system 1 will be described. 9 is a flowchart showing the procedure of a display method for displaying the surrounding environment of the vehicle 2, which is executed by the processor 20, which is the computer of the display system 1. This process is executed repeatedly.

[0037] When the process starts, first, the position acquisition unit 23 acquires the current position of the vehicle 2 (S100). Next, the environment image generation unit 25 generates a virtual environment image, which is a virtual image showing the surrounding environment of the vehicle 2, based on the current position of the vehicle 2 and map information (S104). The map information can be acquired from the navigation device 7, for example.

[0038] Next, the partial image extraction unit 26 acquires a real-world image of the surroundings of the vehicle 2 from the on-board camera 3, and extracts a participant image, which is an image portion of traffic participants, from the real-world image (S106). Furthermore, the vehicle detection unit 27 detects the positions of surrounding vehicles, which are vehicles within the surrounding environment, and vehicle attributes, including the vehicle type, size, and / or color, from the real-world image (S108). At this time, the vehicle detection unit 27 may determine whether or not there is a possibility that the detected surrounding vehicles will come into contact with the vehicle.

[0039] Then, the display control unit 28 generates a composite image by fitting the surrounding vehicle display showing the detected surrounding vehicles and the extracted participant video of at least the pedestrian into the corresponding positions on the virtual environment image (S110), displays the generated composite image on the display device 12 (S112), and ends this process.

[0040] After this process is completed, the processor 20 returns to step S100 and repeats this process, displaying the composite image at the current time on the display device 12 in real time. In parallel with this process, the display control unit 28 can move the center position of the viewpoint of the composite image and / or increase the display magnification of the composite image in response to touching of a part of the composite image displayed in step S112.

[0041] 5. Other Embodiments In the above-described embodiment, the real environment image is acquired from the camera 3 mounted on the vehicle 2, but it may also be acquired from a street light camera present in the surrounding environment via road-to-vehicle communication, etc. The real environment video may also be acquired from an on-board camera provided on a vehicle around the vehicle 2 by communication via a communication network or by vehicle-to-vehicle communication.

[0042] In the above-described embodiment, the display control unit 28 highlights the participant images of traffic participants such as pedestrians and cyclists, but it may also highlight only pedestrians who require special attention, such as children and elderly people. The highlighting may be in the form described above, or may be a blinking display or an enlarged display.

[0043] The display control unit 28 can cause the display device 12 to display a composite image based on a clear virtual environment image, regardless of environmental conditions, even when direct visibility is poor, such as at night or in rainy weather.

[0044] The camera 3 may be an infrared camera, which can notify the driver D of the presence of a pedestrian that cannot be seen with the naked eye in the dark by displaying the composite image.

[0045] In response to a touch at any position on the composite image displayed on the display device 12, the display control unit 28 may further embed and composite a partial image of the real environment image at the touched position onto the virtual environment image to generate a composite image.

[0046] The present invention is not limited to the configurations of the above-described embodiments, and can be implemented in various forms without departing from the spirit of the invention.

[0047] 6. Configurations Supported by the Above Embodiments The above-described embodiment supports the following configurations.

[0048] (Configuration 1) A display system comprising: a position acquisition unit that acquires the current position of the vehicle; an environmental image generation unit that generates a virtual environment image, which is a virtual image showing the environment around the vehicle, based on the current position of the vehicle and map information; a partial image extraction unit that acquires a real environment image around the vehicle and extracts a participant image, which is a video portion of traffic participants, from the real environment image; and a display control unit that generates a composite image by fitting each of the extracted participant images into the virtual environment image at a corresponding position on the virtual environment image and displaying the composite image on a display device. According to the display system of configuration 1, by using a virtual environment image, information that is unnecessary for driving in the real world is deleted, and necessary information is simply displayed, while by embedding and synthesizing participant images, the presence of traffic participants that should be taken into consideration when driving can be communicated to the driver in an easily recognizable and realistic manner.

[0049] (Configuration 2) The display system according to Configuration 1, wherein the display control unit highlights, in the composite image, a participant image of the traffic participant who is a pedestrian. The display system of configuration 2 can more reliably and realistically notify the driver of the presence of pedestrians, which are easily overlooked by the driver.

[0050] (Configuration 3) A display system as described in Configuration 1 or 2, which includes a vehicle detection unit that detects the positions of surrounding vehicles, which are vehicles within the surrounding environment, and vehicle attributes including vehicle type, size, and / or color from the real environment image, and the display control unit, for traffic participants that are surrounding vehicles, inserts and synthesizes a virtual vehicle representation, which is a graphic representation corresponding to the vehicle attributes of the surrounding vehicles, into the corresponding position on the virtual environment image as a surrounding vehicle display to generate the composite image. According to the display system of configuration 3, vehicles for which detailed information can be easily represented using graphic representations are displayed using virtual vehicle representations, thereby reducing the processing load required to generate composite images and output them to the display device.

[0051] (Configuration 4) A display system described in any one of configurations 1 to 3, wherein the display device is a touch panel, and the display control unit, in response to a user's operation on the display device, moves the viewpoint of the composite image so that the position indicated by the operation is the center and displays it on the display device, and / or enlarges the composite image by a predetermined magnification and displays it on the display device. According to the display system of configuration 4, the driver D can freely change the center position and / or display magnification of the composite image as needed, and can more easily grasp the surrounding environment.

[0052] (Configuration 5) A display system described in Configuration 3 or 4, wherein the vehicle detection unit determines whether or not there is a possibility that the surrounding vehicle will come into contact with the host vehicle, and when there is a possibility that the surrounding vehicle will come into contact with the host vehicle, the display control unit highlights the surrounding vehicle display corresponding to the surrounding vehicle in the composite image. According to the display system of configuration 5, the presence of a nearby vehicle with which there is a possibility of contact or collision can be more reliably notified to the driver.

[0053] (Configuration 6) A display system described in any of configurations 1 to 5, wherein the display control unit generates a composite image based on the virtual environment image at the current time and the participant image at a predetermined time interval, and displays the composite image at the current time on the display device in real time. According to the display system of configuration 6, the appearance and movement of traffic participants in an ever-changing traffic environment can be communicated to the driver in a manner that allows for easy spatial recognition and gives the traffic participants a realistic sense of presence.

[0054] (Configuration 7) The display system according to any one of configurations 1 to 6, wherein the display device is disposed in front of a pillar on the driver's seat side of the vehicle. According to the display system of configuration 7, the driver can obtain information from the composite image displayed on the display device with minimal eye movement.

[0055] (Configuration 8) A display system described in any of configurations 1 to 7, wherein the virtual environment image is an overhead image of the surrounding environment including the current position of the vehicle, and a virtual vehicle representation, which is a graphic representation of the vehicle, is superimposed at a position corresponding to the vehicle on the virtual environment image. According to the display system of configuration 8, the display system is based on a virtual environment image that provides a bird's-eye view of the area around the current position of the vehicle, including a virtual vehicle representation of the vehicle itself. This makes it easier for the driver to grasp the positional relationship between traffic participants and the vehicle itself, as well as the positional relationship between traffic participants, compared to a bird's-eye view in which the image synthesized from multiple camera images is prone to distortion.

[0056] (Configuration 9) A display method executed by a computer provided in a display system, comprising the steps of: acquiring the current position of the vehicle; generating a virtual environment image, which is a virtual image showing the surrounding environment of the vehicle, based on the current position of the vehicle and map information; acquiring a real environment image around the vehicle and extracting participant images, which are image portions of traffic participants, from the real environment image; and generating a composite image by inserting each of the extracted participant images into the corresponding positions on the virtual environment image and displaying the composite image on a display device. According to the display method of configuration 9, it is easy to grasp the three-dimensional positional relationship of the traffic environment including traffic participants based on the virtual environment image, and by embedding and synthesizing images of the traffic participants into the virtual environment image, it is possible to realistically convey to the driver the presence of the traffic participants and details of their movements. [Explanation of symbols]

[0057] 1...display system, 2...own vehicle, 3...camera, 3a...front camera, 3b...left side camera, 3c...right side camera, 4...object detection device, 5...vehicle monitoring device, 6...GNSS receiver, 7...navigation device, 10...driver's seat, 11, 11a, 11b...pillar, 12, 14...display device, 13...instrument panel, 20...processor, 21...memory, 22...display program, 23...position acquisition unit, 25...environment image generation unit, 26...partial image extraction unit, 27...vehicle detection unit, 28...display control unit, 30, 30a, 30b, 30c, 30d...synthetic image, 31...virtual environment image, 32...virtual own vehicle representation, 33...participant image, 34...surrounding vehicle display, D...driver, P1, P2...position.

Claims

1. a position acquisition unit that acquires the current position of the vehicle; an environmental image generating unit that generates a virtual environmental image, which is a virtual image showing the surrounding environment of the vehicle, based on the current position of the vehicle and map information; a partial image extraction unit that acquires a real environment image around the vehicle and extracts a participant image, which is an image portion of a traffic participant, from the real environment image; a display control unit that generates a composite image by fitting each of the extracted participant videos into the virtual environment image at a corresponding position on the virtual environment image and displays the composite image on a display device; A display system comprising:

2. the display control unit highlights the participant image of the traffic participant who is a pedestrian in the composite image. The display system of claim 1 .

3. a vehicle detection unit that detects the position of a surrounding vehicle that is a vehicle in the surrounding environment from the real environment image and vehicle attributes including a vehicle type, a size, and / or a color; and the display control unit generates the composite image by fitting a virtual vehicle representation, which is a graphic representation according to the vehicle attributes of the surrounding vehicle, into a corresponding position on the virtual environment image as a surrounding vehicle display, for the traffic participant that is the surrounding vehicle. The display system of claim 1 .

4. The display device is a touch panel, The display control unit, in response to a user operation on the display device, Displaying the composite image on the display device by moving the viewpoint of the composite image so that the position designated by the operation becomes the center, and / or the composite image is enlarged at a predetermined magnification and displayed on the display device; The display system of claim 1 .

5. the vehicle detection unit determines whether or not there is a possibility that the surrounding vehicle will come into contact with the host vehicle; the display control unit, when there is a possibility that the surrounding vehicle may come into contact with the host vehicle, highlights the surrounding vehicle display corresponding to the surrounding vehicle in the composite image. The display system of claim 3 .

6. the display control unit generates a composite image based on the virtual environment image and the participant image at a current time at a predetermined time interval, and displays the composite image at the current time on the display device in real time. The display system of claim 1 .

7. The display device is disposed in front of a pillar on the driver's seat side of the vehicle. The display system of claim 1 .

8. The virtual environment image is an image overlooking the surrounding environment including the current position of the vehicle, and a virtual vehicle representation, which is a graphic representation of the vehicle, is superimposed and displayed at a position on the virtual environment image corresponding to the vehicle. A display system according to any one of claims 1 to 7.

9. A display method executed by a computer included in a display system, acquiring a current position of the vehicle; generating a virtual environment image that is a virtual image showing an environment surrounding the vehicle based on a current position of the vehicle and map information; acquiring a real-world image of the surroundings of the vehicle and extracting a participant image, which is an image portion of a traffic participant, from the real-world image; a step of generating a composite image by fitting each of the extracted participant images into the virtual environment image at a corresponding position on the virtual environment image, and displaying the composite image on a display device; A display method having the following.

Images