Display system, mobile object, image processing device, and display device

Abstract

To improve the convenience of a technology for displaying images of an external area of a moving object.SOLUTION: A display system includes a display device provided in a vehicle and a control unit that causes the display device to display a marker corresponding to an object located to the right rear or left rear of the vehicle, and the object is at a distance from the vehicle that is less than a predetermined distance.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display system, a mobile object, an image processing device, and a display device. [Background technology]

[0002] Conventionally, there are known techniques for displaying an image of an external area of ​​a moving body such as a vehicle. For example, Patent Document 1 discloses a technique for controlling the power supply to a monitor that displays an image captured by a camera mounted on a vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-40113 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, there has been room for improvement in the technology for displaying images of the external area of ​​a moving object.

[0005] The present disclosure relates to an image processing device, an imaging device, a display device, and a display system that improve the convenience of technology for displaying an image of an area outside a moving object. [Means for solving the problem]

[0006] A display system according to one embodiment of the present disclosure includes a display device provided in a vehicle, and a control unit that causes the display device to display a marker corresponding to an object located to the right rear or left rear of the vehicle, the object being located at a distance from the vehicle that is less than a predetermined distance.

[0007] A moving object according to an embodiment of the present disclosure includes the above-described display system.

[0008] An image processing device according to one embodiment of the present disclosure is an image processing device mounted on a vehicle, and includes a control unit that causes a display device to display a marker corresponding to an object located to the right rear or left rear of the vehicle, the object being at a distance from the vehicle that is less than a predetermined distance.

[0009] A display device according to one embodiment of the present disclosure includes a communication unit that acquires an image of the area behind a vehicle, and a control unit that displays, on the image, a marker corresponding to an object located to the right or left rear of the vehicle and whose distance from the vehicle is less than a predetermined distance. [Effects of the Invention]

[0010] The image processing device, imaging device, display device, and display system according to an embodiment of the present disclosure improve the convenience of the technology for displaying an image of an area outside a moving object. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of a display system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a view of a vehicle equipped with a display system as seen from the left side. [Figure 3] FIG. 2 is a diagram schematically illustrating the appearance of the display device of FIG. [Figure 4] FIG. 2 is a diagram showing a first example of a first video. [Figure 5] 5 is a diagram showing a first example of a second image corresponding to the display area of ​​the first image in FIG. 4. FIG. [Figure 6] 10A and 10B are diagrams illustrating examples of third markers superimposed on a detection target. [Figure 7] FIG. 2 is a diagram showing a first example of a first marker and a second marker displayed around a detection target. [Figure 8] FIG. 10 is a diagram showing a second example of the first marker and the second marker displayed around the detection target. [Figure 9] FIG. 10 is a diagram showing a third example of first and second markers displayed around a detection target. [Figure 10] FIG. 10 is a diagram showing a second example of the first video. [Figure 11] 11 is a diagram showing a second example of a second image corresponding to the display area of ​​the first image in FIG. 10. FIG. [Figure 12] FIG. 10 is a diagram showing a third example of the first video. [Figure 13] 13 is a diagram showing a third example of the second image corresponding to the display area of ​​the first image in FIG. 12. FIG. [Figure 14] FIG. 10 is a diagram showing a fourth example of the first video. [Figure 15] 15 is a diagram showing a fourth example of the second image corresponding to the display area of ​​the first image in FIG. 14. FIG. [Figure 16] FIG. 10 is a diagram showing a fifth example of the first video. [Figure 17] 17 is a diagram showing a fifth example of the second image corresponding to the display area of ​​the first image in FIG. 16. FIG. [Figure 18] 17 is a diagram showing another example of the second image corresponding to the display area of ​​the first image in FIG. 16. FIG. [Figure 19] FIG. 10 is a diagram showing a sixth example of the first image. [Figure 20] 20 is a diagram showing a sixth example of the second image corresponding to the display area of ​​the first image in FIG. 19. FIG. [Figure 21] FIG. 10 is a diagram showing another example of the second image. [Figure 22] FIG. 10 is a diagram showing a first modified example of the sixth example of the second image. [Figure 23] FIG. 10 is a diagram showing a second modified example of the sixth example of the second image; [Figure 24] FIG. 10 is a diagram showing a third modified example of the sixth example of the second image. [Figure 25] FIG. 10 is a diagram showing a seventh example of the first image. [Figure 26] FIG. 26 is a diagram showing a seventh example of the second image corresponding to the display area of ​​the first image in FIG. 25. [Figure 27] FIG. 10 is a diagram showing an eighth example of the first image. [Figure 28] FIG. 2 is a diagram showing the positional relationship between a vehicle and an object to be detected. [Figure 29] FIG. 25 is a diagram showing an eighth example of the second image corresponding to the display area of ​​the first image in FIG. 24. [Figure 30] FIG. 13 is a diagram showing a modified example of the eighth example of the second image. [Figure 31] FIG. 31 is a diagram showing an example in which the display range of FIG. 30 is changed to a wide view display. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] (Display System) Referring to FIG. 1, a display system 10 according to one embodiment of the present invention will be described.

[0014] 1, the display system 10 includes an imaging device 20, an image processing device 30, and a display device 40. The imaging device 20 and the components of the display system 10 can transmit and receive information via, for example, a network 51. The network 51 may include, for example, a wireless network, a wired network, a CAN (Controller Area Network), or the like.

[0015] In other embodiments, some or all of the components of the display system 10 may be integrated into a single device. For example, the image processing device 30 may be built into the imaging device 20 or the display device 40.

[0016] As shown in FIG. 2 , the imaging device 20, the image processing device 30, and the display device 40 may be provided in a mobile object 50. In the present disclosure, the term "mobile object" may include, for example, vehicles, ships, and aircraft. Examples of vehicles include automobiles, industrial vehicles, railroad vehicles, utility vehicles, and fixed-wing aircraft that travel on runways. Examples of automobiles include passenger cars, trucks, buses, motorcycles, and trolleybuses. Examples of industrial vehicles include agricultural and construction vehicles. Examples of industrial vehicles include forklifts and golf carts. Examples of agricultural industrial vehicles include tractors, cultivators, transplanters, binders, combines, and lawn mowers. Examples of construction industrial vehicles include bulldozers, scrapers, excavators, crane trucks, dump trucks, and road rollers. Examples of vehicles include human-powered vehicles. The classification of vehicles is not limited to the above examples. For example, automobiles may include industrial vehicles that can travel on roads. The same vehicle may be included in multiple categories. Marine vehicles may include, for example, watercraft, boats, and tankers. Aircraft may include, for example, fixed-wing aircraft and rotary-wing aircraft.

[0017] The imaging device 20 is capable of capturing an image of the external area of ​​the moving body 50. The position of the imaging device 20 may be any position inside or outside the moving body 50. For example, as shown in FIG. 2, the imaging device 20 is located behind the moving body 50 and is capable of capturing an image of the external area behind the moving body 50. The position of the image processing device 30 may be any position within the moving body 50. The display device 40 is viewable by the subject 60. The position of the display device 40 may be any position within the moving body 50. For example, as shown in FIG. 2, the display device 40 is located in the dashboard of the moving body 50.

[0018] (imaging device) The imaging device 20 will now be described in detail. For example, as shown in FIG.

[0019] The imaging optical system 21 forms an image of a subject. For example, the imaging optical system 21 may include a diaphragm and one or more lenses.

[0020] The imaging element 22 has a plurality of pixels arranged two-dimensionally. The imaging element 22 may include, for example, a charge coupled device (CCD) imaging element or a complementary metal oxide semiconductor (CMOS) imaging element. The imaging element 22 can capture an image of a subject formed by the imaging optical system 21 and generate a captured image.

[0021] The communication unit 23 may include a communication interface capable of communicating with an external device. The communication unit 23 may be capable of transmitting and receiving information via a network 51. The external device may include, for example, the image processing device 30. The "communication interface" in this disclosure may include, for example, a physical connector and a wireless communication device. The physical connector may include an electrical connector compatible with transmission of electrical signals, an optical connector compatible with transmission of optical signals, and an electromagnetic connector compatible with transmission of electromagnetic waves. The electrical connector may include a connector compliant with IEC 60603, a connector compliant with the USB standard, a connector compatible with an RCA terminal, a connector compatible with an S terminal specified in EIAJ CP-1211A, a connector compatible with a D terminal specified in EIAJ RC-5237, a connector compliant with the HDMI (registered trademark) standard, and a connector compatible with a coaxial cable, including a BNC (British Naval Connector or Baby-series N Connector, etc.). The optical connector may include various connectors compliant with IEC 61754. The wireless communication device may include a wireless communication device compatible with various standards, including Bluetooth (registered trademark) and IEEE 802.11. The radio includes at least one antenna.

[0022] The control unit 24 includes one or more processors. In this disclosure, the term "processor" may include a dedicated processor specialized for a specific process and a general-purpose processor that executes a specific function by loading a specific program. Dedicated processors may include digital signal processors (DSPs) and application-specific integrated circuits (ASICs). Processors may include programmable logic devices (PLDs). PLDs may include field-programmable gate arrays (FPGAs). The control unit 24 may be either a system-on-a-chip (SoC) or a system-in-a-package (SiP) in which one or more processors work together.

[0023] The control unit 24 controls the overall operation of the imaging device 20. The control unit 24 may cause the imaging element 22 to generate captured images at an arbitrary frame rate. The frame rate may, for example, be approximately equal to the frame rate at which the display device 40 can display images. The control unit 24 may perform predetermined image processing on the generated captured images. The image processing may include, for example, exposure adjustment processing, white balance processing, and distortion correction processing. The control unit 24 outputs the captured images to the image processing device 30 via the communication unit 23. For example, the control unit 24 may sequentially output the captured images at the above-mentioned frame rate. Hereinafter, each captured image output at the above-mentioned frame rate will also be simply referred to as a frame. The multiple captured images output from the imaging device 20 will also be referred to as a first image. For example, if the frame rate is 60 fps (frames per second), 60 captured images per second are output as the first image.

[0024] (Image processing device) The image processing device 30 will be described in detail below. The image processing device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0025] The communication unit 31 may include a communication interface capable of communicating with various external devices. The external devices may include, for example, the imaging device 20, the display device 40, an ECU (Electronic Control Unit or Engine Control Unit) provided in the mobile object 50, a speed sensor, an acceleration sensor, a rotation angle sensor, a steering angle sensor, an engine rotation speed sensor, an accelerator sensor, a brake sensor, an illuminance sensor, a raindrop sensor, a mileage sensor, an obstacle detection device using millimeter-wave radar, ultrasonic sonar, or the like, an ETC (Electronic Toll Collection System) receiving device, a GPS (Global Positioning System) device, a navigation device, an Internet server, a mobile phone, and the like.

[0026] The communication unit 31 may include a communication interface for pedestrian-to-vehicle communication, road-to-vehicle communication, and vehicle-to-vehicle communication. The communication unit 31 may include a receiver compatible with optical beacons of DSRC (Dedicated Short-Range Communication) and VICS (Vehicle Information and Communication System) provided in Japan. The communication unit 31 may include a receiver compatible with road traffic information systems of other countries.

[0027] The communication unit 31 may be capable of acquiring various types of information from external devices. For example, the communication unit 31 may be capable of acquiring mobile object information and environmental information.

[0028] The mobile object information may include any information related to the mobile object 50. The mobile object information may include, for example, the speed, acceleration, turning gravity, inclination, direction, and turning status of the mobile object 50, the steering angle of the steering wheel, the temperature of the coolant, the remaining amount of fuel, the remaining amount of battery, the battery voltage, the engine speed, the gear position, whether or not a reverse signal is present, whether or not the accelerator is operated, the accelerator opening, whether or not the brakes are operated, the brake pedal depression amount, whether or not the parking brake is activated, the difference in the rotation speeds of the front and rear wheels or all four wheels, the tire pressure, the amount of expansion and contraction of the dampers, the spatial position of the driver's eyes, the number and seat positions of occupants, seat belt fastening information, whether the doors are open or closed, whether or not the windows are open or closed, the temperature inside the vehicle, whether or not the air conditioning is operating, the set temperature of the air conditioning, the air conditioning air volume, the outside air circulation setting, the operation status of the wipers, the driving mode, connection information with external devices, the current time, the average fuel consumption, the instantaneous fuel consumption, the lighting status of various lamps, the location information of the mobile object 50, and route information to the destination of the mobile object 50. The various lamps may include, for example, headlamps, fog lamps, backup lamps, position lamps, and turn signals.

[0029] The environmental information may include any information related to the external environment of the mobile object 50. The environmental information may include, for example, the brightness around the mobile object 50, the weather, the atmospheric pressure, the outside temperature, map information, traffic information, road construction information, temporary changes in the speed limit of the road, objects detected by other vehicles, and the lighting status of traffic lights.

[0030] The storage unit 32 may include a primary storage device and a secondary storage device. The storage unit 32 may be configured using, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape, etc. The optical memory may include, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), and a BD (Blu-ray Disc) (registered trademark), etc. The storage unit 32 stores various information and programs necessary for the operation of the image processing device 30.

[0031] The control unit 33 includes one or more processors and controls the overall operation of the image processing device 30.

[0032] The control unit 33 may acquire moving object information and environmental information from an external device via the communication unit 31. The control unit 33 may determine a predicted path of the moving object 50, for example, based on the moving object information. Hereinafter, the predicted path of the moving object 50 will also be referred to as a first predicted path.

[0033] The control unit 33 may acquire the first image from the imaging device 20 via the communication unit 31. The first image includes a detection area and a display area.

[0034] The control unit 33 may detect at least a portion of the detection target within a detection area on the acquired first image. The detection area on the first image may be at least a portion of an area on a captured image, which is each frame of the first image. Each frame of the first image may be referred to as a captured image. The detection area on the first image may be larger than the display area. The detection area on the first image may encompass the display area. The control unit 33 may detect the detection target inside the display area. The control unit 33 may detect the detection target outside the display area but inside the detection area. The area inside the detection area and the display area may be referred to as a first area. The area inside the detection area but outside the display area may be referred to as a second area.

[0035] The detection target may include multiple types of objects. The types of objects may include, for example, people, other moving objects, roads, lanes, white lines, gutters, sidewalks, crosswalks, road signs, traffic signs, guardrails, walls, and traffic lights. The types of detection targets that the control unit 33 can detect are not limited to these. For example, when a portion of the detection target in the first video is hidden behind another object, at least a portion of the detection target may include a portion of the detection target that is not hidden behind the other object. For example, when the lower half of a pedestrian's body is hidden behind an obstacle in the first video, the control unit 33 may be able to detect the upper half of the pedestrian. Any object detection algorithm may be employed to detect at least a portion of the detection target. For example, the control unit 33 may detect at least a portion of the detection target using an algorithm such as pattern matching or feature point extraction using captured images that are each frame of the first video.

[0036] When at least a portion of the detection target is detected in the first video, the control unit 33 may determine a predicted path of the detection target based on the first video. Hereinafter, the predicted path of the detection target will also be referred to as a second predicted path. Any algorithm may be employed to determine the second predicted path. For example, the control unit 33 may determine the second predicted path based on changes in the orientation and position of the detection target in the captured image, which is each frame of the first video.

[0037] When at least a portion of the detection target is detected in the first video, the control unit 33 may estimate the relative positional relationship between the moving body 50 and the detection target based on the first video. The relative positional relationship may include, for example, the distance between the moving body 50 and the detection target, and whether or not the first predicted path of the moving body 50 overlaps with the second predicted path of the detection target. Any algorithm may be employed to estimate the distance between the moving body 50 and the detection target. For example, the control unit 33 may estimate the distance between the moving body 50 and the detection target by a motion stereo method using captured images that are each frame of the first video signal. In another embodiment, the control unit 33 may acquire information indicating the relative positional relationship between the moving body 50 and the detection target from an external device via the communication unit 31.

[0038] When the distance between the moving body 50 and the detection target decreases, the control unit 33 may determine which of the moving body 50 and the detection target contributes more to the decrease in distance. Any algorithm may be employed to determine the contributions of the moving body 50 and the detection target to the decrease in distance. In one example, the control unit 33 may detect the movement speed of the moving body 50 based on the moving body information. The control unit 33 may detect the movement speed of the detection target based on, for example, a change in the position of the detection target in a captured image, which is each frame of the first video. The control unit 33 may determine that the one of the moving body 50 and the detection target, which has a higher movement speed, contributes more to the decrease in distance. In another example, when the movement speed of the moving body 50 is less than a reference value, the control unit 33 may determine that the detection target contributes more to the decrease in distance. When the movement speed of the moving body 50 is equal to or greater than a reference value, the control unit 33 may determine that the moving body 50 contributes more to the decrease in distance. The reference value may be set arbitrarily, for example, to approximately zero. The operation of the image processing device 30 according to the contribution of the moving body 50 and the detection target to the decrease in distance will be described in detail later.

[0039] When at least a portion of the detection target is detected on the first video, the control unit 33 may determine, based on the first video, whether or not there is a possibility that the moving object 50 will come into contact with the detection target. Any algorithm may be employed to determine the possibility that the moving object 50 will come into contact with the detection target. For example, the control unit 33 may determine that there is a possibility that the moving object 50 will come into contact with the detection target when at least one of the following conditions is satisfied: the distance between the moving object 50 and the detection target is less than a predetermined threshold; and the rate at which the distance is decreasing is equal to or greater than a predetermined threshold. Details of the operation of the image processing device 30 depending on whether or not there is such a possibility will be described later.

[0040] The control unit 33 may cause the display device 40 to display a second image corresponding to the display area on the first image acquired from the imaging device 20. Specifically, the control unit 33 may output the second image to the display device 40 via the communication unit 31. For example, when the control unit 33 detects that the mobile object 50 is moving backward based on the mobile object information, the control unit 33 may cause the display device 40 to display the second image. For example, the control unit 33 may detect moving backward based on the shift position of the transmission gear. For example, the control unit 33 may detect moving backward based on a reverse signal output from the mobile object when moving backward. The second image may be, for example, an image obtained by clipping a display area on a captured image, which is each frame of the first image. The display area on the first image may be at least a portion of the captured image, which is each frame of the first image. The display area may be smaller than the detection area. The display area may be included in the detection area. The position, shape, and size of the display area may be determined arbitrarily. The control unit 33 may change the position, shape, and size of the display area. The position, shape, and size of the display area may be changed so that, for example, the display area and the detection area substantially coincide with each other.

[0041] The control unit 33 may composite various markers into the second image and display them on the display device 40. Compositing includes overwriting and mixing. The markers may include, for example, one or more images. The control unit 33 may dynamically change the display mode of at least a portion of the markers superimposed on the second image. The display mode may include, for example, the position, size, shape, color, and shade of at least a portion of the marker on the second image. When displaying a marker corresponding to a detection target detected on the first image, the control unit 33 may determine the display mode of the marker depending on the type of the detection target. Details of the operation of the image processing device 30 that displays various markers on the display device 40 will be described later.

[0042] (display device) The display device 40 will be described in detail. The display device 40 may include, for example, a liquid crystal display and an organic EL (Electroluminescence) display. The display device 40 may display, for example, a second image input from the image processing device 30 via a network 51. The display device 40 may function as a touch screen capable of accepting user operations. The display device 40 may include switches and keys capable of accepting user operations. The switches may include physical switches and electronic switches. The keys may include physical keys and electronic keys. When the display device 40 accepts a user operation on the display device itself, it may transmit a user input based on the operation to the image processing device 30.

[0043] The display device 40 can be arranged in various locations on the vehicle 50. FIGS. 3A to 3E are diagrams illustrating examples of arrangements of the display device 40 when the vehicle 50 is a vehicle. FIG. 3A illustrates an in-dashboard display device 40a housed within the vehicle's dashboard. FIG. 3B illustrates an on-dashboard display device 40b located on the dashboard. In FIG. 3B, the display device 40b is built into the vehicle 50. The display device 40b may be detachably attached to the dashboard. FIG. 3C illustrates a display device 40c built into a rearview mirror and capable of displaying images as needed. FIG. 4D illustrates a display device 40d built into an instrument panel. In FIG. 4D, the display device 40d is located adjacent to instruments such as a speedometer and a tachometer. In one embodiment, the entire instrument panel may be used as the display device 40d, such as an LCD, and a second image may be displayed on the display device 40e along with images of the speedometer, tachometer, and the like. FIG. 4E illustrates a display device 40e using a portable information terminal, such as a tablet terminal. The display device 40e may be a mobile phone display.

[0044] 4 to 18, the second video and various markers that the image processing device 30 causes the display device 40 to display will be specifically described. In this disclosure, the terms "up and down direction" and "left and right direction" for a video or an image correspond to two-dimensional directions in the video or the image. In this disclosure, the terms "height direction," "horizontal direction," and "depth direction" for a video or an image correspond to three-dimensional directions in the space in which the video or the image is displayed.

[0045] (1st example (reference example)) 4 shows a first example of a detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. In the example shown in FIG. 4, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63 and a vehicle 64 that appear inside the display area 62 in the first image.

[0046] The control unit 33 determines whether one or more conditions are satisfied based on the relative positional relationship between the moving object 50 and the detection target detected inside the display area 62 on the first image. The one or more conditions may include, for example, a first condition that the detection target is located within the first predicted path 65 of the moving object 50. The one or more conditions may include, for example, a second condition that at least a portion of the first predicted path 65 of the moving object 50 overlaps with at least a portion of the second predicted path of the detection target. If it is determined that the one or more conditions are satisfied, the control unit 33 may cause the display device 40 to display a predetermined marker corresponding to the detection target superimposed on the second image. The predetermined marker may include a first marker, a second marker, and a third marker.

[0047] In a first example, the control unit 33 may determine that the one or more conditions are satisfied for the pedestrian 63. In such a case, the control unit 33 may display a marker corresponding to the pedestrian 63. The control unit 33 may determine that the one or more conditions are not satisfied for the vehicle 64. In such a case, the control unit 33 does not display a marker corresponding to the vehicle 64.

[0048] Fig. 5 shows an example of the second image corresponding to display area 62 of the first image shown in Fig. 4. When the aspect ratio of display area 62 of the first image differs from the aspect ratio of the screen of display device 40, control unit 33 may crop display area 62 of the first image, and then output the second image that has been transformed to match the aspect ratio of the screen of display device 40 to display device 40. As shown in Fig. 5, a pedestrian 63 and a vehicle 64 are shown in the second image.

[0049] The control unit 33 may, for example, superimpose guide lines 66 indicating at least a part of the first predicted path 65 of the moving object 50 shown in Fig. 4 on the second image and display the same on the display device 40. The control unit 33 may dynamically change the guide lines 66 in accordance with, for example, a change in the steering angle of the steering wheel.

[0050] The first image has a larger range than the display area 62. The control unit 33 may change the range of the display area 62. The control unit 33 may superimpose an icon image 67 on the second image and display it on the display device 40. For example, the outline 67a of the icon image 67 shown in FIG. 5 indicates the maximum range when the range of the display area 62 is changed. The white rectangle 67b of the icon image 67 indicates the display area 62. The icon image 67 shown in FIG. 5 indicates the relative position and size of the display area 62 with respect to the maximum range of the display area 62.

[0051] FIG. 6 shows an example of a marker superimposed on a pedestrian 63 in the second image. Hereinafter, this marker will also be referred to as a third marker 68. A contour 69 of the third marker 68 may substantially match the contour of the pedestrian 63 detected in the second image. A region 70 inside the contour 69 of the third marker 68 may be filled with a color or pattern corresponding to, for example, a "person" type of detection target. When a pedestrian 63 is detected in the first image, the control unit 33 may superimpose the third marker 68 on the pedestrian 63 in the second image and display the same on the display device 40. This configuration allows the target person 60 to easily recognize the pedestrian 63 in the second image. The control unit 33 may hide the third marker 68 after a predetermined time has elapsed since the third marker 68 was displayed.

[0052] 7 shows an example of two types of markers superimposed around pedestrian 63 on the second image. Hereinafter, these two types of markers will also be referred to as first marker 71 and second marker 72. For example, control unit 33 may hide third marker 68, and then cause display device 40 to display first marker 71 and second marker 72 superimposed on the second image.

[0053] The control unit 33 may move the position of the first marker 71 to follow the pedestrian 63 on the second image. Since the first marker 71 follows the pedestrian 63, the target person 60 can easily catch the pedestrian 63. The first marker 71 is displayed around the pedestrian 63 at a distance from the pedestrian 63. When the first marker 71 is displayed on the display device 40, the target person 60 can easily understand the behavior of the pedestrian 63. The control unit 33 may change the superimposed position of the second marker 72 relative to the superimposed position of the first marker 71 on the second image. The control unit 33 may move the second marker relatively based on the position of the first marker 71.

[0054] For example, when the distance between the moving object 50 and the pedestrian 63 is decreasing, the control unit 33 may determine that the moving object 50 has made a large contribution to the decrease in the distance. In such a case, the control unit 33 may move the second marker 72 toward the first marker 71. First, the control unit 33 displays the second marker 72 at a position away from the first marker 71. Next, the control unit 33 moves the second marker 72 toward the first marker 71 until the distance between the second marker 72 and the first marker 71 reaches a predetermined distance. Next, the control unit 33 erases the second marker 72. Next, the control unit 33 displays the second marker 72 again at a position away from the first marker 71, and repeats the above-mentioned process. In this example, the second marker 72 is moving toward the object, the first marker 71, and the subject 60 may understand that the second marker 72 is approaching the first marker 71.

[0055] For example, when the distance between the moving object 50 and the pedestrian 63 is decreasing, the control unit 33 may determine that the pedestrian 63 has made a significant contribution to the decrease in distance. In such a case, the control unit 33 may move the second marker 72 away from the first marker 71. First, the control unit 33 displays the second marker 72 at a position close to the first marker 71. Next, the control unit 33 moves the second marker 72 away from the first marker 71 until the distance between the second marker 72 and the first marker 71 reaches a predetermined distance. Next, the control unit 33 erases the second marker 72. Next, the control unit 33 displays the second marker 72 again at a position close to the first marker 71, and repeats the above-described process. In this example, the second marker 72 has moved away from the object, the first marker 71, so the subject 60 may understand that the second marker 72 is moving away toward the first marker 71.

[0056] The control unit 33 changes the movement direction of the second marker 72 relative to the first marker 71 depending on the degree of contribution of the moving body 50 and the pedestrian 63 to the decrease in the distance between the moving body 50 and the pedestrian 63. Depending on the movement direction of the second marker 72, the subject 60 can identify, for example, whether the moving body 50 is approaching the pedestrian 63 or the pedestrian 63 is approaching the moving body 50.

[0057] The control unit 33 may repeatedly enlarge or reduce the second marker 72 around the first marker 71 on the second image as a center, for example, as shown in Fig. 8. The control unit 33 may superimpose the first marker 71 and the second marker 72 having a shape similar to the outline 69 of the pedestrian 63 on the second image as shown in Fig. 9. The control unit 33 may repeatedly enlarge or reduce the second marker 72 around the first marker 71 as a center. The control unit 33 switches between enlarging and reducing the second marker 72 depending on the degree of contribution of the moving object 50 and the pedestrian 63 to the reduction in the distance between the moving object 50 and the pedestrian 63.

[0058] When the distance between the moving body 50 and the detection target displaying the first marker 71 and the second marker 72 becomes less than a predetermined threshold, the control unit 33 may superimpose a new marker on the second image. Hereinafter, the new marker is also referred to as a fourth marker. The fourth marker may include any image. For example, the fourth marker may include an image showing an exclamation point (!). With this configuration, for example, when the pedestrian 63 displaying the first marker 71 and the second marker 72 approaches the moving body 50 by a certain amount or more, the fourth marker is superimposed and displayed on the second image. The fourth marker allows the target 60 to identify, for example, that the pedestrian 63 is located near the moving body 50. In another embodiment, when the distance between the moving body 50 and the detection target displaying the first marker 71 and the second marker 72 becomes less than a predetermined threshold, the control unit 33 may change the display mode of the first marker 71 and the second marker 72. The control unit 33 may, for example, change the colors of the first marker 71 and the second marker 72. Even with this configuration, the target person 60 can identify that, for example, a pedestrian 63 is located near the moving object 50 by the change in the color of the markers.

[0059] The control unit 33 may detect two detection targets lined up front and behind each other in the depth direction. The control unit 33 may display a first marker 71 and a second marker 72 on each of the two detection targets located in front and behind. The control unit 33 may assign different first markers 71 and second markers 72 to the two detection targets. For example, the control unit 33 may assign first markers 71 and second markers 72 to the first detection target located in the back that are less noticeable than the first markers 71 and second markers 72 assigned to the second detection target located in the front. For example, the control unit 33 may change the first markers 71 and second markers 72 assigned to the first detection target located in the back by darkening the color, increasing the transmittance, or making the lines thinner than the first markers 71 and second markers 72 assigned to the second detection target located in the front.

[0060] (Example 2-4 (reference example)) FIG. 10 shows a second example of a detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. In the example shown in FIG. 10, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63a that appears inside the display area 62 in the first image, and pedestrians 63b and 63c that appear outside the display area 62 but inside the detection area 61. The processing of the control unit 33 regarding the pedestrian 63a is similar to the processing regarding the pedestrian 63 shown in FIG. 4, for example.

[0061] When the detection position on the first image where the detection target is detected is outside the display area 62 but inside the detection area 61, the control unit 33 may superimpose a marker corresponding to the detection target on the second image and display it on the display device 40. Hereinafter, the marker will also be referred to as a fifth marker. The control unit 33 may display the fifth marker when it is determined that there is a possibility of contact between the moving object 50 and the detection target. When the detection position on the first image where the detection target is detected is to the right of the display area 62, the control unit 33 may superimpose the fifth marker on the right edge of the second image and display it on the display device 40. When the detection position on the first image where the detection target is detected is to the left of the display area 62, the control unit 33 may superimpose the fifth marker on the left edge of the second image and display it on the display device 40.

[0062] In the second example, the detection position where pedestrian 63b is detected is to the right of display region 62. Controller 33 may determine that there is a possibility of contact between moving body 50 and pedestrian 63b. In such a case, controller 33 may cause display device 40 to display a fifth marker corresponding to pedestrian 63b superimposed on the right edge of the second image. Details of the fifth marker corresponding to pedestrian 63b will be described later. The detection position where pedestrian 63c is detected is to the left of display region 62. Controller 33 may determine that there is no possibility of contact between moving body 50 and pedestrian 63c. In such a case, controller 33 may not cause display device 40 to display the fifth marker corresponding to pedestrian 63b.

[0063] Fig. 11 shows an example of the second image corresponding to display area 62 of the first image shown in Fig. 10. As shown in Fig. 11, pedestrian 63a appears in the second image. Pedestrians 63b and 63c do not appear in the second image.

[0064] As shown in FIG. 11 , the control unit 33 may superimpose an obstacle image 74 on the second video and display it on the display device 40. The obstacle image 57 indicates the detection result of an obstacle detection device using an ultrasonic sonar or the like provided on the moving object 50. The obstacle image 74 may include an image 74a, an image 74b, and an image 74c. The image 74a is an image obtained by viewing the moving object 50 from above. The image 74b is an image indicating that an obstacle has been detected to the left rear of the moving object 50. The image 74c is an image indicating that an obstacle has been detected to the right rear of the moving object 50. The detection result of the obstacle detection device and the detection result of the detection target by the control unit 33 do not necessarily have to match. For example, in the example shown in FIG. 11 , the obstacle image 74 indicates that obstacles have been detected to the right rear and left rear of the moving object 50. On the other hand, the control unit 33 may determine that there is no possibility of contact between the moving object 50 and the pedestrian 63c present to the left rear of the moving object 50. In such a case, the control unit 33 does not need to display the fifth marker corresponding to the pedestrian 63c.

[0065] FIG. 11 shows an example of a fifth marker 73 corresponding to a pedestrian 63b. The fifth marker 73 may include an icon image 73a and a band image 73b. The icon image 73a may be an image corresponding to a "person" type of detection target. The subject 60 who views the icon image 73a can recognize that a person is present to the right of the second image. The band image 73b is, for example, a band-shaped image extending in the vertical direction on the second image. The band image 73b may be filled with, for example, a color or pattern corresponding to a "person" type of detection target. The control unit 33 may move the band image 73b within a right edge region 73c on the second image. The control unit 33 may change the movement speed and width of the band image 73b.

[0066] The fifth marker 73 will be described in detail. The control unit 33 may determine the width of the band image 73b according to the distance between the moving object 50 and the pedestrian 63b. For example, the control unit 33 may increase the width of the band image 73b as the distance decreases. The subject 60 who views the band image 73b can recognize the distance between the moving object 50 and the pedestrian 63b based on the width of the band image 73b.

[0067] The control unit 33 may determine that, for example, the contribution of the moving object 50 to the decrease in the distance between the moving object 50 and the pedestrian 63b is large. In this case, the control unit 33 repeatedly moves the band image 73b in a first direction within the right edge region 73c of the second image. The first direction may be, for example, a direction from the outside to the inside in the left-right direction of the second image. The control unit 33 may determine that, for example, the contribution of the pedestrian 63b to the decrease in the distance between the moving object 50 and the pedestrian 63b is large. In this case, the control unit 33 repeatedly moves the band image 73b in a second direction within the right edge region 73c of the second image. The second direction may be, for example, a direction from the inside to the outside in the left-right direction of the second image. The subject 60 who views the band image 73b can recognize whether the moving object 50 is approaching the pedestrian 63b or the pedestrian 63b is approaching the moving object 50 based on the movement direction of the band image 73b.

[0068] The control unit 33 may determine the movement speed of the band image 73b according to the rate at which the distance between the moving object 50 and the pedestrian 63b is decreasing. For example, the faster the rate at which the distance is decreasing, the faster the movement speed of the band image 73b may be set. The subject 60 who views the band image 73b can recognize the rate at which the distance between the moving object 50 and the pedestrian 63b is decreasing based on the movement speed of the band image 73b.

[0069] With fifth marker 73 displayed, when control unit 33 detects a user input corresponding to a predetermined user operation, for example, control unit 33 may change display area 62 so that the detected position of pedestrian 63b on the first image is included inside display area 62. For example, as shown in Fig. 12, control unit 33 may extend display area 62 on the first image in the left-right direction and move display area 62 to the right side of detection area 61. With this configuration, pedestrian 63b appears on the second image, as shown in Fig. 13, for example.

[0070] The above-mentioned predetermined user operation may include any user operation. For example, the above-mentioned predetermined user operation may include a first user operation for changing the steering angle of the moving object 50. The fifth marker 73 may function as a GUI (Graphical User Interface) that accepts a second user operation. Hereinafter, the GUI is also referred to as an interface image. In such a case, the above-mentioned predetermined user operation may include the second user operation.

[0071] Control unit 33 may automatically change display area 62 so that the detected position of pedestrian 63b on the first image is included inside display area 62. In this case, control unit 33 may maintain the automatic change of display area 62 until pedestrian 63b is no longer detected in detection area 61 of the first image.

[0072] The control unit 33 may change the icon image 67 in accordance with the change in the display area 62, for example, as shown in FIG.

[0073] The control unit 33 may change the display area 62 on the first image in response to, for example, a pinch-in operation or a pinch-out operation on the display device 40. For example, as shown in Fig. 14, the control unit 33 may make the display area 62 approximately coincide with the detection area 61. In such a case, all detection targets within the detection area 61 are displayed on the display device 40, as shown in Fig. 15, for example.

[0074] (5th example (reference example)) 16 shows a fifth example of a detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. In the example shown in FIG. 16, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a vehicle 64a that appears within the first predicted path 65 of the moving object 50, and a vehicle 64b and a pedestrian 63d that appear outside the first predicted path 65.

[0075] In the fifth example, a case where the external area of ​​the moving object 50 is dark, such as at night or inside a tunnel, will be described. When the external area of ​​the moving object 50 is dark, the characteristic values ​​of the first image and the second image may decrease. The characteristic values ​​may include any parameter related to the visibility of the image. For example, the characteristic values ​​may include at least one of the brightness value and the contrast ratio of the image. When the characteristic value of the second image decreases, the visibility of the second image may decrease.

[0076] The control unit 33 may perform specific image processing on a region corresponding to the detection target in the second image. The specific image processing may include a first process of superimposing a marker corresponding to the detection target on the region. Hereinafter, the marker will also be referred to as a sixth marker. The sixth marker may include, for example, an image that substantially matches the outline shape of the detection target in the second image. With this configuration, the sixth marker is displayed superimposed on the detection target in the second image. Therefore, even if the characteristic value of the second image is low, the subject 60 can easily recognize the detection target in the second image. The specific image processing may include a second process of changing the characteristic value of the region corresponding to the detection target in the second image. For example, the control unit 33 may change the characteristic value of the region so as to improve the visibility of the region in the second image. With this configuration, the visibility of the detection target in the second image is improved. Therefore, even if the characteristic value of the second image is low, the subject 60 can easily recognize the detection target in the second image.

[0077] The control unit 33 may execute the above-described specific image processing when one or more conditions are satisfied. The one or more conditions may include a condition that the detection target is located within the first predicted path 65 of the moving object 50. The one or more conditions may include a condition that the first predicted path 65 of the moving object 50 overlaps with the second predicted path of the detection target. The one or more conditions may include a condition that the distance between the moving object 50 and the detection target is less than a predetermined threshold. The one or more conditions may include a condition that a characteristic value of at least a part of an area of ​​the second image is less than a predetermined threshold.

[0078] In the fifth example, the control unit 33 may determine that one or more of the above-described conditions are satisfied for each of the vehicles 64a, 64b, and the pedestrian 63d. In this case, the control unit 33 may cause the display device 40 to display three sixth markers 75a, 75b, and 75c corresponding to the vehicles 64a, 64b, and the pedestrian 63d, respectively, superimposed on the second image, as shown in Fig. 17 . The control unit 33 may cause the display device 40 to display the sixth markers superimposed on the detection targets in a bright place.

[0079] The control unit 33 may change the shape of the guide line 66. The control unit 33 may change the shape of the guide line 66 in the area where the guide line 66 overlaps with the detection target. FIG. 18 shows one example of the shape of the guide line 66. In the guide line 66 in FIG. 18, the guide line 66 is not displayed in the area where the guide line 66 overlaps with the sixth marker 75a. The shape example of the guide line 66 is not limited to being removed, and other design changes are permitted. Design changes include changing the color, the transparency, the type to a dashed line, the line thickness, and blinking. The control unit 33 may change the shape of the guide line 66 when the sixth marker is not displayed. The control unit 33 may change the shape of the guide line 66 when the first marker 71 and the second marker 72 are displayed on the detection target.

[0080] As described above, according to the display system 10 of one embodiment, various markers corresponding to the detection target detected on the first image are superimposed on the second image and displayed on the display device 40. The subject 60 viewing the markers can recognize at a glance the relative positional relationship between the moving object 50 and the detection target. This improves the convenience of the technology for displaying an image of the area outside the moving object 50.

[0081] 19 to 31, examples of various images that the image processing device 30 displays on the display device 40 by combining the guide wall image 80 and other images in the display area on the first video will be specifically described below.

[0082] (Example 6) FIG. 19 illustrates a sixth example of a detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. The imaging device 20 may be disposed to capture an image of the rear of a moving object 50, which is a vehicle. In the example illustrated in FIG. 19, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The detection area 61 is not limited to a shape longer in the left-right direction than in the up-down direction, and various shapes are possible, such as a square, a vertically elongated rectangle, or a circle. The display area 62 may have various positions, sizes, and shapes. For example, the display area 62 is not limited to being located at the center of the detection area 61, and may be located to the left or right. The display area 62 is not limited to a portion of the detection area 61, and the entire detection area 61 may be the display area 62. The control unit 33 may change the display area 62 in the first image in response to a pinch-in operation, a pinch-out operation, or the like performed on the display device 40.

[0083] The control unit 33 generates a second image by superimposing a guide wall image 80 indicating the first predicted path 65 of the moving object 50 onto a display area on the first image acquired from the imaging device 20. The control unit 33 may output an image in which the guide wall image 80 is superimposed on the second image and display it on the display device 40. The guide wall image 80 can indicate the predicted path of the moving object 50 when reversing. The guide wall image 80 is a marker that gives the subject 60 a three-dimensional impression. The guide wall image 80 can be considered to be an image of a virtual guide wall that is placed in the real space shown in the first image and guides the predicted path, projected onto the first image captured by the imaging device 20. The guide wall image 80 can be viewed within the field of view of the subject 60 as consisting of multiple virtual translucent walls extending from the road surface to a predetermined height in the vertical direction. The guide wall image 80 can also be considered to be a three-dimensional wall-like representation of the guide lines displayed on the display device 40 when the vehicle is reversing. Wall-like displays include displays in other forms such as a surface, a film, a board, etc. The three-dimensional display of the guide wall image 80 makes it easier for the subject 60 to grasp the location of a parking space on the road surface, the distance to an obstacle, etc.

[0084] The control unit 33 may recognize a detection target within the display area 62 of the first image. The control unit 33 may detect a detection target that is inside the display area 62 on the first image and is displayed within the first predicted path 65. When the detection target is detected, the control unit 33 may synthesize a virtual plane as a first recognition wall image 83 on the moving body 50 side of the recognized detection target. When the detection target moves, the first recognition wall image 83 moves together with the detection target.

[0085] FIG. 20 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. 19. The guide wall image 80 is positioned above and away from the bottom edge of the second image. The guide wall image 80 may include two side wall images 81a and 81b extending in the height and depth directions. The side wall image 81a is located on the left side in the horizontal direction, and the side wall image 81b is located on the right side in the horizontal direction. The area between the two side wall images 81a and 81b indicates the first predicted path 65 along which the moving object 50, which is a vehicle, will travel. The distance between the side wall images 81a and 81b may be set to indicate the width of the moving object 50. In FIG. 20, the side wall images 81a and 81b are displayed linearly. The control unit 33 may acquire moving object information from the communication unit 31 and curve the side wall images 81a and 81b according to the predicted path. In this case, the moving object information includes the steering angle of the steering wheel.

[0086] The guide wall image 80 may include multiple distance wall images 82a, 82b, and 82c extending in the height and horizontal directions and indicating the distance in the depth direction from the moving object 50. The distance wall images 82a, 82b, and 82c may connect the side wall images 81a and 81b. Three distance wall images 82a, 82b, and 82c may be shown in FIG. 20. The number of distance wall images is not limited to three and may be two or four or more. The distance in the depth direction between the distance wall images 82a and 82b may be narrower than the distance in the depth direction between the distance wall images 82b and 82c. The height direction lengths of the distance wall images 82a, 82b, and 82c may be longer or shorter than the height direction lengths of the side wall images 81a and 81b.

[0087] A frame may be displayed around the periphery of each of the side wall images 81a, 81b and distance wall images 82a, 82b, and 82c of the guide wall image 80. The guide wall image 80 may include at least one of auxiliary lines 84a extending in the depth direction, auxiliary lines 84b extending in the horizontal direction, and auxiliary lines 84c extending in the height direction. The guide wall image 80 may include all of the auxiliary lines 84a extending in the depth direction, auxiliary lines 84b extending in the horizontal direction, and auxiliary lines 84c extending in the height direction. The auxiliary line 84a extending in the depth direction may be displayed on the wall surfaces of the side wall images 81a and 81b. The auxiliary line 84b extending in the horizontal direction may be displayed on the wall surfaces of the distance wall images 82a, 82b, and 82c. The auxiliary line 84c extending in the height direction may be displayed on the wall surfaces of the side wall images 81a, 81b and distance wall images 82a, 82b, and 82c. Each of the auxiliary lines 84a, 84b, and 84c can be displayed in any number greater than or equal to one. When there are multiple auxiliary lines 84a, 84b, and 84c, the intervals between the auxiliary lines 84a, 84b, and 84c can be equal. The intervals between the auxiliary lines 84a, 84b, and 84c may vary depending on conditions such as the height from the road surface and the distance from the moving object 50. The frames displayed on the peripheries of the side wall images 81a and 81b and the distance wall images 82a, 82b, and 82c can be considered part of the multiple auxiliary lines 84a, 84b, and 84c. In FIG. 20 and the following figures, even when multiple auxiliary lines 84a, 84b, and 84c exist, only one of each is labeled as appropriate.

[0088] The distance wall image 82a is a first distance wall image that is closer to the moving object 50 than the other distance wall images 82b and 82c when mapped onto the real space shown in the first video. The distance wall image 82a can be displayed in a different color from the other distance wall images 82b and 82c. For example, the distance wall images 82b and 82c can be displayed in a semi-transparent white, and the distance wall image 82a can be displayed in a semi-transparent red. The side wall images 81a and 81b can be displayed in a semi-transparent white. The above color settings are merely examples, and the color of the side wall images 81a and 81b may be different from that of the distance wall images 82b and 82c.

[0089] The transmittance of the sidewall images 81a, 81b may be changed according to the position in the depth direction. The transmittance of the sidewall images 81a, 81b may be set to be lower toward the front in the depth direction and higher toward the back. The sidewall images 81a, 81b may have different saturation or brightness according to the position in the depth direction. For example, the saturation can be made higher toward the front in the depth direction. The saturation of an image can also be said to be the color intensity of the image. Continuous changes in transmittance, saturation, brightness, chromaticity, etc. can be said to have a gradation.

[0090] The transmittance of the distance wall images 82a, 82b, and 82c may be changed according to the height from the road surface. The transmittance of the distance wall images 82a, 82b, and 82c may be set to increase as the distance from the road surface increases in the height direction. The saturation or brightness of the distance wall images 82a, 82b, and 82c may be varied according to the height from the road surface. For example, the saturation may be decreased as the distance from the road surface increases in the height direction.

[0091] The auxiliary lines 84b, 84c on the distance wall image 82a, which is the first distance wall image, may be the same color as the auxiliary lines 84b, 84c on the other distance wall images 82b, 82c. The auxiliary lines 84a, 84b, 84c may be any color. For example, if the color of the walls of the guide wall image 80 other than the distance wall image 82a is white, the color of the auxiliary lines 84a, 84b, 84c may be white with higher brightness or lower transmittance than these walls.

[0092] The control unit 33 may detect a vehicle 64 displayed in the display area 62 of the first image as a detection target. In the example of FIG. 20 , the vehicle 64 is located within the first predicted path 65. The detection target detected by the control unit 33 is not limited to a vehicle, but may also be a building, a fence, an obstacle on the road, a person, an animal, etc. When a detection target is detected, the control unit 33 may superimpose a first recognition wall image 83 on the side of the recognized moving object 50 of the detection target. When a detection target is detected, the control unit 33 may change the height of the guide wall image 80. When a detection target is detected, the control unit 33 can shorten the height of the guide wall image 80. This allows the control unit 33 to make the display easier to see even when the first recognition wall image 83 is displayed, and to direct the subject 60's attention to the detection target.

[0093] The first recognition wall image 83 of the detection target located within the first predicted path 65 extends in the height direction and the horizontal direction. The first recognition wall image 83 may be displayed as an opaque or semi-transparent surface of any color. In FIG. 20 , the first recognition wall image 83 is displayed as a semi-transparent surface having a rectangular shape on the moving object 50 side of the vehicle 64. The horizontal width of the first recognition wall image 83 may correspond to the width of the vehicle 64. The horizontal width of the first recognition wall image 83 may be approximately equal to the width of the vehicle 64. The height of the first recognition wall image 83 may be set to a predetermined value. The transmittance of the first recognition wall image 83 may be changed depending on the height direction from the road surface. The transmittance of the first recognition wall image 83 may be set to increase as the distance from the road surface in the height direction increases. Furthermore, the saturation or brightness of the first recognition wall image 83 may vary depending on the distance from the road surface in the height direction. For example, the saturation may be decreased as the distance from the road surface in the height direction increases. The first recognition wall image 83 may include auxiliary lines extending in the horizontal and vertical directions.

[0094] The control unit 33 may move the auxiliary lines extending horizontally of the first recognition wall image 83 in the height direction within the frame of the first recognition wall image 83. The auxiliary lines may be displayed so as to appear from the bottom end of the frame of the first recognition wall image 83 and disappear at the top end. The control unit 33 may display both or either the relative position and the relative distance change between the moving object 50 and the vehicle 64 by moving the auxiliary lines extending horizontally of the first recognition wall image 83 in the height direction. For example, when the distance between the moving object 50 and the vehicle 64 is narrowing, the control unit 33 may move the auxiliary lines extending horizontally faster in the height direction. The control unit 33 may move the auxiliary lines vertically faster as the distance between the moving object 50 and the vehicle 64 becomes shorter. The control unit 33 may move the auxiliary lines horizontally of the first recognition wall image 83 in the height direction regardless of the position of the detection target and the distance from the detection target.

[0095] The color of the first recognition wall image 83 can be different depending on the distance in the depth direction from the moving object 50 to the vehicle 64. For example, as shown in Fig. 20, when the distance between the moving object 50 and the vehicle 64 is greater than a predetermined distance, the control unit 33 can display the color of the first recognition wall image 83 in blue. When the moving object 50 moves backward and the distance to the vehicle 64 becomes shorter, the control unit 33 can change the color of the first recognition wall image 83 from blue to yellow, or from yellow to red, for example.

[0096] When the distance between the moving object 50 and the vehicle 64 is equal to or greater than a predetermined distance, the control unit 33 may not display the first recognition wall image 83. When the moving object 50 and the vehicle 64 become relatively close and are within the predetermined distance, the control unit 33 may, for example, highlight the vehicle 64 in the second image and then display the first recognition wall image 83. Examples of highlighting include displaying the outline of the vehicle 64 in a thick line, flashing the outline of the vehicle 64, filling in the image of the vehicle 64, and the like. The control unit 33 may change the display mode in accordance with a change in the relative position between the moving object 50 and the vehicle 64. For example, when the vehicle 64 is moving relatively away from the moving object 50, the control unit 33 may not display the first recognition wall image 83.

[0097] 21 is a diagram showing an example of the second image in a state where the distance between the moving object 50 and the vehicle 64 has narrowed as the moving object 50 moves backward. The first recognition wall image 83 is displayed in red. This makes it possible to warn the subject 60 that there is a risk of collision with the vehicle 64 if the subject 60 moves backward any further.

[0098] When the detection object and the guide wall image 80 overlap, the control unit 33 can change the display of the guide wall image 80. For example, as shown in FIG. 21 , the control unit 33 can at least partially not display the guide wall image 80 that overlaps with the vehicle 64 in the second video. More specifically, the control unit 33 does not cause the display device 40 to display the guide wall image 80 that overlaps with the vehicle 64 and is located farther from the moving object 50 in the depth direction than the vehicle 64 or the first recognition wall image 83. In this way, the display device 40 can display an image that is easy for the subject 60 to see and intuitively understand. The method of changing the display of the guide wall image 80 is not limited to hiding the overlapping portion. For example, the control unit 33 can increase the transmittance of the guide wall image 80 in the portion that overlaps with the detection object.

[0099] (Variation) The guide wall image 80 is not limited to the example shown in Figures 20 and 21. Figures 22 to 24 show modified examples of the guide wall image 80.

[0100] (First Modification) In the example shown in FIG. 22, the horizontally extending auxiliary lines 84b of the distance wall images 82a, 82b, and 82c may display only one of the lowest positions in the height direction. The distance wall image 82a displaying only the lowest auxiliary line 84b may make it easier to recognize the height of the ground. The distance wall image 82a displaying only the lowest auxiliary line 84b may make it easier to understand the distance from the recognition wall image. Auxiliary lines 84c extending in the height direction may be displayed at both ends of the horizontal direction of the distance wall images 82a, 82b, and 82c. The periphery of the distance wall images 82a, 82b, and 82c may be displayed with an auxiliary line 84b located at the bottom and an auxiliary line 84c extending in the height direction located at both ends of the horizontal direction. The distance wall images 82a, 82b, and 82c may be displayed with an auxiliary line 84b extending in the horizontal direction at the bottom of the periphery and an auxiliary line 84c extending in the height direction at both left and right ends.

[0101] The distance wall images 82a, 82b, and 82c may be displayed including a surface having a translucent color. The distance wall images 82a, 82b, and 82c may have increasing transparency or lighter color density as they move away from the road surface in the vertical direction. The auxiliary lines 84b and 84c located on the periphery of the distance wall image 82a closest to the moving object 50 may be displayed in a color different from the other auxiliary lines. The different color may be, for example, red. The other auxiliary lines may be white, but is not limited to this. The first recognition wall image 83 displayed in front of the vehicle 64 may be displayed as a translucent surface of any color, and may have increasing transparency or lighter color density as they move away from the road surface in the vertical direction. The first recognition wall image 83 may be displayed so that the upper boundary is not visible and gradually disappears in the vertical direction.

[0102] In the example shown in FIG. 22, only two auxiliary lines 84a extending in the depth direction of the sidewall images 81a and 81b of the guide wall image 80 are displayed at the upper and lower ends of the sidewall images 81a and 81b. The lower auxiliary line 84a1 can be displayed more emphasized than the upper auxiliary line 84a2. Highlighting includes displaying using a thicker line, a line with lower transmittance, a line with higher brightness, etc. The upper auxiliary line 84a2 and the lower auxiliary line 84a1 may be displayed in the same manner. Lines in the same manner are lines that are similar in thickness, transmittance, brightness, etc., for example.

[0103] The auxiliary line 84a extending in the depth direction may be displayed using a display method in which it gradually fades and disappears at the farthest position in the depth direction. The sidewall images 81a and 81b may be displayed so that the farthest end portions in the depth direction are visible.

[0104] Auxiliary lines 84c extending in the height direction may be displayed on the sidewall images 81a and 81b at least at the foremost end and at positions where the lines intersect with the distance wall images 82a, 82b, and 82c. These auxiliary lines 84c extending in the height direction may be displayed more emphasized than auxiliary lines 84c extending in other height directions. The auxiliary lines 84c extending in the height direction may each have a different thickness. The auxiliary lines 84c displayed at the position where the lines intersect with the distance wall image 82a may be displayed thicker than the auxiliary lines 84c displayed at the positions where the lines intersect with the distance wall image 82b and 82c.

[0105] (Second Modification) The guide wall image 80 shown in FIG. 23 differs from the guide wall image 80 in FIG. 22 in the side wall images 81a and 81b. In FIG. 23, two or more auxiliary lines 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 may be displayed, including the top and bottom ones. Of the auxiliary lines 84a extending in the depth direction, the auxiliary line located lowest in the height direction may be displayed more emphasized than the other auxiliary lines. Multiple auxiliary lines 84a may all be displayed as lines of the same type. The guide wall image 80 in FIG. 23 may employ the same variations in transmittance, color, and color scheme as those that may be employed in the guide wall image 80 in FIG. 22.

[0106] (Third Modification) The guide wall image 80 shown in FIG. 24 differs from the guide wall image 80 in FIG. 22 in the side wall images 81a and 81b. In FIG. 24, only the auxiliary lines 84a located at the bottom of the guide wall image 80 extending in the depth direction of the side wall images 81a and 81b are displayed. The auxiliary lines 84a located at the bottom of the guide wall image 80 extending in the depth direction, together with the auxiliary lines 84b extending in the horizontal direction, may correspond to lines on the road surface when mapped onto the real space shown in the first video. The guide wall image 80 in FIG. 24 may employ the same changes in transmittance, color, and color scheme as those employed in the guide wall image 80 in FIG. 22. The auxiliary lines 84c extending in the height direction may each have a different thickness. The auxiliary lines 84c aligned from the front to the back in the depth direction may be thinner as they move further back.

[0107] In the guide wall image 80 shown in each of FIGS. 20 to 24, the display device 40 does not need to display some or all of the auxiliary lines 84a, 84b, and 84c.

[0108] (Example 7) 25 shows a seventh example of a detection area 61 in the first video acquired by the image processing device 30 from the imaging device 20. In the example shown in FIG. 25, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The display area 62 is not limited to a part of the detection area 61, and the entire detection area 61 may be used as the display area 62.

[0109] In the seventh example, the moving object 50 may be a vehicle that is reversing toward a parking space in a parking lot. The imaging device 20 may be located behind the moving object 50 and capture an image of the rear. The control unit 33 may detect, from the detection area 61, a vehicle 64b and a pedestrian 63 that are outside the first predicted path 65 and at least partially displayed within the display area 62, as detection targets. In the example shown in FIG. 25 , the control unit 33 determines that the distance in the depth direction of the vehicle 64a displayed within the first predicted path 65 of the moving object 50 is farther than a predetermined distance, and does not consider the vehicle 64a to be a detection target.

[0110] FIG. 26 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. 25. The control unit 33 recognizes the vehicle 64b and pedestrian 63 as detection targets in the display area 62 of the first image and displays first recognition wall images 83a and 83b, which are virtual planes, on the moving object 50 side of the vehicle 64b and the pedestrian 63, respectively. The first recognition wall images 83a and 83b may be displayed as semi-transparent surfaces parallel to the side wall images 81a and 81b extending in the height and depth directions. The control unit 33 may acquire a change in the distance between the vehicle 64b and the pedestrian 63 and the moving object 50, and display the side wall images 81a and 81b only when the distance is changing in a direction that shortens. As described above, any algorithm may be used to estimate the distance between the moving object 50 and the detection targets.

[0111] The control unit 33 may estimate the horizontal distance of the vehicle 64a and pedestrian 63 to be detected from the guide wall image 80 and determine the display colors of the first recognition wall images 83a and 83b according to the estimated distance. Here, the distance between the detection target and the guide wall image 80 refers to the distance between the detection target and the virtual guide wall displayed by the guide wall image 80, assuming that the virtual guide wall exists in the real space displayed by the first video. The control unit 33 may estimate a first distance between the detection target and the position of the guide wall image 80, where the guide wall image 80 is mapped in the real space displayed by the first video. The control unit 33 may set a threshold value for the first distance and display the first recognition wall images 83a and 83b on the display device 40 in different colors, such as blue, yellow, and red, in order from the longest to the shortest horizontal distance. The first recognition wall images 83a and 83b may have auxiliary lines extending in the depth direction. The control unit 33 can move the auxiliary lines extending in the depth direction of the first recognition wall images 83a, 83b in the height direction within the frame of the first recognition wall image 83, similar to the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in Figure 20.

[0112] The control unit 33 estimates a first position as the position in the depth direction of the vehicle 64b and the pedestrian 63 to be detected. The control unit 33 may at least partially change the display of the sidewall images 81a and 81b facing the vehicle 64b and the pedestrian 63, respectively, in accordance with the first position. The manner of changing the display includes changing the color, brightness, saturation, etc. of the wall surface, changing the color, thickness, etc. of the border line and / or auxiliary line, or a combination thereof. For example, in FIG. 26 , the display color of a portion 81a1 of the sidewall image 81a facing the vehicle 64b and a portion 81b1 of the sidewall image 81b facing the pedestrian 63 is changed. The portions of the sidewall images 81a and 81b may be regions separated by an auxiliary line 84c extending in the height direction on the sidewall images 81a and 81b. The display colors of the portion 81a1 of the sidewall image 81a and the portion 81b1 of the sidewall image 81b may differ depending on the first distance. When the detected obstacle is larger than the length represented by the side wall image 81 a, 81 b, the control unit 33 may change the display color of the entire side wall image 81 a or 81 b. Examples of large obstacles include large vehicles such as trailers, buildings, etc.

[0113] When the detection target moves, the control unit 33 may estimate a second predicted path of the detection target and estimate a first position as a position where the second predicted path intersects with the sidewall images 81a and 81b. The control unit 33 may estimate the first position as a position where the second predicted path in the real space shown in the first video intersects with the mapping of the sidewall images 81a and 81b onto the real space. The control unit 33 may at least partially change the display of the sidewall images 81a and 81b in accordance with the first position. In this case, the display color of the portions 81a1 and 81b1 of the sidewall images 81a and 81b whose display is changed may also be different depending on the first distance. When the detection target is a very large obstacle, the control unit 33 may change the display color of the entire sidewall image 81a or 81b.

[0114] As another display mode, the control unit 33 may change the display of portions of the side wall images 81a and 81b that are located further in the depth direction than the first recognition wall images 83a and 83b. As yet another display mode, the control unit 33 may change the display of portions of the side wall images 81a and 81b that are located further in the depth direction than the position where the second predicted path intersects with the side wall images 81a and 81b.

[0115] The display of the first recognition wall images 83a and 83b may be changed depending on the height from the road surface. As with the first recognition wall image 83 in the sixth example, the display of the first recognition wall images 83a and 83b may be set to different values ​​for transmittance, saturation, brightness, or the like depending on the height from the road surface.

[0116] As described above, the display system 10 can spatially and three-dimensionally represent the distance and / or position of a detection object that may be an obstacle to reversing, using the guide wall image 80 and the first recognition wall images 83a, 83b. Furthermore, it can three-dimensionally display positions where there is a risk of collision based on the predicted path of a moving detection object. Thus, the display system 10 can spatially and three-dimensionally represent the detection object and the warning content by displaying the guide wall image 80 and the first recognition wall images 83a, 83b as walls (or planes) having a height direction. This makes it easier for the target person 60 to understand the display and the warning content.

[0117] (Example 8) FIG. 27 shows an eighth example of a detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. The imaging device 20 may be disposed so as to capture an image of the rear of a moving body 50, which is a vehicle. In the example shown in FIG. 27, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63 and a vehicle 64b that are displayed in a second area, which is an area outside the display area 62 on the first image but inside the detection area 61. In the eighth example, the control unit 33 determines that the distance in the depth direction of the vehicle 64a displayed in the display area 62 is farther than a predetermined distance, and does not regard the vehicle 64a as a detection target.

[0118] 28 is a diagram showing a schematic arrangement of moving body 50 and detection target pedestrian 63 and vehicle 64b as viewed from above along the height direction in the real space shown in the first video. Image capture device 20 can capture images of subjects within the ranges included in areas F1 and F2 in the real space. Areas F1 and F2 correspond to detection area 61 in the first video. Area F1 is a first area corresponding to display area 62 in the first video. Area F2 corresponds to a second area inside detection area 61 and outside display area 62 in the first video.

[0119] 28, for the sake of explanation, a virtual guide wall 90, which is a real-space mapping of the guide wall image 80 displayed in the second video, is displayed by a dashed line. Virtual side walls 91a and 91b correspond to the side wall images 81a and 81b displayed in the second video. Virtual distance walls 92a, 92b, and 92c correspond to the distance wall images 82a, 82b, and 82c displayed in the second video.

[0120] 28, for the sake of explanation, virtual lines 95l1-95l5 and 95r1-95r5 indicating the horizontal distance of the detection target from imaginary side walls 91a and 91b of the moving object 50 are shown by dashed lines. This horizontal distance is also referred to as the second distance. The virtual lines 95l1-95l5 are located on the left side of the moving object 50. The virtual lines 95r1-95r5 are located on the right side of the moving object 50. The intervals between the virtual lines 95l1-95l5 and the virtual lines 95r1-95r5 can be set arbitrarily. The intervals between the virtual lines 95l1-95l5 and the virtual lines 95r1-95r5 may be equal. The control unit 33 can determine from the first image to which of the virtual lines 95l1 to 95l5 and 95r1 to 95r5 the detection target is closest, or within which two of the virtual lines 95l1 to 95l5 and 95r1 to 95r5 the detection target is located.

[0121] FIG. 29 illustrates a second image corresponding to the first image in FIG. 27. In the eighth example, the guide wall image 80 may employ a display method similar to that described in the sixth and seventh examples. When a detection target is located in the second region of the first image, the control unit 33 superimposes second recognition wall images 89a and 89b on the edge of the second image on the side where the detection target is located. The second recognition wall images 89a and 89b may include wall portions 96a and 96b extending in the height and depth directions, respectively, and floor portions 97a and 97b extending in the depth and horizontal directions. In FIG. 29, the wall portions 96a and 96b extend from the bottom to the top of the left and right edges of the second image. The wall portions 96a and 96b may extend vertically along only a portion of the left and right edges of the second image, rather than the entirety of the entirety. Similarly, the floor portions 97a and 97b may extend horizontally along the entire or part of the bottom edge of the second image. The floor portions 97a and 97b may be displayed between the guide wall image 80 and the lower end of the second image.

[0122] The control unit 33 may change the display of the second recognition wall images 89a and 89b depending on the second distance of the detection target. The display changes may include changes in color, transmittance, saturation, brightness, size, etc., as well as dynamic changes in display methods such as flashing and movement. For example, in one of several embodiments, the display changes may be represented by a change in color. The display color may be red for a high level of danger, yellow for a medium level of danger, and blue for a low level of danger. The pedestrian 63 shown in FIG. 28 is located on the third virtual line 95r3 horizontally to the right of the virtual side wall 91b of the moving object 50. In this case, the control unit 33 may determine the level of danger to be medium and change the color of the right-side second recognition wall image 89b to yellow. The vehicle 64b shown in FIG. 28 is located farther away from the fifth virtual line 95l5 horizontally to the left of the virtual side wall 91a of the moving object 50. In this case, the control unit 33 may determine that the level of risk is low and change the color of the second recognition wall image 89a on the left side to blue. In one embodiment, the change in display may include horizontal movement of the vertical auxiliary lines 96a1, 96b1 displayed on the walls 96a, 96b. The control unit 33 may change the speed of the movement of the walls 96a, 96b depending on the length of the second distance.

[0123] The second recognition wall images 89a and 89b may be semi-transparent. The wall portions 96a and 96b of the second recognition wall images 89a and 89b may have display characteristics that change vertically. For example, the wall portions 96a and 96b of the second recognition wall images 89a and 89b may have lower transmittance toward the bottom. Furthermore, the saturation of the second recognition wall images 89a and 89b may be increased toward the bottom. The floor portions 97a and 97b of the second recognition wall image 89a may have display characteristics that change horizontally. For example, the floor portions 97a and 97b of the second recognition wall image 89a may have lower transmittance toward the left and right ends and higher transmittance toward the center of the second image. Furthermore, the floor portions 97a and 97b of the second recognition wall image 89a may have lower saturation from the left and right ends toward the center of the second image. The saturation of an image can also be expressed as the color intensity of the image.

[0124] The control unit 33 may determine the type of the detection target and display icon images 99a and 99b corresponding to the type on the floors 97a and 97b of the second recognition wall images 89a and 89b. The type of the detection target can be determined using a known method based on the image of the detection target detected from the first video. For example, the control unit 33 may determine the object type by matching the contour shape of the detection target with a model pattern. For example, icons representing a person and a vehicle may be selected for a pedestrian 63 and a vehicle 64b, respectively. The control unit 33 may estimate a second distance, which is the horizontal distance between the detection target in real space and virtual side walls 91a and 91b, which are obtained by mapping the side wall images 81a and 81b onto real space. The second distance may be determined from the relationship between the positions of the images of the detection targets and the lines, which are obtained by mapping the virtual lines 95l1 to 95l5 and 95r1 to 95r5 in real space onto the detection area 61 in the first video. The control unit 33 can change the positions at which the icon images 99a and 99b are displayed, depending on the second distance.

[0125] 29, first distance identification lines 98l1-98l5 and 98r1-98r5 are provided corresponding to virtual lines 95l1-95l5 and 95r1-95r5 in real space, respectively. The first distance identification lines 98l1-98l5 are located on the left side of sidewall image 81a and are numbered 98l1, 98l2, 98l3, 98l4, and 98l5 from the center to the left. The first distance identification lines 98r1-98r5 are located on the right side of sidewall image 81b and are numbered 98r1, 98r2, 98r3, 98r4, and 98r5 from the center to the right. The first distance identification lines 98l1-98l5 and 98r1-98r5 indicate the positions where icon images 99a and 99b are displayed, and do not necessarily have to be displayed in the second video.

[0126] In FIG. 28, the pedestrian 63 is located on the third virtual line 95r3 to the right of the virtual side wall 91b of the moving object 50 in the horizontal direction. Therefore, the control unit 33 may place the icon image 99b of the pedestrian 63 on the third first distance identification line 98r3 on the right in the second image of FIG. 29. As shown in FIG. 29, the icon image 99b may be displayed three-dimensionally as a plane having lengths in the depth and height directions on the first distance identification line 98r3 extending in the depth direction in the second image. When viewed stereoscopically, the icon image 99b may be displayed as a plane parallel to the side wall images 81a, 81b and the second recognition wall images 89a, 89b. Similar to the second recognition wall image 89b, the display mode of the icon image 99b, such as color, saturation, brightness, etc., may be changed depending on the second distance. For example, the icon image 99b may be displayed in yellow.

[0127] In FIG. 28, the vehicle 64b is located farther from the virtual side wall 91a of the moving object 50 than the fifth virtual line 95l5 on the left side in the horizontal direction from the virtual side wall 91a. In this case, the control unit 33 may place the icon image 99a of the vehicle 64b on the left first distance identification line 98l5 in the second image of FIG. 29. Alternatively, the control unit 33 may place the icon image 99a so that it appears to be on the same plane as the wall portion 96a of the second recognition wall image 89a when the second image is viewed three-dimensionally. Similar to the icon image 99b, the display mode of the icon image 99a can be changed depending on the second distance. For example, the icon image 99a can be displayed in blue.

[0128] When the relative position of the detection target as seen from the moving object 50 changes due to movement of both or either of the moving object 50 and the detection target, the positions and display manner of the second recognition wall images 89a, 89b and the icon images 99a, 99b change. Furthermore, when the moving object 50 enters the display area 62, the detection target may be displayed in the manner described in the seventh example. An example of a change in display when the vehicle 64b moves from area F2 to area F1 in the real space shown in FIG. 28 will be described.

[0129] First, in the real space of Fig. 28, the vehicle 64b is located to the left of the virtual line 95l5. At this time, as described above, the second recognition wall image 89a is displayed in the second video of Fig. 29, and the icon image 99a is displayed on the leftmost side of the floor portion 98a of the second recognition wall image 89a. At this time, the second recognition wall image 89a and the icon image 99a can be displayed in blue.

[0130] In the real space of FIG. 28, as the vehicle 64b approaches the virtual guide wall 90, the second recognition wall image 89a displayed in the second video of FIG. 29 changes its display mode. For example, the color of the second recognition wall image 89a changes sequentially from blue to yellow to red. At the same time, the display position of the icon image 99a moves from above the first distance identification line 98l5 to above the first distance identification line 98l1. The color of the icon image 99a may change sequentially from blue to yellow to red. This allows the subject 60 to recognize that the vehicle 64b is approaching outside, which is not displayed in the second video. Furthermore, using a stereoscopic display method allows the subject 60 to spatially grasp the approach of the vehicle 64b and reduces the risk of overlooking information.

[0131] The icon image 99a may move toward the center of the screen when the vehicle 64b approaches the moving object 50. The target person 60 may recognize the moving direction of the vehicle 64b through the movement of the icon image 99a.

[0132] Furthermore, when the vehicle 64b enters from area F2 to area F1 in the real space of FIG. 28, the vehicle 64b enters the display target area of ​​the first image and is therefore included in the second image and displayed on the display device 40. At the same time, the display of the second recognition wall image 89a disappears, and the first recognition wall image 83a is displayed on the guide wall image 80 side of the vehicle 64b, as shown in FIG. 26. When the second recognition wall image 89a disappears, the control unit 33 may flash the second recognition wall image 89a multiple times to draw the attention of the subject 60. The control unit 33 displays the first recognition wall image 83a on the moving object 50 side of the vehicle 64b. Furthermore, the control unit 33 estimates a second predicted path of the vehicle 64b and estimates a first position as a position where the second predicted path intersects with the side wall images 81a and 81b. The control unit 33 may change the display of a portion 81a1 of the side wall image 81a that includes the first position. For example, the control unit 33 can change the color of the part 81a1 of the sidewall image 81a sequentially from blue to yellow to red as the vehicle 64b approaches, thereby enabling the subject 60 to easily recognize that the vehicle 64b is approaching and that the second predicted path of the vehicle 64b overlaps with the first predicted path 65 of the moving object 50.

[0133] As described above, according to one of the embodiments of the present disclosure, the spatial understanding of the position of the detection target is facilitated by the three-dimensional display of the guide wall image, the first recognition wall image, the second recognition wall image, the icon image, etc. Furthermore, according to one of the embodiments of the present disclosure, by changing the display mode of each wall surface, it is possible to provide information such as cautions and warnings in an easily understandable manner. Furthermore, by using three-dimensional display, it is possible to reduce the risk of overlooking information.

[0134] (Variation) FIG. 30 shows another example of the second image corresponding to the first image in FIG. 27. In FIG. 30, view display areas 101, 102 are added to the lower left corner, indicating the selection status between "Normal View Display" and "Wide View Display." "Normal View Display" and "Wide View Display" will be described later. In FIG. 30, the display manner of the guide wall image 80, second recognition wall images 89a, 89b, and icon images 99a, 99b may be partially different from that in FIG. 29. For example, in FIG. 30, the guide wall image 80 is displayed in the same manner as in FIG. 22.

[0135] The wall portions 96a, 96b of the second recognition wall images 89a, 89b may have one or more auxiliary lines 96a1, 96b1 extending in the height direction. In FIG. 30, the wall portions 96a, 96b have multiple auxiliary lines 96a1, 96b1, but only one of them is labeled. The wall portions 96a, 96b of the second recognition wall images 89a, 89b may indicate the distance to the detection target with color, and the transmittance may vary from the front to the back in the depth direction. The transmittance of the wall portions 96a, 96b may gradually increase or decrease in density in the height direction, and may disappear at a predetermined position in the height direction.

[0136] The control unit 33 can indicate the presence of an object within a predetermined distance range, the distance to the detection object, or a change in the distance to the detection object by changing the transmittance or color density of the wall portions 96a and 96b and the auxiliary lines 96a1 and 96a2. The change in the transmittance or color density of the wall portions 96a and 96b can also be considered a change in gradation. For example, the control unit 33 can indicate that the detection object is within a predetermined distance by moving the auxiliary lines 96a1 and 96a2 on the wall portions 96a and 96b. The control unit 33 can indicate that the detection object is within a predetermined distance by making the color of the near side of the wall portions 96a and 96b the darkest and changing the gradation in the depth direction. The change in the auxiliary lines 96a1 and 96b1 and transmittance of the wall portions 96a and 96b is not essential. The control unit 33 can also prevent the auxiliary lines 96a1 and 96b1 from being displayed on the wall portions 96a and 96b. The control unit 33 can prevent the walls 96a and 96b from having a change (gradation) in transmittance.

[0137] In addition to first distance identification lines 98l and 98r, one or more second distance identification lines 100l and 100r are provided on the floors 97a and 97b of the second recognition wall images 89a and 89b. Although multiple second distance identification lines 100l and 100r are present in FIG. 30, only one of each is shown with a reference symbol. The first distance identification lines 98l and 98r and the second distance identification lines 100l and 100r do not necessarily need to be displayed on the display. The first distance identification lines 98l and 98r correspond to the horizontal distance from the moving object 50 to the detection target, whereas the second distance identification lines 100l and 100r may correspond to the depth distance. Each of the multiple second distance identification lines 100l and 100r may correspond to the depth distance when the auxiliary lines 84c extending in the height direction on the sidewall images 81a and 81b are mapped onto real space.

[0138] The subject 60 can grasp the horizontal and depth directions of the detection target based on the horizontal and depth directions of the icon images 99a and 99b on the floors 97a and 97b. The icon images 99a and 99b are three-dimensional images corresponding to the type of detection target. The floors 97a and 97b may be positioned closer to the user than the foremost end of the sidewall images 81a and 81b and inside a line extending from the sidewall images 81a and 81b. The positional relationship between the second recognition wall images 89a and 89b and the guide wall image 80 is not limited to this and can be changed as appropriate.

[0139] In FIG. 30 , view display areas 101 and 102, which indicate the selection status of “Normal View Display” and “Wide View Display,” are shown in the lower right portion of display device 40. “Normal View Display” and “Wide View Display” differ in the angle of view of the displayed image. In one embodiment, “Normal View Display” corresponds to the angle of view of display area 62 of the second image shown in FIG. 4 . In one embodiment, “Wide View Display” corresponds to the angle of view of detection area 61 of the first image shown in FIG. 4 . “Wide View Display” does not need to be the entire image of detection area 61 of the first image; it may be an image cropped from detection area 61 with an angle of view that includes the “second image.” The second images shown in FIG. 30 and those shown previously are both normal view displays. Whether the image being displayed is “Normal View Display” or “Wide View Display” is indicated by changing the display of view display areas 101 and 102, which indicate the selection status. For example, changing the display of the area showing the selection status includes changing the color, the density, or the color of the characters in the view display areas 101 and 102.

[0140] The control unit 33 can switch between "normal view display" and "wide view display" depending on the result of processing the image of the first video or upon receiving input from the subject 60 via the display device 40 or the like. For example, the control unit 33 may switch the displayed video to "wide view display" when it detects that a detection target is approaching in the "normal view display" state. The control unit 33 may switch the displayed video to "normal view display" when it detects that no detection target is present within a predetermined range in the "wide view display" state. If the display device 40 has a touch panel screen, the control unit 33 may switch the display upon receiving a signal from the display device 40 indicating that the subject 60 has touched and selected the view display area 101 or 102 on the screen.

[0141] FIG. 31 is a diagram showing an example of an image obtained by switching from the "normal view display" shown in FIG. 30 to the "wide view display." The angle of view of the image in the "wide view display" in FIG. 31 is wider in the horizontal direction than the angle of view of the image in the "normal view display" in FIG. 30. In FIG. 31, a pedestrian 63 and a vehicle 64b that are located outside the display range of the "normal view display" shown in FIG. 30 are displayed. Furthermore, because the horizontal angle of view of the image displayed in FIG. 31 is wider than that of the "normal view display" in FIG. 30, the distance between the left and right side wall images 81a and 81b of the guide wall image 80 is displayed relatively narrow.

[0142] In FIG. 31, the pedestrian 63 and the vehicle 64b have entered the display range, and therefore the second recognition wall images 89a and 89b have disappeared. The control unit 33 recognizes the vehicle 64b and the pedestrian 63 to be detected within the detection area 61 of the first image, and can display first recognition wall images 83a and 83b, which are virtual planes, on the moving object 50 side of the vehicle 64b and the pedestrian 63, respectively. The control unit 33 estimates the positions of the vehicle 64b and the pedestrian 63 to be detected in the depth direction. The control unit 33 may change the display of side wall images 81a and 81b facing the vehicle 64b and the pedestrian 63, respectively, based on the estimated positions. In FIG. 31, the display of a portion 81b1 of the side wall image 81b and a portion 81a1 of the side wall image 81a are changed to correspond to the pedestrian 63 and the vehicle 64b, respectively. The change in display may be performed in various ways. For example, changes to the display include changing the color, changing the thickness of the surrounding auxiliary lines, changing the line type, starting and stopping blinking, and changing the blinking cycle, etc. The first recognition wall images 83a and 83b and parts 81a1 and 81b1 of the side wall images 81a and 81b may employ a display method and display mode similar to the example in FIG.

[0143] As shown in FIGS. 30 and 31 , according to one of the embodiments, the image captured by the imaging device 20 can be displayed on the display device 40 by switching between a “normal view display” and a “wide view display” with different angles of view. In the “normal view display,” the subject 60 can easily recognize the presence and position of a detection target located outside the range of the displayed image by using the second recognition wall images 89a and 89b and the icon images 99a and 99b. By automatically or manually switching the display device 40 from the “normal view display” to the “wide view display,” the subject 60 can confirm the detection target detected outside the range of the image displayed in the “normal view display” by viewing the image. In the “wide view display,” the display of the guide wall image 80 and the first recognition wall image 83b allows the subject 60 to grasp the positional relationship with the detection target in three dimensions. In this way, the display system 10 according to one of the embodiments allows the subject 60 to easily understand the surrounding situation and sense danger.

[0144] Although the present invention has been described based on the drawings and embodiments, it should be noted that those skilled in the art would easily be able to make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present invention. For example, the functions included in each means, step, etc. can be rearranged so as not to be logically inconsistent, and multiple means, steps, etc. can be combined or divided into one.

[0145] For example, the components and functions of the display system 10 according to the above-described embodiment may be rearranged. For example, some or all of the configuration and functions of the image processing device 30 may be included in at least one of the imaging device 20 and the display device 40.

[0146] Some of the components of the display system 10 according to the above-described embodiment may be located outside the mobile object 50. For example, the image processing device 30 or the like may be realized as a communication device such as a mobile phone or an external server, and may be connected to the other components of the display system 10 by wire or wirelessly. [Explanation of symbols]

[0147] 10 Display System 20 Imaging device 21 Imaging optical system 22 Image sensor 23 Communications Department 24 Control Unit 30 Image processing device 31 Communications Department 32 Storage section 33 Control Unit 40 Display device 50 Mobile 51 Network 60 Target Audience 61 Detection Area 62 Display area 63, 63a, 63b, 63c, 63d Pedestrians 64, 64a, 64b vehicles 65 First predicted course 66 Guide Lines 67 Icon Images 68 Third Marker 69 Contour 70 areas 71 First Marker 72 Second Marker 73 5th marker 73a Icon Image 73b Obi image 73c Right edge area 74 Obstacle Images 75a, 75b, 75c 6th marker 80 Guide Wall Images 81a, 81b Side wall images 82a, 82b, 82c distance wall images 83, 83a, 83b 1st recognition wall image 84a, 84b, 84c auxiliary line 89a, 89b 2nd recognition wall image 90 Virtual Guide Wall 91a, 91b Virtual side walls 92a, 92b, 92c Virtual distance wall 95l1~95l5, 95r1~95r5 virtual lines 96a, 96b wall section 97a, 97b floor part 98l1~98l5, 98r1~98r5 1st distance identification line 99a, 99b icon images 100l, 100r Second distance discrimination line 101, 102 View display area

Claims

1. a display device provided in the vehicle; a control unit that determines that a distance between the vehicle and an object to the right rear or left rear of the vehicle, which is included in a first image (excluding an overhead image obtained by combining multiple images) acquired from an imaging device that images the rear of the vehicle and which is not included in a second image displayed on the display device, is less than a predetermined distance, and when the distance between the vehicle and the object decreases and the control unit determines that the moving speed of the vehicle is less than a predetermined reference value, composites a marker corresponding to the object onto the second image and displays it on the display device; The control unit is capable of acquiring a moving speed of the vehicle. Display system.

2. The display system according to claim 1 , wherein the control unit changes a display mode of the marker depending on a distance between the vehicle and the object.

3. The display system according to claim 2 , wherein the display mode is a width of the marker.

4. The display system described in claim 1, wherein the control unit causes the display device to display the marker in a manner indicating that the object is approaching the vehicle when the distance between the vehicle and the object decreases and the control unit determines that the vehicle's movement speed is less than a predetermined reference value.

5. The imaging device is further provided, the display device displays the second image, which is a part of the first image captured by the imaging device of the imaging area; the control unit detects an object within the imaging area to the right rear or left rear of the vehicle and outside the second image, and when the control unit determines that there is a possibility of contact between the vehicle and the object because the distance between the vehicle and the object is less than a predetermined distance, causes the display device to display the marker. A display system according to any one of claims 1 to 3.

6. A mobile object comprising a display system according to any one of claims 1 to 5.

7. An image processing device mounted on a vehicle, a control unit that determines that a distance between the vehicle and an object to the right rear or left rear of the vehicle, which is included in a first image (excluding an overhead image obtained by combining multiple images) acquired from an imaging device that images the rear of the vehicle and which is not included in a second image displayed on the display device, is less than a predetermined distance, and when the control unit determines that the distance between the vehicle and the object is decreasing and that the moving speed of the vehicle is less than a predetermined reference value, composites a marker corresponding to the object onto the second image and displays the composited image on a display device external to the image processing device; The control unit is capable of acquiring a moving speed of the vehicle. Image processing device.

8. A communication unit that acquires a first image (excluding an overhead image synthesized from multiple images) of the rear of the vehicle from an imaging device; a control unit that determines that a distance between the vehicle and an object to the right rear or left rear of the vehicle, which is included in the first image acquired from the imaging device via the communication unit and is not included in the second image displayed on the display device, is less than a predetermined distance, and when the distance between the vehicle and the object decreases and the control unit determines that the moving speed of the vehicle is less than a predetermined reference value, composites a marker corresponding to the object onto the second image and displays it; The control unit is capable of acquiring a moving speed of the vehicle. A display device comprising:

Images