Display system, mobile body, image processing apparatus, and display system

Abstract

To improve the convenience of a technology for displaying images of an external area of a moving object.SOLUTION: An imaging device 20 includes an imaging element 22 and a control unit 33. The imaging element 22 captures an image behind a vehicle 50 to generate an image. The control unit 33 displays a recognition image on the image at a position in the image corresponding to a position between the vehicle and a detection target, the recognition image extending a predetermined distance in the height direction relative to the road surface and the transmittance increasing with increasing distance in the height direction from the road surface.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to Display system, mobile body, image processing apparatus, and display system .

Background Art

[0002] Conventionally, a technique for displaying an image of an external area of a moving body such as a vehicle has been known. For example, Patent Document 1 discloses a technique for controlling power supply to a monitor that displays an image of a camera provided in a vehicle.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventionally, there has been room for improvement in the technique for displaying an image of an external area of a moving body.

[0005] The present disclosure relates to an image processing apparatus, an imaging apparatus, a display apparatus, and a display system that improve the convenience of a technique for displaying an image of an external area of a moving body.

Means for Solving the Problems

[0006] According to an embodiment of the present disclosure, The display system is a display system mounted on a vehicle, and control means for causing a marker to be displayed on a display means when an object behind the vehicle is detected while the vehicle is moving at a moving speed less than a reference value is provided.

[0007] According to an embodiment of the present disclosure, The mobile body is the above display system is provided.

[0008] According to an embodiment of the present disclosure, The image processing apparatus is an image processing apparatus mounted on a vehicle, and a control unit for causing a marker to be displayed on a display means when an object behind the vehicle is detected while the vehicle is moving at a moving speed less than a reference value is provided.

[0009] A display system according to an embodiment of the present disclosure includes A display system mounted on a vehicle, imaging means for imaging the rear of the vehicle and generating a second video, and control means for superimposing a marker on the second video when an object behind the vehicle is detected while the vehicle is moving at a moving speed less than a reference value and includes.

[0010] According to an embodiment of the present disclosure The image processing apparatus is an image processing apparatus mounted on a vehicle having a display means for displaying a video, and control means for causing a marker indicating that an object behind the vehicle moving at a moving speed less than a reference value has been detected to be displayed on the display means and includes.

Advantages of the Invention

[0012] According to an image processing apparatus, an imaging apparatus, a display apparatus, and a display system according to an embodiment of the present disclosure, the convenience of a technique for displaying an image of an external area of a moving body is improved.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0015] (Display system) With reference to FIG. 1, a display system 10 according to an embodiment of the present invention will be described.

[0016] As shown in FIG. 1, the display system 10 includes an imaging device 20, an image processing device 30, and a display device 40. Each component of the imaging device 20 and the display system 10 can transmit and receive information via, for example, a network 51. The network 51 may include, for example, wireless, wired, or CAN (Controller Area Network), etc.

[0017] In other embodiments, some or all of the components of the display system 10 may be integrally configured as one device. For example, a configuration in which the image processing device 30 is built into the imaging device 20 or the display device 40 is conceivable.

[0018] As shown in FIG. 2, the imaging device 20, the image processing device 30, and the display device 40 may be provided in the moving body 50. The "moving body" in the present disclosure may include, for example, vehicles, ships, and aircrafts. Vehicles may include, for example, automobiles, industrial vehicles, railway vehicles, living vehicles, and fixed-wing aircrafts running on runways. Automobiles may include, for example, passenger cars, trucks, buses, two-wheeled vehicles, and trolley buses. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction. Industrial vehicles may include, for example, forklifts and golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, and lawn mowers. Industrial vehicles for construction may include, for example, bulldozers, scrapers, shovel cars, crane cars, dump trucks, and road rollers. Vehicles may include those that run manually. The classification of vehicles is not limited to the examples described above. For example, automobiles may include industrial vehicles that can run on roads. The same vehicle may be included in multiple classifications. Ships may include, for example, marine jets, boats, and tankers. Aircrafts may include, for example, fixed-wing aircrafts and rotary-wing aircrafts.

[0019] The imaging device 20 can image the external area of the moving body 50. The position of the imaging device 20 is arbitrary inside and outside the moving body 50. For example, as shown in FIG. 2, the imaging device 20 is located behind the moving body 50 and can image the external area behind the moving body 50. The position of the image processing device 30 is arbitrary inside the moving body 50. The display device 40 is visible to the subject 60. The position of the display device 40 is arbitrary in the moving body 50. For example, as shown in FIG. 2, the display device 40 is located in the dashboard of the moving body 50.

[0020] (Imaging device) The imaging device 20 will be described in detail. For example, as shown in FIG. 1, the imaging device 20 includes an imaging optical system 21, an image sensor 22, a communication unit 23, and a control unit 24.

[0021] The imaging optical system 21 forms an image of the subject. For example, the imaging optical system 21 may include a diaphragm and one or more lenses.

[0022] The image sensor 22 has a plurality of pixels arranged in a two-dimensional array. The image sensor 22 may include, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 22 can image the subject image formed by the imaging optical system 21 and generate an imaging image.

[0023] The communication unit 23 may include a communication interface capable of communicating with an external device. The communication unit 23 may be capable of transmitting and receiving information via the network 51. The external device may include, for example, the image processing device 30. The "communication interface" in the present disclosure may include, for example, a physical connector and a wireless communication device. The physical connector may include an electrical connector corresponding to transmission by an electrical signal, an optical connector corresponding to transmission by an optical signal, and an electromagnetic connector corresponding to transmission by an electromagnetic wave. The electrical connector may include a connector compliant with IEC60603, a connector compliant with the USB standard, a connector corresponding to an RCA terminal, a connector corresponding to an S terminal defined in EIAJ CP-1211A, a connector corresponding to a D terminal defined in EIAJ RC-5237, a connector compliant with the HDMI (registered trademark) standard, and a connector corresponding to a coaxial cable including a BNC (British Naval Connector or Baby-series N Connector, etc.). The optical connector may include various connectors compliant with IEC 61754. The wireless communication device may include a wireless communication device compliant with each standard including Bluetooth (registered trademark) and IEEE802.11. The wireless communication device includes at least one antenna.

[0024] The control unit 24 includes one or more processors. The "processor" in the present disclosure may include a dedicated processor specialized for a specific process and a general-purpose processor that executes a specific function by loading a specific program. The dedicated processor may include a DSP (Digital Signal Processor) and an ASIC (Application Specific Integrated Circuit). The processor may include a programmable logic device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The control unit 24 may be either a SoC (System-on-a-Chip) or a SiP (System In a Package) in which one or more processors cooperate.

[0025] The control unit 24 controls the operation of the entire imaging device 20. The control unit 24 may cause the imaging element 22 to generate an imaging image at an arbitrary frame rate. The frame rate may, for example, substantially match the frame rate that the display device 40 can display. The control unit 24 may execute predetermined image processing on the generated imaging image. The image processing may include, for example, exposure adjustment processing, white balance processing, and distortion correction processing. The control unit 24 outputs the imaging image to the image processing device 30 via the communication unit 23. For example, the control unit 24 may sequentially output the imaging images at the above-mentioned frame rate. Hereinafter, each imaging image output at the above-mentioned frame rate is also simply referred to as a frame. The plurality of imaging images output from the imaging device 20 is also referred to as a first video. For example, when the frame rate is 60 fps (Frames per Seconds), 60 imaging images per second are output as the first video.

[0026] (Image processing device) The image processing device 30 will be described in detail. The image processing device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0027] The communication unit 31 may include a communication interface capable of communicating with various external devices. The external devices may include, for example, an imaging device 20, a display device 40, an ECU (Electronic Control Unit or Engine Control unit) provided in a moving body 50, a speed sensor, an acceleration sensor, a rotation angle sensor, a steering wheel angle sensor, an engine speed sensor, an accelerator sensor, a brake sensor, an illuminance sensor, a raindrop sensor, a travel distance sensor, an obstacle detection device using a millimeter wave radar, an ultrasonic sonar, etc., an ETC (Electronic Toll Collection System) receiver, a GPS (Global Positioning System) device, a navigation device, a server on the Internet, a mobile phone, and the like.

[0028] The communication unit 31 may include a communication interface for vehicle-to-pedestrian communication, vehicle-to-roadside communication, and vehicle-to-vehicle communication. The communication unit 31 may include a receiver corresponding to optical beacons of DSRC (Dedicated Short-Range Communication) and VICS (registered trademark) (Vehicle Information and Communication System) provided in Japan. The communication unit 31 may include a receiver corresponding to a road traffic information providing system in other countries.

[0029] The communication unit 31 may be able to acquire various information from external devices. For example, the communication unit 31 may be able to acquire mobile body information and environmental information.

[0030] The mobile body information may include any information regarding the mobile body 50. The mobile body information may include, for example, the speed, acceleration, turning gravity, inclination, direction, and turning situation of the mobile body 50, the steering angle of the steering wheel, the temperature of the cooling water, the remaining fuel amount, the remaining battery amount, the battery voltage, the engine speed, the gear position, the presence or absence of a reverse signal, the presence or absence of an accelerator operation, the accelerator opening, the presence or absence of a brake operation, the brake depression degree, the operation status of the parking brake, the rotational speed difference between the front and rear wheels or the four wheels, the tire air pressure, the expansion and contraction amount of the damper, the spatial position of the driver's eyes, the number of passengers and the seat position, the seat belt wearing information, the opening and closing of the door, the opening and closing of the window, the interior temperature, the operation status of the air conditioner, the set temperature of the air conditioner, the air volume of the air conditioner, the setting of the outside air circulation, the operation status of the wiper, the driving mode, the connection information with external devices, the current time, the average fuel consumption, the instantaneous fuel consumption, the lighting status of various lamps, the position information of the mobile body 50, and the route information to the destination of the mobile body 50, etc. The various lamps may include, for example, headlamps, fog lamps, back lamps, position lamps, and turn signals.

[0031] The environmental information may include any information regarding the external environment of the mobile body 50. The environmental information may include, for example, the brightness around the mobile body 50, the weather, the atmospheric pressure, the outside air temperature, the map information, the traffic information, the road construction information, the temporary change of the speed limit on the driving road, the objects detected by other vehicles, and the lighting status of traffic signals, etc.

[0032] The storage unit 32 may include a temporary storage device and a secondary storage device. The storage unit 32 may be configured using, for example, a semiconductor memory, a magnetic memory, and an optical memory, etc. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape, etc. The optical memory may include, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), and a BD (Blu-ray Disc) (registered trademark), etc. The storage unit 32 stores various information and programs necessary for the operation of the image processing apparatus 30.

[0033] The control unit 33 includes one or more processors. The control unit 33 controls the operation of the entire image processing apparatus 30.

[0034] The control unit 33 may acquire moving object information and environmental information from an external device via the communication unit 31. The control unit 33 may determine, for example, a predicted travel route of the moving object 50 based on the moving object information. Hereinafter, the predicted travel route of the moving object 50 is also referred to as a first predicted travel route.

[0035] The control unit 33 may acquire a first video from the imaging device 20 via the communication unit 31. The first video includes a detection area and a display area.

[0036] The control unit 33 may detect at least a part of a detection target within the detection area on the acquired first video. The detection area on the first video may be at least a part of an area on the imaging image that is each frame of the first video. Each frame of the first video may be referred to as an imaging image. The detection area on the first video may be larger than the display area. The detection area on the first video may encompass the display area. The control unit 33 may detect the detection target inside the display area. The control unit 33 may detect the detection target outside the display area and inside the detection area. The area inside the detection area and the display area may be referred to as a first area. The area inside the detection area and outside the display area may be referred to as a second area.

[0037] The detection target may include a plurality of types of objects. The types of objects may include, for example, people, other moving bodies, roadways, lanes, white lines, gutters, sidewalks, crosswalks, road signs, traffic signs, guardrails, walls, and traffic lights, etc. The types of detection targets that the control unit 33 can detect are not limited to these. At least a part of the detection target may include, for example, when a part of the detection target on the first video is hidden by the shadow of another object, the part of the detection target that is not hidden by the shadow of the other object. For example, when the lower body of a pedestrian is hidden by the shadow of an obstacle on the first video, the control unit 33 may be able to detect the upper body of the pedestrian. Any object detection algorithm may be adopted for the detection of at least a part of the detection target. For example, the control unit 33 may detect at least a part of the detection target by an algorithm such as pattern matching or feature point extraction using the captured image that is each frame of the first video.

[0038] When at least a part of the detection target is detected on the first video, the control unit 33 may determine the predicted path of the detection target based on the first video. Hereinafter, the predicted path of the detection target is also referred to as the second predicted path. Any algorithm may be adopted for the determination of the second predicted path. For example, the control unit 33 may determine the second predicted path based on the changes in the orientation and position of the detection target on the captured image that is each frame of the first video.

[0039] When at least a part of the detection target is detected on the first video, the control unit 33 may estimate the relative positional relationship between the moving body 50 and the detection target based on the first video. The relative positional relationship may include, for example, the distance between the moving body 50 and the detection target, and the presence or absence of overlap between the first predicted path of the moving body 50 and the second predicted path of the detection target. Any algorithm may be adopted for the estimation of the distance between the moving body 50 and the detection target. For example, the control unit 33 may estimate the distance between the moving body 50 and the detection target by the motion stereo method using the captured image that is each frame of the first video signal. In other embodiments, the control unit 33 may obtain information indicating the relative positional relationship between the moving body 50 and the detection target from an external device via the communication unit 31.

[0040] When the distance between the moving body 50 and the detection target decreases, the control unit 33 may determine which of the moving body 50 and the detection target contributes more to the decrease in the distance. Any algorithm may be adopted to determine the contributions of the moving body 50 and the detection target to the decrease in the distance. In one example, the control unit 33 may detect the moving speed of the moving body 50 based on the moving body information. The control unit 33 may detect the moving speed of the detection target based on, for example, the change in the position of the detection target on the imaging image that is each frame of the first video. The control unit 33 may determine that the contribution of the one with the higher moving speed among the moving body 50 and the detection target is greater with respect to the decrease in the distance. In another example, when the moving speed of the moving body 50 is less than the reference value, the control unit 33 may determine that the contribution of the detection target is greater with respect to the decrease in the distance. When the moving speed of the moving body 50 is greater than or equal to the reference value, the control unit 33 may determine that the contribution of the moving body 50 is greater with respect to the decrease in the distance. The reference value may be arbitrarily determined, and may be determined to be, for example, substantially zero. Details of the operation of the image processing apparatus 30 according to the contributions of the moving body 50 and the detection target to the decrease in the distance will be described later.

[0041] When at least a part of the detection target is detected on the first video, the control unit 33 may determine whether there is a possibility that the moving body 50 contacts the detection target based on the first video. Any algorithm may be adopted to determine the possibility of contact between the moving body 50 and the detection target. For example, when at least one of the condition that the distance between the moving body 50 and the detection target is less than a predetermined threshold value and the condition that the decreasing speed of the distance is greater than or equal to a predetermined threshold value is satisfied, the control unit 33 may determine that there is a possibility that the moving body 50 contacts the detection target. Details of the operation of the image processing apparatus 30 according to the presence or absence of the possibility will be described later.

[0042] The control unit 33 may cause the display device 40 to display a second video corresponding to the display area on the first video acquired from the imaging device 20. Specifically, the control unit 33 may output the second video to the display device 40 via the communication unit 31. For example, when the control unit 33 detects the reverse movement of the moving body 50 based on the moving body information, the control unit 33 may cause the display device 40 to display the second video. For example, the control unit 33 may detect the reverse movement based on the shift position of the transmission gear. For example, the control unit 33 may detect the reverse movement based on the reverse signal output from the moving body during reverse movement. The second video may be, for example, a video obtained by cutting out the display area on the captured image which is each frame of the first video. The display area on the first video may be at least a part of the area on the captured image which is each frame of the first video. The display area may be smaller than the detection area. The display area may be included in the detection area. The position, shape, and size of the display area may be arbitrarily determined. The control unit 33 may change the position, shape, and size of the display area. By changing the position, shape, and size of the display area, for example, the display area and the detection area may substantially coincide.

[0043] The control unit 33 may synthesize various markers into the second video and cause the display device 40 to display them. The synthesis includes overwriting and mixing. The marker may include, for example, one or more images. The control unit 33 may dynamically change at least a part of the display mode of the marker superimposed on the second video. The display mode may include, for example, the position, size, shape, color, and shading, etc. of at least a part of the marker on the second video. When the control unit 33 displays a marker corresponding to the detection target detected on the first video, the control unit 33 may determine the display mode of the marker according to the type of the detection target. Details of the operation of the image processing device 30 that causes the display device 40 to display various markers will be described later.

[0044] (Display device) The display device 40 will be described in detail. The display device 40 may include, for example, a liquid crystal display and an organic EL (Electroluminescence) display. The display device 40 may display a second video input from the image processing device 30 via, for example, the network 51. The display device 40 may function as a touch screen capable of receiving operations by the user. The display device 40 may include switches and keys capable of receiving operations by the user. The switches may include physical switches and electronic switches. The keys may include physical keys and electronic keys. When the display device 40 receives an operation by the user on the device itself, it may transmit a user input based on the operation to the image processing device 30.

[0045] The display device 40 can be arranged at various locations of the moving body 50. FIGS. 3A to 3E are diagrams showing examples of the arrangement of the display device 40 when the moving body 50 is a vehicle. FIG. 3A shows an in-dash type display device 40a stored in the dashboard of the vehicle. FIG. 3B shows an on-dash type display device 40b arranged on the dashboard. In FIG. 3B, the display device 40b is incorporated in the vehicle 50. The display device 40b may be detachably attached to the dashboard. FIG. 3C shows a display device 40c built into the rearview mirror and capable of displaying a video as needed. FIG. 4D shows a display device 40d built into the instrument panel. In FIG. 4D, the display device 40d is arranged adjacent to instruments such as a speedometer and a tachometer. In one embodiment, the entire instrument panel may be a display device 40d such as an LCD, and a second video may be displayed therein together with images of a speedometer and a tachometer. FIG. 4E shows a display device 40e using a portable information terminal, for example, a tablet terminal. A mobile phone display may be used for the display device 40e.

[0046] With reference to FIGS. 4 to 18, the second video and various markers that the image processing apparatus 30 causes the display apparatus 40 to display will be specifically described. In the present disclosure, the terms "vertical direction" and "horizontal direction" with respect to a video or an image correspond to two-dimensional directions in the video or the image. In the present disclosure, the terms "height direction", "horizontal direction", and "depth direction" with respect to a video or an image correspond to three-dimensional directions of the space projected by the video or the image.

[0047] (First Example (Reference Example)) FIG. 4 shows a first example of the detection region 61 in the first video acquired by the image processing apparatus 30 from the imaging apparatus 20. In the example shown in FIG. 4, the detection region 61 is longer in the horizontal direction than in the vertical direction. The display region 62 is located at the center in the horizontal direction of the detection region 61. The control unit 33 may detect each of the pedestrian 63 and the vehicle 64 reflected inside the display region 62 on the first video as a detection target.

[0048] The control unit 33 determines whether or not one or more conditions are satisfied based on the relative positional relationship between the detection target detected inside the display region 62 on the first video and the moving body 50. The one or more conditions may include, for example, a first condition that the detection target is located within the first predicted path 65 of the moving body 50. The one or more conditions may include, for example, a second condition that at least a part of the first predicted path 65 of the moving body 50 overlaps with at least a part of the second predicted path of the detection target. When it is determined that the one or more conditions are satisfied, the control unit 33 may superimpose a predetermined marker corresponding to the detection target on the second video and cause the display apparatus 40 to display it. The predetermined marker may include a first marker, a second marker, and a third marker.

[0049] In the first example, the control unit 33 may determine that the one or more conditions are satisfied for the pedestrian 63. In such a case, the control unit 33 may display a marker corresponding to the pedestrian 63. The control unit 33 may determine that the one or more conditions are not satisfied for the vehicle 64. In such a case, the control unit 33 does not display a marker corresponding to the vehicle 64.

[0050] FIG. 5 shows an example of a second video corresponding to the display area 62 of the first video shown in FIG. 4. When the aspect ratio of the display area 62 of the first video is different from the aspect ratio of the screen of the display device 40, the control unit 33 may cut out the display area 62 of the first video and then output a second video deformed according to the aspect ratio of the screen of the display device 40 to the display device 40. As shown in FIG. 5, a pedestrian 63 and a vehicle 64 are shown on the second video.

[0051] The control unit 33 may superimpose, on the second video, a guide line 66 indicating at least a part of the first predicted path 65 of the moving body 50 shown in FIG. 4, for example, and display it on the display device 40. The control unit 33 may dynamically change the guide line 66, for example, according to a change in the steering angle of the steering wheel.

[0052] The first video has a wider range than the display area 62. The control unit 33 can change the range of the display area 62. The control unit 33 may superimpose an icon image 67 on the second video and display it on the display device 40. For example, the contour 67a of the icon image 67 shown in FIG. 5 indicates the maximum range when the range of the display area 62 is changed. The white rectangle 67b of the icon image 67 indicates the display area 62. The icon image 67 shown in FIG. 5 indicates the relative position and size of the display area 62 with respect to the maximum range of the display area 62.

[0053] FIG. 6 shows an example of a marker superimposed on the pedestrian 63 on the second video, for example. Hereinafter, the marker is also referred to as a third marker 68. The contour line 69 of the third marker 68 may substantially coincide with the contour line of the pedestrian 63 detected on the second video. The area 70 inside the contour line 69 of the third marker 68 may be filled with a color or pattern corresponding to, for example, the type of detection target, "person". When the pedestrian 63 is detected on the first video, the control unit 33 may superimpose the third marker 68 on the pedestrian 63 on the second video and display it on the display device 40. According to such a configuration, the target person 60 can easily visually recognize the pedestrian 63 on the second video. The control unit 33 may turn off the third marker 68 when a predetermined time has elapsed since the third marker 68 was displayed.

[0054] FIG. 7 shows an example of two types of markers superimposed on the periphery of the pedestrian 63 on the second video. Hereinafter, each of the two types of markers is also referred to as a first marker 71 and a second marker 72. The control unit 33 may superimpose the first marker 71 and the second marker 72 on the second video and display them on the display device 40, for example, after making the third marker 68 non-displayed.

[0055] The control unit 33 may move the position of the first marker 71 following the pedestrian 63 on the second video. Since the first marker 71 follows the pedestrian 63, the subject 60 can easily capture the pedestrian 63. The first marker 71 is displayed away from the pedestrian 63 around the pedestrian 63. When the first marker 71 is displayed on the display device 40, the subject 60 can easily grasp the behavior of the pedestrian 63. The control unit 33 may relatively change the superimposed position of the second marker 72 with respect to the superimposed position of the first marker 71 on the second video. The control unit 33 may relatively move the second marker based on the position of the first marker 71.

[0056] For example, when the distance between the moving body 50 and the pedestrian 63 is decreasing, the control unit 33 may determine that the contribution of the moving body 50 to the decrease in the distance is large. In such a case, the control unit 33 may move the second marker 72 toward the first marker 71. First, the control unit 33 displays the second marker 72 at a position away from the first marker 71. Next, the control unit 33 moves the second marker 72 toward the first marker 71 until the distance from the first marker 71 reaches a predetermined distance. Next, the control unit 33 turns off the second marker 72. Next, the control unit 33 displays the second marker 72 again at a position away from the first marker 71 and repeats the above-described process. In this example, since the second marker 72 is moving toward the object of the first marker 71, the subject 60 can understand that the second marker 72 is approaching the first marker 71.

[0057] For example, when the distance between the mobile body 50 and the pedestrian 63 is decreasing, the control unit 33 may determine that the contribution of the pedestrian 63 to the decrease in the distance is large. In such a case, the control unit 33 may move the second marker 72 away from the first marker 71. First, the control unit 33 causes the second marker 72 to be displayed at a position close to the first marker 71. Next, the control unit 33 moves the second marker 72 away from the first marker 71 until the distance from the first marker 71 reaches a predetermined distance. Next, the control unit 33 deletes the second marker 72. Next, the control unit 33 causes the second marker 72 to be displayed again at a position close to the first marker 71 and repeats the above-described processing. In this example, since the second marker 72 is moving away from the object of the first marker 71, the subject 60 can understand that the second marker 72 is moving away from the first marker 71.

[0058] The control unit 33 changes the moving direction of the second marker 72 with respect to the first marker 71 according to the degree of contribution of the mobile body 50 and the pedestrian 63 to the decrease in the distance between the mobile body 50 and the pedestrian 63. The subject 60 can identify whether, for example, the mobile body 50 is approaching the pedestrian 63 or the pedestrian 63 is approaching the mobile body 50 according to the moving direction of the second marker 72.

[0059] The control unit 33 may repeat the enlargement or reduction of the second marker 72 centered on the first marker 71 on the second video, for example, as shown in FIG. 8. The control unit 33 may superimpose the first marker 71 and the second marker 72 having the same shape as the contour line 69 of the pedestrian 63 on the second video, for example, as shown in FIG. 9. The control unit 33 may repeat the enlargement or reduction of the second marker 72 centered on the first marker 71. The control unit 33 switches the enlargement or reduction of the second marker 72 according to the degree of contribution of the mobile body 50 and the pedestrian 63 to the decrease in the distance between the mobile body 50 and the pedestrian 63.

[0060] When the distance between the detection target on which the first marker 71 and the second marker 72 are displayed and the moving body 50 becomes less than a predetermined threshold value, the control unit 33 may superimpose a new marker on the second video. Hereinafter, the new marker is also referred to as a fourth marker. The fourth marker may include an arbitrary image. For example, the fourth marker may include an image showing an exclamation mark "!". According to such a configuration, for example, when the pedestrian 63 on which the first marker 71 and the second marker 72 are displayed and the moving body 50 approach each other to a certain extent or more, the fourth marker is superimposed and displayed on the second video. The subject 60 can identify, by the fourth marker, that, for example, the pedestrian 63 is located in the vicinity of the moving body 50. In another embodiment, when the distance between the detection target on which the first marker 71 and the second marker 72 are displayed and the moving body 50 becomes less than a predetermined threshold value, the control unit 33 may change the display mode of the first marker 71 and the second marker 72. The control unit 33 may change, for example, the colors of the first marker 71 and the second marker 72. Also according to such a configuration, the subject 60 can identify, by the change in the color of the marker, that, for example, the pedestrian 63 is located in the vicinity of the moving body 50.

[0061] The control unit 33 can detect two detection targets arranged in the front-back direction in the depth direction. The control unit 33 may display the first marker 71 and the second marker 72 on each of the two detection targets located in the front and back. The control unit 33 may attach different first markers 71 and second markers 72 to the two detection targets. For example, the control unit 33 may attach the first marker 71 and the second marker 72 that are less conspicuous than the first marker 71 and the second marker 72 attached to the second detection target located on the front side to the first detection target located on the back side. For example, the control unit 33 may change the first marker 71 and the second marker 72 attached to the first detection target located on the back side to be darker in color, higher in transmittance, thinner in line, etc. than the first marker 71 and the second marker 72 attached to the second detection target located on the front side.

[0062] (Second - Fourth Examples (Reference Examples)) FIG. 10 shows a second example of the detection area 61 in the first video acquired by the image processing apparatus 30 from the imaging apparatus 20. In the example shown in FIG. 10, the detection area 61 is longer in the left - right direction than in the up - down direction. The display area 62 is located at the center in the left - right direction of the detection area 61. The control unit 33 may detect the pedestrian 63a reflected inside the display area 62 on the first video, and the pedestrians 63b and 63c reflected outside the display area 62 and inside the detection area 61 as detection targets respectively. The processing of the control unit 33 regarding the pedestrian 63a is the same as the processing regarding the pedestrian 63 shown in FIG. 4, for example.

[0063] When the detection position on the first video where the detection target is detected is outside the display area 62 and inside the detection area 61, the control unit 33 may superimpose a marker corresponding to the detection target on the second video and display it on the display device 40. Hereinafter, the marker is also referred to as the fifth marker. When it is determined that there is a possibility that the moving body 50 and the detection target may come into contact, the control unit 33 may display the fifth marker. When the detection position on the first video where the detection target is detected is on the right side of the display area 62, the control unit 33 may superimpose the fifth marker on the right end of the second video and display it on the display device 40. When the detection position on the first video where the detection target is detected is on the left side of the display area 62, the control unit 33 may superimpose the fifth marker on the left end of the second video and display it on the display device 40.

[0064] In the second example, the detection position where the pedestrian 63b is detected is on the right side of the display area 62. The control unit 33 may determine that there is a possibility that the moving body 50 and the pedestrian 63b may come into contact. In such a case, the control unit 33 may superimpose the fifth marker corresponding to the pedestrian 63b on the right end of the second video and display it on the display device 40. Details of the fifth marker corresponding to the pedestrian 63b will be described later. The detection position where the pedestrian 63c is detected is on the left side of the display area 62. The control unit 33 may determine that there is no possibility that the moving body 50 and the pedestrian 63c may come into contact. In such a case, the control unit 33 does not have to display the fifth marker corresponding to the pedestrian 63b.

[0065] FIG. 11 shows an example of a second video corresponding to the display area 62 of the first video shown in FIG. 10. As shown in FIG. 11, a pedestrian 63a is shown on the second video. Pedestrians 63b and 63c are not shown on the second video.

[0066] As shown in FIG. 11, the control unit 33 may, for example, superimpose the obstacle image 74 on the second video and display it on the display device 40. The obstacle image 57 shows the detection result of an obstacle detection device using an ultrasonic sonar or the like provided in the moving body 50. The obstacle image 74 may include an image 74a, an image 74b, and an image 74c. The image 74a is an image of the moving body 50 viewed from above. The image 74b is an image indicating that an obstacle has been detected behind the left rear of the moving body 50. The image 74c is an image indicating that an obstacle has been detected behind the right rear of the moving body 50. The detection result of the obstacle by the obstacle detection device and the detection result of the detection target by the control unit 33 do not necessarily have to match. For example, in the example shown in FIG. 11, the obstacle image 74 indicates that obstacles have been detected behind the right rear and the left rear of the moving body 50, respectively. On the other hand, the control unit 33 may determine that there is no possibility of contact with the moving body 50 for the pedestrian 63c existing behind the left rear of the moving body 50. In such a case, the control unit 33 does not have to display the fifth marker corresponding to the pedestrian 63c.

[0067] An example of the fifth marker 73 corresponding to the pedestrian 63b is shown in FIG. 11. The fifth marker 73 may include an icon image 73a and a band image 73b. The icon image 73a may be an image corresponding to the type of detection target, "person". The subject 60 who has visually recognized the icon image 73a can recognize that there is a person to the right of the second video. The band image 73b is, for example, a strip-shaped image extending in the vertical direction on the second video. The band image 73b may be filled with a color or pattern corresponding to the type of detection target, "person", for example. The control unit 33 may move the band image 73b within the right end area 73c on the second video. The control unit 33 may change the moving speed and width of the band image 73b.

[0068] A detailed description will be given of the fifth marker 73. The control unit 33 may determine the width of the strip image 73b according to the distance between the moving body 50 and the pedestrian 63b. For example, the control unit 33 may increase the width of the strip image 73b as the distance gets closer. The subject 60 who visually recognizes the strip image 73b can recognize the distance between the moving body 50 and the pedestrian 63b based on the width of the strip image 73b.

[0069] The control unit 33 may determine that, for example, the contribution of the moving body 50 is large with respect to the decrease in the distance between the moving body 50 and the pedestrian 63b. In such a case, the control unit 33 repeatedly moves the strip image 73b in the first direction within the right end region 73c on the second video. The first direction may be, for example, the direction from the outside to the inside in the left - right direction on the second video. The control unit 33 may determine that, for example, the contribution of the pedestrian 63b is large with respect to the decrease in the distance between the moving body 50 and the pedestrian 63b. In such a case, the control unit 33 repeatedly moves the strip image 73b in the second direction within the right end region 73c on the second video. The second direction may be, for example, the direction from the inside to the outside in the left - right direction on the second video. The subject 60 who visually recognizes the strip image 73b can recognize whether the moving body 50 is approaching the pedestrian 63b or the pedestrian 63b is approaching the moving body 50 based on the moving direction of the strip image 73b.

[0070] The control unit 33 may determine the moving speed of the strip image 73b according to the decreasing speed of the distance between the moving body 50 and the pedestrian 63b. For example, the faster the decreasing speed of the distance, the faster the moving speed of the strip image 73b may be. The subject 60 who visually recognizes the strip image 73b can recognize the decreasing speed of the distance between the moving body 50 and the pedestrian 63b based on the moving speed of the strip image 73b.

[0071] When the control unit 33 detects a user input in response to a predetermined user operation, for example, while the fifth marker 73 is being displayed, the control unit 33 may change the display area 62 so that the detection position of the pedestrian 63b on the first video is included inside the display area 62. For example, as shown in FIG. 12, the control unit 33 may lengthen the display area 62 on the first video in the left-right direction and move the display area 62 closer to the right side within the detection area 61. With such a configuration, as shown in FIG. 13 for example, the pedestrian 63b is shown on the second video.

[0072] The above-described predetermined user operation may include any user operation. For example, the above-described predetermined user operation may include a first user operation for changing the steering angle of the steering of the moving body 50. The fifth marker 73 may function as a GUI (Graphic User Interface) that accepts a second user operation. Hereinafter, the GUI is also referred to as an interface image. In such a case, the above-described predetermined user operation may include the second user operation.

[0073] The control unit 33 may automatically change the display area 62 so that the detection position of the pedestrian 63b on the first video is included inside the display area 62. In this case, the control unit 33 may maintain the automatic change of the display area 62 until the pedestrian 63b is no longer detected in the detection area 61 of the first video.

[0074] The control unit 33 may change the icon image 67 in response to a change in the display area 62, as shown in FIG. 13 for example.

[0075] The control unit 33 may change the display area 62 on the first video in response to, for example, a pinch-in operation and a pinch-out operation on the display device 40. For example, as shown in FIG. 14, the control unit 33 may make the display area 62 substantially coincide with the detection area 61. In such a case, as shown in FIG. 15 for example, all detection targets within the detection area 61 are displayed on the display device 40.

[0076] (Fifth Example (Reference Example)) FIG. 16 shows a fifth example of the detection area 61 in the first video acquired by the image processing apparatus 30 from the imaging apparatus 20. In the example shown in FIG. 16, the detection area 61 is longer in the horizontal direction than in the vertical direction. The display area 62 is located at the center of the detection area 61 in the horizontal direction. The control unit 33 may detect the vehicle 64a shown in the first predicted path 65 of the moving body 50 and the vehicles 64b and the pedestrian 63d shown outside the first predicted path 65 as detection targets, respectively.

[0077] In the fifth example, the case where the external area of the moving body 50 is dark, such as at night or in a tunnel, will be described. When the external area of the moving body 50 is dark, the characteristic values of the first video and the second video may decrease. The characteristic value may include any parameter related to the visibility of the video. For example, the characteristic value may include at least one of, for example, the luminance value and the contrast ratio of the video. When the characteristic value of the second video decreases, the visibility of the second video may decrease.

[0078] The control unit 33 may execute specific image processing on the area corresponding to the detection target on the second video. The specific image processing may include a first process of superimposing a marker corresponding to the detection target on the area. Hereinafter, the marker is also referred to as the sixth marker. The sixth marker may include, for example, an image that substantially matches the contour shape of the detection target on the second video. According to such a configuration, the sixth marker is superimposed and displayed on the detection target on the second video. Therefore, even when the characteristic value of the second video is low, the subject 60 can easily recognize the detection target on the second video. The specific image processing may include a second process of changing the characteristic value of the area corresponding to the detection target on the second video. For example, the control unit 33 may change the characteristic value of the area so as to improve the visibility of the area on the second video. According to such a configuration, the visibility of the detection target on the second video is improved. Therefore, even when the characteristic value of the second video is low, the subject 60 can easily recognize the detection target on the second video.

[0079] The control unit 33 may execute the above-described specific image processing when one or more conditions are satisfied. The one or more conditions may include the condition that the detection target is located within the first predicted path 65 of the moving body 50. The one or more conditions may include the condition that the first predicted path 65 of the moving body 50 and the second predicted path of the detection target overlap. The one or more conditions may include the condition that the distance between the moving body 50 and the detection target is less than a predetermined threshold value. The one or more conditions may include the condition that the characteristic value of at least a part of the area of the second video is less than a predetermined threshold value.

[0080] In the fifth example, the control unit 33 may determine that the above-described one or more conditions are satisfied for each of the vehicles 64a and 64b and the pedestrian 63d. In such a case, the control unit 33 may cause the display device 40 to display, for example, as shown in FIG. 17, three sixth markers 75a, 75b, and 75c corresponding to the vehicles 64a and 64b and the pedestrian 63d, respectively, superimposed on the second video. The control unit 33 may cause the sixth marker to be superimposed and displayed on the detection target in a bright place.

[0081] The control unit 33 may change the shape of the guide line 66. The control unit 33 may change the shape of the guide line 66 in the region where the guide line 66 and the detection target overlap. FIG. 18 shows one example of the shape of the guide line 66. In the guide line 66 of FIG. 18, the guide line is not displayed in the region where the guide line 66 and the sixth marker 75a overlap. The example of the shape of the guide line 66 is not limited to deletion, and other design changes are allowed. The design changes include color change, transmittance change, type change to a dashed line or the like, line thickness change, and blinking. The control unit 33 may change the shape of the guide line 66 when the sixth marker is not displayed. The control unit 33 may change the shape of the guide line 66 when the first marker 71 and the second marker 72 are displayed on the detection target.

[0082] As described above, according to the display system 10 according to one embodiment, various markers corresponding to a detection target detected on the first video are superimposed on the second video and displayed on the display device 40. The subject 60 who visually recognizes the marker can recognize at a glance the relative positional relationship between the moving body 50 and the detection target. Therefore, the convenience of the technology for displaying the video of the external area of the moving body 50 is improved.

[0083] Hereinafter, with reference to FIGS. 19 to 31, examples of various videos in which the image processing device 30 synthesizes the guide wall image 80 and other images in the display area on the first video and causes the display device 40 to display them will be specifically described.

[0084] (Sixth example) FIG. 19 shows a sixth example of the detection area 61 in the first video acquired by the image processing device 30 from the imaging device 20. The imaging device 20 may be arranged to image the rear of the moving body 50 which is a vehicle. In the example shown in FIG. 19, the detection area 61 is longer in the left - right direction than in the up - down direction. The display area 62 is located at the center in the left - right direction of the detection area 61. The detection area 61 is not limited to a shape that is longer in the left - right direction than in the up - down direction, and various shapes such as a square, a vertically long rectangle, a circle, etc. are possible. The display area 62 can have various positions, sizes, and shapes. For example, the display area 62 is not necessarily located at the center of the detection area 61 and may be closer to the left or right. The display area 62 is not limited to a part of the detection area 61, and the entire detection area 61 may be used as the display area 62. The control unit 33 may change the display area 62 on the first video according to a pinch - in operation and a pinch - out operation on the display device 40, etc.

[0085] The control unit 33 generates a second video by synthesizing a guide wall image 80 indicating the first predicted path 65 of the moving body 50 on the display area on the first video acquired from the imaging device 20. The control unit 33 may output an image with the guide wall image 80 superimposed on the second video and display it on the display device 40. The guide wall image 80 can indicate the predicted path when the moving body 50 moves backward. The guide wall image 80 is a marker that gives a three-dimensional impression to the subject 60. The guide wall image 80 can be considered as an image projected onto the first video when the imaging device 20 captures a virtual guide wall that guides the predicted path and is arranged in the real space shown in the first video. The guide wall image 80 can be visually recognized as being composed of a plurality of virtual translucent wall surfaces extending from the road surface to a predetermined height in the height direction within the visual field of the subject 60. The guide wall image 80 can also be said to be a three-dimensional display of the guide line displayed on the display device 40 when the vehicle moves backward by means of a wall-like display. The wall-like display includes displays in other forms such as surfaces, films, and plates. The three-dimensional display of the guide wall image 80 makes it easier for the subject 60 to grasp the position of the parking space on the road surface, the distance to obstacles, etc.

[0086] The control unit 33 can recognize a detection target within the display area 62 of the first video. The control unit 33 may detect a detection target that is inside the display area 62 on the first video and is reflected within the first predicted path 65. When detecting a detection target, the control unit 33 can synthesize a virtual plane as a first recognition wall image 83 on the side of the moving body 50 of the recognized detection target. The first recognition wall image 83 moves together with the detection target when the detection target moves.

[0087] FIG. 20 shows an example of a second video corresponding to the display area 62 of the first video shown in FIG. 19. The guide wall image 80 is arranged away from the lower end of the second video upward. The guide wall image 80 may include two side wall images 81a and 81b extending in the height direction and the depth direction. The side wall image 81a is located on the left side in the horizontal direction, and the side wall image 81b is located on the right side in the horizontal direction. The area between the two side wall images 81a and 81b shows a first predicted path 65 through which the moving body 50, which is a vehicle, passes. The interval between the side wall images 81a and 81b may be set to indicate the vehicle width of the moving body 50. In FIG. 20, the side wall images 81a and 81b are displayed linearly. The control unit 33 may acquire moving body information from the communication unit 31 and curve the side wall images 81a and 81b according to the predicted path. In this case, the moving body information includes the steering angle of the steering wheel.

[0088] The guide wall image 80 may extend in the height direction and the horizontal direction and include a plurality of distance wall images 82a, 82b, 82c indicating the depth direction distance from the moving body 50. The distance wall images 82a, 82b, 82c may connect between the side wall images 81a and 81b. Three distance wall images 82a, 82b, and 82c may be shown in FIG. 20. The number of distance wall images is not limited to three and may be two or four or more. The depth direction interval between the distance wall images 82a and 82b may be narrower than the depth direction interval between the distance wall images 82b and 82c. The height direction length of the distance wall images 82a, 82b, 82c may be longer or shorter than the height direction length of the side wall images 81a and 81b.

[0089] A frame may be displayed on the outer periphery of each side wall image 81a, 81b of the guide wall image 80 and the distance wall images 82a, 82b, 82c. The guide wall image 80 may include at least one type of auxiliary line 84a extending in the depth direction, an auxiliary line 84b extending in the horizontal direction, and an auxiliary line 84c extending in the height direction. The guide wall image 80 may include all of the auxiliary line 84a extending in the depth direction, the auxiliary line 84b extending in the horizontal direction, and the auxiliary line 84c extending in the height direction. The auxiliary line 84a extending in the depth direction may be displayed on the wall surfaces of the side wall images 81a, 81b. The auxiliary line 84b extending in the horizontal direction may be displayed on the wall surfaces of the distance wall images 82a, 82b, 82c. The auxiliary line 84c extending in the height direction may be displayed on the wall surfaces of the side wall images 81a, 81b and the distance wall images 82a, 82b, 82c. Each of the auxiliary lines 84a, 84b, 84c can be displayed in any number of 1 or more. When there are a plurality of auxiliary lines 84a, 84b, 84c, the intervals between the respective auxiliary lines 84a, 84b, 84c can be equal intervals. The intervals between each of the auxiliary lines 84a, 84b, 84c may vary according to conditions such as the height from the road surface and the distance from the moving body 50. The frame displayed on the outer periphery of each side wall image 81a, 81b and the distance wall images 82a, 82b, 82c can be regarded as a part of the plurality of auxiliary lines 84a, 84b, 84c. In FIG. 20 and the following figures, even when there are a plurality of each of the auxiliary lines 84a, 84b, 84c, only one of them is appropriately labeled with a reference numeral.

[0090] The distance wall image 82a is a first distance wall image closer to the moving body 50 than the other distance wall images 82b, 82c when mapped to the real space projected by the first video. The distance wall image 82a can have a color different from that of the other distance wall images 82b, 82c. For example, the distance wall images 82b, 82c can be displayed in translucent white, and the distance wall image 82a can be displayed in translucent red. The side wall images 81a, 81b can be displayed in translucent white. The above color settings are exemplary, and the colors of the side wall images 81a, 81b may be different from those of the distance wall images 82b, 82c.

[0091] The transmittance of the side wall images 81a and 81b may be changed according to the position in the depth direction. The transmittance of the side wall images 81a and 81b may be set to be lower on the front side in the depth direction and higher on the back side. The side wall images 81a and 81b may have different saturation or lightness according to the position in the depth direction. For example, the saturation can be increased on the front side in the depth direction. The saturation of an image can also be described as the intensity of the color of the image. The fact that the transmittance, saturation, lightness, chromaticity, etc. change continuously can be said to have a gradation.

[0092] The transmittance of the distance wall images 82a, 82b, and 82c may be changed according to the height from the road surface. The transmittance of the distance wall images 82a, 82b, and 82c may be set to be higher as the distance from the road surface increases in the height direction. The distance wall images 82a, 82b, and 82c may have different saturation or lightness according to the height from the road surface. For example, the saturation can be decreased as the distance from the road surface increases in the height direction.

[0093] The auxiliary lines 84b and 84c on the distance wall image 82a, which is the first distance wall image, may have the same color as the auxiliary lines 84b and 84c of the other distance wall images 82b and 82c. The auxiliary lines 84a, 84b, and 84c can be of any color. For example, when the color of the wall surface of the guide wall image 80 other than the distance wall image 82a is white, the color of the auxiliary lines 84a, 84b, and 84c can be white with higher luminance or lower transmittance than these wall surfaces.

[0094] The control unit 33 may detect the vehicle 64 shown within the display area 62 of the first video as a detection target. In the example of FIG. 20, the vehicle 64 is located within the first predicted path 65. The detection target detected by the control unit 33 is not limited to a vehicle, and may be a building, a fence, an obstacle on the road, a person, an animal, or the like. When detecting a detection target, the control unit 33 may synthesize the first recognition wall image 83 on the moving body 50 side of the recognized detection target. When detecting a detection target, the control unit 33 may change the length of the guide wall image 80 in the height direction. When detecting a detection target, the control unit 33 can shorten the length of the guide wall image 80 in the height direction. Thereby, even when the first recognition wall image 83 is displayed, the control unit 33 can make the display easier to view and direct the attention of the target person 60 to the detection target.

[0095] The first recognition wall image 83 of the detection target located within the first predicted path 65 extends in the height direction and the horizontal direction. The first recognition wall image 83 may be displayed as an opaque or translucent surface of any color. In FIG. 20, the first recognition wall image 83 is displayed as a translucent surface having a rectangular shape on the moving body 50 side of the vehicle 64. The horizontal width of the first recognition wall image 83 may correspond to the lateral width of the vehicle 64. The horizontal width of the first recognition wall image 83 can be made to substantially match the lateral width of the vehicle 64. The height of the first recognition wall image 83 can be set to a predetermined value. The transmittance of the first recognition wall image 83 may be changed according to the height direction from the road surface. The transmittance of the first recognition wall image 83 can be set to increase as the distance in the height direction from the road surface increases. Also, the first recognition wall image 83 may have different saturation or brightness according to the distance in the height direction from the road surface. For example, the saturation can be lowered as the distance in the height direction from the road surface increases. The first recognition wall image 83 may include auxiliary lines extending in the horizontal direction and the height direction.

[0096] The control unit 33 can move the auxiliary line extending in the horizontal direction of the first recognition wall image 83 within the frame of the first recognition wall image 83 in the height direction. The auxiliary line can be displayed so as to appear from the lower end of the frame of the first recognition image 83 and disappear at the upper end. By moving the auxiliary line extending in the horizontal direction of the first recognition wall image 83 in the height direction, the control unit 33 can display both or either of the relative position and the relative distance change between the moving body 50 and the vehicle 64. For example, when the distance between the moving body 50 and the vehicle 64 is decreasing, the control unit 33 can move the auxiliary line extending in the horizontal direction quickly in the height direction. The control unit 33 may move the auxiliary line faster in the height direction as the distance between the moving body 50 and the vehicle 64 is closer. The control unit 33 may move the auxiliary line extending in the horizontal direction of the first recognition wall image 83 in the height direction regardless of the position of the detection target and the distance from the detection target.

[0097] The color of the first recognition wall image 83 can be made different colors according to the distance in the depth direction from the moving body 50 to the vehicle 64. For example, as shown in FIG. 20, when the distance between the moving body 50 and the vehicle 64 is farther than a predetermined distance, the control unit 33 can display the color of the first recognition wall image 83 in blue. When the distance to the vehicle 64 is shortened by the backward movement of the moving body 50, the control unit 33 may change the color of the first recognition wall image 83, such as from blue to yellow and from yellow to red.

[0098] When the distance between the moving body 50 and the vehicle 64 is separated by a predetermined distance or more, the control unit 33 cannot display the first recognition wall image 83. When the moving body 50 and the vehicle 64 relatively approach and are within a predetermined distance, the control unit 33, for example, first highlights the vehicle 64 in the second video and then displays the first recognition wall image 83. The highlighting includes, for example, thickening the contour line of the vehicle 64, blinking the contour line of the vehicle 64, filling the image of the vehicle 64, and the like. The control unit 33 may change the display mode according to the change in the relative position with the vehicle 64. For example, when the vehicle 64 moves relatively away from the moving body 50, the control unit 33 can refrain from displaying the first recognition wall image 83.

[0099] FIG. 21 is a diagram showing an example of the second video in a state where the distance between the moving body 50 and the vehicle 64 is reduced due to the backward movement of the moving body 50. The first recognition wall image 83 is displayed in red. Thereby, it is possible to warn the subject 60 that there is a risk of colliding with the vehicle 64 if moving backward any further.

[0100] When the detection object and the guide wall image 80 overlap, the control unit 33 can change the display of the guide wall image 80. For example, as shown in FIG. 21, the control unit 33 can at least partially not display the guide wall image 80 that overlaps with the vehicle 64 in the second video. More specifically, the control unit 33 does not display on the display device 40 the guide wall image 80 that overlaps with the vehicle 64 and is located farther from the moving body 50 in the depth direction than the vehicle 64 or the first recognition wall image 83. By doing so, the display device 40 can display the image in an easy-to-see and intuitive way for the subject 60 to understand. The method of changing the display of the guide wall image 80 is not limited to non-display of the overlapping part. For example, the control unit 33 can increase the transmittance of the guide wall image 80 in the part that overlaps with the detection object.

[0101] (Modification example) The guide wall image 80 is not limited to the example shown in FIGS. 20 and 21. FIGS. 22 to 24 show modification examples of the guide wall image 80.

[0102] (First modification example) In the example shown in FIG. 22, the auxiliary line 84b extending horizontally in the distance wall images 82a, 82b, 82c can display only one of the positions that is the lowest in the height direction. The distance wall image 82a that displays only the auxiliary line 84b at the lowest position can facilitate the recognition of the ground surface height. The distance wall image 82a that displays only the auxiliary line 84b at the lowest position can facilitate the grasping of the distance from the recognition wall image. Auxiliary lines 84c extending in the height direction can be displayed at both horizontal ends of the distance wall images 82a, 82b, 82c. On the outer periphery of the distance wall images 82a, 82b, 82c, the auxiliary line 84b located at the lower end and the auxiliary lines 84c extending in the height direction located at both horizontal ends can be displayed. The distance wall images 82a, 82b, 82c may be displayed by the auxiliary line 84b extending horizontally at the lower end of the outer periphery and the auxiliary lines 84c extending in the height direction at both left and right ends.

[0103] The distance wall images 82a, 82b, 82c can be displayed including a surface having a translucent color. The distance wall images 82a, 82b, 82c can increase the transmittance or reduce the color density as they move away from the road surface in the height direction. The auxiliary lines 84b, 84c located on the outer periphery of the distance wall image 82a closest to the moving body 50 may be displayed in a color different from other auxiliary lines. The different color can be, for example, red. Also, other auxiliary lines can be white, but not limited to this. The first recognition wall image 83 displayed in front of the vehicle 64 may be displayed as a surface of an arbitrary translucent color, and may be displayed so that the transmittance increases or the color density decreases as it moves away from the road surface in the height direction. The first recognition wall image 83 may not display the upper boundary visually recognizable and may be displayed so as to gradually disappear in the height direction.

[0104] In the example shown in FIG. 22, the auxiliary line 84a extending in the depth direction of the side wall images 81a, 81b of the guide wall image 80 has only two displayed at the upper and lower ends of the side wall images 81a, 81b respectively The lower auxiliary line 84a1 can be highlighted more than the upper auxiliary line 84a2. Highlighting includes displaying using a thicker line, a line with a lower transmittance, a line with a higher luminance, etc. No. The upper auxiliary line 84a2 and the lower auxiliary line 84a1 may be lines in the same display mode. . Lines in the same mode are, for example, lines that are similar in terms of thickness, transmittance, luminance, etc.

[0105] The auxiliary line 84a extending in the depth direction may be displayed using a display method that gradually becomes thinner and disappears at the farthest position in the depth direction. The side wall images 81a, 81b may be displayed so that the end portions at the farthest position in the depth direction are visible.

[0106] Auxiliary lines 84c extending in the height direction may be displayed on the side wall images 81a, 81b at least at the frontmost end portion and the positions where they intersect the distance wall images 82a, 82b, 82c. These auxiliary lines 84c extending in the height direction may be highlighted more than other auxiliary lines 84c extending in the height direction. The auxiliary lines 84c extending in the height direction may have different thicknesses respectively. The auxiliary line 84c displayed at the position where it intersects the distance wall image 82a may be displayed thicker than the auxiliary lines 84c displayed at the positions where they intersect the distance wall images 82b, 82c.

[0107] (Second Modification Example) The guide wall image 80 shown in FIG. 23 is different from the guide wall image 80 and the side wall images 81a, 81b in FIG. 22. In FIG. 23, two or more auxiliary lines 84a extending in the depth direction of the side wall images 81a, 81b of the guide wall image 80 may be displayed, including the uppermost and lowermost ones. The auxiliary line 84a extending in the depth direction may highlight the auxiliary line located at the bottommost in the height direction more than other auxiliary lines. The plurality of auxiliary lines 84a may all be displayed as lines in the same mode. The guide wall image 80 in FIG. 23 may adopt changes in transmittance, changes in color, color schemes, etc. that can be adopted by the guide wall image 80 in FIG. 22.

[0108] (Third Modification Example) The guide wall image 80 shown in FIG. 24 is different from the guide wall image 80 and the side wall images 81a and 81b in FIG. 22. In FIG. 24, the auxiliary line 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 displays only the auxiliary line 84a located at the lower end. The auxiliary line 84a extending in the depth direction located at the lower end of the guide wall image 80 may correspond to a line on the road surface when mapped to the real space projected by the first video together with the auxiliary line 84b extending in the horizontal direction. The guide wall image 80 in FIG. 24 may adopt changes in transmittance, color changes, color schemes, etc. that can be adopted in the guide wall image 80 in FIG. 22. The auxiliary lines 84c extending in the height direction may have different thicknesses for each. Among the auxiliary lines 84c arranged from the front side to the back side in the depth direction, the lines located on the back side may be thinner.

[0109] In the guide wall image 80 shown in each of FIGS. 20 to 24, the display device 40 may not display some or all of the auxiliary lines 84a, 84b, and 84c.

[0110] (Example 7) FIG. 25 shows a seventh example of the detection area 61 in the first video acquired by the image processing device 30 from the imaging device 20. In the example shown in FIG. 25, the detection area 61 is longer in the left - right direction than in the up - down direction. The display area 62 is located at the center in the left - right direction of the detection area 61. The display area 62 is not limited to a part of the detection area 61 and may be the entire detection area 61.

[0111] In Example 7, the moving body 50 is a vehicle and can be assumed to be moving backward toward a parking space in a parking lot. The imaging device 20 is located behind the moving body 50 and can be assumed to be imaging the rear. The control unit 33 may detect, as detection targets, the vehicle 64b and the pedestrian 63 that are outside the first predicted path 65 and at least partially reflected within the display area 62 from the detection area 61. In the example shown in FIG. 25, for the vehicle 64a reflected within the first predicted path 65 of the moving body 50, the control unit 33 determines that the distance in the depth direction is farther than a predetermined distance and does not regard it as a detection target.

[0112] FIG. 26 shows an example of a second video corresponding to the display area 62 of the first video shown in FIG. 25. The control unit 33 recognizes the vehicle 64b and the pedestrian 63 to be detected within the display area 62 of the first video, and displays first recognition wall images 83a and 83b, which are virtual planes, on the moving body 50 side of the vehicle 64b and the pedestrian 63, respectively. The first recognition wall images 83a and 83b can be displayed as semi-transparent surfaces parallel to the side wall images 81a and 81b extending in the height direction and the depth direction. The control unit 33 acquires the distance change in the relative position between the vehicle 64b and the pedestrian 63 and the moving body 50, and may display the side wall images 81a and 81b only when the distance is changing in the direction of shortening. For estimating the distance between the moving body 50 and the detection target, any algorithm can be adopted as described above.

[0113] The control unit 33 may estimate the horizontal distance of the vehicle 64a and the pedestrian 63 to be detected from the guide wall image 80, and determine the display colors of the first recognition wall images 83a and 83b according to the estimated distance. Here, the distance between the detection target and the guide wall image 80 is the distance between the detection target and the virtual guide wall when the virtual guide wall displayed by the guide wall image 80 is present in the real space projected by the first video. The control unit 33 can estimate the first distance between them from the position where the guide wall image 80 is mapped to the real space projected by the first video and the position of the detection target. The control unit 33 sets a threshold for the first distance, and for each of the first recognition wall images 83a and 83b, different display colors such as blue, yellow, and red may be used in order from the longer horizontal distance to the shorter horizontal distance and displayed on the display device 40. The first recognition wall images 83a and 83b may have auxiliary lines extending in the depth direction. The control unit 33 can move the auxiliary lines extending in the depth direction of the first recognition wall images 83a and 83b in the height direction within the frame of the first recognition wall image 83 in the same manner as the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in FIG. 20.

[0114] The control unit 33 estimates the first position as the position of the vehicle 64b and the pedestrian 63 to be detected in the depth direction. The control unit 33 faces the vehicle 64b and the pedestrian 63 respectively The display of the sidewall images 81a and 81b corresponding to the first position is at least partially changed. The display may be changed in various ways, including changing the color, brightness, saturation, etc. of the wall surface, changing the frame line, / or changes in the color, thickness, etc. of auxiliary lines, or combinations thereof. For example, in Figure 26 In the figure, a part 81a1 of the side wall image 81a facing the vehicle 64b and a part 81a2 of the side wall image 81a facing the pedestrian 63 are The display color of a part 81b1 of the sidewall image 81b is changed. The side wall images 81a and 81b are divided into areas by auxiliary lines 84c extending in the height direction. The display colors of a part 81a1 of the sidewall image 81a and a part 81b1 of the sidewall image 81b may be The control unit 33 detects that the detection target is the side wall images 81a, 8 When the obstacle is large compared to the length represented by the side wall image 81a or 81b, The display color may be changed. Large obstacles include large vehicles such as trailers and buildings. It can be done.

[0115] When the detection target moves, the control unit 33 estimates a second predicted path of the detection target, The first position may be estimated as a position where the second predicted path intersects with the side wall images 81a and 81b. The control unit 33 calculates the first position based on the second predicted path in the real space shown in the first image and the side wall image. It can be estimated as the intersection position of the mapping of 81a and 81b onto the real space. 33 at least partially changes the display of the sidewall images 81a, 81b in accordance with the first position. In this case, the display of the parts 81a1 and 81b of the side wall images 81a and 81b may be changed. The display color of the first distance may be different depending on the first distance. When the obstacle is large, the display color of the entire side wall image 81a or 81b may be changed.

[0116] As another display mode, the control unit 33 may change the display of the portions of the side wall images 81a and 81b that are located deeper in the depth direction than the first recognition wall images 83a and 83b. As yet another display mode, the control unit 33 may change the display of the portions of the side wall images 81a and 81b that are located deeper in the depth direction than the position where the second predicted path intersects the side wall images 81a and 81b.

[0117] The display of the first recognition wall images 83a and 83b may be changed according to the height from the road surface. Similar to the first recognition wall image 83 in the sixth example, the display of the first recognition wall images 83a and 83b may be set to different values for transmittance, chroma, or lightness, etc. according to the height from the road surface.

[0118] As described above, the display system 10 can spatially and three-dimensionally represent the distance and / or position of a detection target that may be an obstacle to backward movement, using the guide wall image 80 and the first recognition wall images 83a and 83b. Also, it is possible to three-dimensionally display the position where there is a risk of collision, from the predicted path of the detection target during movement. In this way, the display system 10 enables a spatial and three-dimensional representation of the detection target and the warning content by representing the guide wall image 80 and the first recognition wall images 83a and 83b as walls (or planes) having a height direction. This facilitates the display and understanding of the warning content for the target person 60.

[0119] (Eighth Example) FIG. 27 shows an eighth example of the detection area 61 in the first video acquired by the image processing device 30 from the imaging device 20. The imaging device 20 may be arranged to image the rear of the moving body 50 which is a vehicle. In the example shown in FIG. 27, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center in the left-right direction of the detection area 61. The control unit 33 may detect the pedestrian 63 and the vehicle 64b that appear in the second area which is outside the display area 62 on the first video and inside the detection area 61, as detection targets respectively. In the eighth example, for the vehicle 64a that appears in the display area 62, the control unit 33 determines that the distance in the depth direction is farther than a predetermined distance and does not regard it as a detection target.

[0120] FIG. 28 schematically shows the arrangement of the moving body 50, the pedestrian 63 to be detected, and the vehicle 64b as seen from above along the height direction in the real space shown by the first video. The imaging device 20 can image a subject in a range included in the regions F1 and F2 in the real space. The regions F1 and F2 correspond to the detection region 61 of the first video. The region F1 is a first region corresponding to the display region 62 of the first video. The region F2 corresponds to a second region that is inside the detection region 61 of the first video and outside the display region 62. For the sake of explanation, FIG. 28 shows a virtual guide wall 90 mapped to the real space of the guide wall image 80 displayed in the second video by a dashed line. The virtual side walls 91a, 91b correspond to the side wall images 81a, 81b displayed in the second video. The virtual distance walls 92a, 92b, 92c correspond to the distance wall images 82a, 82b, 82c displayed in the second video.

[0121]

[0122] Also, for the sake of explanation, FIG. 28 shows virtual lines 95l1 to 95l5 and 95r1 to 95r5 indicating the horizontal distance of the detection target with respect to the virtual side walls 91a, 91b of the moving body 50 by dashed lines. This horizontal distance is also referred to as the second distance. The virtual lines 95l1 to 95l5 are located on the left side of the moving body 50. The virtual lines 5r1 to 95r5 are located on the right side of the moving body 50. The intervals between the virtual lines 95l1 to 95l5 and the virtual lines 95r1 to 95r5 can be arbitrarily set respectively. The intervals between the virtual lines 95l1 to 95l5 and the virtual lines 95r1 to 95r5 may be equally spaced respectively. The control unit 33 determines which of the virtual lines 95l1 to 95l5 and 95r1 to 95r5 the detection target is closest to from the first video, or ​​​​​​​​​​It is possible to determine whether it is located within the region sandwiched between any two of 5l5 and 95r1 to 95r5. It can be done.

[0123] A second video corresponding to the first video in FIG. 27 is illustrated in FIG. 29. In the eighth example, the guide wall image 80 may adopt the same display method as that described in the sixth and seventh examples. When the detection target is located in the second region of the first video, the control unit 33 synthesizes the second recognition wall images 89a and 89b at the end on the side where the detection target exists in the second video. The second recognition wall images 89a and 89b may each include wall portions 96a and 96b extending in the height direction and the depth direction, and floor portions 97a and 97b extending in the depth direction and the horizontal direction. In FIG. 29, the wall portions 96a and 96b extend from the lower end to the upper end of the left and right ends of the second video. The wall portions 96a and 96b may extend vertically along only a part, not the whole, of the left and right ends of the second video. Similarly, the floor portions 97a and 97b may extend horizontally along the whole or a part of the lower end portion of the second video. The floor portions 97a and 97b may be displayed between the guide wall image 80 and the lower end of the second video.

[0124] The control unit 33 may change the display of the second recognition wall images 89a and 89b according to the second distance of the detection target. The change in the display may include changes in color, transmittance, saturation, brightness, size, etc., and changes in dynamic display methods such as fading and moving. For example, in one of the plurality of embodiments, the change in the display may be represented by a change in color. The display color may be red when the risk level is high, yellow when the risk level is medium, blue when the risk level is low, etc. The pedestrian 63 shown in FIG. 28 is located on the third virtual line 95r3 on the right side in the horizontal direction from the virtual side wall 91b of the moving body 50. In this case, the control unit 33 may set the color of the second recognition wall image 89b on the right side to yellow with a medium risk level. Also, the vehicle 64b shown in FIG. 28 is horizontal from the virtual side wall 91a of the moving body 50. In this case, the control unit 33 detects the danger by The degree of danger may be determined to be low, and the color of the second recognition wall image 89a on the left side may be set to blue. In one embodiment, the change in display is a vertical auxiliary line displayed on the walls 96a, 96b. The control unit 33 may include a lateral movement of the first and second distances 96a1, 96b1. The speed of movement of the walls 96a, 96b may be varied accordingly.

[0125] The second recognition wall images 89a and 89b may be semi-transparent. The wall portions 96a and 96b of the second recognition wall images 89a and 89b may have display characteristics that change in the vertical direction. For example, the wall portions 96a and 96b of the second recognition wall images 89a and 89b may have a lower transmittance toward the lower portion. Also, the second recognition wall images 89a and 89b may have a higher saturation toward the lower portion. The floor portions 97a and 97b of the second recognition wall image 89a may have display characteristics that change in the horizontal direction. For example, the floor portions 97a and 97b of the second recognition wall image 89a may have a lower transmittance toward the left and right ends and a higher transmittance toward the center of the second image. Also, the floor portions 97a and 97b of the second recognition wall image 89a may have a lower saturation from the left and right ends toward the center of the second image. The saturation of an image can also be expressed as the color density of the image.

[0126] The control unit 33 determines the type of the detection target, and displays an icon image 99a, 99b may be displayed on the floor portions 97a and 97b of the second recognition wall images 89a and 89b. The type of the object is determined by a known method based on the image of the object detected in the first video. For example, the control unit 33 may match the contour shape of the detection target with a model pattern. For example, the object type can be determined by detecting a pedestrian 63 and a vehicle 64. For b, icons representing a person and a vehicle can be selected. The virtual side walls 91a and 91b obtained by mapping the wall images 81a and 81b into the real space can estimate the second distance, which is the horizontal distance between the virtual side walls and the detection target in the real space. The second distance can be determined from the relationship between the lines obtained by mapping the virtual lines 95l1 to 95l5 and 95r1 to 95r5 in the real space to the detection area 61 of the first video and the positions of the images of the respective detection targets. The control unit 33 can change the positions where the icon images 99a and 99b are displayed according to the second distance. In the display example of the second video shown in FIG. 29, first distance discrimination lines 98l1 to 98l5 and 98r1 to 98r5 are provided corresponding to the virtual lines 95l1 to 95l5 and 95r1 to 95r5 in the real space, respectively. The first distance discrimination lines 98l1 to 98l5 are located on the left side of the side wall image 81a and are numbered in the order of 98l1, 98l2, 98l3, 98l4, and 98l5 from the center to the left side. The lines 98r1 to 98r5 are located on the right side of the side wall image 81b and are numbered in the order of 98r1, 98r2, 98r3, 98r4, and 98r5 from the center to the right side. The first distance discrimination lines 98l1 to 98l5 and 98r1 to 98r5 indicate the positions where the icon images 99a and 99b are displayed and may not be displayed in the second video. In FIG. 28, the pedestrian 63 is located on the third virtual line 95r3 on the right side in the horizontal direction from the virtual side wall 91b of the moving body 50. Therefore, the control unit 33 may arrange the icon image 99b of the pedestrian 63 on the third first distance discrimination line 98r3 on the right side in the second video shown in FIG. 29. As shown in FIG. 29, the icon image 99b is three-dimensionally represented as a surface having lengths in the depth direction and the height direction on the first distance discrimination line 98r3 extending in the depth direction on the second video.

[0127] In the display example of the second video shown in FIG. 29, first distance discrimination lines 98l1 to 98l5 and 98r1 to 98r5 are provided corresponding to the virtual lines 95l1 to 95l5 and 95r1 to 95r5 in the real space, respectively. The first distance discrimination lines 98l1 to 98l5 and 98r1 to 98r5 are located on the left side of the side wall image 81a and are numbered in the order of 98l1, 98l2, 98l3, 98l4, and 98l5 from the center to the left side. The lines 98r1 to 98r5 are located on the right side of the side wall image 81b and are numbered in the order of 98r1, 98r2, 98r3, 98r4, and 98r5 from the center to the right side. The first distance discrimination lines 98l1 to 98l5 and 98r1 to 98r5 indicate the positions where the icon images 99a and 99b are displayed and may not be displayed in the second video.

[0128] In FIG. 28, the pedestrian 63 is located on the third virtual line 95r3 on the right side in the horizontal direction from the virtual side wall 91b of the moving body 50. Therefore, the control unit 33 may arrange the icon image 99b of the pedestrian 63 on the third first distance discrimination line 98r3 on the right side in the second video shown in FIG. 29. As shown in FIG. 29, the icon image 99b is three-dimensionally represented as a surface having lengths in the depth direction and the height direction on the first distance discrimination line 98r3 extending in the depth direction on the second video. may be shown. When viewed stereoscopically, the icon image 99b may be displayed as a plane parallel to the side wall images 81a, 81b and the second recognition wall images 89a, 89b. The icon image 99b , similar to the second recognition wall image 89b, may have different display modes such as color, saturation, lightness, etc. according to the second distance. For example, the icon image 99b may be displayed in yellow.

[0129] In FIG. 28, the vehicle 64b is located farther from the virtual side wall 91a than the fifth virtual line 95l5 on the left side in the horizontal direction from the virtual side wall 91a of the moving body 50. In this case, the control unit 33 may arrange the icon image 99a of the vehicle 64b on the left first distance discrimination line 98l5 in the second video of FIG. 29. Alternatively, the control unit 33 may arrange the icon image 99a so as to appear on the same plane as the wall portion 96a of the second recognition wall image 89a when the second video is viewed stereoscopically. The icon image 9 9a, similar to the icon image 99b, may have different display modes according to the second distance. For example, the icon image 99a may be displayed in blue.

[0130] When both the moving body 50 and the detection target move or either one of them moves, and the relative position of the detection target as viewed from the moving body 50 changes, the positions and display modes of the second recognition wall images 89a, 89b and the icon images 99a, 99b change. Further, when the moving body 50 enters the display area 62, the detection target may be displayed in the manner described in the seventh example. An example of the change in display when the vehicle 64b moves from the region F2 to the region F1 in the real space shown in FIG. 28 will be described. First, in the real space of FIG. 28, the vehicle 64b is located on the left side of the virtual line 95l5. In the space shown, an example of the change in display when the vehicle 64b moves from the region F2 to the region F1 will be described.

[0131] First, in the real space of FIG. 28, the vehicle 64b is located on the left side of the virtual line 95l5. ​Let's assume this. At this time, in the second video of FIG. 29, as described above, the second recognition wall image 89a is shown, and an icon image 99a is displayed at the leftmost side of the floor portion 98a of the second recognition wall image 89a. At this time, the second recognition wall image 89a and the icon image 99a can be shown in blue.

[0132] In the real space of FIG. 28, as the vehicle 64b approaches the virtual guide wall 90, the second recognition wall image 89a displayed in the second video of FIG. 29 changes its display mode. For example, the color of the second recognition wall image 89a sequentially changes from blue to yellow and then to red. At the same time, the icon image 99a moves from above the first distance discrimination line 98l5 to above the first distance discrimination line 98l1 in terms of the display position. The color of the icon image 99a may sequentially change from blue to yellow and then to red. Thus, the subject 60 can recognize that the vehicle 64b is approaching outside the area not displayed in the second video. Also, by using a three-dimensional display method, the subject 60 can spatially understand the approach of the vehicle 64b and can reduce the risk of overlooking information.

[0133] When the vehicle 64b is approaching the moving body 50, the icon image 99a can move toward the center of the screen. The subject 60 can recognize the moving direction of the vehicle 64b through the movement of the icon image 99a.

[0134] Furthermore, when the vehicle 64b enters from the region F2 to F1 in the real space of FIG. 28, since the vehicle 64b enters the display target region of the first video, it is included in the second video and displayed on the display device 40. At the same time, the display of the second recognition wall image 89a disappears, and as shown in FIG. 26, the vehicle 64b ​​​​​​​​​​​​On the side of the guide wall image 80, the first recognition wall image 83a is displayed. When the second recognition wall image 89a disappears, the control unit 33 may blink the second recognition wall image 89a a plurality of times to prompt the attention of the subject 60. The control unit 33 displays the first recognition wall image 83a on the side of the moving body 50 of the vehicle 64b. Further, the control unit 33 estimates the second predicted travel route of the vehicle 64b and estimates the first position as the position where the second predicted travel route intersects the side wall images 81a and 81b. The control unit 33 may change the display of a part 81a1 of the side wall image 81a including the first position. For example, the control unit 33 can sequentially change the color of a part 81a1 of the side wall image 81a to blue, yellow, and red as the vehicle 64b approaches. Thereby, the subject 60 can easily grasp that the vehicle 64b is approaching and that the second predicted travel route of the vehicle 64b overlaps the first predicted travel route 65 of the moving body 50. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information. As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information.

[0135] As described above, according to one of the plurality of embodiments of the present disclosure, the three-dimensional display such as the guide wall image, the first recognition wall image, the second recognition wall image, and the icon image facilitates the spatial understanding of the position of the detection target. Further, according to one of the plurality of embodiments of the present disclosure, by changing the display mode of each wall surface, information such as attention and warning can be provided in an easily understandable manner. Also, by using three-dimensional display, it is possible to reduce the risk of overlooking information.

[0136] (Modification example) Figure 30 shows another example of the second video corresponding to the first video in Figure 27. In Figure 30, view display areas 101 and 102 indicating the selection status between "Normal View display" and "Wide View display" are added at the lower left. The "Normal View display" and "Wide View display" will be described later. Figure 30 may be partially different from Figure 29 in the display modes of the guide wall image 80, the second recognition wall images 89a and 89b, and the icon images 99a and 99b. For example, in Figure 30, a guide wall image 80 with the same display mode as in Figure 22 is displayed.

[0137] The wall portions 96a and 96b of the second recognition wall images 89a and 89b can have one or more auxiliary lines 96a1 and 96b1 extending in the height direction. In Figure 30, the wall portions 96a and 96b have a plurality of auxiliary lines 96a1 and 96b1, but only one of them is labeled for clarity. The wall portions 96a and 96b of the second recognition wall images 89a and 89b can represent the distance to the detection target in color and may have different transmittances from the front to the back in the depth direction. The wall portions 96a and 96 b may gradually increase in transmittance or gradually decrease in density in the height direction and disappear at a predetermined position in the height direction.

[0138] The control unit 33 can display the presence of an object within a predetermined distance range, the distance to the detection target, or the change in the distance to the detection target based on the change in the transmittance or color density of the wall portions 96a and 96b, and the change in the auxiliary lines 96a1 and 96a2. The change in the transmittance or color density of the wall portions 96a and 96b can also be referred to as a gradient change. For example, the control unit 33 can display that the detection target exists within a predetermined distance by the movement of the auxiliary lines 96a1 and 96a2 of the wall portions 96a and 96b. The control unit 33 can The color of the front side is made the darkest, and it can be displayed that the detection target exists within a predetermined distance due to the change in the gradient in the depth direction. The auxiliary lines 96a1 and 96b of the wall portions 96a and 96b and the change in transmittance are not essential. The control unit 33 can prevent the auxiliary lines 96a1 and 96b1 from being displayed on the wall portions 96a and 96b. The control unit 33 can prevent the wall portions 96a and 96b from having a change in transmittance (gradient).

[0139] On the floor portions 97a and 97b of the second recognition wall images 89a and 89b, in addition to the first distance discrimination lines 98l and 98r, one or more second distance discrimination lines 100l and 100r are provided respectively. In FIG. 30, there are a plurality of second distance discrimination lines 100l and 100r, but only one of each is shown with a reference numeral. The first distance discrimination lines 98l and 98r and the second distance discrimination lines 100l and 100r may not be displayed on the display. The first distance discrimination lines 98l and 98r correspond to the horizontal distance from the moving body 50 to the detection target, while the second distance discrimination lines 100l and 100r can correspond to the distance in the depth direction. Each of the plurality of second distance discrimination lines 100l and 100r may correspond to the distance in the depth direction when the respective auxiliary lines 84c extending in the height direction on the side wall images 81a and 81b are mapped into the real space.

[0140] The subject 60 can grasp the horizontal and depth direction positions to the detection target according to the horizontal and depth direction positions of the icon images 99a and 99b on the floor portions 97a and 97b. As the icon images 99a and 99b, three-dimensional images corresponding to the type of the detection target are used. The floor portions 97a and 97b may be on the front side of the most front end of the side wall images 81a and 81b and may be arranged inside the lines extending the side wall images 81a and 81b. The positional relationship between the second recognition wall images 89a and 89b and the guide wall image 80 is not limited to this and can be changed as appropriate.

[0141] In FIG. 30, view display areas 101 and 102 indicating the selection status between "Normal View display" and "Wide View display" are shown in the lower right part of the display device 40. The "Normal View display" and the "Wide View display" have different viewing angles of the video to be displayed. In one of the plurality of embodiments, the "Normal View display" corresponds to the viewing angle of the display area 62 of the second video shown in FIG. 4. In one of the plurality of embodiments, the "Wide View display" corresponds to the viewing angle of the detection area 61 of the first video shown in FIG. 4. The "Wide View display" does not necessarily need to be the video of the entire detection area 61 of the first video, and may be a video cut out from the detection area 61 with an angle of view including the "second video". FIG. 30 and the second video shown earlier are both in the normal view display. Whether the currently displayed video is in the "Normal View display" or the "Wide View display" is indicated by changing the display of the view display areas 101 and 102 indicating the selection status. For example, changing the display of the area indicating the selection status includes changing the color of the view display areas 101 and 102, changing the density, changing the color of the characters, etc.

[0142] The control unit 33 can switch between the "Normal View display" and the "Wide View display" according to the result of processing the image of the first video or in response to an input from the subject 60 via the display device 40 or the like. For example, when the control unit 33 detects that the detection target is approaching in the state of the "Normal View display", it may switch the displayed video to the "Wide View display". When the control unit 33 detects that there is no detection target within a predetermined range in the state of the "Wide View display", it may switch the displayed video to the "Normal View display". When the display device 40 is provided with a touch panel type screen, the control unit 33 may receive a signal indicating that the subject 60 has touched and selected a part of the view display area 101 or 102 on the screen from the display device 40 and switch the display.

[0143] FIG. 31 is a diagram showing an example of an image switched from the "Normal View Display" shown in FIG. 30 to the "Wide View Display". The viewing angle of the image in the "Wide View Display" of FIG. 31 is wider in the horizontal direction than the viewing angle of the image in the "Normal View Display" of FIG. 30. In FIG. 31, a pedestrian 63 and a vehicle 64b located outside the display range of the "Normal View Display" shown in FIG. 30 are displayed. Also, in the image shown in FIG. 31, since the viewing angle in the horizontal direction is wider than that in the "Normal View Display" of FIG. 30, the interval between the left and right side wall images 81a and 81b of the guide wall image 80 is relatively narrowed and displayed.

[0144] In FIG. 31, since the pedestrian 63 and the vehicle 64b have entered the display range, the second recognition wall images 89a, 89b have disappeared. The control unit 33 recognizes the vehicle 64b and the pedestrian 63 within the detection area 61 of the first image, and can display the first recognition wall images 83a and 83b, which are virtual planes, on the moving body 50 side of the vehicle 64b and the pedestrian 63, respectively. The control unit 33 recognizes the vehicle 64b and the pedestrian 63 within the detection area 61 of the first image, and on the moving body 50 side of the vehicle 64b and the pedestrian 63 respectively, can display the first recognition wall images 83a and 83b, which are virtual planes. The control unit 33 estimates the positions of the detected vehicle 64b and pedestrian 63 in the depth direction. Based on the estimated positions, the control unit 33 may change the display of the side wall images 81a and 81b facing the vehicle 64b and the pedestrian 63, respectively. In FIG. 31, corresponding to the pedestrian 63 and the vehicle 64b respectively, the display of a part 81b1 of the side wall image 81b and a part 81a1 of the side wall image 81a is changed. The change in display includes various modes. For example, the change in display includes a change in color, a change in the thickness of the auxiliary line surrounding the periphery, a change in the line type, the start and stop of blinking, and a change in the blinking period, etc. The first recognition wall images 83a, 83b, and also, the parts 81a1, 81b1 of the side wall images 81a, 81b can adopt a display method and display mode similar to the example of FIG. 26. For example, the change in display includes a change in color, a change in the thickness of the auxiliary line surrounding the periphery, a change in the line type, the start and stop of blinking, and a change in the blinking period, etc. The first recognition wall images 83a, 83b, and also, the parts 81a1, 81b1 of the side wall images 81a, 81b can adopt a display method and display mode similar to the example of FIG. 26. also, the parts 81a1, 81b1 of the side wall images 81a, 81b can adopt a display method and display mode similar to the example of FIG. 26. and display mode can be adopted.

[0145] As shown in FIGS. 30 and 31 above, according to one of the plurality of embodiments, it is possible to switch and display the video captured by the imaging device 20 between "Normal View Display" and "Wide View Display" with different viewing angles on the display device 40. In the "Normal View Display", the subject 60 can easily grasp the presence and position of the detection target located outside the range of the displayed video by the second recognition wall images 89a, 89b and the icon images 99a, 99b. When the display device 40 automatically or manually switches from the "Normal View Display" to the "Wide View Display", the subject 60 can confirm the detection target detected outside the range of the video in the "Normal View Display" on the video. In the "Wide View Display", it becomes possible to three-dimensionally grasp the positional relationship with the detection target by the display of the guide wall image 80 and the first recognition wall image 83b. Thus, the display system 10 in one of the plurality of embodiments can easily allow the subject to grasp the surrounding situation and detect the danger.

[0146] Although the present invention has been described based on the drawings and embodiments, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present invention. For example, the functions and the like included in each means, each step, etc. can be rearranged so as not to be logically contradictory, and a plurality of means, steps, etc. can be combined into one or divided.

[0147] For example, each component and function of the display system 10 according to the above-described embodiment may be rearrangeable. For example, part or all of the configuration and functions of the image processing device 30 may be included in at least one of the imaging device 20 and the display device 40.

[0148] Some of the components of the display system 10 according to the above-described embodiment may be located outside the moving body 50. For example, the image processing device 30 or the like may be realized as a communication device such as a mobile phone or an external server, and may be connected to other components of the display system 10 by wire or wirelessly.

Explanation of Symbols

[0149] 10 Display system 20 Imaging device 21 Imaging optical system 22 Image sensor 23 Communication unit 24 Control unit 30 Image processing device 31 Communication unit 32 Memory unit 33 Control unit 40 Display device 50 Moving body 51 Network 60 Target person 61 Detection area 62 Display area 63, 63a, 63b, 63c, 63d Pedestrians 64, 64a, 64b Vehicles 65 First predicted path 66 Guide line 67 Icon image 68 Third marker 69 Contour line 70 Area 71 First marker 72 Second marker 73 Fifth marker 73a Icon image 73b Band image 73c Right end area 74 Obstacle image 75a, 75b, 75c Sixth marker 80 Guide wall image 81a, 81b Side wall image 82a, 82b, 82c Distance wall image 83, 83a, 83b First recognition wall image 84a, 84b, 84c Auxiliary line 89a, 89b Second recognition wall image 90 Virtual guide wall 91a, 91b Virtual side wall 92a, 92b, 92c Virtual distance wall Virtual lines 95l1~95l5, 95r1~95r5 Wall portions 96a, 96b Floor portions 97a, 97b First distance identification lines 98l1~98l5, 98r1~98r5 Icon images 99a, 99b Second distance identification lines 100l, 100r View display areas 101, 102

Claims

1. A display system mounted on a vehicle, comprising: an imaging device that images the rear of the vehicle; a display device that displays a second video which is a part of a first video obtained by imaging an imaging area by the imaging device; a control unit that detects an object within the imaging area on the right rear or left rear of the vehicle and outside the second video, and determines that there is a possibility of contact between the vehicle and the object because the distance between the vehicle and the object is less than a predetermined distance, and when the distance between the vehicle and the object is decreasing and the vehicle is moving at a moving speed less than a reference value, a marker is displayed in a display mode corresponding to the object in the second video displayed on the display device and recognizable that the object is approaching the vehicle; the control unit is capable of acquiring the moving speed of the vehicle; a display system.

2. The display system according to claim 1, wherein the control unit displays the marker by changing its width.

3. The display system according to claim 1 or 2, wherein the control unit displays the marker by changing its width according to the distance between the object and the vehicle.

4. The display system according to any one of claims 1 to 3, wherein the control unit displays the marker at the right end or the left end of the second video according to the position of the object.

5. The control unit elongates the display area of the second video to be displayed on the display device in the left - right direction according to a user operation. The display system according to any one of claims 1 to 4.

6. A moving body comprising the display system according to claims 1 to 5.

7. An image processing device mounted on a vehicle, which causes a display device to display a second video which is a part of a first video obtained by imaging an imaging area by an imaging device acquired from an imaging device that images the rear of the vehicle, comprising: a control unit that acquires the first video and cuts out the second video from the first video; the control unit detects an object outside the second video within the imaging area on the right rear or left rear of the vehicle; When the control unit determines that there is a possibility of contact between the vehicle and the object because the distance between the vehicle and the object is less than a predetermined distance, and the distance between the vehicle and the object is decreasing and the vehicle is moving at a moving speed less than a reference value, a marker is displayed in a display mode corresponding to the detected object and recognizable that the object is approaching the vehicle on the second video displayed on the display device. The control unit can acquire the moving speed of the vehicle. Image processing device.

8. A program for an image processing device mounted on the vehicle that causes a display device to display a second video that is a part of a first video obtained by imaging an imaging area with an imaging device that images the rear of the vehicle. An acquisition step in which a control unit mounted on the image processing device acquires the first video imaged by the imaging device. A step in which the control unit cuts out the second video from the first video. A detection step in which the control unit detects an object outside the second video within the imaging area on the right rear or left rear of the vehicle. When the control unit detects the object in the detection step, when it is determined that there is a possibility of contact between the vehicle and the object because the distance between the vehicle and the object is less than a predetermined distance, and the distance between the vehicle and the object is decreasing and the vehicle is moving at a moving speed less than a reference value, a marker is displayed in a display mode corresponding to the object and recognizable that the object is approaching the vehicle on the second video displayed on the display device. The control unit can acquire the moving speed of the vehicle. Program.

Images