Display system, mobile body, image processing device, and display system

Abstract

To improve the convenience of a technology for displaying images of an external area of a moving object.SOLUTION: An imaging device 20 includes an imaging element 22 and a control unit 33. The imaging element 22 captures an image behind a vehicle 50 to generate an image. The control unit 33 displays a recognition image on the image at a position in the image corresponding to a position between the vehicle and a detection target, the recognition image extending a predetermined distance in the height direction relative to the road surface and the transmittance increasing with increasing distance in the height direction from the road surface.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging device, an image processing device, a display device, a display system, and a vehicle. [Background technology]

[0002] Conventionally, there are known techniques for displaying an image of an external area of ​​a moving body such as a vehicle. For example, Patent Document 1 discloses a technique for controlling the power supply to a monitor that displays an image captured by a camera provided in a vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-40113 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, there has been room for improvement in the technology for displaying images of the external area of ​​a moving object.

[0005] The present disclosure relates to an image processing device, an imaging device, a display device, and a display system that improve the convenience of a technology for displaying an image of an area outside a moving object. [Means for solving the problem]

[0006] An imaging device according to one embodiment of the present disclosure includes an imaging element that captures an image behind a vehicle to generate an image, and a control unit that displays a recognition image on the image at a position within the image that corresponds to a position between the vehicle and a detection target, the recognition image extending a predetermined distance in the height direction relative to the road surface and having a higher transmittance the greater the distance in the height direction from the road surface.

[0007] An image processing device according to one embodiment of the present disclosure includes a communication unit that acquires an image captured behind a vehicle, and a control unit that displays a recognition image on the image at a position within the image corresponding to a position between the vehicle and a detection target, the recognition image extending a predetermined distance in the height direction relative to the road surface and having a higher transmittance the greater the distance in the height direction from the road surface.

[0008] A display device according to one embodiment of the present disclosure includes a communication unit that acquires an image captured behind a vehicle, and a control unit that displays a recognition image on the image at a position within the image that corresponds to a position between the vehicle and a detection target, the recognition image extending a predetermined distance in the height direction relative to the road surface and having a higher transmittance the greater the distance in the height direction from the road surface.

[0009] A display system according to one embodiment of the present disclosure includes a communication unit that acquires an image of the area behind the vehicle, and a control unit that displays a recognition image on the image at a position within the image that corresponds to a position between the vehicle and a detection target, the recognition image extending a predetermined distance in the height direction relative to the road surface and having a higher transmittance the greater the distance in the height direction from the road surface.

[0010] A vehicle according to an embodiment of the present disclosure includes a display system including an imaging element that captures an image behind the vehicle to generate an image, and a control unit that displays, at a position in the image that corresponds to a position between the vehicle and a detection target, a recognition image that extends a predetermined distance in a height direction relative to a road surface and has a higher transmittance as the distance in the height direction from the road surface increases.

[0011] An image processing device according to one embodiment of the present disclosure acquires an image of the external area of ​​a moving body captured by an imaging device, and displays on the image a recognition image that extends in a height direction relative to the road surface and has a higher transmittance as the distance in the height direction relative to the road surface increases at a position in the image corresponding to the position between the moving body and a detection target. Effect of the Invention

[0012] According to an image processing device, an imaging device, a display device, and a display system according to an embodiment of the present disclosure, the convenience of the technology for displaying an image of an area outside a moving object is improved. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a schematic configuration of a display system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a view of a vehicle equipped with a display system as viewed from the left side. [Diagram 3] FIG. 2 is a diagram illustrating a schematic external view of the display device of FIG. [Figure 4] FIG. 2 is a diagram showing a first example of a first image. [Diagram 5] 5 is a diagram showing a first example of a second image corresponding to the display region of the first image in FIG. 4. [Figure 6] 13 is a diagram showing an example of a third marker superimposed on a detection target. FIG. [Figure 7] 4 is a diagram showing a first example of a first marker and a second marker displayed around a detection target. FIG. [Figure 8] 13 is a diagram showing a second example of a first marker and a second marker displayed around a detection target. FIG. [Figure 9] 13 is a diagram showing a third example of a first marker and a second marker displayed around a detection target. FIG. [Figure 10] FIG. 13 is a diagram showing a second example of the first image. [Figure 11] 11 is a diagram showing a second example of a second image corresponding to the display region of the first image in FIG. 10. FIG. [Figure 12] FIG. 13 is a diagram showing a third example of the first image. [Figure 13] 13 is a diagram showing a third example of the second image corresponding to the display region of the first image in FIG. 12. FIG. [Figure 14] FIG. 13 is a diagram showing a fourth example of the first image. [Figure 15] 15 is a diagram showing a fourth example of the second image corresponding to the display region of the first image in FIG. 14. FIG. [Figure 16] FIG. 13 is a diagram showing a fifth example of the first image. [Figure 17] 17 is a diagram showing a fifth example of the second image corresponding to the display region of the first image in FIG. 16. FIG. [Figure 18]17 is a diagram showing another example of the second image corresponding to the display region of the first image in FIG. 16. FIG. [Figure 19] FIG. 13 is a diagram showing a sixth example of the first image. [Figure 20] 20 is a diagram showing a sixth example of the second image corresponding to the display region of the first image in FIG. 19. FIG. [Figure 21] FIG. 13 is a diagram showing another example of the second image. [Figure 22] FIG. 13 is a diagram showing a first modified example of the sixth example of the second image. [Figure 23] FIG. 11 is a diagram showing a second modified example of the sixth example of the second image; [Figure 24] FIG. 13 is a diagram showing a third modified example of the sixth example of the second image. [Diagram 25] A figure showing a seventh example of the first image. [Figure 26] FIG. 27 is a diagram showing a seventh example of the second image corresponding to the display region of the first image in FIG. [Figure 27] FIG. 13 is a diagram showing an eighth example of the first image. [Figure 28] FIG. 2 is a diagram showing the positional relationship between a vehicle and an object to be detected; [Figure 29] FIG. 25 is a diagram showing an eighth example of the second image corresponding to the display region of the first image in FIG. 24. [Diagram 30] FIG. 13 is a diagram showing a modified example of the eighth example of the second image. [Diagram 31] FIG. 31 is a diagram showing an example in which the display range of FIG. 30 is changed to a wide view display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] (Display System) Referring to FIG. 1, a display system 10 according to an embodiment of the present invention will be described.

[0016] 1, the display system 10 includes an imaging device 20, an image processing device 30, and a display device 40. The imaging device 20 and each component of the display system 10 can transmit and receive information via, for example, a network 51. The network 51 may include, for example, a wireless network, a wired network, a CAN (Controller Area Network), or the like.

[0017] In another embodiment, some or all of the components of the display system 10 may be integrated into a single device. For example, the image processing device 30 may be built into the imaging device 20 or the display device 40.

[0018] As shown in FIG. 2, the imaging device 20, the image processing device 30, and the display device 40 may be provided in a moving object 50. The "moving object" in the present disclosure may include, for example, vehicles, ships, and aircraft. The vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, residential vehicles, and fixed-wing aircraft that run on runways. The automobiles may include, for example, passenger cars, trucks, buses, motorcycles, and trolley buses. The industrial vehicles may include, for example, industrial vehicles for agriculture and construction. The industrial vehicles may include, for example, forklifts and golf carts. The industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, and lawnmowers. The industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, and road rollers. The vehicles may include those that run by human power. The classification of vehicles is not limited to the above examples. For example, the automobiles may include industrial vehicles that can run on roads. The same vehicle may be included in multiple classifications. Marine vehicles may include, for example, watercraft, boats, tankers, etc. Aircraft may include, for example, fixed-wing aircraft and rotorcraft, etc.

[0019] The imaging device 20 is capable of capturing an image of the external area of ​​the moving body 50. The position of the imaging device 20 may be any position inside or outside the moving body 50. For example, as shown in FIG. 2, the imaging device 20 is located at the rear of the moving body 50 and is capable of capturing an image of the external area behind the moving body 50. The position of the image processing device 30 may be any position within the moving body 50. The display device 40 is viewable by the subject 60. The position of the display device 40 may be any position within the moving body 50. For example, as shown in FIG. 2, the display device 40 is located in the dashboard of the moving body 50.

[0020] (Imaging device) The imaging device 20 will be described in detail below. For example, as shown in FIG.

[0021] The imaging optical system 21 forms an image of a subject. For example, the imaging optical system 21 may include a diaphragm and one or more lenses.

[0022] The imaging element 22 has a plurality of pixels arranged two-dimensionally. The imaging element 22 may include, for example, a charge coupled device (CCD) imaging element or a complementary metal oxide semiconductor (CMOS) imaging element. The imaging element 22 can capture an image of a subject formed by the imaging optical system 21 to generate a captured image.

[0023] The communication unit 23 may include a communication interface capable of communicating with an external device. The communication unit 23 may be capable of transmitting and receiving information via a network 51. The external device may include, for example, an image processing device 30. The "communication interface" in the present disclosure may include, for example, a physical connector and a wireless communication device. The physical connector may include an electrical connector compatible with transmission by electrical signals, an optical connector compatible with transmission by optical signals, and an electromagnetic connector compatible with transmission by electromagnetic waves. The electrical connector may include a connector conforming to IEC60603, a connector conforming to the USB standard, a connector compatible with an RCA terminal, a connector compatible with an S terminal defined in EIAJ CP-1211A, a connector compatible with a D terminal defined in EIAJ RC-5237, a connector conforming to the HDMI (registered trademark) standard, and a connector compatible with a coaxial cable including BNC (British Naval Connector or Baby-series N Connector, etc.). The optical connector may include various connectors conforming to IEC 61754. The wireless communication device may include a wireless communication device conforming to each standard including Bluetooth (registered trademark) and IEEE802.11. The radio includes at least one antenna.

[0024] The control unit 24 includes one or more processors. In the present disclosure, the term "processor" may include a dedicated processor specialized for a specific process, and a general-purpose processor that executes a specific function by loading a specific program. The dedicated processor may include a digital signal processor (DSP) and an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 24 may be either a system-on-a-chip (SoC) in which one or more processors work together, or a system in a package (SiP).

[0025] The control unit 24 controls the operation of the entire imaging device 20. The control unit 24 may cause the imaging element 22 to generate an image at an arbitrary frame rate. The frame rate may be, for example, approximately equal to the frame rate at which the display device 40 can display. The control unit 24 may execute a predetermined image processing on the generated image. The image processing may include, for example, exposure adjustment processing, white balance processing, and distortion correction processing. The control unit 24 outputs the image to the image processing device 30 via the communication unit 23. For example, the control unit 24 may sequentially output the image at the above-mentioned frame rate. Hereinafter, each image output at the above-mentioned frame rate is also simply referred to as a frame. The multiple images output from the imaging device 20 are also referred to as a first image. For example, when the frame rate is 60 fps (Flame per Seconds), 60 images per second are output as the first image.

[0026] (Image Processing Device) A detailed description will be given of the image processing device 30. The image processing device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0027] The communication unit 31 may include a communication interface capable of communicating with various external devices. The external devices may include, for example, the imaging device 20, the display device 40, an ECU (Electronic Control Unit or Engine Control unit) provided in the moving body 50, a speed sensor, an acceleration sensor, a rotation angle sensor, a steering angle sensor, an engine speed sensor, an accelerator sensor, a brake sensor, an illuminance sensor, a raindrop sensor, a mileage sensor, an obstacle detection device using a millimeter wave radar, an ultrasonic sonar, or the like, an ETC (Electronic Toll Collection System) receiving device, a GPS (Global Positioning System) device, a navigation device, a server on the Internet, a mobile phone, or the like.

[0028] The communication unit 31 may include a communication interface for pedestrian-vehicle communication, road-to-vehicle communication, and vehicle-to-vehicle communication. The communication unit 31 may include a receiver compatible with optical beacons of DSRC (Dedicated Short-Range Communication) and VICS (Vehicle Information and Communication System) (registered trademark) provided in Japan. The communication unit 31 may include a receiver compatible with road traffic information systems of other countries.

[0029] The communication unit 31 may be capable of acquiring various information from an external device. For example, the communication unit 31 may be capable of acquiring moving object information and environmental information.

[0030] The mobile body information may include any information related to the mobile body 50. The mobile body information may include, for example, the speed, acceleration, turning gravity, inclination, direction, and turning status of the mobile body 50, the steering angle of the steering wheel, the temperature of the cooling water, the remaining amount of fuel, the remaining amount of battery, the battery voltage, the engine speed, the gear position, the presence or absence of a reverse signal, the presence or absence of an accelerator operation, the accelerator opening, the presence or absence of a brake operation, the brake depression, the presence or absence of an operation of the parking brake, the difference in the rotation speed of the front and rear wheels or the four wheels, the tire pressure, the expansion and contraction amount of the damper, the spatial position of the driver's eyes, the number and seat position of the passengers, the fastening information of the seat belt, the opening and closing of the doors, the opening and closing of the windows, the temperature inside the vehicle, the presence or absence of the operation of the air conditioning, the set temperature of the air conditioning, the blowing amount of the air conditioning, the setting of the outside air circulation, the operation status of the wipers, the driving mode, the connection information with the external device, the current time, the average fuel consumption, the instantaneous fuel consumption, the lighting state of various lamps, the position information of the mobile body 50, and the route information to the destination of the mobile body 50. The various lamps may include, for example, headlamps, fog lamps, backup lamps, position lamps, and turn signals.

[0031] The environmental information may include any information related to the external environment of the moving body 50. The environmental information may include, for example, the brightness around the moving body 50, the weather, the air pressure, the outside temperature, map information, traffic information, road construction information, temporary changes in the speed limit of the road, objects detected by other vehicles, and the lighting status of traffic lights.

[0032] The storage unit 32 may include a temporary storage device and a secondary storage device. The storage unit 32 may be configured using, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape, etc. The optical memory may include, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), and a BD (Blu-ray Disc) (registered trademark), etc. The storage unit 32 stores various information and programs necessary for the operation of the image processing device 30.

[0033] The control unit 33 includes one or more processors and controls the operation of the entire image processing device 30.

[0034] The control unit 33 may acquire moving object information and environmental information from an external device via the communication unit 31. The control unit 33 may determine a predicted path of the moving object 50, for example, based on the moving object information. Hereinafter, the predicted path of the moving object 50 is also referred to as a first predicted path.

[0035] The control unit 33 may acquire a first image from the imaging device 20 via the communication unit 31. The first image includes a detection area and a display area.

[0036] The control unit 33 may detect at least a portion of the detection target within the detection area on the acquired first image. The detection area on the first image may be at least a portion of the area on the captured image, which is each frame of the first image. Each frame of the first image may be referred to as a captured image. The detection area on the first image may be larger than the display area. The detection area on the first image may include the display area. The control unit 33 may detect the detection target inside the display area. The control unit 33 may detect the detection target outside the display area and inside the detection area. The detection area and the area inside the display area may be referred to as a first area. The area inside the detection area and outside the display area may be referred to as a second area.

[0037] The detection target may include a plurality of types of objects. The types of objects may include, for example, a person, another moving object, a road, a lane, a white line, a gutter, a sidewalk, a crosswalk, a road sign, a traffic sign, a guardrail, a wall, and a traffic light. The types of detection targets that the control unit 33 can detect are not limited to these. For example, when a part of the detection target on the first image is hidden behind another object, at least a part of the detection target may include a part of the detection target that is not hidden behind the other object. For example, when the lower half of the pedestrian's body is hidden behind an obstacle on the first image, the control unit 33 may be able to detect the upper half of the pedestrian. Any object detection algorithm may be employed to detect at least a part of the detection target. For example, the control unit 33 may detect at least a part of the detection target by an algorithm such as pattern matching or feature point extraction using the captured images that are each frame of the first image.

[0038] When at least a part of the detection target is detected on the first video, the control unit 33 may determine a predicted path of the detection target based on the first video. Hereinafter, the predicted path of the detection target is also referred to as a second predicted path. Any algorithm may be adopted for determining the second predicted path. For example, the control unit 33 may determine the second predicted path based on changes in the orientation and position of the detection target on the captured image, which is each frame of the first video.

[0039] When at least a part of the detection target is detected on the first image, the control unit 33 may estimate the relative positional relationship between the moving body 50 and the detection target based on the first image. The relative positional relationship may include, for example, the distance between the moving body 50 and the detection target, and the presence or absence of overlap between the first predicted path of the moving body 50 and the second predicted path of the detection target. Any algorithm may be adopted for estimating the distance between the moving body 50 and the detection target. For example, the control unit 33 may estimate the distance between the moving body 50 and the detection target by a motion stereo method using the captured images that are each frame of the first video signal. In another embodiment, the control unit 33 may acquire information indicating the relative positional relationship between the moving body 50 and the detection target from an external device via the communication unit 31.

[0040] When the distance between the moving body 50 and the detection target decreases, the control unit 33 may determine which of the moving body 50 and the detection target contributes more to the decrease in the distance. Any algorithm may be adopted to determine the contribution of the moving body 50 and the detection target to the decrease in the distance. In one example, the control unit 33 may detect the moving speed of the moving body 50 based on the moving body information. The control unit 33 may detect the moving speed of the detection target based on a change in the position of the detection target on the captured image, which is each frame of the first video, for example. The control unit 33 may determine that the contribution of either the moving body 50 or the detection target, which has a higher moving speed, is greater to the decrease in the distance. In another example, the control unit 33 may determine that the contribution of the detection target is greater to the decrease in the distance when the moving speed of the moving body 50 is less than a reference value. The control unit 33 may determine that the contribution of the moving body 50 is greater to the decrease in the distance when the moving speed of the moving body 50 is equal to or greater than a reference value. The reference value may be set arbitrarily, and may be set to approximately zero, for example. The operation of the image processing device 30 according to the contribution of the moving body 50 and the detection target to the reduction in distance will be described in detail later.

[0041] When at least a part of the detection target is detected on the first image, the control unit 33 may determine whether or not there is a possibility that the moving body 50 and the detection target will come into contact with each other based on the first image. Any algorithm may be adopted for determining the possibility that the moving body 50 and the detection target will come into contact with each other. For example, the control unit 33 may determine that there is a possibility that the moving body 50 and the detection target will come into contact with each other when at least one of the following conditions is satisfied: the distance between the moving body 50 and the detection target is less than a predetermined threshold value, and the speed at which the distance is decreasing is equal to or greater than a predetermined threshold value. Details of the operation of the image processing device 30 according to the presence or absence of the possibility will be described later.

[0042] The control unit 33 may display the second image corresponding to the display area on the first image acquired from the imaging device 20 on the display device 40. Specifically, the control unit 33 may output the second image to the display device 40 via the communication unit 31. For example, when the control unit 33 detects the backward movement of the moving body 50 based on the moving body information, the control unit 33 may display the second image on the display device 40. For example, the control unit 33 may detect the backward movement based on the shift position of the transmission gear. For example, the control unit 33 may detect the backward movement based on a reverse signal output from the moving body when the moving body is reversed. The second image may be, for example, an image obtained by cutting out the display area on the captured image, which is each frame of the first image. The display area on the first image may be at least a part of the area on the captured image, which is each frame of the first image. The display area may be smaller than the detection area. The display area may be included in the detection area. The position, shape, and size of the display area may be determined arbitrarily. The control unit 33 may change the position, shape, and size of the display area. The position, shape, and size of the display area may change so that, for example, the display area and the detection area substantially coincide with each other.

[0043] The control unit 33 may composite various markers into the second image and display them on the display device 40. Compositing includes overwriting and mixing. The markers may include, for example, one or more images. The control unit 33 may dynamically change the display mode of at least a part of the markers superimposed on the second image. The display mode may include, for example, the position, size, shape, color, and shade of at least a part of the marker on the second image. When displaying a marker corresponding to a detection target detected on the first image, the control unit 33 may determine the display mode of the marker depending on the type of the detection target. Details of the operation of the image processing device 30 that displays various markers on the display device 40 will be described later.

[0044] (display device) The display device 40 will be described in detail. The display device 40 may include, for example, a liquid crystal display and an organic EL (Electroluminescence) display. The display device 40 may display, for example, a second image input from the image processing device 30 via a network 51. The display device 40 may function as a touch screen capable of receiving an operation by a user. The display device 40 may include a switch and a key capable of receiving an operation by a user. The switch may include a physical switch and an electronic switch. The key may include a physical key and an electronic key. When the display device 40 receives an operation by a user on the display device itself, the display device 40 may transmit a user input based on the operation to the image processing device 30.

[0045] The display device 40 can be arranged in various places in the moving object 50. FIGS. 3A to 3E are diagrams showing examples of the arrangement of the display device 40 when the moving object 50 is a vehicle. FIG. 3A shows an in-dashboard display device 40a stored in the dashboard of the vehicle. FIG. 3B shows an on-dashboard display device 40b arranged on the dashboard. In FIG. 3B, the display device 40b is built into the vehicle 50. The display device 40b may be detachably attached to the dashboard. FIG. 3C shows a display device 40c that is built into the rearview mirror and can display an image as needed. FIG. 4D shows a display device 40d built into the instrument panel. In FIG. 4D, the display device 40d is arranged adjacent to instruments such as a speedometer and a tachometer. In one embodiment, the entire instrument panel may be a display device 40d such as an LCD, and a second image may be displayed therein together with images of the speedometer and the tachometer. FIG. 4E shows a display device 40e using a portable information terminal, for example, a tablet terminal. The display device 40e may be a mobile phone display.

[0046] 4 to 18, the second image and various markers that the image processing device 30 causes the display device 40 to display will be specifically described. In this disclosure, the terms "up-down direction" and "left-right direction" for an image or picture correspond to two-dimensional directions in the image or picture. In this disclosure, the terms "height direction," "horizontal direction," and "depth direction" for an image or picture correspond to three-dimensional directions in the space in which the image or picture is projected.

[0047] (1st example (reference example)) Fig. 4 shows a first example of a detection area 61 in a first image acquired by the image processing device 30 from the imaging device 20. In the example shown in Fig. 4, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63 and a vehicle 64 that are shown inside the display area 62 on the first image.

[0048] The control unit 33 determines whether or not one or more conditions are satisfied based on the relative positional relationship between the detection target detected inside the display area 62 on the first image and the moving body 50. The one or more conditions may include, for example, a first condition that the detection target is located within the first predicted path 65 of the moving body 50. The one or more conditions may include, for example, a second condition that at least a part of the first predicted path 65 of the moving body 50 overlaps with at least a part of the second predicted path of the detection target. When it is determined that the one or more conditions are satisfied, the control unit 33 may cause the display device 40 to display a predetermined marker corresponding to the detection target by superimposing it on the second image. The predetermined marker may include a first marker, a second marker, and a third marker.

[0049] In the first example, the control unit 33 may determine that the one or more conditions are satisfied for the pedestrian 63. In such a case, the control unit 33 may display a marker corresponding to the pedestrian 63. The control unit 33 may determine that the one or more conditions are not satisfied for the vehicle 64. In such a case, the control unit 33 does not display a marker corresponding to the vehicle 64.

[0050] Fig. 5 shows an example of the second image corresponding to display area 62 of the first image shown in Fig. 4. When the aspect ratio of display area 62 of the first image differs from the aspect ratio of the screen of display device 40, control unit 33 may cut out display area 62 of the first image, and then output the second image that has been transformed to match the aspect ratio of the screen of display device 40 to display device 40. As shown in Fig. 5, a pedestrian 63 and a vehicle 64 are shown in the second image.

[0051] The control unit 33 may, for example, superimpose a guide line 66 indicating at least a part of a first predicted path 65 of the moving object 50 shown in Fig. 4 on the second image and display it on the display device 40. The control unit 33 may dynamically change the guide line 66 in response to a change in the steering angle of the steering wheel, for example.

[0052] The first image has a larger range than the display area 62. The control unit 33 may change the range of the display area 62. The control unit 33 may superimpose an icon image 67 on the second image and display it on the display device 40. For example, the outline 67a of the icon image 67 shown in FIG. 5 indicates the maximum range when the range of the display area 62 is changed. The white rectangle 67b of the icon image 67 indicates the display area 62. The icon image 67 shown in FIG. 5 indicates the relative position and size of the display area 62 with respect to the maximum range of the display area 62.

[0053] FIG. 6 shows an example of a marker superimposed on a pedestrian 63 on the second image, for example. Hereinafter, the marker is also referred to as a third marker 68. A contour line 69 of the third marker 68 may approximately match a contour line of the pedestrian 63 detected on the second image. An area 70 inside the contour line 69 of the third marker 68 may be filled with a color or pattern corresponding to a "person" which is a type of detection target. When the pedestrian 63 is detected on the first image, the control unit 33 may superimpose the third marker 68 on the pedestrian 63 on the second image and display it on the display device 40. With this configuration, the subject 60 can easily view the pedestrian 63 on the second image. When a predetermined time has elapsed since the control unit 33 displayed the third marker 68, the control unit 33 may hide the third marker 68.

[0054] 7 shows an example of two types of markers superimposed around pedestrian 63 on the second image. Hereinafter, the two types of markers are also referred to as first marker 71 and second marker 72. For example, after hiding third marker 68, control unit 33 may cause display device 40 to display first marker 71 and second marker 72 superimposed on the second image.

[0055] The control unit 33 may move the position of the first marker 71 so as to follow the pedestrian 63 on the second image. Since the first marker 71 follows the pedestrian 63, the target person 60 can easily catch the pedestrian 63. The first marker 71 is displayed around the pedestrian 63 at a distance from the pedestrian 63. When the first marker 71 is displayed on the display device 40, the target person 60 can easily grasp the behavior of the pedestrian 63. The control unit 33 may change the superimposed position of the second marker 72 relatively to the superimposed position of the first marker 71 on the second image. The control unit 33 may move the second marker relatively based on the position of the first marker 71.

[0056] For example, when the distance between the moving object 50 and the pedestrian 63 is decreasing, the control unit 33 may determine that the contribution of the moving object 50 to the decrease in the distance is large. In such a case, the control unit 33 may move the second marker 72 toward the first marker 71. First, the control unit 33 displays the second marker 72 at a position away from the first marker 71. Next, the control unit 33 moves the second marker 72 toward the first marker 71 until the distance between the second marker 72 and the first marker 71 becomes a predetermined distance. Next, the control unit 33 erases the second marker 72. Next, the control unit 33 displays the second marker 72 again at a position away from the first marker 71, and repeats the above-mentioned process. In this example, since the second marker 72 is moving toward the object, the first marker 71, the subject 60 may understand that the second marker 72 is approaching the first marker 71.

[0057] For example, when the distance between the moving object 50 and the pedestrian 63 is decreasing, the control unit 33 may determine that the pedestrian 63 has contributed greatly to the decrease in the distance. In such a case, the control unit 33 may move the second marker 72 away from the first marker 71. First, the control unit 33 displays the second marker 72 at a position close to the first marker 71. Next, the control unit 33 moves the second marker 72 away from the first marker 71 until the distance between the second marker 72 and the first marker 71 becomes a predetermined distance. Next, the control unit 33 erases the second marker 72. Next, the control unit 33 displays the second marker 72 again at a position close to the first marker 71, and repeats the above-mentioned process. In this example, since the second marker 72 has moved away from the object, the first marker 71, the subject 60 may understand that the second marker 72 is moving away toward the first marker 71.

[0058] The control unit 33 changes the moving direction of the second marker 72 relative to the first marker 71 depending on the degree of contribution of the moving body 50 and the pedestrian 63 to the reduction in the distance between the moving body 50 and the pedestrian 63. Depending on the moving direction of the second marker 72, the subject 60 can identify, for example, whether the moving body 50 is approaching the pedestrian 63 or the pedestrian 63 is approaching the moving body 50.

[0059] The control unit 33 may repeatedly enlarge or reduce the second marker 72 around the first marker 71 on the second image as a center, as shown in Fig. 8 for example. The control unit 33 may superimpose the first marker 71 and the second marker 72 having a shape similar to the contour line 69 of the pedestrian 63 on the second image as shown in Fig. 9 for example. The control unit 33 may repeatedly enlarge or reduce the second marker 72 around the first marker 71 as a center. The control unit 33 switches between enlarging and reducing the second marker 72 depending on the degree of contribution of the moving object 50 and the pedestrian 63 to the reduction in the distance between the moving object 50 and the pedestrian 63.

[0060] When the distance between the detection target displaying the first marker 71 and the second marker 72 and the moving body 50 becomes less than a predetermined threshold, the control unit 33 may superimpose a new marker on the second image. Hereinafter, the new marker is also referred to as a fourth marker. The fourth marker may include any image. For example, the fourth marker may include an image showing an exclamation mark "!". According to this configuration, for example, when the pedestrian 63 displaying the first marker 71 and the second marker 72 and the moving body 50 approach each other to a certain degree or more, the fourth marker is superimposed and displayed on the second image. The subject 60 can identify, for example, that the pedestrian 63 is located in the vicinity of the moving body 50 by the fourth marker. In another embodiment, the control unit 33 may change the display mode of the first marker 71 and the second marker 72 when the distance between the detection target displaying the first marker 71 and the second marker 72 and the moving body 50 becomes less than a predetermined threshold. The control unit 33 may, for example, change the colors of the first marker 71 and the second marker 72. Even with this configuration, the subject 60 can identify that, for example, a pedestrian 63 is located near the moving object 50 by the change in the color of the markers.

[0061] The control unit 33 may detect two detection targets arranged front and back in the depth direction. The control unit 33 may display a first marker 71 and a second marker 72 on each of the two detection targets located front and back. The control unit 33 may attach different first markers 71 and second markers 72 to the two detection targets. For example, the control unit 33 may attach a first marker 71 and a second marker 72 to the first detection target located at the back side that are less noticeable than the first marker 71 and the second marker 72 attached to the second detection target located at the front side. For example, the control unit 33 may change the first marker 71 and the second marker 72 attached to the first detection target located at the back side to a darker color, a higher transmittance, a thinner line, etc., than the first marker 71 and the second marker 72 attached to the second detection target located at the front side.

[0062] (Example 2-4 (reference example)) 10 shows a second example of a detection area 61 in a first image acquired by the image processing device 30 from the imaging device 20. In the example shown in FIG. 10, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63a that is shown inside the display area 62 in the first image, and pedestrians 63b and 63c that are shown outside the display area 62 but inside the detection area 61. The process of the control unit 33 regarding the pedestrian 63a is similar to the process regarding the pedestrian 63 shown in FIG. 4, for example.

[0063] When the detection position on the first image where the detection target is detected is outside the display area 62 and inside the detection area 61, the control unit 33 may superimpose a marker corresponding to the detection target on the second image and display it on the display device 40. Hereinafter, the marker is also referred to as a fifth marker. When it is determined that there is a possibility that the moving body 50 and the detection target may come into contact with each other, the control unit 33 may display the fifth marker on the display device 40 by superimposing the fifth marker on the right edge of the second image when the detection position on the first image where the detection target is detected is to the right of the display area 62. When the detection position on the first image where the detection target is detected is to the left of the display area 62, the control unit 33 may superimpose the fifth marker on the left edge of the second image and display it on the display device 40.

[0064] In the second example, the detection position where the pedestrian 63b is detected is to the right of the display area 62. The control unit 33 may determine that there is a possibility that the moving object 50 and the pedestrian 63b will come into contact with each other. In such a case, the control unit 33 may cause the display device 40 to display a fifth marker corresponding to the pedestrian 63b by superimposing it on the right edge of the second image. Details of the fifth marker corresponding to the pedestrian 63b will be described later. The detection position where the pedestrian 63c is detected is to the left of the display area 62. The control unit 33 may determine that there is no possibility that the moving object 50 and the pedestrian 63c will come into contact with each other. In such a case, the control unit 33 may not cause the display device 40 to display the fifth marker corresponding to the pedestrian 63b.

[0065] Fig. 11 shows an example of the second image corresponding to display area 62 of the first image shown in Fig. 10. As shown in Fig. 11, pedestrian 63a appears in the second image. Pedestrians 63b and 63c are not shown in the second image.

[0066] As shown in FIG. 11, the control unit 33 may superimpose an obstacle image 74 on the second image and display it on the display device 40. The obstacle image 57 shows the detection result of an obstacle detection device using an ultrasonic sonar or the like provided on the moving body 50. The obstacle image 74 may include an image 74a, an image 74b, and an image 74c. The image 74a is an image of the moving body 50 viewed from above. The image 74b is an image showing that an obstacle has been detected to the left rear of the moving body 50. The image 74c is an image showing that an obstacle has been detected to the right rear of the moving body 50. The detection result of the obstacle by the obstacle detection device and the detection result of the detection target by the control unit 33 do not necessarily have to match. For example, in the example shown in FIG. 11, the obstacle image 74 shows that obstacles have been detected to the right rear and left rear of the moving body 50. On the other hand, the control unit 33 may determine that there is no possibility of contact between the moving body 50 and the pedestrian 63c present to the left rear of the moving body 50. In this case, the control unit 33 does not need to display the fifth marker corresponding to the pedestrian 63c.

[0067] FIG. 11 shows an example of the fifth marker 73 corresponding to the pedestrian 63b. The fifth marker 73 may include an icon image 73a and a band image 73b. The icon image 73a may be an image corresponding to a "person" which is a type of detection target. The subject 60 who views the icon image 73a can recognize that a person is present to the right of the second image. The band image 73b is, for example, a band-shaped image extending in the vertical direction on the second image. The band image 73b may be filled with a color or pattern corresponding to a "person" which is a type of detection target. The control unit 33 may move the band image 73b within a right end region 73c on the second image. The control unit 33 may change the moving speed and width of the band image 73b.

[0068] The fifth marker 73 will be described in detail. The control unit 33 may determine the width of the band image 73b according to the distance between the moving object 50 and the pedestrian 63b. For example, the control unit 33 may increase the width of the band image 73b as the distance becomes closer. The subject 60 who visually recognizes the band image 73b can recognize the distance between the moving object 50 and the pedestrian 63b based on the width of the band image 73b.

[0069] The control unit 33 may determine that, for example, the contribution of the moving body 50 is large with respect to the reduction in the distance between the moving body 50 and the pedestrian 63b. In this case, the control unit 33 repeatedly moves the band image 73b in a first direction in the right end region 73c on the second image. The first direction may be, for example, a direction from the outside to the inside in the left-right direction on the second image. The control unit 33 may determine that, for example, the contribution of the pedestrian 63b is large with respect to the reduction in the distance between the moving body 50 and the pedestrian 63b. In this case, the control unit 33 repeatedly moves the band image 73b in a second direction in the right end region 73c on the second image. The second direction may be, for example, a direction from the inside to the outside in the left-right direction on the second image. The subject 60 who views the band image 73b can recognize whether the moving body 50 is approaching the pedestrian 63b or the pedestrian 63b is approaching the moving body 50 based on the movement direction of the band image 73b.

[0070] The control unit 33 may determine the moving speed of the band image 73b according to the speed at which the distance between the moving object 50 and the pedestrian 63b decreases. For example, the moving speed of the band image 73b may be made faster as the speed at which the distance decreases increases. The subject 60 who visually recognizes the band image 73b can recognize the speed at which the distance between the moving object 50 and the pedestrian 63b decreases based on the moving speed of the band image 73b.

[0071] When the control unit 33 detects a user input corresponding to a predetermined user operation while the fifth marker 73 is displayed, the control unit 33 may change the display area 62 so that the detected position of the pedestrian 63b on the first image is included inside the display area 62. For example, as shown in Fig. 12, the control unit 33 may lengthen the display area 62 on the first image in the left-right direction and move the display area 62 to the right side within the detection area 61. With this configuration, the pedestrian 63b is shown on the second image, as shown in Fig. 13, for example.

[0072] The above-mentioned predetermined user operation may include any user operation. For example, the above-mentioned predetermined user operation may include a first user operation for changing the steering angle of the moving body 50. The fifth marker 73 may function as a GUI (Graphic User Interface) for accepting a second user operation. Hereinafter, the GUI is also referred to as an interface image. In this case, the above-mentioned predetermined user operation may include the second user operation.

[0073] Control unit 33 may automatically change display area 62 so that the detection position of pedestrian 63b on the first image is included inside display area 62. In this case, control unit 33 may maintain the automatic change of display area 62 until pedestrian 63b is no longer detected in detection area 61 of the first image.

[0074] The control unit 33 may change the icon image 67 in response to a change in the display area 62, for example, as shown in FIG.

[0075] The control unit 33 may change the display area 62 on the first image in response to, for example, a pinch-in operation and a pinch-out operation on the display device 40. For example, as shown in Fig. 14, the control unit 33 may cause the display area 62 to approximately coincide with the detection area 61. In such a case, all detection targets within the detection area 61 are displayed on the display device 40, as shown in Fig. 15, for example.

[0076] (5th example (reference example)) Fig. 16 shows a fifth example of a detection area 61 in a first image acquired by the image processing device 30 from the imaging device 20. In the example shown in Fig. 16, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a vehicle 64a that appears within a first predicted path 65 of the moving object 50, and a vehicle 64b and a pedestrian 63d that appear outside the first predicted path 65.

[0077] In the fifth example, a case will be described in which the external area of ​​the moving body 50 is dark, such as at night or inside a tunnel. When the external area of ​​the moving body 50 is dark, the characteristic values ​​of the first image and the second image may decrease. The characteristic values ​​may include any parameter related to the visibility of the images. For example, the characteristic values ​​may include at least one of the brightness value and the contrast ratio of the images. When the characteristic value of the second image decreases, the visibility of the second image may decrease.

[0078] The control unit 33 may execute specific image processing on a region corresponding to the detection target on the second image. The specific image processing may include a first process of superimposing a marker corresponding to the detection target on the region. Hereinafter, the marker is also referred to as a sixth marker. The sixth marker may include, for example, an image that substantially matches the contour shape of the detection target on the second image. According to this configuration, the sixth marker is displayed superimposed on the detection target on the second image. Therefore, even if the characteristic value of the second image is low, the subject 60 can easily recognize the detection target on the second image. The specific image processing may include a second process of changing the characteristic value of the region corresponding to the detection target on the second image. For example, the control unit 33 may change the characteristic value of the region so as to improve the visibility of the region on the second image. According to this configuration, the visibility of the detection target on the second image is improved. Therefore, even if the characteristic value of the second image is low, the subject 60 can easily recognize the detection target on the second image.

[0079] The control unit 33 may execute the above-mentioned specific image processing when one or more conditions are satisfied. The one or more conditions may include a condition that the detection target is located within the first predicted path 65 of the moving object 50. The one or more conditions may include a condition that the first predicted path 65 of the moving object 50 overlaps with the second predicted path of the detection target. The one or more conditions may include a condition that the distance between the moving object 50 and the detection target is less than a predetermined threshold. The one or more conditions may include a condition that a characteristic value of at least a part of an area of ​​the second image is less than a predetermined threshold.

[0080] In the fifth example, the control unit 33 may determine that one or more of the above conditions are satisfied for each of the vehicles 64a, 64b, and the pedestrian 63d. In this case, the control unit 33 may cause the display device 40 to display three sixth markers 75a, 75b, and 75c corresponding to the vehicles 64a, 64b, and the pedestrian 63d, respectively, superimposed on the second image, as shown in Fig. 17 for example. The control unit 33 may cause the display device 40 to display the sixth markers superimposed on the detection target in a bright place.

[0081] The control unit 33 may change the shape of the guide line 66. The control unit 33 may change the shape of the guide line 66 in the area where the guide line 66 and the detection target overlap. FIG. 18 shows one example of the shape of the guide line 66. The guide line 66 in FIG. 18 is not displayed in the area where the guide line 66 and the sixth marker 75a overlap. The example of the shape of the guide line 66 is not limited to being erased, and other design changes are allowed. Design changes include changing the color, changing the transparency, changing the type to a broken line, changing the line thickness, and blinking. The control unit 33 may change the shape of the guide line 66 when the sixth marker is not displayed. The control unit 33 may change the shape of the guide line 66 when the first marker 71 and the second marker 72 are displayed on the detection target.

[0082] As described above, according to the display system 10 according to one embodiment, various markers corresponding to the detection target detected on the first image are superimposed on the second image and displayed on the display device 40. The subject 60 who views the markers can recognize the relative positional relationship between the moving object 50 and the detection target at a glance. This improves the convenience of the technology for displaying an image of the external area of ​​the moving object 50.

[0083] Examples of various images that the image processing device 30 synthesizes with the guide wall image 80 and other images in the display area on the first video and causes the display device 40 to display them will be specifically described below with reference to FIGS.

[0084] (Example 6) FIG. 19 shows a sixth example of the detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. The imaging device 20 may be disposed so as to capture an image of the rear of the moving body 50, which is a vehicle. In the example shown in FIG. 19, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The detection area 61 is not limited to a shape longer in the left-right direction than in the up-down direction, and various shapes such as a square, a vertically elongated rectangle, and a circle are possible. The display area 62 can have various positions, sizes, and shapes. For example, the display area 62 is not limited to being located at the center of the detection area 61, and may be shifted to the left or right. The display area 62 is not limited to a part of the detection area 61, and the entire detection area 61 may be the display area 62. The control unit 33 may change the display area 62 on the first image in response to a pinch-in operation, a pinch-out operation, or the like on the display device 40.

[0085] The control unit 33 generates a second image by synthesizing a guide wall image 80 indicating the first predicted path 65 of the moving object 50 in a display area on the first image acquired from the imaging device 20. The control unit 33 may output an image in which the guide wall image 80 is superimposed on the second image and display it on the display device 40. The guide wall image 80 can indicate the predicted path of the moving object 50 when moving backward. The guide wall image 80 is a marker that gives the subject 60 a three-dimensional impression. The guide wall image 80 can be considered to be an image in which a virtual guide wall that guides the predicted path and is arranged in the real space reflected by the first image is projected onto the first image captured by the imaging device 20. The guide wall image 80 can be visually recognized as being composed of a plurality of virtual semi-transparent walls extending from the road surface to a predetermined height in the height direction within the field of view of the subject 60. The guide wall image 80 can also be said to be a three-dimensional display of the guide line displayed on the display device 40 when the vehicle moves backward, displayed in a wall-like manner. The wall-like display includes other types of display such as a surface, a film, a board, etc. The three-dimensional display of the guide wall image 80 makes it easier for the subject 60 to grasp the position of the parking space on the road surface, the distance to an obstacle, etc.

[0086] The control unit 33 may recognize a detection target within the display area 62 of the first image. The control unit 33 may detect a detection target that is inside the display area 62 on the first image and is reflected within the first predicted path 65. When a detection target is detected, the control unit 33 may synthesize a virtual plane as a first recognition wall image 83 on the moving object 50 side of the recognized detection target. When the detection target moves, the first recognition wall image 83 moves together with the detection target.

[0087] FIG. 20 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. 19. The guide wall image 80 is disposed away from the lower end of the second image to the upper side. The guide wall image 80 may include two side wall images 81a and 81b extending in the height direction and the depth direction. The side wall image 81a is located on the left side in the horizontal direction, and the side wall image 81b is located on the right side in the horizontal direction. The area between the two side wall images 81a and 81b shows the first predicted course 65 along which the moving body 50, which is a vehicle, passes. The interval between the side wall images 81a and 81b may be set to show the vehicle width of the moving body 50. In FIG. 20, the side wall images 81a and 81b are displayed linearly. The control unit 33 may acquire moving body information from the communication unit 31 and curve the side wall images 81a and 81b according to the predicted course. In this case, the moving body information includes the steering angle of the steering wheel.

[0088] The guide wall image 80 may include a plurality of distance wall images 82a, 82b, 82c extending in the height direction and the horizontal direction and indicating the distance in the depth direction from the moving body 50. The distance wall images 82a, 82b, 82c may connect between the side wall images 81a and 81b. Three distance wall images 82a, 82b, and 82c may be shown in FIG. 20. The number of distance wall images is not limited to three, and may be two or four or more. The distance between the distance wall images 82a and 82b in the depth direction may be narrower than the distance between the distance wall images 82b and 82c in the depth direction. The height length of the distance wall images 82a, 82b, 82c may be longer or shorter than the height length of the side wall images 81a and 81b.

[0089] A frame may be displayed on the outer periphery of each of the side wall images 81a, 81b and the distance wall images 82a, 82b, and 82c of the guide wall image 80. The guide wall image 80 may include at least one of the auxiliary lines 84a extending in the depth direction, the auxiliary lines 84b extending in the horizontal direction, and the auxiliary lines 84c extending in the height direction. The guide wall image 80 may include all of the auxiliary lines 84a extending in the depth direction, the auxiliary lines 84b extending in the horizontal direction, and the auxiliary lines 84c extending in the height direction. The auxiliary lines 84a extending in the depth direction may be displayed on the wall surfaces of the side wall images 81a and 81b. The auxiliary lines 84b extending in the horizontal direction may be displayed on the wall surfaces of the distance wall images 82a, 82b, and 82c. The auxiliary lines 84c extending in the height direction may be displayed on the wall surfaces of the side wall images 81a, 81b and the distance wall images 82a, 82b, and 82c. Each of the auxiliary lines 84a, 84b, and 84c can be displayed in any number of numbers equal to or greater than one. When the auxiliary lines 84a, 84b, and 84c are multiple, the intervals between the auxiliary lines 84a, 84b, and 84c can be equal. The intervals between the auxiliary lines 84a, 84b, and 84c may be different depending on conditions such as the height from the road surface and the distance from the moving body 50. The frames displayed on the periphery of each of the side wall images 81a and 81b and the distance wall images 82a, 82b, and 82c can be regarded as a part of the multiple auxiliary lines 84a, 84b, and 84c. In FIG. 20 and the following figures, even when multiple auxiliary lines 84a, 84b, and 84c exist, only one of each is appropriately labeled.

[0090] The distance wall image 82a is a first distance wall image that is closer to the moving object 50 than the other distance wall images 82b and 82c when projected onto the real space shown by the first video. The distance wall image 82a can be different in color from the other distance wall images 82b and 82c. For example, the distance wall images 82b and 82c can be displayed in semi-transparent white, and the distance wall image 82a can be displayed in semi-transparent red. The side wall images 81a and 81b can be displayed in semi-transparent white. The above color settings are merely examples, and the colors of the side wall images 81a and 81b may be different from those of the distance wall images 82b and 82c.

[0091] The transmittance of the sidewall images 81a, 81b may be changed according to the position in the depth direction. The transmittance of the sidewall images 81a, 81b may be set to be lower toward the front in the depth direction and higher toward the back. The sidewall images 81a, 81b may have different saturation or brightness according to the position in the depth direction. For example, the saturation can be made higher toward the front in the depth direction. The saturation of an image can also be rephrased as the color density of the image. Continuous changes in the transmittance, saturation, brightness, chromaticity, etc. can be said to have a gradation.

[0092] The transmittance of the distance wall images 82a, 82b, and 82c may be changed according to the height from the road surface. The transmittance of the distance wall images 82a, 82b, and 82c may be set to be higher as the distance from the road surface increases in the height direction. The saturation or brightness of the distance wall images 82a, 82b, and 82c may be different according to the height from the road surface. For example, the saturation can be lowered as the distance from the road surface increases in the height direction.

[0093] The auxiliary lines 84b, 84c on the distance wall image 82a, which is the first distance wall image, may be the same color as the auxiliary lines 84b, 84c on the other distance wall images 82b, 82c. The auxiliary lines 84a, 84b, 84c may be any color. For example, if the color of the walls of the guide wall image 80 other than the distance wall image 82a is white, the color of the auxiliary lines 84a, 84b, 84c may be white with a higher luminance or lower transmittance than these walls.

[0094] The control unit 33 may detect a vehicle 64 shown in the display area 62 of the first image as a detection target. In the example of FIG. 20, the vehicle 64 is located within the first predicted path 65. The detection target detected by the control unit 33 is not limited to a vehicle, and may be a building, a fence, an obstacle on the road, a person, an animal, or the like. When the detection target is detected, the control unit 33 may synthesize a first recognition wall image 83 on the side of the recognized moving body 50 of the detection target. When the detection target is detected, the control unit 33 may change the length of the guide wall image 80 in the height direction. When the detection target is detected, the control unit 33 can shorten the length of the guide wall image 80 in the height direction. Thereby, the control unit 33 can make the display easier to see even when the first recognition wall image 83 is displayed, and can direct the attention of the subject 60 to the detection target.

[0095] The first recognition wall image 83 of the detection target located in the first predicted course 65 extends in the height direction and the horizontal direction. The first recognition wall image 83 may be displayed as an opaque or semi-transparent surface of any color. In FIG. 20, the first recognition wall image 83 is displayed as a semi-transparent surface having a rectangular shape on the moving body 50 side of the vehicle 64. The horizontal width of the first recognition wall image 83 may correspond to the width of the vehicle 64. The horizontal width of the first recognition wall image 83 may be approximately equal to the width of the vehicle 64. The height of the first recognition wall image 83 may be set to a predetermined value. The transmittance of the first recognition wall image 83 may be changed according to the height direction from the road surface. The transmittance of the first recognition wall image 83 may be set so that it becomes higher as it moves away from the road surface in the height direction. In addition, the saturation or brightness of the first recognition wall image 83 may be different according to the distance in the height direction from the road surface. For example, the saturation can be lowered as it moves away from the road surface in the height direction. The first recognition wall image 83 may include auxiliary lines extending in the horizontal and vertical directions.

[0096] The control unit 33 may move the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in the height direction within the frame of the first recognition wall image 83. The auxiliary lines may be displayed so as to appear from the lower end of the frame of the first recognition wall image 83 and disappear at the upper end. The control unit 33 may display both or either of the relative position and the relative distance change between the moving body 50 and the vehicle 64 by moving the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in the height direction. For example, when the distance between the moving body 50 and the vehicle 64 is narrowing, the control unit 33 may move the auxiliary lines extending in the horizontal direction faster in the height direction. The closer the distance between the moving body 50 and the vehicle 64 is, the faster the control unit 33 may move the auxiliary lines in the height direction. The control unit 33 may move the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in the height direction regardless of the position of the detection target and the distance from the detection target.

[0097] The color of the first recognition wall image 83 may be different depending on the distance in the depth direction from the moving object 50 to the vehicle 64. For example, as shown in Fig. 20, when the distance between the moving object 50 and the vehicle 64 is greater than a predetermined distance, the control unit 33 may display the color of the first recognition wall image 83 in blue. When the moving object 50 moves backwards and the distance to the vehicle 64 becomes shorter, the control unit 33 may change the color of the first recognition wall image 83 from blue to yellow, and from yellow to red, for example.

[0098] When the distance between the moving body 50 and the vehicle 64 is equal to or greater than a predetermined distance, the control unit 33 may not display the first recognition wall image 83. When the moving body 50 and the vehicle 64 relatively approach each other and become within a predetermined distance, the control unit 33 may, for example, highlight the vehicle 64 in the second image and then display the first recognition wall image 83. Examples of highlighting include displaying the outline of the vehicle 64 in a thick line, blinking the outline of the vehicle 64, filling in the image of the vehicle 64, and the like. The control unit 33 may change the display mode according to a change in the relative position with the vehicle 64. For example, when the vehicle 64 moves relatively away from the moving body 50, the control unit 33 may not display the first recognition wall image 83.

[0099] 21 is a diagram showing an example of the second image in a state in which the distance between the moving body 50 and the vehicle 64 is narrowed due to the moving body 50 moving backward. The first recognition wall image 83 is displayed in red. This makes it possible to warn the subject 60 that there is a risk of colliding with the vehicle 64 if the subject 60 moves backward any further.

[0100] When the detection object and the guide wall image 80 overlap, the control unit 33 can change the display of the guide wall image 80. For example, as shown in FIG. 21, the control unit 33 can at least partially not display the guide wall image 80 overlapping the vehicle 64 of the second image. More specifically, the control unit 33 does not display the guide wall image 80 that overlaps with the vehicle 64 and is located farther from the moving body 50 in the depth direction than the vehicle 64 or the first recognition wall image 83 on the display device 40. In this way, the display device 40 can display an image that is easy for the subject 60 to see and intuitively understand. The method of changing the display of the guide wall image 80 is not limited to hiding the overlapping portion. For example, the control unit 33 can increase the transmittance of the guide wall image 80 in the portion overlapping with the detection object.

[0101] (Modification) The guide wall image 80 is not limited to the example shown in Figures 20 and 21. Figures 22 to 24 show modified examples of the guide wall image 80.

[0102] (First Modification) In the example illustrated in FIG. 22, the horizontally extending auxiliary lines 84b of the distance wall images 82a, 82b, and 82c may display only one of the lowest positions in the height direction. The distance wall image 82a displaying only the lowest position auxiliary line 84b may facilitate recognition of the ground level. The distance wall image 82a displaying only the lowest position auxiliary line 84b may facilitate understanding of the distance from the recognition wall image. An auxiliary line 84c extending in the height direction may be displayed at both ends of the horizontal direction of the distance wall images 82a, 82b, and 82c. An auxiliary line 84b located at the lower end and an auxiliary line 84c extending in the height direction located at both ends of the horizontal direction may be displayed on the periphery of the distance wall images 82a, 82b, and 82c. The distance wall images 82a, 82b, and 82c may be displayed by an auxiliary line 84b extending in the horizontal direction at the lower end of the periphery and an auxiliary line 84c extending in the height direction at both left and right ends.

[0103] The distance wall images 82a, 82b, and 82c may be displayed including a surface having a semi-transparent color. The distance wall images 82a, 82b, and 82c may have a higher transmittance or a lighter color density as they move away from the road surface in the height direction. The auxiliary lines 84b and 84c located on the outer periphery of the distance wall image 82a closest to the moving body 50 may be displayed in a color different from the other auxiliary lines. The different color may be, for example, red. The other auxiliary lines may be white, but is not limited to this. The first recognition wall image 83 displayed in front of the vehicle 64 may be displayed as a semi-transparent surface of any color, and may be displayed so that the transmittance increases or the color density decreases as it moves away from the road surface in the height direction. The first recognition wall image 83 may be displayed so that the boundary at the upper end is not displayed visibly, but gradually disappears in the height direction.

[0104] In the example shown in FIG. 22, the side wall images 81a and 81b of the guide wall image 80 extend in the depth direction. Only two auxiliary lines 84a are displayed at the top and bottom ends of the sidewall images 81a and 81b. The lower auxiliary line 84a1 can be displayed more highlighted than the upper auxiliary line 84a2. This includes displaying using thicker lines, lines with lower transmittance, lines with higher brightness, etc. The upper auxiliary line 84a2 and the lower auxiliary line 84a1 may be displayed in the same manner. Lines of the same aspect are lines that are similar in terms of, for example, thickness, transmittance, brightness, and the like.

[0105] The auxiliary line 84a extending in the depth direction may be displayed using a display method in which it gradually becomes thinner and disappears at the furthest position in the depth direction. The sidewall images 81a and 81b may be displayed so that the ends at the furthest position in the depth direction are visible.

[0106] In the sidewall images 81a and 81b, auxiliary lines 84c extending in the height direction may be displayed at least at the end on the nearest side and at the positions where the auxiliary lines 84c intersect with the distance wall images 82a, 82b, and 82c. These auxiliary lines 84c extending in the height direction may be displayed more emphasized than the auxiliary lines 84c extending in the other height directions. The auxiliary lines 84c extending in the height direction may each have a different thickness. The auxiliary lines 84c displayed at the positions where the auxiliary lines 84c intersect with the distance wall image 82a may be displayed thicker than the auxiliary lines 84c displayed at the positions where the auxiliary lines 84c intersect with the distance wall images 82b and 82c.

[0107] (Second Modification) The guide wall image 80 shown in FIG. 23 differs from the guide wall image 80 in FIG. 22 in the side wall images 81a and 81b. In FIG. 23, the auxiliary lines 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 may be displayed in two or more lines, including the top and bottom lines. The auxiliary lines 84a extending in the depth direction may be highlighted in the lowermost line in the height direction more than the other auxiliary lines. The multiple auxiliary lines 84a may all be displayed as lines of the same type. The guide wall image 80 in FIG. 23 may adopt the same changes in transmittance, color changes, color schemes, and the like that may be adopted in the guide wall image 80 in FIG. 22.

[0108] (Third Modification) The guide wall image 80 shown in FIG. 24 differs from the guide wall image 80 in FIG. 22 in side wall images 81a and 81b. In FIG. 24, the auxiliary lines 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 are displayed only at the lower end. The auxiliary lines 84a extending in the depth direction located at the lower end of the guide wall image 80, together with the auxiliary lines 84b extending in the horizontal direction, may correspond to lines on the road surface when projected onto the real space shown by the first video. The guide wall image 80 in FIG. 24 may adopt the changes in transmittance, color changes, color schemes, and the like that may be adopted in the guide wall image 80 in FIG. 22. The auxiliary lines 84c extending in the height direction may each have a different thickness. The auxiliary lines 84c aligned from the front side to the back side in the depth direction may be thinner as they are positioned further back.

[0109] In the guide wall image 80 shown in each of FIGS. 20 to 24, the display device 40 does not need to display some or all of the auxiliary lines 84a, 84b, and 84c.

[0110] (Example 7) Fig. 25 shows a seventh example of a detection area 61 in a first image acquired by the image processing device 30 from the imaging device 20. In the example shown in Fig. 25, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The display area 62 is not limited to a part of the detection area 61, and the entire detection area 61 may be the display area 62.

[0111] In the seventh example, the moving object 50 may be a vehicle that is moving backward toward a parking space in a parking lot. The imaging device 20 may be located behind the moving object 50 and capture an image of the rear. The control unit 33 may detect, from the detection area 61, a vehicle 64b and a pedestrian 63 that are outside the first predicted path 65 and at least partially shown in the display area 62, as detection targets. In the example shown in FIG. 25, the control unit 33 determines that the distance in the depth direction of the vehicle 64a shown in the first predicted path 65 of the moving object 50 is farther than a predetermined distance, and does not detect the vehicle 64a.

[0112] FIG. 26 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. 25. The control unit 33 recognizes the vehicle 64b and the pedestrian 63 as detection targets in the display area 62 of the first image, and displays the first recognition wall images 83a and 83b, which are virtual planes, on the moving object 50 side of the vehicle 64b and the pedestrian 63, respectively. The first recognition wall images 83a and 83b may be displayed as semi-transparent surfaces parallel to the side wall images 81a and 81b extending in the height direction and the depth direction. The control unit 33 may obtain a change in the distance of the relative positions between the vehicle 64b and the pedestrian 63 and the moving object 50, and may display the side wall images 81a and 81b only when the distance is changing in a direction to become shorter. Any algorithm may be adopted for estimating the distance between the moving object 50 and the detection target, as described above.

[0113] The control unit 33 may estimate the horizontal distance of the vehicle 64a and the pedestrian 63 to be detected from the guide wall image 80, and determine the display color of the first recognition wall images 83a and 83b according to the estimated distance. Here, the distance between the detection target and the guide wall image 80 is the distance between the detection target and the virtual guide wall displayed by the guide wall image 80 when the virtual guide wall exists in the real space displayed by the first video. The control unit 33 may estimate a first distance between the guide wall image 80 and the detection target from a position where the guide wall image 80 is mapped in the real space displayed by the first video. The control unit 33 may set a threshold value for the first distance, and display the first recognition wall images 83a and 83b on the display device 40 in different display colors such as blue, yellow, and red in order from the longest to the shortest horizontal distance. The first recognition wall images 83a and 83b may have auxiliary lines extending in the depth direction. The control unit 33 may move the auxiliary lines extending in the depth direction of the first recognition wall images 83a, 83b in the height direction within the frame of the first recognition wall image 83, in the same manner as the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in Figure 20.

[0114] The control unit 33 determines the positions of the vehicle 64b and the pedestrian 63 to be detected in the depth direction. The control unit 33 estimates the first position by detecting the vehicle 64b and the pedestrian 63. The display of the sidewall images 81a and 81b corresponding to the first position is at least partially changed. The display may be changed in various ways, including changing the color, brightness, saturation, etc. of the wall surface, changing the frame line, / or changes in the color, thickness, etc. of auxiliary lines, or combinations thereof. For example, in Figure 26 In the figure, a part 81a1 of the side wall image 81a facing the vehicle 64b and a part 81a2 of the side wall image 81a facing the pedestrian 63 are The display color of a part 81b1 of the sidewall image 81b is changed. The side wall images 81a and 81b are divided into areas by auxiliary lines 84c extending in the height direction. The display colors of a part 81a1 of the sidewall image 81a and a part 81b1 of the sidewall image 81b may be The control unit 33 detects that the detection target is the side wall images 81a, 8 When the obstacle is large compared to the length represented by the side wall image 81a or 81b, The display color may be changed. Large obstacles include large vehicles such as trailers and buildings. It can be done.

[0115] When the detection target moves, the control unit 33 estimates a second predicted path of the detection target, The first position may be estimated as a position where the second predicted path intersects with the side wall images 81a and 81b. The control unit 33 calculates the first position based on the second predicted path in the real space shown in the first image and the side wall image. It can be estimated as the intersection position of the mapping of 81a and 81b onto the real space. 33 at least partially changes the display of the sidewall images 81a, 81b in accordance with the first position. In this case, the display of the parts 81a1 and 81b of the side wall images 81a and 81b may be changed. The display color of the first distance may be different depending on the first distance. When the obstacle is large, the display color of the entire side wall image 81a or 81b may be changed.

[0116] As another display mode, the control unit 33 may change the display of the portions of the sidewall images 81a, 81b located on the far side in the depth direction from the first recognition wall images 83a, 83b. As yet another display mode, the control unit 33 may change the display of the portions of the sidewall images 81a, 81b located on the far side in the depth direction from the position where the second predicted path intersects with the sidewall images 81a, 81b.

[0117] The display of the first recognition wall images 83a and 83b may be changed according to the height from the road surface. As with the first recognition wall image 83 of the sixth example, the display of the first recognition wall images 83a and 83b may be set to different values ​​of transmittance, saturation, brightness, or the like according to the height from the road surface.

[0118] As described above, the display system 10 can spatially and three-dimensionally display the distance and / or position of a detection object that may be an obstacle to reverse movement using the guide wall image 80 and the first recognition wall images 83a, 83b. In addition, it can three-dimensionally display positions where there is a risk of collision based on the predicted path of the detection object while moving. In this way, the display system 10 can spatially and three-dimensionally display the detection object and the warning content by displaying the guide wall image 80 and the first recognition wall images 83a, 83b as walls (or planes) having a height direction. This makes it easier for the subject 60 to understand the display and the warning content.

[0119] (Example 8) FIG. 27 shows an eighth example of the detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. The imaging device 20 may be disposed so as to capture an image of the rear of the moving body 50, which is a vehicle. In the example shown in FIG. 27, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63 and a vehicle 64b that are reflected in a second area that is outside the display area 62 on the first image and is an area inside the detection area 61. In the eighth example, the control unit 33 determines that the distance in the depth direction of the vehicle 64a reflected in the display area 62 is farther than a predetermined distance, and does not detect the vehicle 64a.

[0120] FIG. 28 shows a moving object 50 as seen from above along the height direction in the real space shown in the first image. 13 is a schematic diagram showing the arrangement of a pedestrian 63 and a vehicle 64b to be detected. The device 20 can capture an image of a subject within the ranges included in the areas F1 and F2 in real space. The area F1 and the area F2 correspond to the detection area 61 of the first image. The first area corresponds to the display area 62 of the first image. The area F2 is inside the detection area 61 of the first image. and corresponds to a second region outside the display region 62.

[0121] 28, for the sake of explanation, a virtual guide wall 90 obtained by mapping the guide wall image 80 displayed in the second video onto real space is displayed by a dashed line. The virtual side walls 91a and 91b correspond to the side wall images 81a and 81b displayed in the second video. The virtual distance walls 92a, 92b, and 92c correspond to the distance wall images 82a, 82b, and 82c displayed in the second video.

[0122] For the sake of explanation, FIG. 28 also shows detection of the moving object 50 with respect to virtual side walls 91a and 91b. The imaginary lines 95l1-95l5 and 95r1-95r5, which indicate the horizontal distance of the object, This horizontal distance is also called the second distance. The imaginary lines 5r1 to 5r5 are located on the left side of the moving body 50. The intervals between the virtual lines 95l1 to 95l5 and the virtual lines 95r1 to 95r5 are The distance between the virtual lines 95l1 to 95l5 and the virtual lines 95r1 to 95r5 can be set arbitrarily. The control unit 33 determines which virtual line 95 the detection target is located on from the first image. The closest to l1~95l5 and 95r1~95r5, or the virtual line 95l1~9 Determine which two of 5l5 and 95r1 to 95r5 it is located in. It is possible.

[0123] The second image corresponding to the first image in FIG. 27 is illustrated in FIG. 29. In the eighth example, the guide wall image 80 may adopt a display method similar to that described in the sixth and seventh examples. When the detection target is located in the second region of the first image, the control unit 33 synthesizes the second recognition wall images 89a and 89b at the end of the second image on the side where the detection target exists. The second recognition wall images 89a and 89b may include wall portions 96a and 96b extending in the height direction and the depth direction, respectively, and floor portions 97a and 97b extending in the depth direction and the horizontal direction. In FIG. 29, the wall portions 96a and 96b extend from the lower end to the upper end of the left and right ends of the second image. The wall portions 96a and 96b may extend up and down along only a part of the left and right ends of the second image, rather than the entirety of the left and right ends. Similarly, the floor portions 97a and 97b may extend in the left and right direction along the entirety or a part of the lower end of the second image. The floor portions 97a and 97b may be displayed between the guide wall image 80 and the lower end of the second image.

[0124] The control unit 33 displays the second recognition wall images 89a and 89b according to the second distance of the detection target. The display may be changed in color, transmittance, saturation, brightness, size, etc., and in points. This includes dynamic changes in the display method, such as disappearance, movement, etc. The change in display may be indicated by a change in color. The display color may be red for high risk and red for low risk. The color may be yellow when the danger level is medium, and blue when the danger level is low. , which is located on the third imaginary line 95r3 on the horizontal right side from the imaginary side wall 91b of the moving body 50. In this case, the control unit 33 determines the danger level to be medium and changes the color of the second recognition wall image 89b on the right side to 28 is horizontally away from the imaginary side wall 91a of the moving body 50. In this case, the control unit 33 detects the danger by The degree of danger may be determined to be low, and the color of the second recognition wall image 89a on the left side may be set to blue. In one embodiment, the change in display is a vertical auxiliary line displayed on the walls 96a, 96b. The control unit 33 may include a lateral movement of the first and second distances 96a1, 96b1. The speed of movement of the walls 96a, 96b may be varied accordingly.

[0125] The second recognition wall images 89a and 89b may be semi-transparent. The wall portions 96a and 96b of the second recognition wall images 89a and 89b may have display characteristics that change in the vertical direction. For example, the wall portions 96a and 96b of the second recognition wall images 89a and 89b may have a lower transmittance toward the lower portion. Also, the second recognition wall images 89a and 89b may have a higher saturation toward the lower portion. The floor portions 97a and 97b of the second recognition wall image 89a may have display characteristics that change in the horizontal direction. For example, the floor portions 97a and 97b of the second recognition wall image 89a may have a lower transmittance toward the left and right ends and a higher transmittance toward the center of the second image. Also, the floor portions 97a and 97b of the second recognition wall image 89a may have a lower saturation from the left and right ends toward the center of the second image. The saturation of an image can also be expressed as the color density of the image.

[0126] The control unit 33 determines the type of the detection target, and displays an icon image 99a, 99b may be displayed on the floor portions 97a and 97b of the second recognition wall images 89a and 89b. The type of the object is determined by a known method based on the image of the object detected in the first video. For example, the control unit 33 may match the contour shape of the detection target with a model pattern. For example, the object type can be determined by detecting a pedestrian 63 and a vehicle 64. For b, icons representing a person and a vehicle can be selected. Virtual side walls 91a and 91b are obtained by mapping the wall images 81a and 81b onto the real space, and detection The second distance may be a horizontal distance to the target object. The virtual lines 95l1 to 95l5 and 95r1 to 95r5 are mapped onto the detection area 61 of the first image. The control unit 33 can determine the second distance from the relationship between each line and the position of the image of each detection target. The positions at which the icon images 99a and 99b are displayed may be changed depending on the situation.

[0127] In the display example of the second image shown in FIG. 29, virtual lines 95l1 to 95l5, 95 The first distance discrimination lines 98l1 to 98l5, 98r1 to 95r5, respectively The first distance discrimination lines 98l1 to 98l5 are provided on the left side of the side wall image 81a. From the center to the left, the order is 98l1, 98l2, 98l3, 98l4, and 98l5. 98r1 to 98r5 are located on the right side of the side wall image 81b, and are numbered from the center to the right. The first distance is numbered 98r1, 98r2, 98r3, 98r4, and 98r5. The identification lines 98l1 to 98l5 and 98r1 to 98r5 are used to display the icon images 99a and 99b. This indicates the position where the image is to be displayed, and does not have to be displayed in the second image.

[0128] In FIG. 28, the pedestrian 63 is at the third intersection on the horizontal right side from the imaginary side wall 91b of the moving object 50. Since the eye is located on the virtual line 95r3 of the eye, the control unit 33 detects the pedestrian The icon image 99b of 63 may be placed on the third first distance identification line 98r3 on the right side. As shown in FIG. 29, the icon image 99b is a first distance image extending in the depth direction on the second image. On the separation line 98r3, the surface is three-dimensionally represented as a surface having lengths in the depth direction and height direction. When viewed stereoscopically, the icon image 99b may be displayed in the side wall images 81a, 81b, and The icon image 99b may be displayed as a plane parallel to the second recognition wall images 89a and 89b. Similar to the second recognition wall image 89b, the display mode of the color, saturation, brightness, etc. is changed according to the second distance. For example, the icon image 99b may be displayed in yellow.

[0129] In FIG. 28, the vehicle 64b is located at the fifth position on the left side in the horizontal direction from the imaginary side wall 91a of the moving body 50. The virtual eye line 95l5 is located farther from the virtual side wall 91a. In the second image of FIG. 29, the icon image 99a of the vehicle 64b is displayed on the left side of the first distance discrimination Alternatively, the control unit 33 may arrange the second image on the line 98l5 when the second image is viewed stereoscopically. The icon image 9 may be arranged so that it appears flush with the wall portion 96a of the recognition wall image 89a. 9a, similar to the icon image 99b, can change the display mode according to the second distance. For example, the icon image 99a may be displayed in blue.

[0130] When the moving object 50 and / or the detection target move, When the relative position of the detection target as seen from the eye changes, the second recognition wall images 89a, 89b and the eye The positions and display modes of the images 99a and 99b change. When entering 62, the detection target may be displayed in the manner described in the seventh example. An example of a change in display when a vehicle 64b moves from an area F2 to an area F1 in space. This article explains:

[0131] First, in the real space of FIG. 28, the vehicle 64b is located to the left of the imaginary line 95l5. At this time, the second recognition wall image 89a is displayed in the second image of FIG. Also, an icon image 99a is displayed on the leftmost side of the floor portion 98a of the second recognition wall image 89a. At this time, the second recognition wall image 89a and the icon image 99a are displayed in blue. It can be shown.

[0132] In the real space of FIG. 28, as the vehicle 64b approaches the virtual guide wall 90, The second recognition wall image 89a displayed in the second image of 29 changes its display mode. For example, the color of the second recognition wall image 89a changes from blue to yellow to red in sequence. The display position of the cone image 99a is changed from above the first distance discrimination line 98l5 to above the first distance discrimination line 98l1. The color of the icon image 99a will change from blue to yellow to red. This allows the target person 60 to see that a vehicle 64b is approaching outside, which is not displayed in the second image. In addition, by using a three-dimensional display method, the user can easily see the object. The person 60 can spatially grasp the approach of the vehicle 64b, and the risk of overlooking the information is reduced. It becomes possible to reduce

[0133] The icon image 99a may move toward the center of the screen when the vehicle 64b approaches the moving object 50. The subject 60 may recognize the moving direction of the vehicle 64b through the movement of the icon image 99a.

[0134] Furthermore, when the vehicle 64b enters the area F1 from the area F2 in the real space of FIG. Since 4b is within the display target area of ​​the first image, it is included in the second image and displayed on the display device 40. At the same time, the second recognition wall image 89a disappears, and the vehicle 64b The first recognition wall image 83a is displayed on the guide wall image 80 side. The second recognition wall image 89a disappears. When the target person 60 is about to enter the image, the control unit 33 displays the second recognition wall image 89a multiple times to draw the target person 60's attention. The control unit 33 displays the first recognition wall image 83a on the moving object 50 side of the vehicle 64b. Further, the control unit 33 estimates a second predicted path of the vehicle 64b, and The control unit 33 estimates a first position as a position where the wall images 81a and 81b intersect. The control unit 33 may change the display of a part 81a1 of the sidewall image 81a including the position of the For example, the color of a part 81a1 of the sidewall image 81a changes from blue to yellow as the vehicle 64b approaches. and red in sequence. This allows the subject 60 to know that the vehicle 64b is and the second predicted path of the vehicle 64b is closer to the first predicted path 65 of the moving body 50. It becomes easy to understand that there is an overlap with

[0135] As described above, according to one of the embodiments of the present disclosure, the spatial understanding of the position of the detection target is facilitated by the three-dimensional display of the guide wall image, the first recognition wall image, the second recognition wall image, the icon image, etc. Furthermore, according to one of the embodiments of the present disclosure, information such as cautions and warnings can be provided in an easily understandable manner by changing the display mode of each wall surface. In addition, the use of three-dimensional display makes it possible to reduce the risk of overlooking information.

[0136] (Modification) FIG. 30 shows another example of the second image corresponding to the first image in FIG. 27. In FIG. 30, view display areas 101, 102 are added at the bottom left to show the selection status between "Normal View Display" and "Wide View Display". The "Normal View Display" and "Wide View Display" will be described later. In FIG. 30, the display manner of the guide wall image 80, the second recognition wall images 89a, 89b, and the icon images 99a, 99b may be partially different from that in FIG. 29. For example, in FIG. 30, the guide wall image 80 is displayed in the same manner as in FIG. 22.

[0137] The wall portions 96a and 96b of the second recognition wall images 89a and 89b are each formed of one or more vertically extending In FIG. 30, the walls 96a and 96b can have auxiliary lines 96a1 and 96b1. Although there are a plurality of auxiliary lines 96a1, 96b1, only one of them is labeled. The wall portions 96a and 96b of the second recognition wall images 89a and 89b are used to detect the distance to the detection target. The transmittance may be different from the front to the back in the depth direction. b: The transmittance gradually increases in the height direction, or the density gradually decreases in the height direction. It may disappear in place.

[0138] The control unit 33 controls the transmittance or color depth of the walls 96a and 96b, and the auxiliary line 96a1 , 96a2 indicates that an object is present within a predetermined distance range, and The distance or the change in distance to the detected object can be displayed. The change in the transmittance or color density of the light can be called a change in gradation. For example, The control unit 33 detects the detection object by the movement of the auxiliary lines 96a1 and 96a2 of the walls 96a and 96b. The control unit 33 may indicate that the wall portions 96a, 96b are within a predetermined distance. The color of the near side is the darkest, and the detection target is determined by the change in gradation in the depth direction. The auxiliary lines 96a1, 96b of the walls 96a, 96b can be used to indicate that the wall is within the distance. The control unit 33 may be provided with auxiliary walls 96a and 96b. The control unit 33 can prevent the lines 96a1 and 96b1 from being displayed. It is possible to have no change in transmittance (gradation).

[0139] In addition to the first distance identification lines 98l, 98r, one or more second distance identification lines 100l, 100r are provided on the floors 97a, 97b of the second recognition wall images 89a, 89b. In FIG. 30, there are a plurality of second distance identification lines 100l, 100r, but only one of each is indicated by a reference symbol. The first distance identification lines 98l, 98r and the second distance identification lines 100l, 100r do not have to be displayed on the display. The first distance identification lines 98l, 98r correspond to the horizontal distance from the moving object 50 to the detection target, whereas the second distance identification lines 100l, 100r can correspond to the distance in the depth direction. Each of the plurality of second distance identification lines 100l, 100r may correspond to the distance in the depth direction when the auxiliary lines 84c extending in the height direction on the sidewall images 81a, 81b are mapped onto the real space.

[0140] The subject 60 can grasp the horizontal and depth directions of the detection target according to the horizontal and depth directions of the icon images 99a and 99b on the floors 97a and 97b. As the icon images 99a and 99b, three-dimensional images according to the type of the detection target are used. The floors 97a and 97b may be disposed on the near side of the end of the sidewall images 81a and 81b on the near side and inside the line extending from the sidewall images 81a and 81b. The positional relationship between the second recognition wall images 89a and 89b and the guide wall image 80 is not limited to this and can be changed as appropriate.

[0141] In FIG. 30, the display device 40 shows view display areas 101 and 102 in the lower right portion, which indicate the selection status of "Normal View Display" and "Wide View Display". The "Normal View Display" and "Wide View Display" have different angles of view of the images displayed. In one of a plurality of embodiments, the "Normal View Display" corresponds to the angle of view of the display area 62 of the second image shown in FIG. 4. In one of a plurality of embodiments, the "Wide View Display" corresponds to the angle of view of the detection area 61 of the first image shown in FIG. 4. The "Wide View Display" does not need to be the image of the entire detection area 61 of the first image, but may be an image with an angle of view including the "second image" cut out from the detection area 61. The second images shown in FIG. 30 and those shown before are both normal view displays. Whether the image being displayed is "Normal View Display" or "Wide View Display" is indicated by changing the display of the view display areas 101 and 102 indicating the selection status. For example, changing the display of the area showing the selection status includes changing the color or darkness of the view display areas 101 and 102, changing the color of the characters, and the like.

[0142] The control unit 33 can switch between the "normal view display" and the "wide view display" according to the result of processing the image of the first video, or by receiving an input from the subject 60 via the display device 40 or the like. For example, the control unit 33 may switch the displayed video to the "wide view display" when it detects that a detection target is approaching in the "normal view display" state. The control unit 33 may switch the displayed video to the "normal view display" when it detects that a detection target is not present within a predetermined range in the "wide view display" state. When the display device 40 has a touch panel type screen, the control unit 33 may switch the display upon receiving a signal from the display device 40 indicating that the subject 60 has touched and selected a portion of the view display area 101 or 102 on the screen.

[0143] FIG. 31 is a diagram showing an example of an image switched from the "normal view display" shown in FIG. 30 to the "wide view display." The angle of view of the image in the "wide view display" in FIG. 31 is wider in the horizontal direction than the angle of view of the image in the "normal view display" in FIG. 30. In FIG. 31, a pedestrian 63 and a vehicle 64b located outside the display range of the "normal view display" shown in FIG. 30 are displayed. Also, since the horizontal angle of view of the image displayed in FIG. 31 is wider than that of the "normal view display" in FIG. 30, the distance between the left and right side wall images 81a and 81b of the guide wall image 80 is displayed relatively narrower.

[0144] In FIG. 31, a pedestrian 63 and a vehicle 64b enter the display range, and thus the second recognition wall The images 89a and 89b have disappeared. The control unit 33 detects the detection object in the detection area 61 of the first image. The vehicle 64b and the pedestrian 63 are detected, and the vehicle 64b and the pedestrian 63 are moved to the moving object 50 side. , first recognition wall images 83a and 83b, which are virtual planes, can be displayed. The control unit 33 determines the positions of the vehicle 64b and the pedestrian 63 to be detected in the depth direction. The control unit 33 estimates the positions of the vehicle 64b and the pedestrian 63 based on the estimated positions. The display of the side wall images 81a and 81b facing each other may be changed. 63 and vehicle 64b, a part 81b1 of the side wall image 81b and a part 81b2 of the side wall image The display of the part 81a1 of the image 81a is changed. The display change may be performed in various ways. For example, For example, display changes can include changing the color, thickness of the surrounding auxiliary lines, line style, and turning on and off blinking. The first recognition wall images 83a, 83b, and The side wall images 81a and 81b are displayed in a manner similar to that shown in FIG. and display modes may be adopted.

[0145] As shown in FIG. 30 and FIG. 31, according to one of the embodiments, the image captured by the imaging device 20 can be displayed on the display device 40 by switching between the "normal view display" and the "wide view display" having different angles of view. In the "normal view display", the subject 60 can easily grasp the presence and position of a detection target located outside the range of the displayed image by the second recognition wall images 89a, 89b and the icon images 99a, 99b. By automatically or manually switching the display device 40 from the "normal view display" to the "wide view display", the subject 60 can confirm the detection target detected outside the range of the image of the "normal view display" by the image. In the "wide view display", the display of the guide wall image 80 and the first recognition wall image 83b makes it possible to grasp the positional relationship with the detection target in a three-dimensional manner. In this way, the display system 10 in one of the embodiments can make it easy for the subject to grasp the surrounding situation and sense danger.

[0146] Although the present invention has been described based on the drawings and embodiments, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present invention. For example, the functions included in each means, each step, etc. can be rearranged so as not to cause logical inconsistency, and multiple means, steps, etc. can be combined into one or divided into two or more.

[0147] For example, the components and functions of the display system 10 according to the above-described embodiment may be rearranged. For example, a part or all of the configuration and functions of the image processing device 30 may be included in at least one of the imaging device 20 and the display device 40.

[0148] Some of the components of the display system 10 according to the above-described embodiment may be located outside the mobile object 50. For example, the image processing device 30 and the like may be realized as a communication device such as a mobile phone or an external server, and may be connected to other components of the display system 10 by wire or wirelessly. [Explanation of symbols]

[0149] 10 Display System 20 Imaging device 21 Imaging optical system 22 Image sensor 23 Communications Department 24 Control Unit 30 Image processing device 31 Communications Department 32 Storage section 33 Control Unit 40 Display device 50 Mobile 51 Network 60 Target 61 Detection Area 62 Display area 63, 63a, 63b, 63c, 63d Pedestrian 64, 64a, 64b Vehicles 65 First predicted course 66 Guide Line 67 Icon Images 68 3rd Marker 69 Contour 70 areas 71 First Marker 72 Second Marker 73 5th Marker 73a Icon Image 73b Obi image 73c Right edge area 74 Obstacle Images 75a, 75b, 75c 6th marker 80 Guide Wall Images 81a, 81b Sidewall images 82a, 82b, 82c distance wall images 83, 83a, 83b 1st recognition wall image 84a, 84b, 84c auxiliary line 89a, 89b 2nd recognition wall image 90 Virtual Guide Wall 91a, 91b Virtual side walls 92a, 92b, 92c Virtual Distance Wall 95l1~95l5, 95r1~95r5 Virtual lines 96a, 96b wall 97a, 97b floor section 98l1~98l5, 98r1~98r5 1st distance identification line 99a, 99b icon images 100l, 100r 2nd Distance Identification Line 101, 102 View display area

Claims

1. An imaging element that captures an image of the rear of the vehicle and generates an image; and a control unit that displays, on the image, a recognition image that extends a predetermined distance in a height direction relative to a road surface at a position in the image corresponding to a position between the vehicle and a detection target, and the transmittance of the recognition image increases as the distance in the height direction from the road surface increases. Imaging device.

2. 2. The imaging device according to claim 1, The control unit detects at least a part of the detection target on the image. Imaging device.

3. 3. The imaging device according to claim 1, the recognition image represents a surface; The surface of the recognition image is displayed on a portion of the detection target facing the vehicle. Imaging device.

4. 4. The imaging device according to claim 1, The recognition image shows a different color depending on the distance from the vehicle to the detection target, displays the detection target whose distance from the vehicle is within a predetermined range, and does not display the detection target whose distance from the vehicle is outside the predetermined range. Imaging device.

5. 5. The imaging device according to claim 1, The recognition image extends a predetermined distance in a height direction from the road surface. Imaging device.

6. A communication unit that acquires an image of the rear of the vehicle; and a control unit that displays, on the image, a recognition image that extends a predetermined distance in a height direction relative to a road surface at a position in the image corresponding to a position between the vehicle and a detection target, and the transmittance of the recognition image increases as the distance in the height direction from the road surface increases. Image processing device.

7. A communication unit that acquires an image of the rear of the vehicle; and a control unit that displays, on the image, a recognition image that extends a predetermined distance in a height direction relative to a road surface at a position in the image corresponding to a position between the vehicle and a detection target, and the transmittance of the recognition image increases as the distance in the height direction from the road surface increases. Display device.

8. An imaging element that captures an image of the rear of the vehicle and generates an image; and a control unit that displays, on the image, a recognition image that extends a predetermined distance in a height direction relative to a road surface at a position in the image corresponding to a position between the vehicle and a detection target, and the transmittance of the recognition image increases as the distance in the height direction from the road surface increases. Display system.

9. A vehicle equipped with a display system, The display system comprises: An imaging element that captures an image of the rear of the vehicle and generates an image; and a control unit that displays, on the image, a recognition image that extends a predetermined distance in a height direction relative to a road surface at a position in the image corresponding to a position between the vehicle and a detection target, and the transmittance of the recognition image increases as the distance in the height direction from the road surface increases. vehicle.

10. An image of an external area of ​​the moving object captured by an imaging device is acquired, A recognition image is displayed on the image at a position in the image corresponding to a position between the moving body and the detection target, the recognition image extending in a height direction relative to a road surface and the transmittance increasing as the distance in the height direction relative to the road surface increases. Image processing device.

Images