Display system

Abstract

To provide a display system that can ensure real-time visibility around a mobile body and of the bottom of the mobile body.SOLUTION: A display system 10 includes a mobile body 20 and a display device 30. The mobile body 20 includes a surrounding imaging unit 22 that takes an image of the surroundings of the mobile body 20 to obtain a mobile body surrounding image and an underside imaging unit 23 that takes an image of the underside of the vehicle body of the mobile body 20 to obtain a mobile body bottom image. The display device 30 includes a display unit 32. The mobile body 20 or the display device 32 includes a composite image generation unit 24B that generates a composite image by combining the mobile body surrounding image and the mobile body bottom image. The mobile body 20 and the display device 32 each include a communication unit that transmits and receives images.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a display system.

Background Art

[0002] <​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​A display system according to a first aspect of the present invention includes a mobile body including a perimeter imaging unit that photographs the area around the mobile body and acquires a perimeter image of the mobile body, and a lower side imaging unit that photographs the lower side of the mobile body and acquires a lower side image of the mobile body; a display device including a display unit; a composite image generation unit included in the mobile body or the display device, which generates a composite image by combining the perimeter image of the mobile body acquired by the perimeter imaging unit and the lower side image of the mobile body acquired by the lower side imaging unit; and a communication unit included in the mobile body and the display device, respectively, which transmits or receives image data.

[0007] According to the display system of the first aspect of the present invention, a composite image generation unit included in a moving body or display device generates a composite image by combining an image of the moving body's surroundings and an image of the moving body's bottom. Therefore, the composite image generated by the composite image generation unit includes an image of the moving body's surroundings acquired in real time by a surrounding imaging unit and an image of the moving body's bottom acquired in real time by a lower imaging unit, thus ensuring a real-time field of view of the moving body's surroundings and its bottom.

[0008] Furthermore, in the display system of the second embodiment of the present invention, the mobile body includes the composite image generation unit, the communication unit on the mobile body side includes a transmission unit that transmits the composite image generated by the composite image generation unit, and the communication unit on the display device side includes a reception unit that receives the composite image transmitted from the transmission unit.

[0009] According to the display system of the second aspect of the present invention, a composite image is generated in the moving object, and the generated composite image is transmitted to the display device by a transmitting unit. The display device receives the composite image by a receiving unit. Therefore, by combining the image of the moving object's surroundings and the image of the moving object's bottom surface in the composite image such that the data amount is smaller than the total data amount of the combined image of the moving object's surroundings and the image of the moving object's bottom surface, the amount of data transmitted can be reduced compared to when the images of the moving object's surroundings and bottom surface are transmitted and received as they are.

[0010] Furthermore, in the third embodiment of the present invention, the display system, in the display system of the first embodiment, includes a communication unit on the moving body side that transmits the moving body surrounding image acquired by the surrounding imaging unit and the moving body bottom image acquired by the bottom imaging unit, a communication unit on the display device side that receives the moving body surrounding image and the moving body bottom image transmitted from the transmission unit, and the display device includes the composite image generation unit.

[0011] According to a display system of the third aspect of the present invention, a moving object transmits an image of its surroundings and an image of its bottom, and a display device receives the transmitted images of its surroundings and bottom. The display device also combines the received images of its surroundings and bottom to generate a composite image. Therefore, since the mobile object does not perform the compositing process, the display device can receive the images of its surroundings and bottom faster. As a result, if the display device has higher processing power than the mobile object, for example, the images can be displayed on the display unit faster.

[0012] Furthermore, in the display system of the fourth aspect of the present invention, in the display system of the first or second aspect, when the surrounding imaging unit acquires a plurality of images of the surroundings of the moving object, the composite image generation unit overlaps and composites all adjacent images of the surroundings of the moving object and the bottom surface images of the moving object.

[0013] According to the display system of the fourth aspect of the present invention, the composite image is created by overlapping all adjacent images of the moving object's surroundings and its bottom surface, so that the subject is continuously captured even at the boundaries of adjacent images. Therefore, since the composite image contains image information in all directions around and below the moving object, the field of view around and below the moving object can be continuously secured without interruption.

[0014] Furthermore, in the display system of the fifth aspect of the present invention, in the display system of the first or second aspect, when the surrounding imaging unit acquires a plurality of images of the surroundings of the moving object and the lower imaging unit acquires a single image of the bottom surface of the moving object, the composite image generation unit overlaps the bottom surface image of the moving object with all of the surrounding images of the moving object and composites them.

[0015] According to the display system of the fifth aspect of the present invention, when there is only one image of the bottom surface of the moving object, the composite image is created by overlapping the bottom surface image of the moving object with all the images of the moving object's surroundings. As a result, the subject is continuously captured even at the horizontal boundary between the bottom surface of the moving object and the surroundings of the moving object. Therefore, since image information exists at the boundary between the surroundings of the moving object and the bottom surface of the moving object in the composite image, a continuous and uninterrupted field of view can be secured at the boundary between the surroundings of the moving object and the bottom surface of the moving object.

[0016] Furthermore, in the sixth aspect of the present invention, in the display system of the first or second aspect, when the surrounding imaging unit acquires a plurality of images of the surroundings of the moving object and the lower imaging unit acquires two images of the bottom surface of the moving object, the composite image generation unit composites the plurality of images of the surroundings of the moving object by overlapping each of them with at least one of the images of the bottom surface of the moving object.

[0017] According to the display system of the sixth aspect of the present invention, when there are two images of the bottom surface of the moving object, the composite image is created by overlapping multiple images of the surroundings of the moving object onto at least one of the images of the bottom surface of the moving object. As a result, the subject is continuously captured at the entire horizontal boundary between the bottom surface of the moving object and the surroundings of the moving object. Therefore, since image information exists at the entire horizontal boundary between the surroundings of the moving object and the bottom surface of the moving object in the composite image, a continuous and uninterrupted field of view can be secured at the boundary between the surroundings of the moving object and the bottom surface of the moving object.

[0018] Further, in the display system according to the seventh aspect of the present invention, in any of the display systems according to the first to sixth aspects, when the lower imaging unit acquires a plurality of the moving body bottom images, the composite image generation unit overlaps and synthesizes the adjacent moving body bottom images.

[0019] According to the display system of the seventh aspect of the present invention, when there are a plurality of moving body bottom images, the composite image is synthesized by overlapping adjacent moving body bottom images. Therefore, at the boundary of adjacent moving body bottom images on the bottom surface of the moving body, the subject is continuously reflected. Therefore, since image information exists in all directions of the bottom surface of the moving body in the composite image, the field of view of the bottom surface of the moving body can be continuously ensured without interruption.

[0020] Further, the display system according to the eighth aspect of the present invention includes, in any of the display systems according to the first to seventh aspects, a display image generation unit that generates a display image to be displayed on the display unit from the composite image received by the receiving unit.

[0021] According to the display system of the eighth aspect of the present invention, the display device includes a display image generation unit that generates a display image to be displayed on the display unit from the composite image received by the receiving unit. Therefore, on the display device side, the display image generation unit can generate a display image to be displayed on the display unit from the composite image, so that a display image in a desired viewpoint or display mode of the operator of the display device can be displayed on the display unit.

[0022] Further, in the display system according to the ninth aspect of the present invention, in any of the display systems according to the first to eighth aspects, a part of the moving body is included in the moving body surrounding image and the moving body bottom image.

[0023] According to the display system of the ninth aspect of the present invention, since a part of the moving body is included in the moving body surrounding image and the moving body bottom image, it is possible to prevent the occurrence of dead angles on the surrounding and the bottom surface of the moving body in the composite image.

Advantages of the Invention

[0024] As described above, according to the display system of the present invention, there is an excellent effect that the field of view around the moving body in real time and the field of view of the bottom surface of the moving body can be ensured.

Brief Description of the Drawings

[0025] [Figure 1] It is a block diagram showing the hardware configuration of the display system according to the first embodiment of the present invention. [Figure 2] It is a top view showing the position where the camera for photographing the surroundings of the vehicle is arranged. [Figure 3] It is a diagram showing an example of an image of the surroundings of the vehicle photographed by the camera. [Figure 4] It is a left side view showing the position where the camera for photographing the lower side of the vehicle is arranged. [Figure 5] It is an explanatory diagram for explaining the method of image synthesis. [Figure 6] It is an explanatory diagram for explaining the composite image in the equirectangular format. [Figure 7] It is an explanatory diagram for explaining the conversion of the format from equirectangular to spherical in the full sky image. [Figure 8] It is a diagram showing an example of the display image displayed on the display unit. [Figure 9] It is a diagram showing an example of the display image displayed on the display unit. [Figure 10] It is a diagram showing an example of the display image displayed on the display unit. [Figure 11] It is a diagram showing an example of the display image displayed on the display unit. [Figure 12] It is a diagram showing an example of the display image displayed on the display unit. [Figure 13] It is a diagram showing an example of the display image displayed on the display unit. [Figure 14] It is an explanatory diagram for explaining the method of synthesizing the image of the surroundings of the vehicle and two images of the bottom surface of the vehicle. [Figure 15] It is a block diagram showing the hardware configuration of the display system according to the second embodiment of the present invention. [Modes for carrying out the invention]

[0026] A display system 10 according to the first embodiment of the present invention will be described using Figures 1 to 13. In the following description, when the directions of front, rear, left, right, up, and down are indicated, they refer to the front, rear, left, right, up, and down directions of the vehicle.

[0027] Figure 1 is a block diagram showing the hardware configuration of a display system 10 according to the first embodiment of the present invention. As shown in Figure 1, the display system 10 of this embodiment includes a vehicle 20 as a mobile body and a display device 30 mounted on the vehicle 20. In this embodiment, the vehicle 20 is configured to be switchable between manual driving and automatic driving, and when an occupant manually drives the vehicle 20, known driving assistance functions and the like are provided to assist the occupant's driving operations.

[0028] As shown in Figure 1, the vehicle 20 includes a surrounding imaging unit 22, a downward camera 23 as a downward imaging unit, an image processing device 24, a vehicle control device 26, and a communication unit 28. In this embodiment, the surrounding imaging unit 22 includes, as an example, eight cameras: a front camera 22A, a right front camera 22B, a right side camera 22C, a right rear camera 22D, a rear camera 22E, a left rear camera 22F, a left side camera 22G, and a left front camera 22H.

[0029] These eight cameras 22A to 22H and the lower camera 23 each use, for example, a circular fisheye lens with a field of view larger than 180 degrees. The image captured by the circular fisheye lens is a fisheye image, characterized in that subjects near the center of the shooting range appear large, while subjects around the edges of the shooting range appear small, and the shooting range is represented by a circular image. In this embodiment, the "captured image" may be a still image or a video.

[0030] Figure 2 is a top view showing the positions of eight cameras 22A to 22H that capture images around the vehicle 20, and Figure 3 is a diagram showing an example of vehicle surrounding images P1 to P8 captured by the eight cameras 22A to 22H. As shown in Figures 2 and 3, the front camera 22A is mounted on the front of the vehicle 20, and the vehicle surrounding image P1 captured by the front camera 22A shows the front of the vehicle 20. The right front camera 22B is mounted on the right front of the vehicle 20, and the vehicle surrounding image P2 captured by the right front camera 22B shows the right front of the vehicle 20. The right side camera 22C is mounted on the mirror on the right side of the vehicle 20, and the vehicle surrounding image P3 captured by the right side camera 22C shows the right side of the vehicle 20. The right rear camera 22D is mounted on the right rear of the vehicle 20, and the vehicle surrounding image P4 captured by the right rear camera 22D shows the right rear of the vehicle 20.

[0031] The rear camera 22E is mounted on the rear of the vehicle 20, and the vehicle surrounding image P5 captured by the rear camera 22E shows the rear of the vehicle 20. The left rear camera 22F is mounted on the left rear of the vehicle 20, and the vehicle surrounding image P6 captured by the left rear camera 22F shows the left rear of the vehicle 20. The left side camera 22G is mounted on the left side mirror of the vehicle 20, and the vehicle surrounding image P7 captured by the left side camera 22G shows the left side of the vehicle 20. The left front camera 22H is mounted on the left front of the vehicle 20, and the vehicle surrounding image P8 captured by the left front camera 22H shows the left front of the vehicle 20.

[0032] In this embodiment, as an example, the eight cameras 22A to 22H of the surrounding imaging unit 22 are set to have shooting ranges such that overlapping areas are formed where the shooting areas of adjacent cameras overlap. That is, in the vehicle surrounding image P1, a subject common to the vehicle surrounding image P2 is captured at the right edge, and a subject common to the vehicle surrounding image P8 is captured at the left edge.

[0033] Similarly, in the vehicle surrounding images P2 to P8, the same subjects as the adjacent vehicle surrounding images are captured at the left and right edges of each image. In addition, a portion of the vehicle 20 is captured in the captured images P1 to P8. The surrounding imaging unit 22 configured as described above captures the entire horizontal surroundings (all directions) S of the vehicle 20.

[0034] Figure 4 is a left side view showing the position where the downward camera 23, which photographs the underside of the vehicle 20, is positioned. As shown in Figure 4, the downward camera 23 is mounted on the underside of the front of the vehicle 20, and the vehicle bottom image P9 (not shown) captured by the downward camera 23 shows the underside 20A of the vehicle body and the ground G (including roads, etc.) on which the vehicle 20 is located. In this embodiment, as an example, the lower ends of each of the vehicle surrounding images P1 to P8 show subjects that are common to the vehicle bottom image P9. In other words, the periphery of the vehicle bottom image P9 shows subjects that are common to any of the vehicle surrounding images P1 to P8.

[0035] As shown in Figure 1, the image processing unit 24 is mounted on the vehicle 20. The image processing unit 24, although not shown in the figure, consists of a CPU (Central Processing Unit: processor), ROM (Read Only Memory), RAM (Random Access Memory), and storage, and each component is connected to the others via a bus so that they can communicate with each other. The CPU reads a program from the ROM or storage and executes the program using the RAM as a working area.

[0036] Furthermore, the image processing device 24 includes an image acquisition unit 24A and a composite image generation unit 24B as its functional configuration. The image acquisition unit 24A and the composite image generation unit 24B are executed by the CPU reading a program from ROM or storage and executing the program using RAM as a working area.

[0037] The image acquisition unit 24A acquires image data representing eight vehicle surrounding images P1 to P8, each captured by the eight cameras 22A to 22H of the surrounding imaging unit 22, and a vehicle bottom image P9, each captured by the downward camera 23. In this embodiment, as an example, one captured image consists of approximately 2 million pixels. The number of pixels in the captured images can be appropriately changed depending on the performance of the camera used.

[0038] The composite image generation unit 24B generates a composite image CP by combining the eight vehicle surrounding images P1 to P8 and the vehicle bottom image P9, each represented by the image data acquired by the image acquisition unit 24A. In this embodiment, the composite image CP is a 360-degree spherical image representing all directions that can be seen from the vehicle 20, which is the shooting location, i.e., the entire surroundings (all directions) of the vehicle.

[0039] The composite image generation unit 24B has pre-stored information necessary to convert the eight vehicle surrounding images P1-P8, each captured by the eight cameras 22A-22H, and the vehicle bottom image P9, captured by the lower camera 23, into a 360-degree spherical image. This information includes positional information representing the physical arrangement of the circular fisheye lenses of the nine cameras 22A-22H and the lower camera 23; projection information representing the relationship between the image height of the vehicle surrounding images P1-P8 and the vehicle bottom image P9 and the incident angle of the circular fisheye lens; information indicating the shooting direction of the eight cameras 22A-22H and the lower camera 23; and information representing the characteristics of the circular fisheye lens. This information is derived in advance through calibration.

[0040] The composite image generation unit 24B generates a composite image CP by performing projection transformations on eight fisheye images of the vehicle's surroundings P1 to P8 and a vehicle bottom image P9 using known techniques based on the necessary information described above. In this embodiment, when composing the composite image CP, the aforementioned overlapping portions are overlapped.

[0041] In Figure 5, for example, taking the vehicle surrounding image P6 as an example, the overlapping portion between vehicle surrounding image P6 and vehicle surrounding image P5 is shown as the first region OV1, which is shaded, and the overlapping portion between vehicle surrounding image P6 and vehicle surrounding image P7 is shown as the second region OV2, which is shaded. The composite image generation unit 24B composites the vehicle surrounding images P1 to P8 by overlapping these overlapping portions on a pixel-by-pixel basis. Note that known techniques can be used for the overlapping method.

[0042] Furthermore, the vehicle bottom image P9 is composited with the composite vehicle surrounding images P1 to P8. In this embodiment, since the periphery of the vehicle bottom image P9 contains subjects common to each of the vehicle surrounding images P1 to P8, the area in which these common subjects are captured becomes the overlapping portion where adjacent shooting areas overlap. In Figure 5, for example, if we explain using the vehicle surrounding image P6 as an example, the overlapping portion between the vehicle surrounding image P6 and the vehicle bottom image P9 is shown as a third region OV3 indicated by diagonal lines. The composite image generation unit 24B composites the vehicle bottom image P9 below the vehicle surrounding images P1 to P8 by overlapping this overlapping portion on a pixel-by-pixel basis. Note that known techniques can be used for the overlapping method.

[0043] Figure 6 is an explanatory diagram for illustrating the equirectangular composite image. Here, the equirectangular composite image is an image with a vertical-to-horizontal aspect ratio of 1:2. The equirectangular format is a format in which a rectangular composite image with a vertical-to-horizontal aspect ratio of 1:2 is pasted onto a spherical screen, and when the composite image is viewed from the center of the screen, a 360-degree view is correctly displayed.

[0044] In this embodiment, the composite image generation unit 24B generates an equirectangular composite image CP, that is, a composite image CP with a vertical-to-horizontal ratio of 1:2. The equirectangular composite image CP is a planar 360-degree spherical image. In this embodiment, as an example, the composite image CP is composed of image data of 3840 × 1920 pixels, or approximately 7.4 million pixels. The number of pixels in the composite image CP can be appropriately changed according to the performance of the display unit 32 of the display device 30, which will be described later.

[0045] As shown in the right-hand diagram of Figure 6, a 360-degree spherical image is an image represented in a coordinate system that includes angular coordinates around a predetermined axis. With the radial movement set as a constant n (for example, 1), it is represented as an array of pixel values ​​whose coordinates are the vertical angle φ made with respect to the axis in the pitch direction and the horizontal angle λ corresponding to the rotation angle around the axis in the yaw direction. The vertical angle φ is in the range of -90 degrees to +90 degrees (or 0 degrees to 180 degrees), and the horizontal angle λ is in the range of -180 degrees to +180 degrees (or 0 degrees to 360 degrees).

[0046] The composite image generation unit 24B ensures that the horizontal direction of the composite image CP represents the entire horizontal circumference of the vehicle 20, i.e., 360 degrees, and therefore matches the horizontal direction of the composite vehicle surrounding images P1 to P8. Furthermore, the vehicle bottom image P9 is composited to the lower end of the composite vehicle surrounding images P1 to P8, and the lower end of the vehicle bottom image P9 matches the lower end of the composite image CP. In this embodiment, the area directly above the vehicle 20 is not reflected in the vehicle surrounding images P1 to P8. Therefore, the field of view above the vehicle surrounding images P1 to P8 is read from pre-stored information, and the coordinate position of the upper end of the composite vehicle surrounding images P1 to P8 is determined according to the read field of view.

[0047] Furthermore, the coordinates corresponding to the area directly above the vehicle 20 that is not captured by cameras 22A to 22H are represented by a uniform pixel value as a blank, as shown in Figure 6. In this embodiment, as an example, it is represented by a pixel value representing black, and this black area is designated as the blank image BP. The larger the field of view on the upper side of the vehicle surrounding images P1 to P8, the narrower the width of the blank image BP in the pitch direction (vertical direction). In this way, the composite image generation unit 24B generates the composite image CP.

[0048] Returning to Figure 1, the vehicle control device 26 is a device that controls the automatic driving of the vehicle 20 based on information detected by various sensors (not shown) mounted on the vehicle 20, and the vehicle surrounding images P1 to P8 taken by the surrounding imaging unit 22 and the vehicle bottom image P9 taken by the lower camera 23. The vehicle control device 26 can utilize known technologies.

[0049] Furthermore, the vehicle control device 26 is configured to allow setting various modes, such as a normal driving mode, an abnormal driving mode, and a recovery driving mode. Based on the information detected by the sensors described above, as well as the vehicle surrounding images P1 to P8 and the vehicle bottom image P9, the vehicle control device 26 automatically selects one of the above modes and drives the vehicle 20. The vehicle control device 26 selects the normal driving mode when the vehicle is operating normally, and selects the abnormal driving mode when, for example, an object deemed abnormal is detected by notification information from the automatic driving system or sensing by the sensors described above. Specifically, when the vehicle control device 26 selects the abnormal driving mode, the vehicle 20 stops.

[0050] Furthermore, the recovery driving mode is selected by the vehicle control device 26 when the vehicle 20 starts driving after coming to a stop. Specifically, the vehicle control device 26 either confirms that there is no abnormality in the vehicle 20 through the automatic driving system or sensing by the above-mentioned sensors, or transmits a signal to the occupants via the display device 30 to inform them that an abnormal condition is occurring.

[0051] The vehicle control device 26 transmits the driving status of the vehicle 20, that is, the currently selected mode, to the display device 30 via the communication unit 28.

[0052] The communication unit 28 is an interface for communicating with external devices and also functions as a transmitter that transmits the composite image CP generated by the composite image generation unit 24B. Furthermore, the communication unit 28 also functions as a transmitter that transmits the driving status of the vehicle 20, i.e., the currently selected mode.

[0053] Next, the display device 30 will be described. As shown in Figure 1, the display device 30 comprises a display unit 32, a display control device 34, a communication unit 36, and a recording unit 38. The display unit 32 is not shown in the figure, but as an example, it is installed in a position visible to the occupants inside the vehicle 20.

[0054] The display unit 32 displays a display image HP generated based on the composite image CP received from the vehicle 20. Alternatively, the composite image CP can be displayed directly.

[0055] In this embodiment, the display unit 32, for example, has the capability to display video corresponding to 4K resolution, but the performance of the display unit is not limited to this, and may have the capability to display video with a lower resolution than 4K or a higher resolution than 4K, or can be changed as appropriate.

[0056] The communication unit 36 ​​is an interface for communicating with an external device and also functions as a receiver for receiving a composite image CP transmitted from the communication unit 28 of the vehicle 20. Here, the communication unit 28 of the vehicle 20 and the communication unit 36 ​​of the display device 30 may communicate via a wired cable (not shown) wired inside the vehicle 20, or they may communicate with each other via a network. The network consists of a mobile line, which is a narrow-area wireless communication module such as Wi-Fi® or Bluetooth®, or a wide-area wireless communication module such as 4G, LTE, or 5G.

[0057] The recording unit 38 is a non-volatile memory such as flash memory, and records the composite image CP received by the communication unit 36. It also records various data such as the vehicle's driving status, vehicle speed, and steering angle transmitted from the vehicle's communication unit 28.

[0058] The display control device 34 is a device that displays images, information, etc., on the display unit 32. Although not shown in the diagram, it is composed of a CPU (Central Processing Unit: processor), ROM (Read Only Memory), RAM (Random Access Memory), and storage. Each component is connected to the others via a bus so that they can communicate with each other. The CPU reads a program from the ROM or storage and executes the program using the RAM as a working area.

[0059] Furthermore, the display control device 34 includes a composite image acquisition unit 34A and a display image generation unit 34B as its functional configuration. The composite image acquisition unit 34A and the display image generation unit 34B are executed by the CPU reading a program from ROM or storage and executing the program using RAM as a working area.

[0060] The composite image acquisition unit 34A acquires the composite image CP received by the communication unit 36 ​​and stores it in the recording unit 38.

[0061] The display image generation unit 34B generates a display image HP to be displayed on the display unit 32 of the display device 30. Specifically, the display image generation unit 34B generates the display image HP based on the composite image CP.

[0062] Figure 7 is an explanatory diagram illustrating the conversion from equirectangular to spherical format in a full-sphere image. The display image generation unit 34B takes point Pe(λ,φ) on the composite image CP shown in the left figure of Figure 7 and points Ps(x) on the unit sphere shown in the right figure of Figure 7. s ,y s ,z s Convert to ). Here, in the right diagram of Figure 7, λ represents longitude and φ represents latitude.

[0063] In this case, the conversion formula is shown by the following formulas (1) to (3). x s =cos(φ)sin(λ)···(1) y s =sin(φ) ···(2) z s =cos(φ)cos(λ)···(3)

[0064] The display image generation unit 34B converts a planar sphere image into a spherical sphere image using the above conversion formulas (1) to (3). Based on the composite image CP and the composite image RP representing the converted spherical sphere image, the display image generation unit 34B generates a display image HP in the display mode desired by the occupant viewing the display unit 32. Since the equirectangular planar composite image CP contains information for all directions viewed from the origin, it is also possible to generate a composite image of any shape other than a sphere, such as a cylinder viewed from above, based on this composite image CP. Furthermore, the display image generation unit 34B can also use the composite image CP as is as the display image HP.

[0065] Figures 8 to 13 show examples of display images HP displayed on the display unit 32 of the display device 30. In this embodiment, as an example, when the vehicle 20 is being driven automatically by the vehicle control device 26, the display control device 34 displays the display image HP on the display unit 32.

[0066] As shown in Figure 8, the display image generation unit 34B generates a display image HP1, which is a 360-degree overhead view image generated from a viewpoint looking down on the vehicle 20 in the composite image RP representing a spherical panoramic image. In Figure 8, for convenience, the image GP representing the underside of the vehicle 20 is represented by a roughly rectangular area composed of the same color, but in reality, the actual image representing the underside of the vehicle 20 is displayed. The display image generation unit 34B may also superimpose an image A1 (see Figure 9) that simulates the vehicle 20 onto the display image HP1. In this case, in order to make the area where image A1 is superimposed on the display image HP1, i.e., image GP, visible, image A1 is made transparent. The generated display image HP1 is displayed on the display unit 32 by the display control device 34.

[0067] Furthermore, as shown in Figure 9, the display control device 34 may superimpose and combine an image A2 onto the display image HP1, which includes the image A1 displayed on the display unit 32, by superimposing an image A2 that can be moved in the clockwise rotation direction indicated by arrow M1 or the counterclockwise rotation direction indicated by arrow M2. Image A2 can have a shape formed by connecting an arc on the circumference of the display image HP1 with an inner arc formed with a smaller radius than the said arc at both ends. The area on the display image HP1 over which this image A2 overlaps is considered to be the area in the direction that the occupant wants to see.

[0068] In this embodiment, the occupant can change the direction they want to view by moving the image A2 in the clockwise or counterclockwise rotation direction. The display image generation unit 34B generates a display image HP2 that is cropped from the direction the occupant wants to view, that is, from the occupant's desired free viewpoint, as shown in Figure 10. The display image HP2 shown in Figure 10 is an image of the view from the front of the vehicle 20. The generated display image HP2 is displayed on the display unit 32 by the display control device 34.

[0069] Furthermore, for example, if the composite image generation unit 24B acquires information that an occupant wants to view the vehicle 20 from the rear diagonally above by operating a touch panel (not shown), the composite image generation unit 24B generates a display image HP4, which is an overhead view of the vehicle 20 from the rear diagonally above, based on the composite image RP, as shown in Figure 11. The display image generation unit 34B superimposes an image A3 representing the vehicle 20 onto the display image HP4. This image A3 can be an image that simulates the vehicle 20, similar to the display image HP1 shown in Figure 9.

[0070] For example, suppose two people are walking to the left of a vehicle 50 (see Figure 12) traveling in front of vehicle 20. If these two people are detected as abnormal by notification information from the automatic driving system in the vehicle control device 26 on the vehicle 20 side, or by sensing by sensors (not shown) mounted on the vehicle 20, the communication unit 28 on the vehicle 20 side transmits this information, namely the location and size information of the two people walking, to the display device 30, which is then received by the communication unit 36 ​​on the display device 30.

[0071] The display image generation unit 34B, based on the information received by the communication unit 36 ​​of the display device 30, identifies the area in which the two people are captured from the composite image CP using known techniques, and generates a display image HP4 cropped to the angle of view in which the two people 52 are captured, as shown in Figure 12. The display image HP4 shows the view from the front of the vehicle 20. Based on the information received by the communication unit 36, the display image generation unit 34B superimposes an image A4, which shows the direction in which the two people are located as seen from the vehicle 20, onto the display image HP4. The display image HP4, including the generated image A4, is displayed on the display unit 32 by the display control device 34.

[0072] Furthermore, when the display image generation unit 34B identifies the area in which the two people are captured from the composite image CP using known techniques, as shown in Figure 13, it superimposes and composites image A5, which is a 360-degree overhead view image generated from a viewpoint looking down on the vehicle 20 in the composite image RP representing a spherical panoramic image, onto the display image HP5, which is an overhead view image. In this embodiment, image A5 is configured to be emphasized compared to other image regions.

[0073] Note that, as with the display image HP1 shown in Figure 9, the display image HP5 shown in Figure 13 shows the underside of the vehicle 20 in black, but in reality, it displays the actual image of the underside of the vehicle 20. Alternatively, an image A1 simulating the vehicle 20 may be superimposed and combined with the display image HP1. Also, in the display image HP5, the 360-degree overhead view image around image A1 is shown in white, but in reality, it is configured the same as the display image HP1 shown in Figure 8. The display image HP5, including the generated images A1 and A5, is displayed on the display unit 32 by the display control device 34.

[0074] Furthermore, in this embodiment, the display image generation unit 34B generates a display image HP according to the driving status of the vehicle 20 transmitted from the vehicle control device 26, that is, the currently selected mode. Specifically, when the vehicle 20 is in normal driving mode, the display image generation unit 34B generates the display image HP1 shown in Figure 9, and the display control device 34 displays the generated display image HP1 on the display unit 32. Subsequently, as described above, the display image generation unit 34B generates the display image HP2 shown in Figure 10 or the display image HP3 shown in Figure 11 in response to the movement operation of image A2 by the occupant, and the display control device 34 displays the generated display image HP on the display unit 32.

[0075] Furthermore, if the vehicle 20 is in abnormal driving mode, the display image generation unit 34B generates a display image HP4 including the generated image A4 shown in Figure 12, or a display image HP5 including the generated images A1 and A5 shown in Figure 13, based on the notification information of the automatic driving system in the vehicle control device 26 on the vehicle 20 side and the information detected by sensing by sensors (not shown) mounted on the vehicle 20, as described above, and the display control device 34 displays the generated display image on the display unit 32.

[0076] Furthermore, when the vehicle 20 is in the return-to-driving mode, the display image generation unit 34B generates a display image HP1 shown in Figure 9, in which the underside of the vehicle body of the vehicle 20 is visible, and the display control device 34 displays the generated display image HP1 on the display unit 32. Subsequently, as described above, the display image generation unit 34B generates a display image HP2 shown in Figure 10 or a display image HP3 shown in Figure 11 in response to the movement operation of image A2 by the occupant, and the display control device 34 displays the generated display image HP on the display unit 32.

[0077] Next, the operation and effects of this embodiment will be described.

[0078] According to the display system 10 of this embodiment, the composite image generation unit 24B included in the vehicle 20 generates a composite image CP by combining the vehicle surrounding images P1 to P8 and the vehicle bottom image P9. Therefore, the composite image CP received by the communication unit 36, which functions as the receiving unit of the display device 30, includes the vehicle surrounding images P1 to P8 acquired in real time by the surrounding imaging unit 22 and the vehicle bottom image P9 acquired in real time by the lower camera 23, thus ensuring a real-time view of the area around the vehicle and the area under the vehicle.

[0079] Furthermore, according to the display system 10 of this embodiment, a composite image CP is generated in the vehicle 20, and the generated composite image CP is transmitted to the display device 30 by a communication unit 28 that functions as a transmitter. The display device 30 receives the composite image CP by a communication unit 36 ​​that functions as a receiver. Therefore, by combining the vehicle surrounding images P1 to P8 and the vehicle bottom image P9 in the composite image CP such that the data amount is smaller than the total data amount of the combined vehicle surrounding images P1 to P8 and the vehicle bottom image P9, the amount of data transmitted can be reduced compared to when the vehicle surrounding images P1 to P8 and the vehicle bottom image P9 are transmitted and received as they are.

[0080] Furthermore, according to the display system 10 of this embodiment, the composite image CP is created by overlapping all adjacent vehicle surrounding images P1 to P8 and vehicle bottom image P9, so that the subject is continuously captured even at the boundaries of adjacent images. Therefore, since the composite image CP contains image information in all directions around the vehicle and on the underside of the vehicle, it is possible to ensure a continuous and uninterrupted view of the vehicle's surroundings and the underside of the vehicle.

[0081] Furthermore, according to the display system 10 of this embodiment, if there is only one vehicle bottom image P9, the composite image CP is created by overlapping the vehicle bottom image P9 (vehicle bottom image P9) with all of the vehicle surrounding images P1 to P8 (vehicle surrounding images P1 to P8), so that the subject is continuously captured even at the horizontal boundary between the vehicle bottom and the vehicle surroundings. Therefore, since image information exists at the boundary between the vehicle surroundings and the vehicle bottom in the composite image CP, it is possible to ensure a continuous view without interruption at the boundary between the vehicle surroundings and the vehicle bottom.

[0082] Furthermore, according to the display system 10 of this embodiment, the display device 30 includes a display unit 32 and a display image generation unit 34B that generates a display image HP to be displayed on the display unit 32 from a composite image CP received by a communication unit 36 ​​which functions as a receiving unit. Therefore, on the display device 30 side, the display image generation unit 34B can generate the display image HP to be displayed on the display unit 32 from the composite image CP, so that the display image HP can be displayed on the display unit 32 from the viewpoint and display mode desired by the crew member who is operating the display device 30.

[0083] Furthermore, according to the display system 10 of this embodiment, since the vehicle surrounding images P1 to P8 and the vehicle bottom image P9 include a portion of the vehicle 20, it is possible to prevent the occurrence of blind spots around and on the bottom of the vehicle 20 in the composite image CP.

[0084] Furthermore, according to the display system 10 of this embodiment, since the display device 30 is installed on the vehicle 20, the vehicle 20 can secure a real-time view of the area around the vehicle and the underside of the vehicle.

[0085] Furthermore, according to the display system 10 of this embodiment, since the composite image CP is a 360-degree spherical image, the composite image CP captures 360 degrees of the vehicle 20 in both the horizontal and vertical directions, that is, all directions centered on the vehicle 20. In other words, the shooting direction of each camera is associated with the composite image CP. Therefore, when the occupant monitors the vehicle 20 on the display unit 32, it is possible to prevent blind spots from occurring around the vehicle 20.

[0086] Furthermore, in recent years, in autonomous driving systems still under development, if any abnormality occurs while the vehicle 20 is performing autonomous driving, it is necessary to temporarily stop the vehicle 20. According to the display system 10 of this embodiment, the vehicle 20 includes a lower camera 23 whose shooting range is set so that the lower part of the vehicle body is the shooting area, and the composite image generation unit 24B also combines the vehicle bottom image P9 acquired by the lower camera 23 into the composite image CP. Therefore, the monitor can remotely check the area around the vehicle 20, including the underside of the vehicle body, based on the composite image CP, and safely resume (restart) the temporarily stopped vehicle 20.

[0087] Furthermore, according to the display system 10 of this embodiment, the display image generation unit 34B generates a display image HP that matches the mode set in the vehicle 20, so that the display image HP corresponding to the status of the vehicle 20 can be displayed on the display unit 32.

[0088] Furthermore, when the vehicle 20 is in, for example, normal driving mode or remote control mode, a 360-degree overhead view image generated from a viewpoint looking down on the vehicle 20 is displayed in the composite image RP, which represents a spherical panoramic image, as shown in display image HP1 in Figure 9. When the occupant moves image A2, the display unit 32 displays an image from the viewpoint corresponding to this movement operation, such as display image HP2 shown in Figure 10 or display image HP3 shown in Figure 11. This allows the occupant to view an image from the desired viewpoint.

[0089] Furthermore, when the vehicle 20 enters, for example, an abnormal driving mode, if the object of the abnormality is two people 52, as shown in display images HP4 and HP5 in Figure 12 or Figure 13, the direction in which these two people 52 are located relative to the vehicle 20 is presented by image A4 with an arrow and image A5 with an "X" mark. This allows the occupants to visually confirm in which direction the abnormality is occurring relative to the vehicle 20.

[0090] Furthermore, when vehicle 20 is in, for example, recovery driving mode, an image is displayed that allows the underside of vehicle 20 to be seen, as shown in display image HP1 in Figure 9. Therefore, when recovering (restarting) vehicle 20 that has been stopped in abnormal driving mode, the occupants can also check the underside of vehicle 20, allowing them to safely restart vehicle 20.

[0091] Furthermore, according to the display system 10 of this embodiment, since the ratio of the vertical to horizontal directions of the composite image CP is 1:2, the composite image CP can be constructed in equirectangular form, that is, using equirectangular projection.

[0092] (modified version) Figure 14 is an explanatory diagram illustrating the method of combining vehicle surrounding images P1-P8 and two vehicle bottom images P9 and P10. In the modified example, downward cameras 23 are installed at the front and rear of the vehicle 20, and vehicle bottom images P9 and P10 are acquired by these two downward cameras 23, respectively. The two acquired vehicle bottom images P9 and P10 both contain a common subject.

[0093] In this case, the area in which a common subject is captured in the two vehicle bottom images P9 and P10 becomes the overlapping area where adjacent shooting areas overlap. As shown in Figure 14, the overlapping area of ​​the two vehicle bottom images P9 and P10 is indicated by the fourth region OV4, which is shown by the diagonal lines. That is, vehicle bottom images P10 are located on both the left and right sides of vehicle bottom image P9, and each has an overlapping area. The composite image generation unit 24B combines the two vehicle bottom images P9 and P10 by overlapping these overlapping areas on a pixel-by-pixel basis. Known techniques can be used for the overlapping method. Similarly, if three or more vehicle bottom images are acquired, the three or more vehicle bottom images are combined by overlapping the overlapping areas of adjacent vehicle bottom images on a pixel-by-pixel basis.

[0094] Furthermore, the two vehicle bottom images P9 and P10, which are combined with the combined vehicle surrounding images P1 to P8, are further combined to generate a single combined image CP. In this process, the combined vehicle surrounding images P1 to P8 are combined with at least one of the vehicle bottom images P9 and P10 overlapping. Note that in the region including the fourth region OV4 shown in Figure 14, the combined vehicle surrounding images P1 to P8 are combined with both vehicle bottom images P9 and P10 overlapping.

[0095] In the modified example, if there are two vehicle bottom images P9, the composite image CP is created by overlapping multiple vehicle surrounding images P1 to P8 onto at least one of the vehicle bottom images P9 and P10. As a result, the subject is continuously captured at the entire horizontal boundary between the vehicle bottom and the vehicle surroundings. Therefore, since the composite image CP contains image information at the entire horizontal boundary between the vehicle surroundings and the vehicle bottom, a continuous and uninterrupted view at the boundary between the vehicle surroundings and the vehicle bottom can be ensured.

[0096] Furthermore, in the modified example, if there are multiple vehicle bottom images P9, the composite image CP is created by overlapping adjacent vehicle bottom images. Therefore, the subject is continuously captured at the boundaries between adjacent vehicle bottom images on the vehicle bottom. As a result, the composite image CP contains image information in all directions of the vehicle bottom, allowing for continuous viewing of the vehicle bottom image without interruption.

[0097] Next, a display system 10-2 according to the second embodiment of the present invention will be described using Figure 15. Figure 15 is a block diagram showing the hardware configuration of the display system 10-2 according to the second embodiment of the present invention. In this embodiment, parts that are the same as those in the first embodiment shown in Figure 1 are indicated by the same reference numerals, and detailed explanations are omitted.

[0098] As shown in Figure 15, in the display system 10-2 of this embodiment, the image processing device 24-2 of the vehicle 20-2 does not have a composite image generation unit 24B, while the display control device 34-2 of the display device 30-2 has a composite image generation unit 34D. That is, the generation of the composite image CP is performed on the display device 30-2 side, not on the vehicle 20-2 side. Therefore, the communication unit 28-2 of the vehicle 20-2 functions as a transmitting unit that transmits the vehicle surrounding images P1 to P8 and the vehicle bottom image P9. Also, the communication unit 36-2 of the display device 30-2 functions as a receiving unit that receives the vehicle surrounding images P1 to P8 and the vehicle bottom image P9.

[0099] The display control device 34-2 has an image acquisition unit 34C instead of the composite image acquisition unit 34A in the above embodiment. The image acquisition unit 34C acquires the vehicle surrounding images P1 to P8 and the vehicle bottom image P9 received by the communication unit 36-2 and stores them in the recording unit 38.

[0100] Next, the operation and effects of this embodiment will be described.

[0101] According to the display system 10-2 of this embodiment, the composite image generation unit 34D included in the display device 30-2 generates a composite image CP by combining the vehicle surrounding images P1 to P8 and the vehicle bottom image P9. The display image generation unit 34B generates a display image HP based on the composite image CP generated by the composite image generation unit 34D. Therefore, the display image HP displayed on the display unit 32 includes the vehicle surrounding images P1 to P8 acquired in real time by the surrounding imaging unit 22 and the vehicle bottom image P9 acquired in real time by the lower camera 23, thus ensuring a real-time view of the area around the vehicle and the area under the vehicle.

[0102] Furthermore, according to the display system 10-2 of this embodiment, the vehicle 20-2 transmits vehicle surrounding images P1-P8 and vehicle bottom image P9, and the display device 30-2 receives the transmitted vehicle surrounding images P1-P8 and vehicle bottom image P9. The display device 30-2 also combines the received vehicle surrounding images P1-P8 and vehicle bottom image P9 to generate a combined image CP. Therefore, since the vehicle 20-2 does not perform the combining process, the display device 30-2 can receive the vehicle surrounding images P1-P8 and vehicle bottom image P9 faster. As a result, for example, if the display device 30-2 has higher processing power than the vehicle 20-2, the display image HP can be displayed on the display unit 32 faster.

[0103] [Supplementary explanation of the embodiment] In the above embodiment, the cameras 22A to 22H and the downward camera 23 constituting the surrounding imaging unit 22 each use a circular fisheye lens having a field of view greater than 180 degrees, but the present invention is not limited thereto. As long as the imaging range can be set so that an overlapping portion is formed where adjacent imaging areas overlap, a circular fisheye lens with a field of view of, for example, 140 degrees may be used, and the field of view is not limited. In addition, instead of a circular fisheye lens, a diagonal fisheye lens, other wide-angle lenses, ultra-wide-angle lenses, etc. may be used.

[0104] Furthermore, in the above embodiment, one downward camera 23 is provided on the front lower side of the vehicle 20, but the present invention is not limited thereto. For example, a downward camera 23 may also be provided on the rear lower side of the vehicle 20. In other words, there may be two or more cameras for the lower side, and this can be appropriately changed according to the size of the vehicle body of the vehicle 20.

[0105] Furthermore, in the above embodiment, the horizontal cameras 22A to 22H of the vehicle 20 constituting the surrounding imaging unit 22 were set to eight, but the present invention is not limited thereto. As long as the imaging range can be set so that overlapping portions are formed where adjacent imaging areas overlap, there may be four cameras: a front camera 22A, a right-side camera 22C, a rear camera 22E, and a left-side camera 22G, and this can be appropriately changed according to the field of view of each camera. In addition, the surrounding imaging unit 22 may be composed of a 360-degree camera that captures the entire surroundings of the vehicle.

[0106] Furthermore, in the above embodiment, the vehicle bottom image P9 captured by the lower camera 23 was assumed to contain subjects common to each of the vehicle surrounding images P1 to P8 at its periphery, but the present invention is not limited to this. For example, if only one camera for capturing the bottom surface 20A of the vehicle body is provided on the front lower side of the vehicle 20, the vehicle bottom image P9 may contain only subjects common to the vehicle surrounding image P5 captured by the rear camera 22E. In other words, it is sufficient that the image contains subjects common to one or more of the captured images P1 to P8, which capture the horizontal direction of the vehicle 20.

[0107] If the vehicle bottom image P9 contains only subjects that are common to both the vehicle surrounding image P5 and the vehicle underside image P9, then in the generated composite image CP, the vehicle bottom image P9 corresponding to the vehicle bottom image P9 captured by the lower camera 23 is composited at the lower edge of the vehicle surrounding image P5 corresponding to the vehicle surrounding image P5, and the right side of the vehicle bottom image P9 becomes a blank image BP.

[0108] Furthermore, in the above embodiment, the composite image CP was a planar 360-degree spherical image, but the present invention is not limited thereto. The composite image CP may be a full-circumference image, a panoramic image with the top missing, or a panoramic image with only the bottom missing, that is, an image obtained by compositing a vehicle bottom image P9 with a 360-degree panoramic image taken only in the horizontal direction of the vehicle.

[0109] Furthermore, although the equirectangular projection was used as the full-sphere image in the above embodiment, the present invention is not limited to this. For example, a cubemap format could be used in which planar images are pasted onto the six faces of a cube-shaped screen, as if a die had been disassembled, so that 360 degrees can be seen correctly when the image is viewed from the center. Alternatively, a dome master format, such as those used in planetariums, could be used. Known technologies can be used for the format.

[0110] Furthermore, in the above embodiment, the display image generation unit 34B performs a process to convert a planar 360-degree image into a spherical 360-degree image, but the present invention is not limited thereto. For example, the composite image generation unit 24B may perform a process to convert a planar 360-degree image into a spherical 360-degree image, and transmit the composite image RP, represented by the spherical 360-degree image, to the display device 30.

[0111] Furthermore, in the above embodiment, the composite image generation unit 24B performs a projection conversion process from the vehicle surrounding images P1 to P8 and the vehicle bottom image P9 to a composite image CP, but the present invention is not limited thereto. For example, the composite image generation unit 24B may directly generate an overhead image from the vehicle surrounding images P1 to P8 and the vehicle bottom image P9, and use this overhead image as the composite image.

[0112] Furthermore, although the above embodiment includes one display unit 32, the present invention is not limited to this and may include multiple display units 32.

[0113] Furthermore, in the above embodiment, as shown in Figure 9, the direction the occupant wants to view is changed by moving image A2 in a clockwise or counterclockwise rotation direction, but the present invention is not limited thereto. For example, when the display image HP2 shown in Figure 10 is displayed on the display unit 32, the composite image generation unit 24B composites a pointer (not shown) moved by an input means such as the occupant's touch panel onto the display image HP2. The display image generation unit 34B may detect the distance and direction of movement of the display image HP2 of this pointer and determine the cropping range in the composite image CP according to the detected distance and direction of movement.

[0114] Furthermore, in the first embodiment described above, the communication unit 28 has both the function of a transmitting unit for transmitting the composite image CP and the function of a transmitting unit for transmitting other information, but the present invention is not limited thereto. The transmitting unit for transmitting the composite image CP may be provided separately from the communication unit 28. In this way, by providing a transmitting unit dedicated to transmitting the composite image CP, it is possible to make the communication processing speed of the composite image CP faster.

[0115] Furthermore, in the first embodiment described above, the communication unit 36 ​​has both the function of a receiving unit for receiving the composite image CP and the function of a transmitting and receiving unit for sending and receiving other information, but the present invention is not limited thereto. The receiving unit for receiving the composite image CP may be provided separately from the communication unit 36. In this way, by providing a receiving unit specialized for receiving the composite image CP, it is possible to make the communication processing speed of the composite image CP faster.

[0116] Furthermore, in the second embodiment described above, the communication unit 28-2 has both the function of a transmitting unit for transmitting vehicle surrounding images P1 to P8 and vehicle bottom image P9, and the function of a transmitting unit for transmitting other information, but the present invention is not limited thereto. A transmitting unit for transmitting vehicle surrounding images P1 to P8 and vehicle bottom image P9 may be provided separately from the communication unit 28-2. In this way, by providing a transmitting unit specialized for transmitting vehicle surrounding images P1 to P8 and vehicle bottom image P9, it is possible to make the communication processing speed of vehicle surrounding images P1 to P8 and vehicle bottom image P9 faster.

[0117] Furthermore, in the second embodiment described above, the communication unit 36-2 has both the function of a receiving unit that receives vehicle surrounding images P1 to P8 and vehicle bottom image P9, and the function of a transmitting and receiving unit that transmits and receives other information, but the present invention is not limited thereto. The receiving unit that receives vehicle surrounding images P1 to P8 and vehicle bottom image P9 may be provided separately from the communication unit 36-2. In this way, by providing a receiving unit specialized for receiving vehicle surrounding images P1 to P8 and vehicle bottom image P9, it is possible to make the communication processing speed of vehicle surrounding images P1 to P8 and vehicle bottom image P9 faster.

[0118] Furthermore, although the display device 30 is mounted on the vehicle 20 in the above embodiment, the present invention is not limited to this. For example, it may be mounted on an externally provided remote monitoring device (not shown). In this case, the communication unit 28 of the vehicle 20 and the communication unit 36 ​​on the remote monitoring device side, i.e., the display device 30 side, communicate with each other via the network described above. Note that network communication can also be performed via a cloud server (not shown).

[0119] Thus, when the display device 30 is mounted on an externally located remote monitoring device, the remote monitoring device can secure a real-time view of the area around the vehicle and the underside of the vehicle. In this case, the mobile entity does not have to be the vehicle 20. The mobile entity may be, for example, a mobility device that carries luggage or deliveries, or a self-driving robot, or any other form of self-driving device.

[0120] Furthermore, in the above embodiment, the configurations of vehicles 20 and 20-2 and the configurations of display devices 30 and 30-2 mounted on vehicles 20 and 20-2 are connected by separate buses, but the present invention is not limited to this, and they may be connected by the same bus. In this case, the bus functions as the communication unit 28 and 28-2 on the vehicle side and the communication unit 36 ​​and 36-2 on the display device side. That is, the bus functions as a communication unit that transmits or receives image data.

[0121] Although an example of the present invention has been described above, the present invention is not limited to the above, and it is of course possible to implement it in various ways without departing from its spirit. [Explanation of Symbols]

[0122] 10, 10-2 Display System 20, 20-2 Vehicle (mobile) 30, 30-2 Display device 22 Surround Photography Section 23. Downward camera (lower shooting section) 24B Composite image generation section 28, 28-2 Communications Section (Transmitting Section) 30 Display device 34B Display image generation section 34D composite image generation unit 36, 36-2 Communications section (receiving section) P Photographed image P1-P8 Vehicle surroundings images (moving object surroundings images) P9, P10 Vehicle bottom view images (Moving vehicle bottom view images) CP composite image (equirectangular 360-degree spherical image) RP composite image (sphere format) HP display image HP1~HP5 Display Images

Claims

1. A surrounding imaging unit that captures images of the area around a moving object to obtain an image of the area around the moving object, A lower imaging unit that photographs the lower side of the moving object and acquires a bottom image of the moving object, A mobile body includes a composite image generation unit that generates a composite image by combining an image of the mobile body's surroundings acquired by the surrounding imaging unit and an image of the mobile body's bottom surface acquired by the lower imaging unit, A display device, including a display unit, is mounted on an externally located remote monitoring device, A communication unit is included in the aforementioned mobile body and the aforementioned display device, respectively, which transmits or receives image data. Equipped with, The communication unit on the mobile side includes a transmission unit that transmits the composite image generated by the composite image generation unit, The communication unit on the display device side includes a receiving unit that receives the composite image transmitted from the transmitting unit, When the automatic driving mode for the mobile body is selected, or when the recovery driving mode is selected, the display image generated from the composite image is displayed on the display device. A display system in which, when the abnormal driving mode is selected, the display image generated based on notification information from the automatic driving system and information detected by sensing by sensors mounted on the mobile body is displayed, and when the recovery driving mode is selected, the display image in which the lower side of the mobile body is visible is displayed.

2. A surrounding imaging unit that captures images of the area around a moving object to obtain an image of the area around the moving object, A mobile body including a lower imaging unit that photographs the lower side of the mobile body and acquires a bottom image of the mobile body, A display device including a display unit, A composite image generation unit is included in the moving body or the display device, which generates a composite image by converting the moving body's surrounding image acquired by the surrounding imaging unit and the moving body's bottom surface image acquired by the lower imaging unit into a planar 360-degree spherical image; A communication unit is included in the aforementioned mobile body and the aforementioned display device, respectively, which transmits or receives image data. Equipped with, The aforementioned composite image is a display system in which the image includes information in all directions that can be seen from the moving object which is the shooting location.

3. A surrounding imaging unit captures images of the entire surroundings of a moving object to acquire an image of the area around the moving object, A mobile body including a lower imaging unit that photographs the lower side of the mobile body and acquires a bottom image of the mobile body, A display device including a display unit, A composite image generation unit is included in the moving body or the display device, which generates a composite image by combining an image of the moving body's surroundings acquired by the surrounding imaging unit and an image of the moving body's bottom surface acquired by the lower imaging unit. A communication unit is included in the aforementioned mobile body and the aforementioned display device, respectively, which transmits or receives image data. Equipped with, The aforementioned images of the surroundings of the moving body and the aforementioned images of the bottom surface of the moving body include a portion of the moving body. The composite image generation unit is a display system that generates a composite image by overlapping the overlapping portions of adjacent images of the surroundings of the moving object and the bottom surface of the moving object.

4. The moving body includes the composite image generation unit, The communication unit on the mobile side includes a transmission unit that transmits the composite image generated by the composite image generation unit, The display system according to claim 2 or 3, wherein the communication unit on the display device side includes a receiving unit that receives the composite image transmitted from the transmitting unit.

5. The communication unit on the mobile body side includes a transmission unit that transmits the image of the mobile body's surroundings acquired by the surrounding imaging unit and the image of the mobile body's bottom surface acquired by the lower imaging unit. The communication unit on the display device side includes a receiving unit that receives the image of the surroundings of the moving object and the image of the bottom surface of the moving object transmitted from the transmitting unit. The display system according to claim 2 or 3, wherein the display device includes the composite image generation unit.

6. When the surrounding imaging unit acquires multiple images of the surroundings of the moving object, The display system according to any one of claims 1 to 3, wherein the composite image generation unit overlaps and composites all adjacent images of the surroundings of the moving object and the bottom surface of the moving object.

7. When the surrounding imaging unit acquires multiple images of the moving object's surroundings, and the lower imaging unit acquires one image of the moving object's bottom surface, The display system according to any one of claims 1 to 3, wherein the composite image generation unit composites the bottom surface image of the moving body by overlapping each of the surrounding images of the moving body.

8. When the surrounding imaging unit acquires multiple images of the surroundings of the moving object, and the lower imaging unit acquires two images of the bottom surface of the moving object, The display system according to any one of claims 1 to 3, wherein the composite image generation unit composites a plurality of images surrounding the moving body by overlapping each of them with at least one of the images of the bottom surface of the moving body.

9. When the lower imaging unit acquires multiple images of the bottom surface of the moving body, The display system according to any one of claims 1 to 3, wherein the composite image generation unit composites adjacent bottom surface images of the moving body by overlapping them.

10. The aforementioned display device is A display system according to any one of claims 1 to 3, comprising a display image generation unit that generates a display image to be displayed on the display unit from the composite image.

11. The display system according to claim 1 or claim 2, wherein the image of the surrounding area of ​​the moving body and the image of the bottom surface of the moving body include a part of the moving body.

Images