Remote monitoring system

Abstract

To provide a remote monitoring system that allows a remote monitoring device side to make image processing easy.SOLUTION: A remote monitoring system 10 includes a mobile body 20 and a remote monitoring device 30. The mobile body 20 includes four or more photographing units 22A to 22J whose photographing ranges can be set so that overlapping portions where adjacent photographing areas overlap are formed, a composite image generation unit 24B that combines four or more captured images each captured by the four or more photographing units 22A to 22J with overlapping the overlapping portions and generates a composite image including the entire circumference of the mobile body in the horizontal direction of the mobile body, and a transmitting unit (communication unit) 28 that transmits the generated composite image. The remote monitoring device 30 includes a receiving unit (communication unit) 38 that receives the composite image transmitted from the mobile body 20.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a remote monitoring system.

Background Art

[0002] Patent Document 1 discloses a vehicle remote operation system including a vehicle that generates a composite image, that is, an overhead image, showing a peripheral area of the vehicle as seen from a virtual viewpoint, based on a plurality of images captured by a plurality of cameras mounted on the vehicle. In this vehicle remote operation system, on the vehicle side, a display image for display on a monitor of a remote monitoring device (operation terminal) is generated based on the overhead image, and the generated display image is transmitted to the remote monitoring device and displayed on the monitor of the remote monitoring device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the vehicle remote operation system described in Patent Document 1, since the vehicle side generates an overhead image and further generates a display image, there is a problem that it is difficult to perform image processing operations on the remote monitoring device side.

[0005] In view of the above facts, an object of the present invention is to obtain a remote monitoring system that enables easy image processing on the remote monitoring device side.

Means for Solving the Problems

[0006] A remote monitoring system according to a first aspect of the present invention comprises a mobile body including four or more imaging units mounted on the mobile body, the imaging range of which can be set such that an overlapping portion is formed where adjacent imaging areas overlap; a composite image generation unit that combines four or more images captured by each of the four or more imaging units by overlapping the overlapping portion to generate a composite image that includes the entire surroundings of the mobile body in the horizontal direction; a transmission unit that transmits the composite image generated by the composite image generation unit; and a remote monitoring device including a receiving unit that receives the composite image transmitted from the mobile body.

[0007] According to a remote monitoring system of the first aspect of the present invention, the moving object includes a composite image generation unit that generates a composite image including the entire surroundings of the moving object in the horizontal direction by combining four or more captured images, each captured by four or more imaging units, by overlapping the overlapping portions where adjacent imaging areas overlap, and a transmission unit that transmits the generated composite image. The remote monitoring device also includes a receiving unit that receives the composite image transmitted from the moving object. Therefore, the remote monitoring device can acquire a composite image including the entire surroundings of the moving object in the horizontal direction. Since the composite image contains information about the entire surroundings of the moving object in the horizontal direction, the remote monitoring device can perform image processing based on this acquired composite image. This makes image processing easier on the remote monitoring device.

[0008] Furthermore, in the remote monitoring system of the second aspect of the present invention, in the remote monitoring system of the first aspect, each of the four or more captured images is set to include a part of the moving object.

[0009] According to the remote monitoring system of the second aspect of the present invention, since the shooting range of each of the four or more captured images is set to include a part of the moving object, it is possible to prevent the occurrence of blind spots around the moving object in the composite image.

[0010] Furthermore, a remote monitoring system according to a third aspect of the present invention further includes, in the remote monitoring system according to the first or second aspect, a lower side imaging unit whose imaging range is set such that the lower side of the moving body, including a part of the moving body, becomes the imaging area, and the composite image generation unit also composites the lower side image acquired by the lower side imaging unit into the composite image.

[0011] In recent years, in autonomous driving systems still under development, if any abnormality occurs while the moving vehicle is performing autonomous driving, it is necessary to temporarily suspend the vehicle. According to the remote monitoring system of the second aspect of the present invention, the moving vehicle includes a lower-side imaging unit whose imaging range is set so that the area below the moving vehicle, including a part of the vehicle, becomes the imaging area, and the composite image generation unit also combines the lower-side imaging image acquired by the lower-side imaging unit into the composite image. Therefore, the monitor can remotely check the area around the moving vehicle, including the area below the vehicle body, based on the composite image, and the remote monitoring device can determine when the temporarily suspended vehicle should resume driving. Furthermore, since the moving vehicle includes a lower-side imaging unit whose imaging range is set so that the area below the moving vehicle, including a part of the vehicle, becomes the imaging area, the accuracy of combining the lower imaging area into the composite image can be improved.

[0012] Furthermore, in the remote monitoring system of the fourth aspect of the present invention, the composite image is a 360-degree spherical image in any of the remote monitoring systems of the first to third aspects.

[0013] According to the remote monitoring system of the fourth aspect of the present invention, since the composite image is a 360-degree spherical image, the composite image captures 360 degrees in the horizontal and vertical directions of the moving object, that is, all directions centered on the moving object. Therefore, when the remote monitoring device remotely monitors the moving object, it is possible to prevent blind spots from occurring around the moving object, thereby enabling safe mobility control.

[0014] Furthermore, in a remote monitoring system according to a fifth aspect of the present invention, in any of the first to fourth aspects of the remote monitoring system, the remote monitoring device includes a display unit and a display image generation unit that generates a display image to be displayed on the display unit from the composite image received by the receiving unit.

[0015] According to a fifth aspect of the remote monitoring system of the present invention, the remote monitoring device includes a display unit and a display image generation unit that generates a display image to be displayed on the display unit from a composite image received by a receiving unit. Therefore, on the remote monitoring device side, the display image generation unit can generate a display image to be displayed on the display unit from the composite image, so that a display image can be displayed on the display unit from the viewpoint and display mode desired by the monitor.

[0016] Furthermore, in the remote monitoring system according to the sixth aspect of the present invention, the display image generation unit generates the display image based on the driving status of the moving body, in the remote monitoring system according to the fifth aspect.

[0017] According to the remote monitoring system of the sixth aspect of the present invention, the display image generation unit generates a display image based on the driving status of the moving object, so that a display image corresponding to the driving status of the moving object can be displayed on the display unit. As a result, the monitor can visually confirm a display image corresponding to the driving status of the moving object.

[0018] Furthermore, in the remote monitoring system of the seventh aspect of the present invention, in the remote monitoring system of the fifth aspect, the display image generation unit generates the display image that matches the mode set in the mobile body.

[0019] According to the remote monitoring system of the seventh aspect of the present invention, the display image generation unit generates a display image that matches the mode set on the mobile object, so that a display image corresponding to the set mode of the mobile object can be displayed on the display unit. As a result, the monitor can view a display image corresponding to the set mode of the mobile object.

[0020] Further, in the remote monitoring system according to the eighth aspect of the present invention, in any one of the remote monitoring systems according to the first to seventh aspects, the ratio of the vertical direction to the horizontal direction of the composite image is 1:2.

[0021] According to the remote monitoring system of the eighth aspect of the present invention, since the ratio of the vertical direction to the horizontal direction of the composite image is 1:2, the composite image can be configured in an equirectangular format, that is, an orthographic cylindrical projection method.

Advantages of the Invention

[0022] As described above, according to the remote monitoring system of the present invention, there is an excellent effect that image processing can be easily performed on the remote monitoring device side.

Brief Description of the Drawings

[0023] [Figure 1] It is a configuration diagram showing a schematic configuration of a remote monitoring system according to an embodiment of the present invention. [Figure 2] It is a block diagram showing a hardware configuration of a remote monitoring system according to an embodiment of the present invention. [Figure 3] It is a diagram showing the position where the camera is arranged on the vehicle and the captured images at each position. [Figure 4] It is an explanatory diagram for explaining a method of synthesizing images. [Figure 5] It is an explanatory diagram for explaining a composite image in an equirectangular format. [Figure 6] It is an explanatory diagram for explaining the conversion from an equirectangular format to a spherical format in a full-spherical image. [Figure 7] It is a diagram showing an example of a display image displayed on the display unit of the remote monitoring device. [Figure 8] It is a diagram showing an example of a display image displayed on the display unit of the remote monitoring device. [Figure 9] It is a diagram showing an example of a display image displayed on the display unit of the remote monitoring device. [Figure 10]This figure shows an example of a display image shown on the display unit of a remote monitoring device. [Figure 11] This figure shows an example of a display image shown on the display unit of a remote monitoring device. [Figure 12] This figure shows an example of a display image shown on the display unit of a remote monitoring device. [Modes for carrying out the invention]

[0024] A remote monitoring system 10 according to one embodiment of the present invention will be described using Figures 1 to 12. In the following description, when the directions of front, rear, left, right, up, and down are indicated, they refer to the front, rear, left, right, up, and down directions of a vehicle.

[0025] Figure 1 is a schematic diagram showing the configuration of the remote monitoring system 10 according to this embodiment, and Figure 2 is a block diagram showing the hardware configuration of the remote monitoring system 10 according to one embodiment of the present invention. As shown in Figure 1, the remote monitoring system 10 of this embodiment is configured to include a vehicle 20 as a mobile unit, a remote monitoring device 30, and a network 40.

[0026] Here, we will describe the vehicle 20. The vehicle 20 is an autonomously driven vehicle capable of self-driving, and in this embodiment, it is configured to accommodate a person. As shown in Figure 2, the vehicle 20 includes a camera 22, an image processing device 24, a vehicle control device 26, and a communication unit 28. The camera 22 in this embodiment includes, as an example, nine cameras: a front camera 22A, a front right camera 22B, a right side camera 22C, a rear right camera 22D, a rear camera 22E, a rear left camera 22F, a left side camera 22G, a front left camera 22H, and a downward camera 22J.

[0027] These nine cameras 22A to 22J each correspond to the imaging unit of the present invention, and as an example, each uses a circular fisheye lens with a field of view larger than 180 degrees. The image captured by the circular fisheye lens is a fisheye image, characterized in that subjects near the center of the shooting range appear large, while subjects around the edges of the shooting range appear small, and the shooting range is represented by a circular image. In this embodiment, the "captured image" may be a still image or a video.

[0028] Figure 3 shows the positions of the nine cameras 22A to 22J on the vehicle 20 and the images captured at each position. As shown in Figure 3, the front camera 22A is mounted on the front of the vehicle 20, and the image P1 captured by the front camera 22A shows the front of the vehicle 20. The right front camera 22B is mounted on the right front of the vehicle 20, and the image P2 captured by the right front camera 22B shows the right front of the vehicle 20. The right side camera 22C is mounted on the right side of the vehicle 20, and the image P3 captured by the right side camera 22C shows the right side of the vehicle 20. The right rear camera 22D is mounted on the right rear of the vehicle 20, and the image P4 captured by the right rear camera 22D shows the right rear of the vehicle 20.

[0029] The rear camera 22E is mounted on the rear of the vehicle 20, and the image P5 captured by the rear camera 22E shows the rear of the vehicle 20. The left rear camera 22F is mounted on the left rear of the vehicle 20, and the image P6 captured by the left rear camera 22F shows the left rear of the vehicle 20. The left side camera 22G is mounted on the left side of the vehicle 20, and the image P7 captured by the left side camera 22G shows the left side of the vehicle 20. The left front camera 22H is mounted on the left front of the vehicle 20, and the image P8 captured by the left front camera 22H shows the left front of the vehicle 20. The downward camera 22J is mounted on the underside of the front of the vehicle 20, and the image P9 captured by the downward camera 22J shows the bottom surface 20A of the vehicle body and the ground G (including roads, etc.) on which the vehicle 20 is located. This image P9 corresponds to the downward image of the present invention.

[0030] In this embodiment, as an example, the nine cameras 22A to 22J of the imaging device 22 are set to have their imaging ranges configured such that overlapping areas are formed where the imaging areas of adjacent cameras overlap. That is, in the captured image P1, a subject common to both captured image P2 and captured image P8 is captured at the right edge, and a subject common to both captured image P8 and captured image P1 is captured at the left edge.

[0031] Similarly, in each of the captured images P2 to P8, a subject common to the adjacent captured image is captured at the left and right edges. In this embodiment, as an example, a subject common to captured image P9 is captured at the lower edge of each of the captured images P1 to P8. In other words, a subject common to any of the captured images P1 to P8 is captured at the periphery of captured image P9. Also, although not shown in Figure 3, a part of the vehicle 20 is captured in captured images P1 to P8.

[0032] As shown in Figure 2, the image processing unit 24 is mounted on the vehicle 20. The image processing unit 24, although not shown in the figure, consists of a CPU (Central Processing Unit: processor), ROM (Read Only Memory), RAM (Random Access Memory), and storage, and each component is connected to the others via a bus so that they can communicate with each other. The CPU reads a program from the ROM or storage and executes the program using the RAM as a working area.

[0033] Furthermore, the image processing device 24 includes an image acquisition unit 24A and a composite image generation unit 24B as its functional configuration. The image acquisition unit 24A and the composite image generation unit 24B are executed by the CPU reading a program from ROM or storage and executing the program using RAM as a working area.

[0034] The image acquisition unit 24A acquires image data representing the nine captured images P1 to P9, each captured by the nine cameras 22A to 22J of the imaging device 22. In this embodiment, as an example, one captured image consists of approximately 2 million pixels. The number of pixels in the captured images can be appropriately changed depending on the performance of the camera used.

[0035] The composite image generation unit 24B generates a composite image CP by combining the nine captured images P1 to P9, each represented by the image data acquired by the image acquisition unit 24A. In this embodiment, the composite image CP is a 360-degree spherical image representing all directions that can be seen from the vehicle 20, which is the shooting location, i.e., the entire surroundings (omnidirectional) of the vehicle.

[0036] The composite image generation unit 24B has pre-stored information necessary to convert the nine captured images P1 to P9, each captured by one of the nine cameras 22A to 22J, into a full-sphere image. This information includes positional information representing the physical arrangement of the circular fisheye lenses of each of the nine cameras 22A to 22J, projection information representing the relationship between the image height of the captured images P1 to P9 and the angle of incidence of the circular fisheye lens, information indicating the shooting direction of the nine cameras 22A to 22J, and information representing the characteristics of the circular fisheye lens. This information is derived in advance through calibration.

[0037] The composite image generation unit 24B generates a composite image CP by performing projection transformation on nine captured fisheye images P1 to P9 using known techniques based on the necessary information described above. In this embodiment, when compositing the composite image CP, the overlapping portions described above are overlapped. In Figure 4, for example, if we explain using captured image P6 as an example, the overlapping portion between captured image P6 and captured image P5 is shown as the first region OV1, which is shaded, and the overlapping portion between captured image P6 and captured image P7 is shown as the second region OV2, which is shaded. The composite image generation unit 24B combines captured images P1 to P8 by overlapping these overlapping portions on a pixel-by-pixel basis. Note that known techniques can be used for the overlapping process.

[0038] Furthermore, the captured image P9 is composited with the combined captured images P1 to P8. In this embodiment, since the same subject is captured in the peripheral area of ​​captured image P9 as in captured images P1 to P8, the area in which this common subject is captured becomes the overlapping area where adjacent captured areas overlap. In Figure 4, for example, if we explain using captured image P6 as an example, the overlapping area between captured image P6 and captured image P9 is shown as the third region OV3, indicated by diagonal lines. The composite image generation unit 24B composites captured image P9 below captured images P1 to P8 by overlapping this overlapping area on a pixel-by-pixel basis. Note that known techniques can be used for the overlapping method.

[0039] Figure 5 is an explanatory diagram for illustrating the equirectangular composite image. Here, the equirectangular composite image is an image with a vertical-to-horizontal aspect ratio of 1:2. The equirectangular format is a format in which a rectangular composite image with a vertical-to-horizontal aspect ratio of 1:2 is pasted onto a spherical screen, and when the composite image is viewed from the center of the screen, a 360-degree view is correctly displayed.

[0040] In this embodiment, the composite image generation unit 24B generates an equirectangular composite image CP, that is, a composite image CP with a vertical-to-horizontal ratio of 1:2. The equirectangular composite image CP is a planar 360-degree spherical image. In this embodiment, as an example, the composite image CP is composed of image data of 3840 × 1920 pixels, or approximately 7.4 million pixels. The number of pixels in the composite image CP can be appropriately changed according to the performance of the display unit of the remote monitoring device 30, which will be described later.

[0041] As shown in the right-hand diagram of Figure 5, a 360-degree spherical image is an image represented in a coordinate system that includes angular coordinates around a predetermined axis. With the radial movement set as a constant n (for example, 1), it is represented as an array of pixel values ​​whose coordinates are the vertical angle φ made with respect to the axis in the pitch direction and the horizontal angle λ corresponding to the rotation angle around the axis in the yaw direction. The vertical angle φ is in the range of -90 degrees to +90 degrees (or 0 degrees to 180 degrees), and the horizontal angle λ is in the range of -180 degrees to +180 degrees (or 0 degrees to 360 degrees).

[0042] The composite image generation unit 24B determines that the horizontal direction of the composite image CP represents the entire horizontal circumference of the vehicle 20, i.e., 360 degrees, and therefore matches the horizontal direction of the composited captured images P1 to P8. Furthermore, the captured image P9 is composited to the lower end of the composited captured images P1 to P8, and the lower end of this captured image P9 matches the lower end of the composite image CP. In this embodiment, the area directly above the vehicle 20 is not captured in the captured images P1 to P8. Therefore, the composite image generation unit 24B reads the field of view above the captured images P1 to P8 from pre-stored information, and determines the coordinate position of the upper end of the composited captured images P1 to P8 according to the read field of view.

[0043] Furthermore, the coordinates corresponding to the area directly above the vehicle 20 that is not captured by cameras 22A to 22H are represented by a uniform pixel value as a blank, as shown in Figure 5. In this embodiment, as an example, it is represented by a pixel value representing black, and this black area is designated as the blank image BP. The larger the field of view on the upper side of the captured images P1 to P8, the narrower the width of the blank image BP in the pitch direction (vertical direction). In this way, the composite image generation unit 24B generates the composite image CP.

[0044] Returning to Figure 2, the vehicle control device 26 is a device that controls autonomous driving of the vehicle 20 based on information detected by various sensors (not shown) mounted on the vehicle 20 and captured images P1 to P9 taken by the camera 22. The vehicle control device 26 can utilize known technologies. In this embodiment, the vehicle control device 26 also has the function of controlling the driving of the vehicle 20 by operation from the remote monitoring device 30.

[0045] Furthermore, the vehicle control device 26 is configured to allow setting various modes, such as a normal driving mode, an abnormal driving mode, a recovery driving mode, and a remote control mode. Based on the information detected by the sensors and the captured images P1 to P9, the vehicle control device 26 automatically selects one of the above modes to drive the vehicle 20. The vehicle control device 26 selects the normal driving mode when the vehicle is operating normally, and selects the abnormal driving mode when, for example, an object deemed abnormal is detected by notification information from the automatic driving system or sensing by the sensors. Specifically, when the abnormal driving mode is selected by the vehicle control device 26, the vehicle 20 is stopped.

[0046] Furthermore, the recovery driving mode is selected by the vehicle control device 26 when the vehicle 20 starts driving after coming to a stop. Specifically, the vehicle control device 26 either confirms that there is no abnormality in the vehicle 20 through the automatic driving system or sensing by the above-mentioned sensors, or sends a signal to the remote monitoring device 30 to confirm that there is no abnormality.

[0047] The remote control mode is selected by the vehicle control device 26 when the vehicle 20 is to be controlled by the remote monitoring device 30. Specifically, the vehicle control device 26 transmits a signal to the remote monitoring device 30 to initiate remote control.

[0048] The vehicle control device 26 transmits the driving status of the vehicle 20, that is, the currently selected mode, to the remote monitoring device 30 via the communication unit 28. In addition, the vehicle control device 26 may also transmit "abnormality information" indicating that an abnormality has occurred, via the communication unit 28, to the remote monitoring device 30, for example, when it detects an object that is judged to be abnormal through notification information from the automatic driving system or sensing by the above-mentioned sensors.

[0049] The communication unit 28 is an interface for communicating with external devices and also functions as a transmitter that transmits the composite image CP generated by the composite image generation unit 24B. Furthermore, the communication unit 28 also functions as a transmitter that transmits the driving status of the vehicle 20, i.e., the currently selected mode.

[0050] Next, the remote monitoring device 30 will be described. As shown in Figure 2, the remote monitoring device 30 includes a display device 32 as a display unit, a display control device 34, a remote control device 36, a communication unit 38, and a recording unit 39.

[0051] The display device 32 includes, for example, four display units: a first display unit 32A, a second display unit 32B, a third display unit 32C, and a fourth display unit 32D. As shown in Figure 1, the first to fourth display units 32A to 32D are arranged, for example, in pairs vertically and in pairs horizontally. Specifically, as an example, from the perspective of the observer, the first display unit 32A is located on the upper left, the second display unit 32B on the upper right, the third display unit 32C on the lower left, and the fourth display unit 32D on the lower right.

[0052] As an example, the first display unit 32A displays operational information for multiple vehicles 20, and the second display unit 32B displays vehicle information such as the type of vehicle 20, the number of passengers, and the route of the currently operating vehicle 20. The third display unit 32C and the fourth display unit 32D display a display image HP generated based on the composite image CP received from each vehicle 20. The composite image CP can also be displayed as is.

[0053] In this embodiment, the first to fourth display units 32A to 32D have the capability to display video corresponding to 4K resolution, for example, but the capabilities of each display unit are not limited to this, and they do not all have to have the same capabilities, and they may have the capability to display video with a lower resolution than 4K or a higher resolution than 4K, and can be changed as appropriate.

[0054] The communication unit 38 is an interface for communicating with external devices and also functions as a receiver for receiving the composite image CP transmitted from the communication unit 28 of the vehicle 20. Here, the communication unit 28 of the vehicle 20 and the communication unit 38 of the remote monitoring device 30 communicate with each other via the network 40, as shown in Figure 1. The network 40 consists of narrow-area wireless communication modules such as Wi-Fi® and Bluetooth®, and wide-area wireless communication modules such as 4G, LTE, and 5G, such as mobile lines. Note that communication via the network 40 can also be performed via a cloud server (not shown).

[0055] The recording unit 39 is a non-volatile memory such as flash memory, and records the composite image CP received by the communication unit 38. It also records various data such as the vehicle speed and steering angle of the vehicle 20 transmitted from the vehicle's communication unit 28.

[0056] The remote control device 36 has the function of remotely controlling the vehicle 20, and can, for example, stop the vehicle 20 or drive the vehicle 20 under conditions desired by the supervisor. The remote control device 36 can use known technology.

[0057] The display control device 34 is a device that displays images, information, etc., on the display device 32. Although not shown in the diagram, it is composed of a CPU (Central Processing Unit: processor), ROM (Read Only Memory), RAM (Random Access Memory), and storage. Each component is connected to the others via a bus so that they can communicate with each other. The CPU reads a program from the ROM or storage and executes the program using the RAM as a working area.

[0058] Furthermore, the display control device 34 includes a composite image acquisition unit 34A and a display image generation unit 34B as its functional configuration. The composite image acquisition unit 34A and the display image generation unit 34B are executed by the CPU reading a program from ROM or storage and executing the program using RAM as a working area.

[0059] The composite image acquisition unit 34A acquires the composite image CP received by the communication unit 38 and stores it in the recording unit 39.

[0060] The display image generation unit 34B generates a display image HP to be displayed on the display device 32 (in this embodiment, the third display unit 32C and the fourth display unit 32D). Specifically, the display image generation unit 34B generates the display image HP based on the composite image CP.

[0061] Figure 6 is an explanatory diagram illustrating the conversion from equirectangular to spherical format in a full-sphere image. The display image generation unit 34B takes point Pe(λ,φ) on the composite image CP shown in the left figure of Figure 6 and points Ps(x) on the unit sphere shown in the right figure of Figure 6. s ,y s ,z s Convert to ). Here, in the right diagram of Figure 6, λ represents longitude and φ represents latitude.

[0062] In this case, the conversion formula is shown by the following formulas (1) to (3). x s =cos(φ)sin(λ)···(1) y s =sin(φ) ···(2) z s =cos(φ)cos(λ)···(3)

[0063] The display image generation unit 34B converts the planar sphere image into a spherical sphere image using the above conversion formulas (1) to (3). Based on the composite image CP and the composite image RP representing the converted spherical sphere image, the display image generation unit 34B generates a display image HP in the display mode desired by the monitor operating the remote monitoring device 30. Since the equirectangular planar composite image CP contains information for all directions viewed from the origin, it is also possible to generate a composite image of any shape other than a sphere, such as a cylinder viewed from above, based on this composite image CP. Furthermore, the display image generation unit 34B can also use the composite image CP as is as the display image HP.

[0064] Figures 7 to 12 show examples of display images HP displayed on the third display unit 32C or the fourth display unit 32D of the remote monitoring device 30. In this embodiment, as an example, the third display unit 32C and the fourth display unit 32D each display display images HP generated by the display image generation unit 34B based on composite images CP transmitted from different vehicles 20. In this embodiment, the following describes how to display a display image HP generated for a composite image CP transmitted from one vehicle 20 on the third display unit 32C as an example. For the other vehicles 20, display images HP are generated in the same manner as for the one vehicle 20 and displayed on the fourth display unit 32D.

[0065] As shown in Figure 7, the display image generation unit 34B generates a display image HP1, which is a 360-degree overhead view image generated from a viewpoint looking down on the vehicle 20 in the composite image RP representing a spherical panoramic image. In Figure 7, for convenience, the image representing the underside of the vehicle 20 is shown in black, but in reality, the actual image representing the underside of the vehicle 20 is shown. Alternatively, the display image generation unit 34B may superimpose an image A1, which simulates the vehicle 20, onto the display image HP1. In this case, in order to make the area where image A1 is superimposed on the display image HP1 visible, image A1 is made transparent. The generated display image HP1 is displayed on the third display unit 32C by the display control device 34.

[0066] The display control device 34 may superimpose and combine an image A2 onto the display image HP1, which includes the image A1 displayed on the third display unit 32C, by superimposing an image A2 that can be moved in the clockwise rotation direction indicated by arrow M1 or the counterclockwise rotation direction indicated by arrow M2. Image A2 can have a shape formed by connecting an arc on the circumference of the display image HP1 with an inner arc formed with a smaller radius than the said arc at both ends. The area on the display image HP1 over which this image A2 overlaps is considered to be the area in the direction that the monitor wants to see.

[0067] In this embodiment, the observer can change the direction they want to view by moving the image A2 in the clockwise or counterclockwise rotation direction. As shown in Figure 8, the display image generation unit 34B generates a display image HP2 that is cropped in the direction the observer wants to view, that is, from a free viewpoint desired by the observer. The display image HP2 shown in Figure 8 is an image of the view from the front of the vehicle 20. The generated display image HP2 is displayed on the third display unit 32C by the display control device 34.

[0068] Furthermore, for example, if there are multiple directions the monitor wants to view, the monitor can move image A2 in the desired direction and select that direction by performing an operation such as long-pressing, and then move image A2 in the desired direction again to select another direction in the same way. For example, if the monitor wants to view four directions—forward, backward, left, and right—and all four directions are selected, the display image generation unit 34B cuts out images from the viewpoint of each direction, arranges the cut-out images on the same plane, and generates a single display image HP3 as shown in Figure 9. The generated display image HP2 is displayed on the third display unit 32C by the display control device 34. The display control device 34 may also display the display image HP1 shown in Figure 7 on the third display unit 32C and the display images HP2 and HP3 shown in Figure 8 or Figure 9 on the fourth display unit 32D.

[0069] Furthermore, for example, if the composite image generation unit 24B acquires information that the observer wants to view the vehicle 20 from the rear diagonal upper side, the composite image generation unit 24B generates a display image HP4, which is an overhead view of the vehicle 20 from the rear diagonal upper side, based on the composite image RP, as shown in Figure 10. The display image generation unit 34B superimposes an image A3 representing the vehicle 20 onto the display image HP4. This image A3 can be an image that simulates the vehicle 20, similar to the display image HP1 shown in Figure 7.

[0070] For example, suppose two people are walking to the left of a vehicle 50 (see Figure 11) traveling in front of vehicle 20. If these two people are detected as abnormal by notification information from the automatic driving system in the vehicle control device 26 on the vehicle 20 side or by sensing by sensors (not shown) mounted on the vehicle 20, the communication unit 28 on the vehicle 20 side transmits this information, namely the location and size information of the two people walking, to the remote monitoring device 30, which is then received by the communication unit 38 on the remote monitoring device 30.

[0071] The display image generation unit 34B, based on the information received by the communication unit 38 of the remote monitoring device 30, identifies the area in which the two people are captured from the composite image CP using known techniques, and generates a display image HP5 cropped to the angle of view in which the two people 52 are captured, as shown in Figure 11. The display image HP5 shows the view from the vehicle 20 looking forward. Based on the information received by the communication unit 38, the display image generation unit 34B superimposes an image A4, which shows the direction in which the two people are located as seen from the vehicle 20, as an example with an arrow, onto the display image HP5. The display image HP5, including the generated image A4, is displayed on the third display unit 32C by the display control device 34. Note that image A4 may be in a form other than an arrow, or it may be, for example, a frame line surrounding the two people 52.

[0072] The display image generation unit 34B, after identifying the area in which the two individuals are visible from the composite image CP using known techniques, generates a display image HP6, which is a 360-degree overhead view image generated from a viewpoint looking down on the vehicle 20 in the composite image RP representing a spherical panoramic image, as shown in Figure 12. The display image generation unit 34B then superimposes an image A5, which indicates the direction in which the two individuals are located relative to the vehicle 20 using an "x" mark, onto the display image HP6. In this embodiment, image A5 is configured in a manner that is emphasized compared to other image areas. Note that image A5 may be in a manner other than an "x" mark, such as a circle.

[0073] Note that, as with the display image HP1 shown in Figure 7, the display image HP6 shown in Figure 12 shows the underside of the vehicle 20 in black, but in reality, it displays the actual image of the underside of the vehicle 20. Alternatively, an image A1 simulating the vehicle 20 may be superimposed and combined with the display image HP1. Also, in the display image HP6, the 360-degree overhead view image around image A1 is shown in white, but in reality, it is configured the same as the display image HP1 shown in Figure 7. The display image HP6, including the generated images A1 and A5, is displayed on the third display unit 32C by the display control device 34.

[0074] Furthermore, in this embodiment, the display image generation unit 34B generates a display image HP according to the driving status of the vehicle 20 transmitted from the vehicle control device 26, that is, the currently selected mode. Specifically, when the vehicle 20 is in normal driving mode, the display image generation unit 34B generates the display image HP1 shown in Figure 7, and the display control device 34 displays the generated display image HP1 on the third display unit 32C. Subsequently, as described above, the display image generation unit 34B generates one of the display image HP2 shown in Figure 8, the display image HP3 shown in Figure 9, or the display image HP4 shown in Figure 10 in response to the monitoring operator's movement operation of image A2, and the display control device 34 displays the generated display image on the third display unit 32C.

[0075] Furthermore, if the vehicle 20 is in abnormal driving mode, the display image generation unit 34B generates a display image HP5 including the generated image A4 shown in Figure 11, or a display image HP6 including the generated images A1 and A5 shown in Figure 12, based on the notification information of the automatic driving system in the vehicle control device 26 on the vehicle 20 side and the information detected by sensing by sensors (not shown) mounted on the vehicle 20, as described above, and the display control device 34 displays the generated display image on the third display unit 32C.

[0076] Furthermore, when the vehicle 20 is in the return-to-driving mode, the display image generation unit 34B generates a display image HP1 shown in Figure 7, in which the underside of the vehicle body of the vehicle 20 is visible, and the display control device 34 displays the generated display image HP1 on the third display unit 32C. Subsequently, as described above, the display image generation unit 34B generates one of the display images HP2 shown in Figure 8, display image HP3 shown in Figure 9, or display image HP4 shown in Figure 10 in response to the movement operation of image A2 by the monitor, and the display control device 34 displays the generated display image on the third display unit 32C.

[0077] Furthermore, when the vehicle 20 is in remote control mode, the display image generation unit 34B and the display control device 34 perform the same processing as in normal driving mode.

[0078] Next, the operation and effects of this embodiment will be described.

[0079] According to the remote monitoring system 10 of this embodiment, the vehicle 20 includes a composite image generation unit 24B that combines nine captured images P1 to P9, each captured by nine cameras 22A to 22J, by overlapping the overlapping portions where adjacent shooting areas overlap, to generate a composite image CP that includes the entire surroundings of the vehicle in the horizontal direction, and a communication unit 28 that functions as a transmission unit to transmit the generated composite image. The remote monitoring device 30 also includes a communication unit 38 that functions as a reception unit to receive the composite image CP transmitted from the vehicle 20. Therefore, the remote monitoring device 30 can acquire the composite image CP that includes the entire surroundings of the vehicle in the horizontal direction. Furthermore, since the composite image CP contains information about the entire surroundings of the vehicle in the horizontal direction, the remote monitoring device 30 can perform image processing based on this acquired composite image CP. This makes image processing easier on the remote monitoring device 30.

[0080] Furthermore, in conventional technology, for example, if nine cameras are mounted on a vehicle as in this embodiment, the images captured by each of the nine cameras are displayed on the display device of the remote monitoring device, and nine images are monitored. As a result, the number of screens to be monitored on the display device becomes large, making monitoring cumbersome and potentially requiring time for the monitor to make decisions. Also, if there are multiple vehicles to be monitored, it is difficult to instantly determine which vehicle's image and which direction is being displayed.

[0081] Furthermore, since the image data of the captured images from each of the nine cameras is transmitted to the remote monitoring device, for example, if one of the captured images consists of approximately 2 million pixels, as in the above embodiment, the image data transmitted to the remote monitoring device will be approximately 2 million pixels x 9 (number of cameras) = ​​approximately 18 million pixels. Therefore, depending on the performance and number of cameras, the amount of data to be transmitted may increase even further.

[0082] In contrast, according to the remote monitoring system 10 of this embodiment, the composite image generation unit 24B generates a composite image CP by combining nine captured images P1 to P9, each captured by nine cameras 22A to 22J, overlapping the overlapping portions where adjacent captured areas overlap, and transmits this composite image CP to the remote monitoring device 30. In this embodiment, the composite image CP is generated to consist of image data of approximately 7.4 million pixels. Therefore, the amount of data transmitted can be reduced compared to the conventional technology.

[0083] Furthermore, according to the remote monitoring system 10 of this embodiment, since the shooting range of each of the captured images P1 to P9 is set to include a part of the vehicle 20, it is possible to prevent the occurrence of blind spots around the vehicle 20 in the composite image.

[0084] Furthermore, according to the remote monitoring system 10 of this embodiment, since the composite image CP is a 360-degree spherical image, the composite image CP captures 360 degrees of the vehicle 20 in both the horizontal and vertical directions, that is, all directions centered on the vehicle. In other words, the shooting direction of each camera is associated with the composite image CP. Therefore, when the remote monitoring device 30 remotely monitors the vehicle 20, it is possible to prevent blind spots from occurring around the vehicle 20, thereby enabling safe mobility control.

[0085] Furthermore, according to the remote monitoring system 10 of this embodiment, the remote monitoring device 30 includes first to fourth display units 32A to 32D and a display image generation unit 34B that generates a display image HP to be displayed on the third display unit 32C and the fourth display unit 32D from a composite image CP received by the communication unit 38. Therefore, on the remote monitoring device 30 side, the display image generation unit 34B can generate a display image HP to be displayed on the third display unit 32C and the fourth display unit 32D from the composite image CP, so that the display image HP can be displayed on the third display unit 32C and the fourth display unit 32D from the viewpoint and display mode desired by the monitor.

[0086] Furthermore, by making a single display image HP an image that includes images from multiple viewpoints, or an image that allows the entire surroundings of the vehicle 20 to be visible, as shown in display images HP1, HP3, and HP4 in Figures 7, 9, and 10, the burden on the monitor can be reduced. This makes it possible for one or fewer monitors to remotely monitor and control the operation of multiple vehicles 20. Also, if there are multiple vehicles 20 to be monitored, the observer can instantly determine which vehicle and which direction the image being displayed is from by viewing a single display image HP.

[0087] Furthermore, in recent years, in autonomous driving systems still under development, if any abnormality occurs while the vehicle 20 is performing autonomous driving, it is necessary to temporarily stop the vehicle 20. According to the remote monitoring system 10 of this embodiment, the vehicle 20 includes a lower camera 22J whose shooting range is set so that the lower part of the vehicle body is the shooting area, and the composite image generation unit 24B also combines the captured image P9 acquired by the lower camera 22J into the composite image CP. Therefore, the monitor can remotely check the area around the vehicle 20, including the lower part of the vehicle body, based on the composite image CP, and the remote monitoring device 30 can determine when the temporarily stopped vehicle 20 should resume (start driving). In addition, the vehicle 20 includes a lower camera 22J whose shooting range is set so that the lower part of the vehicle 20, including a part of the vehicle 20, is the shooting area. Therefore, when the composite image generation unit 24B combines the captured image P9 into the composite image CP, there is a higher probability that a part of the vehicle 20 exists as a common subject in the captured images P1 to P8 and the captured image P9, thereby improving the accuracy of the composite.

[0088] Furthermore, according to the remote monitoring system 10 of this embodiment, the display image generation unit 34B generates a display image HP that matches the mode set in the vehicle 20, so that the display image HP corresponding to the status of the vehicle 20 can be displayed on the third display unit 32C and the fourth display unit 32D.

[0089] Furthermore, when the vehicle 20 is in, for example, normal driving mode or remote control mode, a 360-degree overhead view image generated from a viewpoint looking down on the vehicle 20 is displayed in the composite image RP, which represents a spherical panoramic image, as shown in display image HP1 in Figure 7. When the monitor performs a movement operation on image A2, one of the display images shown in Figure 8 (shown as HP2), Figure 9 (shown as HP3), or Figure 10 (shown as HP4) is displayed on the third display unit 32C as an image viewed from the viewpoint corresponding to this movement operation. This allows the monitor to view the image from the desired viewpoint.

[0090] Furthermore, when the vehicle 20 enters, for example, an abnormal driving mode, if the object of the abnormality is two people 52, as shown in display images HP5 and HP6 in Figure 11 or Figure 12, the direction in which these two people 52 are located relative to the vehicle 20 is presented by image A4 with an arrow and image A5 with an "X" mark. This allows the observer to visually confirm in which direction the abnormality is occurring relative to the vehicle 20.

[0091] Furthermore, when vehicle 20 is in, for example, recovery driving mode, an image is displayed that allows the underside of vehicle 20 to be seen, as shown in display image HP1 in Figure 7. Therefore, when recovering (restarting) vehicle 20 that has been stopped in abnormal driving mode, the monitor can also check the underside of vehicle 20, allowing the monitor to safely restart vehicle 20.

[0092] Furthermore, according to the remote monitoring system 10 of this embodiment, the display image generation unit 34B generates a display image HP based on the driving status of the vehicle 20, so that the display image HP corresponding to the driving status of the vehicle 20 can be displayed on the third display unit 32C and the fourth display unit 32D. As a result, the monitor can visually confirm the display image HP corresponding to the driving status of the vehicle 20.

[0093] Furthermore, according to the remote monitoring system 10 of this embodiment, since the ratio of the vertical to horizontal directions of the composite image CP is 1:2, the composite image CP can be constructed in equirectangular form, that is, using equirectangular projection.

[0094] [Supplementary explanation of the embodiment] In the above embodiment, the cameras 22A to 22J constituting the imaging device 22 each use a circular fisheye lens having a field of view greater than 180 degrees, but the present invention is not limited thereto. As long as the shooting range can be set so that an overlapping portion is formed where adjacent shooting areas overlap, a circular fisheye lens with a field of view of, for example, 140 degrees may be used, and the field of view is not limited. Furthermore, instead of a circular fisheye lens, a diagonal fisheye lens, other wide-angle lenses, ultra-wide-angle lenses, etc. may be used.

[0095] Furthermore, in the above embodiment, the cameras 22A to 22J constituting the imaging device 22 include one lower camera 22J located on the front lower side of the vehicle 20, but the present invention is not limited to this. For example, a lower camera may also be provided on the rear lower side of the vehicle 20. In other words, there may be two or more lower cameras, and the number can be appropriately changed according to the size of the vehicle body 20.

[0096] Furthermore, in the above embodiment, the cameras 22A to 22J constituting the imaging device 22 were set to nine, but the present invention is not limited to this. For example, the downward camera 22J may not be included. In this case, the image of the underside of the vehicle 20 is not included in the composite image, so the underside of the vehicle body of the vehicle 20 cannot be seen by the remote monitoring device 30, but the horizontal direction of the vehicle 20 can be seen in 360 degrees.

[0097] Furthermore, in the above embodiment, the horizontal cameras 22A to 22H of the vehicle 20 constituting the imaging device 22 were set to eight, but the present invention is not limited thereto. As long as the imaging range can be set so that overlapping portions are formed where adjacent imaging areas overlap, there may be four cameras: a front camera 22A, a right-side camera 22C, a rear camera 22E, and a left-side camera 22G, or this can be appropriately changed according to the field of view of each camera.

[0098] Furthermore, in the above embodiment, the captured image P9 taken by the lower camera 22J was assumed to contain subjects common to each of the captured images P1 to P8 at its periphery, but the present invention is not limited to this. For example, if only one camera for capturing the bottom surface 20A of the vehicle body is provided on the front lower side of the vehicle 20, the captured image P9 may only contain subjects common to the captured image P5 taken by the rear camera 22E. In other words, it is sufficient that the captured image P9 contains subjects common to one or more of the captured images P1 to P1, which capture the horizontal direction of the vehicle 20.

[0099] If only the same subjects as those in captured image P5 are captured in captured image P9, then in the generated composite image CP, captured image P9, taken by the lower camera 22J, is composited at the bottom edge of captured image P5, and the right side of captured image P9 becomes a blank image BP.

[0100] Furthermore, in the above embodiment, the composite image CP was a planar 360-degree spherical image, but the present invention is not limited thereto. The composite image CP may be a full-circumference image, a panoramic image with the top and bottom cut off, i.e., a 360-degree panoramic image capturing only 360 degrees in the horizontal direction of the vehicle, or a panoramic image with only the top or only the bottom cut off.

[0101] Furthermore, although the equirectangular projection was used as the full-sphere image in the above embodiment, the present invention is not limited to this. For example, a cubemap format could be used in which planar images are pasted onto the six faces of a cube-shaped screen, as if a die had been disassembled, so that 360 degrees can be seen correctly when the image is viewed from the center. Alternatively, a dome master format, such as those used in planetariums, could be used. Known technologies can be used for the format.

[0102] Furthermore, in the above embodiment, the display image generation unit 34B performs a process to convert a planar sphere image into a spherical sphere image, but the present invention is not limited thereto. For example, the composite image generation unit 24B may perform a process to convert a planar sphere image into a spherical sphere image, and transmit the composite image RP, represented by the spherical sphere image, to the remote monitoring device 30.

[0103] Furthermore, in the above embodiment, the remote monitoring device 30 is equipped with four display units, the first to fourth display units 32A to 32D, but the present invention is not limited thereto. It may be equipped with fewer than four display units, or more than four display units. The number of display units may be set, for example, according to the number of vehicles 20 to be operated.

[0104] Furthermore, in the above embodiment, as shown in Figure 7, the direction the observer wants to see is changed by moving image A2 in a clockwise or counterclockwise rotation direction, but the present invention is not limited thereto. For example, when the display image HP2 shown in Figure 8 is displayed on the third display unit 32C, the composite image generation unit 24B composites a pointer (not shown) moved by the observer using an input means such as a mouse onto the display image HP2. The display image generation unit 34B may detect the distance and direction of movement of the display image HP2 of this pointer and determine the cropping range in the composite image CP according to the detected distance and direction of movement.

[0105] Furthermore, in the above embodiment, the communication unit 28 has both the function of a transmitting unit for transmitting the composite image CP and the function of a transmitting and receiving unit for transmitting and receiving other information (such as abnormal information and information related to remote control), but the present invention is not limited thereto. The transmitting unit for transmitting the composite image CP may be provided separately from the communication unit 28. In this way, by providing a transmitting unit dedicated to transmitting the composite image CP, it is possible to make the communication processing speed of the composite image CP faster.

[0106] Furthermore, in the above embodiment, the communication unit 38 has both the function of a receiving unit for receiving the composite image CP and the function of a transmitting and receiving unit for sending and receiving other information (such as abnormal information and information related to remote control), but the present invention is not limited thereto. The receiving unit for receiving the composite image CP may be provided separately from the communication unit 38. In this way, by providing a receiving unit specialized for receiving the composite image CP, it is possible to make the communication processing speed of the composite image CP faster.

[0107] Furthermore, although the mobile body is a vehicle 20 in the above embodiment, the present invention is not limited thereto. The mobile body may be, for example, a mobility device that carries luggage or delivered goods, or a self-driving robot, or any other form as long as it is self-driving.

[0108] Although an example of the present invention has been described above, the present invention is not limited to the above, and it is of course possible to implement it in various ways without departing from its spirit. [Explanation of symbols]

[0109] 10 Remote monitoring system 20 Vehicles (mobile vehicles) 22A Front Camera (Shooting Unit) 22B Right front camera (filming unit) 22C Right-side camera (shooting unit) 22D Right rear camera (shooting unit) 22E Rear Camera (Filming Unit) 22F Left rear camera (filming unit) 22G Left-side camera (imaging unit) 22H Left Front Camera (Filming Unit) 22J Downward Camera (Filming Unit) 24B Composite image generation section 28. Communications Section (Transmission Section) 30 Remote monitoring device 34B Display image generation section 38. Communications Section (Receiving Section) P Photographed image P1~P9 Photographed images CP composite image (equirectangular 360-degree spherical image) RP composite image (sphere format) HP display image HP1~HP6 Display Images

Claims

1. A mobile device is equipped with four or more imaging units, each capable of setting its imaging range so that overlapping areas are formed where adjacent imaging areas overlap. A composite image generation unit combines four or more images captured by each of the four or more imaging units by overlapping the overlapping portions, thereby generating a composite image that includes the entire periphery of the moving object in the horizontal direction. A mobile body including a transmission unit that transmits the composite image generated by the composite image generation unit, A receiving unit that receives the composite image transmitted from the moving body, Display unit and A remote monitoring device including a display image generation unit that generates a display image to be displayed on the display unit from the composite image received by the receiving unit, Equipped with, The display image generation unit generates the display image in a viewpoint or display mode desired by the monitor monitoring the display unit of the remote monitoring device, and displays an image indicating the area in the direction the monitor wants to see over the display image, allowing the monitor to change the direction they want to see by moving the image.

2. The remote monitoring system according to claim 1, wherein the display image generation unit generates the display image which is an overhead view taken from a viewpoint desired by the monitor.

3. The remote monitoring system according to claim 1, wherein the display image generation unit generates the display image cropped in the direction desired by the monitor.

4. A mobile device is equipped with four or more imaging units, each capable of setting its imaging range so that overlapping areas are formed where adjacent imaging areas overlap. A composite image generation unit combines four or more images captured by each of the four or more imaging units by overlapping the overlapping portions, thereby generating a composite image that includes the entire periphery of the moving object in the horizontal direction. A mobile body including a transmission unit that transmits the composite image generated by the composite image generation unit, A receiving unit that receives the composite image transmitted from the moving body, Display unit and A remote monitoring device including a display image generation unit that generates a display image from the composite image received by the receiving unit, which is displayed on the display unit and matches the mode set on the mobile device, Equipped with, The aforementioned mobile device is capable of autonomous driving and is configured to allow setting of at least a normal driving mode and an abnormal driving mode. The remote monitoring system includes a display image generation unit that changes the appearance of the display image between the normal driving mode and the abnormal driving mode, generates the display image based on the composite image received by the receiving unit when the normal driving mode is set, and generates the display image based on notification information from the automated driving system and information detected by sensing by sensors mounted on the mobile body when the abnormal driving mode is set.

5. The aforementioned mobile body is configured to allow setting a recovery driving mode in which it starts moving after stopping. The remote monitoring system according to claim 4, wherein the display image generation unit generates the display image in which the lower side of the moving body is visible when the return-to-running mode is set.

6. The remote monitoring system according to claim 1 or 4, wherein each of the four or more captured images is set to include a part of the moving object.

7. The moving body further includes a lower shooting unit whose shooting range is set such that the area below the moving body, including a part of the moving body, becomes the shooting area. The remote monitoring system according to claim 1 or 4, wherein the composite image generation unit also combines the lower side image captured by the lower side imaging unit with the composite image.

8. The remote monitoring system according to claim 1 or claim 4, wherein the composite image is a 360-degree spherical image.

9. The remote monitoring system according to claim 1 or 4, wherein the display image generation unit generates the display image based on the driving status of the moving body.

10. The remote monitoring system according to claim 1, wherein the display image generation unit generates the display image according to the mode set on the mobile body.

11. The remote monitoring system according to claim 1 or claim 4, wherein the composite image has a vertical-to-horizontal ratio of 1:2.

Images