Bird's-eye view video presentation system

Abstract

To display an overhead image which allows an operator to more properly grasp an image of the periphery.SOLUTION: An overhead image presentation system comprises: a movable device which can be remotely controlled; a plurality of cameras which are installed in the movable device and image a periphery at a wide angle; a detection device which detects an object in the periphery of the movable device; and an information processing device which can perform data communication with the movable device. The information processing device comprises: an overhead image generation unit which generates an overhead image by performing view point conversion of the image acquired by the camera into an image viewed from a virtual overhead view point at which the movable device is viewed from above; a storage unit which stores 3D image data of the object; a 3D image creation unit which identifies a position and posture of the object from the detection result of the detection device and creates a 3D image with the position and posture matching the object by using the 3D image data of the storage unit; and a synthesizing unit which creates an image in which the 3D image is superimposed on the overhead image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an overhead image presentation system. [Background technology]

[0002] In recent years, with the use of remotely controlled mobile devices (hereinafter referred to as "remotely controlled mobile devices") in disaster recovery efforts, active development of image presentation technologies for remotely controlled mobile devices has been underway to improve the efficiency of recovery efforts. For example, the bird's-eye view image presentation system described in Patent Document 1 mounts multiple cameras with wide-angle lenses on the remotely controlled mobile device and processes images acquired from the multiple cameras to present a pseudo-bird's-eye view of the remotely controlled mobile device from a third-person perspective above the device. The bird's-eye view image presentation system described in Patent Document 1 also describes placing markers in the target space and performing correction processing on the bird's-eye view image based on the distance to the markers. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-161895 Summary of the Invention [Problem to be solved by the invention]

[0004] The overhead view image presentation system described in Patent Document 1 can correct the depth direction position of the overhead view image by correcting the overhead view image based on the detection results of the markers, thereby making the overhead view image easier to view.

[0005] However, even when using the overhead image described in Patent Document 1, if there is another object, such as an obstacle or an object to be worked on by the remote mobile device, near the reference position, for example, the remote mobile device to be operated, it may be difficult to accurately grasp the position of the other object, and the position may not be accurately conveyed to the operator of the remote mobile device.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an overhead image presentation system that can display an overhead image that allows an operator to more appropriately grasp an image of the surroundings. [Means for solving the problem]

[0007] In order to achieve the above object, one embodiment of the bird's-eye view video presentation system of the present disclosure comprises a mobile device that can be remotely controlled, including multiple cameras that capture wide-angle images of the surroundings and a detection device that detects surrounding objects, and an information processing device that can communicate data with the mobile device, wherein the information processing device comprises: a bird's-eye view video generation unit that generates a bird's-eye view video by converting the image acquired by the cameras into an image seen from a virtual bird's-eye view point that overlooks the mobile device; a memory unit that stores 3D image data of the object; a 3D image creation unit that identifies the position and posture of the object from the detection result of the detection device and creates a 3D image that matches the position and posture of the object using the 3D image data in the memory unit; and a synthesis unit that creates an image by superimposing the 3D image on the bird's-eye view video.

[0008] In a preferred embodiment of the bird's-eye view video presentation system, the detection device includes a sensor that is installed on the mobile device and detects a relative relationship between the mobile device and surrounding objects.

[0009] In a preferred embodiment of the bird's-eye view video presentation system, the detection device communicates with the object and acquires position information and orientation information of the object.

[0010] In a preferred embodiment of the overhead view video presentation system, the mobile device includes four of the cameras.

[0011] In a preferred embodiment of the overhead image presentation system, the 3D image creation unit creates a three-dimensional image of the mobile device, and the synthesis unit superimposes the three-dimensional image of the mobile device on the overhead image. [Effects of the Invention]

[0012] According to the present disclosure, it is possible to display an overhead image that allows the operator to more appropriately grasp the surrounding image. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an overhead image presentation system according to an embodiment. [Figure 2] FIG. 2 is a block diagram schematically illustrating the functional configuration of the overhead image presentation system according to the embodiment. [Figure 3] FIG. 3 is a flowchart illustrating an example of processing performed by the overhead view video presentation system. [Figure 4] FIG. 4 is a flowchart showing an example of a 3D image creation process of the overhead view video presentation system. [Figure 5] FIG. 5 is a schematic diagram showing an example of an overhead view image. [Figure 6] FIG. 6 is a diagram showing the concept of each coordinate system according to the embodiment. [Figure 7] FIG. 7 is a block diagram schematically showing the functional configuration of an overhead image presentation system according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Below, embodiments of the bird's-eye view video presentation system according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the following embodiments. Furthermore, the components in the following embodiments include those that are replaceable and easy for those skilled in the art, or those that are substantially identical. Furthermore, the components in the embodiments described below can be variously omitted, replaced, or modified without departing from the gist of the present invention. In the following embodiments, components required to illustrate one embodiment of the bird's-eye view video presentation system according to the present invention will be described, and other components will be omitted.

[0015] [System Configuration] FIG. 1 is a diagram illustrating an example of the configuration of an overhead image presentation system according to an embodiment. As illustrated in FIG. 1, an overhead image presentation system 10 according to an embodiment presents an overhead image for controlling and monitoring a remotely operable mobile device (remote mobile device) 14. The overhead image presentation system 10 includes an information processing device 12 capable of data communication with the mobile device 14, a remote control device 13 capable of data communication with the mobile device 14, the mobile device 14, a distance sensor 18, and a camera unit 20. In addition, in the overhead image presentation system 10 of this embodiment, a dump truck 8 is placed near the mobile device 14. The information processing device 12, the remote control device 13, the mobile device 14, the distance sensor 18, and the camera unit 20 are connected to a communication network in a state where they can communicate data with each other, enabling data transmission and reception. The communication network may be constructed to include a public communication line, a dedicated communication line, or the like. The distance sensor 18 and the camera unit 20 may be connected to a communication unit of the mobile device 14, and may transmit and receive data to and from other devices via the communication unit of the mobile device 14.

[0016] In this embodiment, the moving device 14 is a hydraulic excavator. A distance sensor 18 and a camera unit 20 are installed on the moving device 14. The moving device 14 is a device that can be remotely controlled by the remote control device 13.

[0017] The distance sensor (detection device) 18 is provided in the direction in which the mobile device 14 works (the direction in which the bucket is installed), for example, in the direction of travel. The distance sensor 18 detects the distance to an object in the direction in which the mobile device 14 works and the posture of the object. The distance sensor 18 detects an object in the direction in which the mobile device 14 works and acquires the distance to each part of the object (depth information in the depth direction). The distance sensor 18 can be, for example, an infrared sensor (a sensor that emits infrared light as detection light and detects the reflection to detect the distance to the object) or LiDAR. The distance sensor 18 may be a stereo camera that simultaneously captures images of an object from multiple different directions to record depth information. It is sufficient for the distance sensor 18 to capture a depth image including at least depth information. The distance sensor 18 may be, for example, an RGB-D sensor. The RGB-D sensor is a sensor that captures RGB images in addition to depth images. The RGB-D sensor includes a twin-lens infrared camera that captures depth images and a single-lens RGB camera that captures RGB images. The horizontal field of view of the distance sensor 18 is, for example, 88.2 degrees or more and 94.2 degrees or less. The vertical field of view is, for example, 62.5 degrees or more and 68.5 degrees or less. The distance sensor 18 can acquire, as three-dimensional information, information on a three-dimensional point cloud, which is a collection of three-dimensional points indicating the position of the surface of an object. Each three-dimensional point constituting the three-dimensional point cloud acquired by the distance sensor 18 is represented by a three-dimensional coordinate value indicating the position of the surface of the object. The three-dimensional coordinate value is composed of coordinate values ​​corresponding to the X-axis, Y-axis, and Z-axis directions at an arbitrary origin.

[0018] The camera unit 20 has four cameras 22, 24, 26, and 28. The cameras 22, 24, 26, and 28 capture images of the surroundings (surrounding environment) of the mobile device 14 at a wider angle than a predetermined capture range. The cameras 22, 24, 26, and 28 are equipped with fisheye lenses with a wide angle of view of, for example, about 180 degrees. In this embodiment, the cameras 22, 24, 26, and 28 are provided on the front, rear, left, and right sides of the working direction of the mobile device 14. The positions of the four cameras 22, 24, 26, and 28 are not particularly limited as long as they can capture images of the entire area around the mobile device 14. When installing cameras 22, 24, 26, and 28 on mobile device 14, they can be installed based on the arrangement proposed in, for example, "Komatsu Ren, Fujii Hiromitsu, Yamashita Jun, Asama Hajime: Development of an overhead image presentation system for remote robot operation in preparation for failures through camera placement design, Journal of the Japan Society for Precision Engineering, 81, 12 (2015) 1206."

[0019] The remote control device 13 remotely controls the moving device 14. The remote control device 13 may be a device integrated with the information processing device 12. The remote control device 13 communicates with the moving device 14 and controls the movement of the moving device 14 and the operation of the work mechanisms (arm, boom, bucket). The method of remotely controlling the remote control device 13 is not particularly limited, and for example, a mechanism similar to the driver's seat of the moving device 14 may be provided to detect input operations, or operations input via a keyboard, mouse, etc. may be detected.

[0020] Next, the information processing device 12 will be described in detail using Fig. 2. The information processing device 12 generates an overhead image that shows the positions of objects around the mobile device 14, using video data received from the mobile device 14 and information from the distance sensor 18. The information processing device 12 includes an image acquisition unit 40, a surrounding information acquisition unit 42, an input unit 44, a display unit 46, a calculation unit 48, and a storage unit 50. The information processing device 12 also includes a communication function for communicating with each unit.

[0021] The image acquisition unit 40 communicates with each of the cameras 22, 24, 26, and 28 of the camera unit 20, and acquires image data acquired by the cameras 22, 24, 26, and 28. The image acquisition unit 40 sends the acquired image data to the calculation unit 48. The surrounding information acquisition unit 42 communicates with the distance sensor 18, and acquires information about the surroundings of the mobile device 14 acquired by the distance sensor 18. The surrounding information acquisition unit 42 sends the data acquired by the distance sensor 18 to the calculation unit 48. The input unit 44 is used by an operator who operates the information processing device 14 to input operations. The input unit 44 is a keyboard, a mouse, a touch panel, or the like.

[0022] The display unit 46 displays various information to an operator who operates the remote control device 13 and the information processing device 14. The display unit 46 can display an overhead image generated by the calculation unit 48, which will be described later. The display unit 46 may include a display device such as a liquid crystal display (LCD), an organic electro-luminescence display (ELD), or an inorganic electro-luminescence display (IELD). The display unit 46 may include an input device such as a touch screen.

[0023] The calculation unit 48 executes various processes related to the mobile device 14 based on the programs and data stored in the storage unit 50. In particular, in this embodiment, the calculation unit 48 executes a process for generating an overhead image that displays the positions of objects around the mobile device 14. The calculation unit 48 may be implemented by including a processor such as a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), or a system large-scale integration (LSI). The calculation unit 48 reads a program stored in the storage unit 50, loads it into a working memory such as a RAM, and causes a processor such as a CPU to execute instructions included in the program loaded into the working memory. This allows the calculation unit 48 to execute various processes based on each function. The calculation unit 48 includes a bird's-eye image creation unit 52, an object extraction unit 54, a 3D image creation unit 56, and a synthesis unit 58. The components of the calculation unit 48 will be described later.

[0024] The storage unit 50 stores programs and data for implementing various processes executed by the calculation unit 48. Functions provided by the programs stored in the storage unit 50 include a function for generating an overhead image that displays the positions of objects around the mobile device 14. Data stored in the storage unit 50 includes image data and three-dimensional information received from the mobile device 14. The storage unit 50 may be implemented by a non-volatile or volatile semiconductor memory such as a random access memory (RAM), a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM®), a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, or a DVD.

[0025] The storage unit 50 includes an image creation program 62, object data 64, and 3D image data 66. When executed by the calculation unit 48, the image creation program 62 realizes the functions of the bird's-eye view image creation unit 52, the object extraction unit 54, the 3D image creation unit 56, and the synthesis unit 58. The object data 64 is information about various objects assumed to be around the mobile device 14. The object data 64 includes information about the dump truck 8. The information about the dump truck 8 includes information necessary to identify the dump truck 8, such as shape, color, and size. The object data 64 may also include information about objects other than the dump truck 8 that are assumed to be around the mobile device 14, such as work machinery, obstacles, buildings, and objects. The 3D image data 66 is data about the three-dimensional shape of the object stored in the object data 64. In other words, the 3D image data 66 includes data about the three-dimensional shape (solid shape) of the dump truck 8.

[0026] Next, the function of each unit of the calculation unit 48 will be described. The overhead image creation unit 52 generates a first overhead image by performing viewpoint conversion on the image data received from the mobile device 14, i.e., the images (camera images) acquired by the cameras 22, 24, 26, and 28, into an image seen from a virtual overhead viewpoint that overlooks the mobile device 14. Under the assumption that all objects captured by the cameras 22, 24, 26, and 28 of the mobile device 14 exist on a certain plane in the robot coordinate system, the overhead image creation unit 52 generates the overhead image by perspectively projecting the camera images onto that plane and then performing viewpoint conversion processing. The overhead image is, for example, an image in which an image is pasted on the surface of a hemisphere (the surface of the hemisphere and the ground). The overhead image creation unit 52 can generate an overhead image by using the conventional method proposed in "Takaaki Sato, Hiromitsu Fujii, Alessandro Moro, Jun Yamashita, Hajime Asama: Construction of a method for presenting superimposed omnidirectional overhead images using multiple fisheye cameras and LRF, Proceedings of the 13th Conference of the System Integration Division of the Society of Instrument and Control Engineers (2012)." The overhead image creation unit 52 may place a mark in the area where the overhead image is to be created, acquire an image including the mark with the camera unit 20, and create an overhead image that has been corrected based on the detected position of the mark, as described in Japanese Patent Application Laid-Open No. 2020-161865.

[0027] The object extraction unit 54 extracts objects contained in the space formed by the overhead video image based on the detection results of the distance sensor 18. Specifically, the object extraction unit 54 extracts objects in the vicinity of the mobile device 14. The object extraction unit 54 compares the extracted object with the object data 64 and determines whether the extracted object is a registered object. If the extracted object is registered, the object extraction unit 54 extracts the object as an object. The object extraction unit 54 also identifies the position and orientation of the object.

[0028] The 3D image creation unit 56 reads out the 3D image of the object extracted by the object extraction unit 54 from the 3D image data 66, and processes it to correspond to the position and posture of the object extracted by the object extraction unit 54.

[0029] The synthesis unit 58 superimposes the 3D image created by the 3D image creation unit 56 on the overhead image created by the overhead image creation unit 52, and creates an image in which the object of the 3D image is placed inside the overhead image.

[0030] Next, the processing of the overhead-view video presentation system 10 will be described with reference to Figs. 3 to 6. Fig. 3 is a flowchart showing an example of the processing of the overhead-view video presentation system. Fig. 4 is a flowchart showing an example of the processing of creating a 3D image of the overhead-view video presentation system. Fig. 5 is a schematic diagram showing an example of an overhead video. Fig. 6 is a diagram showing the concept of each coordinate system according to the embodiment. The processing shown in Figs. 3 and 4 can be realized by executing the processing in the information processing device 12 that acquires information from each unit. The information processing device 12 repeatedly executes the processing in Fig. 3 and updates the overhead image, thereby creating an overhead video in which the displayed image changes in accordance with the surrounding situation and the movement of the mobile device 14.

[0031] 3, information processing device 12 acquires images from cameras 22, 24, 26, and 28 using image acquisition unit 40 (step S12). Next, information processing device 12 creates an overhead image using overhead image creation unit 52 (step S14). Specifically, the images acquired by the multiple cameras 22, 24, 26, and 28 are combined to create an overhead image in which images are pasted on the wall surfaces including the hemisphere and the bottom surface.

[0032] The information processing device 12 determines whether there is an object (step S16). The information processing device 12 determines, by the object extraction unit 54, whether there is an object to be superimposed on the overhead view video.

[0033] The process of determining whether or not an object is present will be described with reference to Fig. 4. The information processing device 12 may execute the process of Fig. 3 and the process of Fig. 4 in parallel, or may execute the process of Fig. 4 as the determination process of step S16 in Fig. 3.

[0034] The information processing device 12 determines whether an object is included in the overhead-view video (step S32). The information processing device 12, for example, analyzes the image of the camera unit 20 to determine whether an object is included. The information processing device 12 may determine whether an object is present in the surroundings based on the result of detection by the distance sensor 18. Note that the determination in step S32 is a determination as to whether an object is included in the overhead-view video, and if an object is included in the overhead-view video, an object that is a predetermined distance away from the mobile device 14 is also detected.

[0035] If the information processing device 12 determines that there is no target object (No in step S32), it notifies the operator that there is no target object (step S34) and returns to step S32. An example of a notification method is to notify the operator by a message on the display unit 46, but this is not limiting. Instead of notifying the operator that there is no target object, the information processing device 12 may output a notification instructing the operator to approach the target object.

[0036] If the information processing device 12 determines that an object is present (Yes in step S32), it detects the type of the object (step S36). The information processing device 12 analyzes the object detected by the object extraction 54, compares it with the object data 64, determines whether the detected object matches the object in the object data 64, and identifies the object.

[0037] The information processing device 12 acquires 3D image data of the identified object (step S38). The object extraction unit 54 of the information processing device 12 acquires data on the three-dimensional shape of the object from the 3D image data.

[0038] Next, the information processing device 12 acquires data from the distance sensor 18 (step S40). The information processing device 12 acquires the data acquired from the distance sensor 18 by the surrounding information acquisition unit 42.

[0039] Next, the information processing device 12 determines whether the position and posture of the object can be identified (step S42). The information processing device 14 accumulates position information corresponding to the object from data from the distance sensor 18 in the object extraction unit 54, and determines whether the shape of the object can be identified. Specifically, the object extraction unit 54 analyzes the point cloud of the object area (detection results at each position of the distance sensor 18), and determines whether a sufficient number of points have been detected to identify the shape of the object.

[0040] If the information processing device 12 determines that the position and posture of the object cannot be identified (No in step S42), it instructs the operator to move the mobile device (step S44). For example, it instructs the operator to approach the object.

[0041] If it is determined that the position and orientation of the object can be identified (Yes in step S42), the information processing device 12 determines the orientation and composition position of the 3D image data of the object (step S46). That is, the object extraction unit 54 determines the position and orientation of the object based on the detection result of the distance sensor 18, and determines the identified orientation and position as the orientation and composition position of the 3D image data of the object in the overhead video.

[0042] 3, the processing of information processing device 12 will be described. If information processing device 12 determines that there is no object (No in step S16), it does not display a 3D image of the object and ends this processing. In this case, information processing device 12 causes display unit 46 to display an overhead video that does not include a 3D image of the object.

[0043] If the information processing device 12 determines that an object is present (Yes in step S16), it creates a 3D image of the object (step S18). Specifically, the information processing device 14 reads out a 3D image of the object from the 3D image data 66 using the 3D image creation unit 56, and processes it into an image corresponding to the posture and position extracted by the object extraction unit 54.

[0044] After creating the 3D image, information processing device 12 superimposes the 3D image on the overhead video (step S20), thereby causing display unit 46 to display the overhead video including the 3D image of the object.

[0045] The information processing device 12 displays an overhead view image 100 as shown in FIGS. 5 and 6. The viewpoint of the overhead view image 100 can be changed to any position, and the image can be displayed from any position within the overhead view image 100. The overhead view image 100 includes a wall image 101 of a hemisphere and a bottom surface, and 3D images 104 and 106 arranged in a space surrounded by the wall image 101. The wall image 101 is an image created by the overhead view image creation unit 52, and the 3D images 104 and 106 are images created by the 3D image creation unit 56. The 3D image 104 is a stereoscopic image of the mobile device 14. The 3D image 106 is a stereoscopic image of the dump truck 8. The 3D images 104 and 106 are stereoscopic images extracted from the 3D image data 66. The 3D image 104 is arranged at the center of the wall image 101. The 3D image 106 is displayed based on the position and orientation acquired by the object extraction unit 54. The 3D images 104, 106 include information on three-dimensional shapes in all directions, and the appearance of the 3D images 104, 106 on the screen changes depending on the position of the viewpoint.

[0046] Here, the relative position between the wall image (bird's-eye view image) created based on the image from the camera unit 20 and the 3D image whose position and attitude are identified based on the results from the distance sensor 18 will be described. The bird's-eye view image presentation system 10 sets a bird's-eye view image coordinate system 120, a range measurement sensor coordinate system 122, and a dump truck coordinate system 124. The bird's-eye view image coordinate system 120 is a coordinate system set in the space of the wall image 101 corresponding to the detection results from the camera unit 20, and the bird's-eye view image coordinate system 120 includes position information of the mobile device 14 and the camera unit 20. The range measurement sensor coordinate system 122 is a coordinate system associated with the results detected by the distance sensor 18, and also includes position information of the distance sensor 18 relative to the mobile device 14. The dump truck coordinate system 124 is a coordinate system set based on the reference position of the dump truck 8.

[0047] In the range sensor coordinate system 122, the relative position of the distance sensor 18 and the camera unit 20 installed on the mobile device 14 is known. The information processing device 12 converts the position information of the range sensor coordinate system 122 into position information of the overhead image coordinate system 120 based on information about the relative positions of the distance sensor 18 and the camera unit 20. Furthermore, the position of the dump truck 8 in the range sensor coordinate system 122 is identified by detecting the position of the dump truck 8 using the range sensor coordinate system 122. Based on information about the position of the dump truck 8 in the range sensor coordinate system 122, the dump truck coordinate system 124 is converted into position information of the range sensor coordinate system 122, and further converted into position information of the overhead image coordinate system 120 based on the relationship between the overhead image coordinate system 120 and the range sensor coordinate system 122. As a result, the information processing device 12 identifies the position of the dump truck 8 in the overhead image coordinate system 120, and can display a three-dimensional 3D image 106 on the overhead image 100.

[0048] As described above, the overhead image presentation system 10 includes a remotely controlled mobile device 14, a plurality of cameras 22, 24, 26, and 28 that are installed on the mobile device 14 and capture wide-angle images of the surroundings, a distance sensor (detection device 18) that detects objects around the mobile device 14, and an information processing device 12 that is capable of data communication with the mobile device 14. The information processing device 12 also includes a bird's-eye view image generation unit 52 that generates an overhead image by converting the images acquired by the cameras 22, 24, 26, and 28 into images seen from a virtual bird's-eye view point that overlooks the mobile device, a memory unit 50 that stores 3D image data of the object, a 3D image creation unit 56 that identifies the position and orientation of the object from the detection results of the distance sensor and creates a 3D image of the position and orientation of the object from the 3D image data in the memory unit 50, and a composition unit 58 that creates an image by superimposing the 3D image on the bird's-eye view image (wall image).

[0049] This allows the overhead-view image presentation system 10 to display a 3D image of the target object within the overhead-view image. Furthermore, by displaying a 3D image 106 of the target dump truck 8 within the overhead-view image, the bird's-eye view image can be generated, allowing the relative positions of the mobile device 14 and the target dump truck 8 to be properly grasped. Furthermore, the overhead-view image presentation system 10 can easily create a three-dimensional shape based on 3D image data 66 prepared in advance. Furthermore, even when the viewpoint of the overhead-view image is changed and an image in a direction that is not visible from the mobile device 14 is displayed, the shape of the target object from the determined viewpoint can be grasped. In other words, since the shape of the target object in a direction that is not visible from the distance sensor 18 and camera unit 20 can also be reproduced using 3D image data 66, the three-dimensional shape of the target object can be displayed in the overhead-view image from any viewpoint.

[0050] Furthermore, in the overhead view video presentation system 10 of the above embodiment, by displaying a 3D image 106 corresponding to the position of the mobile device 14, the relative relationship between the mobile device being operated and the target object can be more accurately grasped. The 3D image 104 of the mobile device 14 can be displayed by storing shape data in the 3D image data and reading it out. Furthermore, the orientation of the mobile device 14 may be identified based on the control information of the mobile device 14 or the image of the camera unit 20, and the shape of the 3D image 104 may be corrected based on the orientation to create a 3D image 104 that corresponds to the actual orientation.

[0051] Furthermore, in the above embodiment, the overhead-view image presentation system 10 has been described as including a single dump truck 8 as the target object, but the target object is not limited to a dump truck. The target object data can include various objects stored in 3D image data. The target object displayed in the overhead-view image is not limited to one. Multiple targets may be detected by the distance sensor 18 (detection device), and 3D images of the multiple targets may be displayed at multiple positions in the overhead-view image.

[0052] Although the embodiments of the present invention have been described above, the embodiments are not limited to the contents of these embodiments. For example, two or more detection devices may be provided. Furthermore, in this embodiment, cameras 22, 24, 26, and 28 are fisheye cameras equipped with fisheye lenses, but stereo cameras may also be used.

[0053] In the above embodiment, the distance sensor 18 is used as a detection device for identifying the position and posture of the object, but the present invention is not limited to this.

[0054] FIG. 7 is a block diagram schematically illustrating the functional configuration of an overhead-view video presentation system according to another embodiment. In the overhead-view video presentation system 10a shown in FIG. 7, components similar to those in the overhead-view video presentation system 10 are denoted by the same reference numerals, and detailed description thereof will be omitted. The overhead-view video presentation system 10a includes an information processing device 12a, a mobile device 14, and a camera unit 20. The overhead-view video presentation system 10a includes, as detection devices, a communication unit 272, a GNSS sensor 274, an attitude detection unit 276, a communication unit 282, a GNSS sensor 284, and an attitude detection unit 286. The communication unit 272, the GNSS sensor 274, and the attitude detection unit 276 are installed in the mobile device 14. The communication unit 282, the GNSS sensor 284, and the attitude detection unit 286 are installed in the dump truck 8. Each of the GNSS sensors 274 and 284 is configured with two GNSS sensors. For example, the two GNSS sensors installed in the mobile device 14 are installed at specified positions on the mobile device 14, and therefore can acquire position information and attitude information of the mobile device 14.

[0055] The communication unit 272 communicates with the surrounding information acquisition unit 42. The communication unit 272 communicates using a public communication network. Note that if the information processing device 12a and the mobile device 14 are within a predetermined distance, the communication unit 272 may communicate using short-range wireless communication. The GNSS sensor 274 detects the position of the mobile device 14 on Earth using a Global Navigation Satellite System that uses artificial satellites (positioning satellites). The attitude detection unit 276 detects the attitude of the mobile device 14. The attitude detection unit 276 may detect the attitude of the mobile device 14 by acquiring operation information and control information of the mobile device 14, or may detect the attitude of the mobile device 14 by detecting the external shape of the mobile device 14 using a camera, sensor, etc.

[0056] The communication unit 282 communicates with the surrounding information acquisition unit 42. The communication unit 282 communicates using a public communication network. Note that, when the information processing device 12a and the dump truck 8 are within a predetermined distance, the communication unit 282 may communicate using short-range wireless communication. The GNSS sensor 284 detects the position of the dump truck 8 on the Earth using a Global Navigation Satellite System that uses artificial satellites (positioning satellites). The attitude detection unit 286 detects the attitude of the dump truck 8. The attitude detection unit 286 may detect the attitude of the mobile device 14 by acquiring operation information and control information of the mobile device 14, or may detect the attitude of the mobile device 14 by detecting the external shape of the dump truck 8 using a camera, sensor, etc.

[0057] The information processing device 12a acquires, via the peripheral information acquisition unit 42, position information and attitude information of the mobile device 14 in the GNSS system, and position information and attitude information of the dump truck 8 in the GNSS system. By acquiring the position information (coordinate information) of the mobile device 14 and the dump truck 8 in the GNSS system, the information processing device 12a detects the relative positions of the mobile device 14 and the dump truck 8, and converts the detected results into an overhead image coordinate system, thereby determining the position of the dump truck 8 on the overhead image. Furthermore, the information processing device 12a can identify the orientation and external shape of the dump truck 8 based on the attitude information, and can correct the 3D image.

[0058] The overhead image presentation system 10a can display a 3D image of an object on the overhead image even if a detection device uses a method for detecting coordinates on the earth, such as a GNSS system. The detection device only needs to be able to detect the relative position between the mobile device 14 and the object, and a method for detecting the relative position through communication between the mobile device 14 and the object may also be used.

[0059] The overhead-view image presentation system 10a may determine whether to communicate with an object based on the results of detecting the presence or absence of an object within the display range of the overhead-view image. In other words, the system may selectively communicate with objects within the display area to detect their relative positions. This prevents an increase in processing load even when there are many candidates for objects that may intrude into the display range of the overhead-view image of the mobile device 14. [Explanation of symbols]

[0060] 8. Dump 10, 10a Bird's-eye view image presentation system 12, 12a Information processing device 13 Remote control device 14 Mobile Devices 18 Distance sensor (detection device) 20 Camera Unit 22, 24, 26, 28 Cameras 40 Image acquisition unit 42 Peripheral information acquisition unit 44 Input section 46 Display section 48 Arithmetic section 50 Storage section 52 Bird's-eye view video production department 54 Object extraction unit 56 3D Image Creation Department 58 Synthesis section 62 Video Creation Program 64 Object Data 66 3D image data 272, 282 Communications Department 286, 284 GNSS sensors 276, 286 Attitude detection unit

Claims

1. a remotely controlled mobile device; a plurality of cameras installed on the mobile device for capturing wide-angle images of the surroundings; a detection device for detecting objects around the mobile device; an information processing device capable of data communication with the mobile device; The information processing device includes: an overhead image generating unit that generates an overhead image by converting an image acquired by the camera into an image viewed from a virtual overhead viewpoint that overlooks the mobile device; a storage unit that stores 3D image data of an object; a 3D image creation unit that identifies the position and orientation of an object from the detection result of the detection device, and creates a 3D image that matches the position and orientation of the object using the 3D image data of the storage unit; a synthesis unit that creates an image by superimposing the 3D image on the overhead image.

2. The overhead image presentation system according to claim 1 , wherein the detection device includes a sensor that is installed on the mobile device and detects a relative relationship between the mobile device and an object around the mobile device.

3. The bird's-eye view video presentation system according to claim 1 , wherein the detection device communicates with the object and acquires position information and orientation information of the object.

4. The overhead image presentation system according to claim 1 , wherein the mobile device includes four of the cameras.

5. the 3D image creation unit creates a three-dimensional image of the mobile device; The overhead view video presentation system according to claim 1 , wherein the synthesizing unit superimposes a three-dimensional image of the mobile device on the overhead view video.

Images