Information display system, information display method, and program

Abstract

An information display system 100 according to the present disclosure is provided with: an acquisition unit 121 for acquiring position information of a display device for displaying a real-space image, an image-capturing device for capturing an image of a predetermined object, and an object in a captured image captured by the image-capturing device; a position alignment unit 122 for aligning the position of the object in the captured image with the position of the display device on the basis of the position information; a generation unit 123 for generating a virtual image of the object on the basis of the position information of the object; and a display control means 124 for controlling the display device so as to display the virtual image of the object aligned with the position of the display device in a superimposed manner on the real-space image.

Description

Information display system, information display method, and program

[0001] The present disclosure relates to an information display system, an information display method, and a program.

[0002] Based on the Aviation Act, restricted surfaces are set around airports to ensure the safe takeoff and landing of aircraft. For this reason, in certain areas around airports, the installation of buildings or trees that extend above the restricted surfaces is prohibited, and monitoring of properties that exceed the restricted surfaces is required.

[0003] As an example of a system used for monitoring operations, Patent Document 1 describes a system using a head-mounted display. Specifically, Patent Document 1 first stores three-dimensional structure data that represents a monitored object in a virtual three-dimensional space. Then, image data is collected from a camera installed within the monitored area, and detection result data indicating an abnormality detected at the monitored location is obtained from the image data. Positioning result data including position data and orientation data for the head-mounted display is then obtained. Three-dimensional integrated data that integrates all positional relationships is then generated based on the three-dimensional structure data, the detection result data, and the positioning result data, and a virtual image is displayed on the head-mounted display during monitoring operations.

[0004] JP 2012-239068 A

[0005] However, the system described in Patent Document 1 simply displays the results of detection of abnormalities in the monitored object, and it is not easy for the monitor to recognize the position of the monitored object in the real space image, which causes a problem that the efficiency of monitoring work performed on site cannot be improved.

[0006] Therefore, an object of the present disclosure is to provide an information display system that can solve the above-mentioned problem of not being able to improve the efficiency of on-site monitoring work.

[0007] An information display system according to one embodiment of the present disclosure includes: a display device that displays a real space image; an imaging device that images a predetermined object; and an acquisition unit that acquires positional information of the object in the image captured by the imaging device; an alignment unit that matches the position of the object in the image to the position of the display device based on the positional information; a generation unit that generates a virtual image of the object based on the positional information of the object; and a display control means that controls the display device to display the virtual image of the object, which has been matched to the position of the display device, superimposed on the real space image.

[0008] Furthermore, an information display method according to one embodiment of the present disclosure includes acquiring positional information of a display device that displays a real space image, an imaging device that images a predetermined object, and the object in an image captured by the imaging device, corresponding the position of the object in the image to the position of the display device based on the positional information, generating a virtual image of the object based on the positional information of the object, and controlling the display device to display the virtual image of the object, corresponding to the position of the display device, superimposed on the real space image.

[0009] Furthermore, a program according to one embodiment of the present disclosure has a configuration that causes a computer to execute the following processes: acquire positional information of a display device that displays a real space image, an imaging device that images a predetermined object, and the object in an image captured by the imaging device; correspond the position of the object in the image to the position of the display device based on the positional information; generate a virtual image of the object based on the positional information of the object; and control the display device to display the virtual image of the object, which has been corresponded to the position of the display device, superimposed on the real space image.

[0010] With the above-described configuration, the present disclosure can improve the efficiency of on-site monitoring work.

[0011] FIG. 1 is a diagram illustrating an overview of an information display system according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a display image on a user terminal disclosed in FIG. 1. FIG. 3 is a diagram illustrating an example of alignment between a user terminal and an imaging device disclosed in FIG. 1. FIG. 4 is a block diagram illustrating the overall configuration of the information display system disclosed in FIG. 1. FIG. 5 is a block diagram illustrating the configuration of a user terminal disclosed in FIG. 1. FIG. 6 is a block diagram illustrating the configuration of an information processing server disclosed in FIG. 1. FIG. 7 is a flowchart illustrating the operation of the information processing server disclosed in FIG. 1. FIG. 8 is a flowchart illustrating the operation of the user terminal disclosed in FIG. 1. FIG. 9 is a diagram illustrating another example of a display image on a user terminal disclosed in FIG. 1. FIG. 10 is a diagram illustrating another example of a display image on a user terminal disclosed in FIG. 1. FIG. 11 is a block diagram illustrating the hardware configuration of an information display system according to a second embodiment of the present disclosure. FIG. 12 is a block diagram illustrating the configuration of an information display system according to a second embodiment of the present disclosure.

[0012] First Embodiment A first embodiment of the present disclosure will be described with reference to Fig. 1 to Fig. 10. First, an overview of an information display system according to this embodiment will be described.

[0013] [Overview] As shown in FIG. 1 , the information processing system of this embodiment includes a user terminal 20, which is a display device used by a user P; a sensor 40, which is an imaging device that captures images of a predetermined object; and an information processing server 10 that processes the displayed image. The information processing system 10 of this embodiment is used by the user P to monitor buildings and trees that protrude beyond a restricted surface in the vicinity of an airport. For example, as shown in FIG. 1 , the user P wears the user terminal 20, which is a head-mounted display, and views the area around the airport that is the location to be monitored, through the user terminal 20. Then, a real space image, such as that shown in the range indicated by the symbol Rd in FIG. 1 , is displayed on the display unit A of the user terminal 20, which is a head-mounted display. The real space image displayed on the user terminal 20 may be an image of the real space viewed through the display unit A, or may be an image of the real space converted into digital data and displayed on the display unit A.

[0014] The sensor 40 then acquires imaging data 30, which is an image captured around the airport, as shown in the range indicated by the symbol Rc in Fig. 1. As will be described later, the imaging data 30 includes three-dimensional point cloud data, which is position information indicating the three-dimensional coordinates of trees T1 and T2 that are the targets of monitoring and are present within the imaging range. This imaging data 30 is transmitted to the information processing server 10 via the user terminal 20 or directly from the imaging device 40.

[0015] The information processing server 10 generates virtual images V1, V2 corresponding to each object from three-dimensional point cloud data included in the imaging data 30 captured by the sensor 40. The information processing server 10 also acquires position information of the user terminal 20 and the sensor 40 and aligns them. That is, the information processing server 10 aligns the position of the imaging data 30 captured by the sensor 40 to correspond to the position of the user terminal 20 based on the difference in position between the user terminal 20 and the sensor 40. The information processing server 10 then transmits virtual images V1, V2 based on the aligned imaging data 30 to the user terminal 20 and controls the display of the virtual images V1, V2 superimposed on the real space image displayed on the user terminal 20.

[0016] As a result, the virtual images V1 and V2 aligned with the real space image in the range indicated by the symbol Rd are displayed on the display unit A of the user terminal 20, as shown in Fig. 2. Each component and operation will be described in detail below with reference to Figs.

[0017] 4 is a system configuration diagram of the entire information processing system. The user terminal 20 acquires photographed data 30 captured by a sensor 40, and shares data with other user terminals 20 in real time via the information processing server 10.

[0018] Here, the sensor 40 is a 3D sensor (three-dimensional sensor) such as a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a ToF camera (Time of Flight Camera), or a stereo camera. However, the sensor 40 is not limited to a 3D sensor, and may be a hyperspectral camera, an RGB camera, or other sensors.

[0019] The user terminal 20 is MR glasses (Mixed Reality Glass) such as a head-mounted display. Therefore, the user terminal 20 is configured to display a real space image and to superimpose virtual image data on the real space image. The user terminal 20 is not limited to MR glasses, but may be any other eyeglass-type device such as VR glasses (Virtual Reality Glass), or any information processing terminal having a display, such as a personal computer, a tablet terminal, or a smartphone. The user terminal 20 may be carried by the user, or may be a system presenting information to the user on-site and exchanging information with the user (for example, a system capable of presenting information to the user via a projector and acquiring information from the user via sensors or voice).

[0020] The user terminal 20 may not be equipped with the sensor 40 and may acquire the image data 30 directly, but may acquire the data only from the information processing server 10. In other words, the user terminal 20 may be connected to the sensor 40 that captures the image data 30, or may display the image data on the head-mounted display described above. The following mainly describes the configuration of the user terminal 20 connected to the sensor 40.

[0021] Fig. 5 is a system configuration diagram of the user terminal 20. The user terminal 20 is configured as an information processing terminal equipped with a calculation device, a storage device, and a display unit. As shown in Fig. 5, the user terminal 20 is equipped with a data collection unit 21, an information display unit 22, an information assignment unit 23, and a data delivery unit 24. Each function of the data collection unit 21, the information display unit 22, the information assignment unit 23, and the data delivery unit 24 can be realized by the calculation device executing a program for realizing each function stored in the storage device.

[0022] The data collection unit 21 acquires data such as the image capture data 30 acquired by the sensor 40 and virtual images stored in the information processing server 10. The data collection unit 21 also acquires location information of the user terminal 20 and the sensor 40. For example, the data collection unit 21 acquires location information including the position and image capture direction of the user terminal 20 from a GPS (Global Positioning System) device or a direction sensor equipped in the user terminal 20. The data collection unit 21 also acquires location information including the position and image capture direction of the sensor 40 from the GPS device or direction sensor equipped in the sensor 40 via the user terminal 20. Note that, as indicated by reference symbol Y1 in FIG. 3 , the data collection unit 21 may use a camera equipped in the user terminal 20 to capture an image of a QR code 41 displayed on the sensor 40 and acquire identification information of the sensor 40 included in the QR code 41. In this case, it is assumed that the identification information of the sensor 40 is previously associated with location information including the position and image capture direction of the sensor 40. The QR code 41 itself may include the position information of the sensor 40 .

[0023] The information display unit 22 controls the display of information acquired by the data collection unit 21 on the display unit A. For example, the information display unit 22 overlays (superimposes) data such as a virtual image transmitted from the information processing server 10, as described below, on a real-space image displayed transparently on the display unit A or converted into digital data and displayed. For example, the virtual image is information generated based on three-dimensional point cloud data, particularly information generated by compressing three-dimensional point cloud data, such as mesh data or bounding boxes that simplify the shapes of objects such as trees T1 and T2 that exist in real space, as described below. The information display unit 22 can use external data to represent a space or an object on the display unit A, or can display an avatar or viewpoint to clearly indicate the user's position and orientation in the space, but the display method is not limited to these. In addition to displaying point cloud data and meshed data, the information display unit 22 can also overlay them to make them easier to see, switch between them, adjust the degree of overlap, and select which to display.

[0024] The information providing unit 23 updates and edits the data acquired by the data collecting unit 21, generates information from other functions of the user terminal 20 (such as a camera, GPS, or IMU sensor), and generates information independently by the user P. For example, as described above, the information providing unit 23 generates information such as a QR code captured by the camera, i.e., identification information of the sensor 40, and information on the positional relationship between the positional information of the sensor 40 and the positional information of the user terminal 20. In addition, the captured data 30 may be pre-processed, such as noise removal, outlier removal, and correction.

[0025] The data delivery unit 24 transmits the above-mentioned photographing data 30, location information, generated information, etc. to the information processing server 10 and stores them in the information processing server 10. The delivery of photographing data, etc. may be all data or only differential data each time it is acquired, and throttling or timing adjustment may be performed in consideration of the transmission load.

[0026] FIG. 6 is a system configuration diagram of the information processing server 10. The information processing server 10 is composed of one or more information processing devices each including a computing device and a storage device. As shown in FIG. 6, the information processing server 10 includes a location information acquisition unit 11, an alignment function unit 12, a data collection unit 13, an information management unit 14, and a data delivery unit 17. The information management unit 14 further includes a mesh processing unit 15. The functions of the location information acquisition unit 11, the alignment function unit 12, the data collection unit 13, the information management unit 14, the mesh processing unit 15, and the data delivery unit 17 can be realized by the computing device executing a program for realizing each function stored in the storage device. The information management unit 14 of the information processing server 10 also includes an information storage unit 16. The information storage unit 16 is composed of a storage device. Each component will be described in detail below.

[0027] First, the data collection unit 13 collects data transmitted from the user terminal 20. For example, the data collection unit 13 collects position information of the user terminal 20 and the sensor 40, and collects three-dimensional point cloud data of the photographed data 30 as position information of objects in the photographed data 30 photographed by the sensor 40. The collected information is acquired by the position information acquisition unit 11 and used in the alignment function unit 12, or passed to the mesh processing unit 15.

[0028] The position information acquisition unit 11 (acquisition unit) acquires position information of the user terminal 20, the sensor 40, objects in the shooting data 30, etc. For example, the position information acquisition unit 11 acquires position information from data obtained from the user terminal 20 or attached information. As an example, the position information acquisition unit 11 acquires position information including the positions and shooting direction of the user terminal 20 and the sensor 40 from information obtained from the GPS and direction sensors of the user terminal 20 and the sensor 40. Furthermore, the position information acquisition unit 11 acquires position information consisting of three-dimensional coordinates of objects such as trees captured in the shooting data 30 from the point cloud data of the shooting data 30. Note that, when the position information acquisition unit 11 acquires identification information of the sensor 40 from the QR code 41 of the sensor 40, it acquires the position information of the sensor 40 that is previously associated with the identification information and stored.

[0029] The alignment function unit 12 (alignment unit) aligns the user terminal 20 and the sensor 40 based on the position information acquired as described above. Specifically, the alignment function unit 12 aligns the position of the image data 30 captured by the sensor 40 to correspond to the position of the user terminal 20 based on the difference in position between the user terminal 20 and the sensor 40. In other words, the alignment function unit 12 aligns the three-dimensional coordinates of the image data 30 acquired by the sensor 40 to the position and image capture direction of the user terminal 20. Note that the user terminal 20 to be aligned is not limited to the user terminal 20 connected to the sensor 40, but also includes other user terminals 20 not connected to the sensor 40. By collecting image data 30 acquired from multiple sensors 40 of the same or different types and aligning it with each user terminal 20, the image data 30 from those sensors 40 can be aligned consistently.

[0030] The mesh processing unit 15 (generation unit) of the information management unit 14 generates compressed data by meshing the three-dimensional point cloud data of the image data 30 captured by the sensor 40, and stores the compressed data in the information storage unit 16. Then, the data delivery unit 17 transmits the stored compressed data to each user terminal 20 in response to a request from each user terminal 20.

[0031] Specifically, the mesh processing unit 15 meshes or polygonizes the three-dimensional point cloud data of objects such as trees T1 and T2 in the photographic data 30, converts it into CAD data, and generates a bounding box virtual image. The objects and granularity of the meshing are determined according to instructions and other information from the user terminal 20. Note that it is also possible to control the granularity and frame rate of the photographic data 30 from the sensor 40 based on instructions and other information from the user terminal 20.

[0032] The mesh processing unit 15 can also select or change the object to be compressed, such as meshed, within the shooting data 30. For example, the mesh processing unit 15 may automatically detect an object, such as a "tree," specified by the user terminal 20 from the shooting data 30 and generate a virtual image by meshing the object. When a range or area for detecting the object is set by the user terminal 20, the mesh processing unit 15 may generate a virtual image of the object limited to that range or area. When an object other than the object is specified by the user terminal 20, such as a "telephone pole or building," the mesh processing unit 15 may exclude the object and mesh other objects (such as trees). The mesh processing unit 15 may also measure the size of the object, such as a tree, from the three-dimensional point cloud data and generate a virtual image including the measurement value.

[0033] The mesh processing unit 15 may also determine the object based on prediction or estimation. For example, as described above, the size of the object may be measured and a virtual image of the object portion predicted to exceed the limiting surface in the future may be generated. A virtual image may be generated in which the predicted portion and the portion that actually exceeds the limiting surface are displayed with different colors or shapes. Furthermore, meshing processing may be performed in cooperation with an external system (e.g., a CAD system, a finished product drawing management system, a drawing management system, a GIS system, etc.). Furthermore, the meshing target may not be limited by distinguishing between objects and non-objects, and the presence or absence of meshing and the meshing granularity (the number of polygons, the number of operations, and other accuracy) may be determined based on processing power, the number of point clouds, density, accuracy, error, etc. Furthermore, the meshing target and the meshing granularity may be determined based on user instructions or cooperation with external data. Furthermore, a temporal history may be maintained, and point cloud processing and meshing processing may be performed based on the history and changes over time. In addition, the meshed data and data that are not subject to meshing may be retained and used to determine whether or not a meshing target is present, detect objects (search for and match objects, etc.), and track objects, and these functions can reduce and simplify point cloud processing. The mesh processing unit 15 may be provided in the user terminal 20, and the user terminal 20 may transmit point cloud data, meshes, or both to the information processing server 10.

[0034] The data delivery unit 17 (display control unit) transmits the virtual images generated by meshing or bounding boxes as described above to the user terminal 20. At this time, the data delivery unit 17 transmits the virtual images to be displayed on the display unit of the user terminal 20 in correspondence with the position of the user terminal 20. As a result, the display unit of the user terminal 20 displays the meshed or bounding boxed virtual images V1 and V2 superimposed on the positions of the trees T1 and T2, which are objects on the real space image, as shown in FIG. 2 , or displays a virtual image V3 including measured height information.

[0035] [Operation] Next, the operation of the above-described information display system will be described with reference to the flowcharts of Figures 7 and 8. First, the operation of the information processing server 10 will be described with reference to the flowchart of Figure 7.

[0036] The information processing server 10 acquires position information (such as the shooting position and direction, shooting settings and conditions, and additional information) of the user terminal 20 and the sensor 40 (step S1). The information processing server 10 then aligns the sensor 40 with the user terminal 20 (step S2). The information processing server 10 also acquires point cloud data, which is the shooting data 30 (step S3). At this time, the information processing server 10 associates the point cloud data with the position of the user terminal 20 according to the above-described alignment. The information processing server 10 may combine point cloud data captured by a moving sensor 40 or synthesize (register) point cloud data captured by multiple sensors 40. When combining several fields of view captured by a 3D sensor, the point cloud data is also aligned using position information, external data, and the like. The alignment may also be performed after compression processing, such as meshing of the three-dimensional point cloud data.

[0037] Next, the information processing server 10 generates a virtual image by meshing the three-dimensional point cloud data so that the portions exceeding the limiting surface are clearly visible (step S4). Furthermore, a virtual image may be generated that displays normal portions as objects, rather than portions exceeding the limiting surface or violations / abnormal portions, or may be generated according to the purpose desired by the user P. Note that the meshing process may be performed on the user terminal 20 at the site, rather than on the information processing server 10. Alternatively, a virtual image that displays portions exceeding the limiting surface by placing a box (bounding box) instead of meshing may be generated, or a virtual image that displays arrows, pin-like shapes, or the like may be generated. Note that the point cloud processing and meshing process may include pre-processing and post-processing, such as noise removal and correction.

[0038] The information processing server 10 then stores necessary or all processing data in the information storage unit 16 (step S5) and updates necessary or all data (step S6), thereby enabling real-time information to be displayed on the user terminal 20. At this time, the information processing server 10 can adjust the granularity and accuracy of the virtual image to be generated based on instructions from users such as user P, collaboration with external systems, and judgments regarding the real-time nature of internal processing and resource availability, thereby displaying information in real time. The information processing server 10 also generates virtual images with additional information, such as measurements of the size of the target object, as necessary (step S7), and transmits the virtual images to each user terminal 20 (step S8).

[0039] Next, the operation of the user terminal 20 will be described with reference to the flowchart of FIG. 8. The user terminal 20 captures the captured image data 30 using the sensor 40 (step S11) and acquires the meshed or bounding boxed virtual image transmitted from the information processing server 10 (step S12). At this time, location information and the like are also acquired. The user terminal 20 then displays a transparent or digitally converted real space image on the display unit A, and superimposes the acquired virtual image on the real space image (step S13). Note that, in the case of a user terminal 20 to which the sensor 40 is not connected, the process of step S11 may be omitted. Note that, in the case where a virtual image is generated by the user terminal 20, the virtual image may be displayed.

[0040] Thereafter, the user terminal 20 may add information such as a message to the photographic data 30 or the acquired data as needed (step S14).The user terminal 20 then transmits the acquired photographic data, location information, and other information to the information processing server 10 (step S15).

[0041] As described above, according to this embodiment, a photographing device such as a 3D LIDAR is used to acquire three-dimensional point cloud data of trees and buildings around an airport, and a virtual image that reveals parts that protrude beyond the restricted surface can be superimposed on a real space image and displayed on a user terminal 20 such as MR glasses. As a result, inspection and monitoring that take into account the Aviation Act can be facilitated, and the efficiency of on-site monitoring work can be improved.

[0042] Another example of the use of the information display system described above will now be described with reference to Figures 9 and 10. Figure 9 shows a case where monitoring work such as measuring the distance between utility poles, power lines, etc. is carried out on a vast site.

[0043] As shown in FIG. 9 , user P wears user terminal 20 and goes to a site to view the target utility poles and power lines. The site is then photographed using three-dimensional sensor 40, and an information processing server (not shown) uses the photographed data, which is three-dimensional point cloud data, to detect the target utility poles and service lines and measure their heights and spacing. The information processing server generates virtual images consisting of "measurement values" and text information about the detected "utility poles" and "service lines," and transmits the images to user terminal 20. As a result, as shown in FIG. 9 , display unit A of user terminal 20 can display a real-space image that has been transmitted through or digitally converted from the display unit A, while superimposing virtual images V11 and V12, such as the measurement values ​​and text information about the target objects, generated by the information processing server, on the real-space image.

[0044] FIG. 10 also illustrates a case where a monitoring operation is performed to detect cracks in the ground at a large area such as an airport, highway, or national highway. A user P wears a user terminal 20 and visits the site to view the target ground. A sensor 40, such as a 3D LIDAR, captures the site, and an information processing server (not shown) detects cracks in the ground and measures their lengths using the captured image data (3D point cloud data). The information processing server generates a virtual image consisting of text information of the "measurement value" and transmits it to the user terminal 20. As a result, as shown in FIG. 10 , the display unit A of the user terminal 20 displays a real-space image that is either transparent or digitally converted, while a virtual image V21, such as text information of the measurement value indicating the crack length generated by the information processing server, is superimposed on the real-space image. The sensor 40 may acquire the captured image data using, for example, a drone or a camera mounted on a vehicle.

[0045] <Embodiment 2> Next, a second embodiment of the present disclosure will be described with reference to Fig. 11 and Fig. 12. Fig. 11 and Fig. 12 are block diagrams showing the configuration of an information display system according to embodiment 2. Note that this embodiment shows an outline of the configuration of the information display system described in the above-mentioned embodiment.

[0046] First, the hardware configuration of the information display system 100 of this embodiment will be described with reference to Fig. 11. The information display system 100 is configured with a general information processing device, and is equipped with the following hardware configuration, for example: - CPU (Central Processing Unit) 101 (arithmetic unit) - ROM (Read Only Memory) 102 (storage device) - RAM (Random Access Memory) 103 (storage device) - Programs 104 loaded into the RAM 103 - Storage device 105 storing the programs 104 - Drive device 106 for reading and writing from / to a storage medium 110 external to the information processing device - Communication interface 107 for connecting to a communication network 111 external to the information processing device - Input / output interface 108 for inputting and outputting data - Bus 109 for connecting the various components

[0047] 11 shows an example of the hardware configuration of an information processing device that is the information display system 100, and the hardware configuration of the information processing device is not limited to the above-described case. For example, the information processing device may be configured with a part of the above-described configuration, such as not including the drive device 106. Furthermore, instead of the above-described CPU, the information processing device may use a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point Number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.

[0048] The information display system 100 can be equipped with an acquisition unit 121, an alignment unit 122, a generation unit 123, and a display control unit 124 shown in FIG. 12 by having the CPU 101 acquire and execute the program group 104. The program group 104 is stored in advance in the storage device 105 or the ROM 102, for example, and is loaded into the RAM 103 and executed by the CPU 101 as needed. The program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, and the drive device 106 may read and supply the programs to the CPU 101. However, the acquisition unit 121, alignment unit 122, generation unit 123, and display control unit 124 described above may be constructed using dedicated electronic circuits for realizing such means.

[0049] The acquisition unit 121 acquires position information of a display device that displays a real space image, an image capture device that captures an image of a predetermined object, and an object in an image captured by the image capture device.

[0050] The positioning unit 122 matches the position of the object in the captured image with the position of the display device based on the position information.

[0051] The generation unit 123 generates a virtual image of the object based on the position information of the object. For example, the generation unit 123 compresses the position information of the object to generate a virtual image in which the shape of the object is simplified.

[0052] The display control unit 124 controls the display device to display a virtual image of the object corresponding to the position of the display device superimposed on the real space image.

[0053] With the above-described configuration, the present disclosure aligns the position information of the image data captured using the image capture device with the position of the user terminal, and further superimposes a virtual image of the object in the image capture data on the real-space image displayed on the user terminal. This allows the user to easily recognize the position and status of the monitored object on the real-space image using the user terminal. As a result, the efficiency of on-site monitoring work can be improved.

[0054] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0055] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that are understandable to those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, at least one or more of the functions of the acquisition unit 121, the alignment unit 122, the generation unit 123, and the display control unit 124 described above may be executed by an information processing device installed and connected anywhere on a network, that is, may be executed by so-called cloud computing.

[0056] <Supplementary Notes> Some or all of the above embodiments can also be described as in the following supplementary notes. Below, an outline of the configurations of the information display system, information display method, and program according to the present disclosure will be described. However, the present disclosure is not limited to the following configurations. (Supplementary Note 1) An information display system comprising: a display device that displays a real space image; a photographing device that photographs a predetermined object; and an acquisition unit that acquires position information of the object in the photographed image photographed by the photographing device; an alignment unit that matches the position of the object in the photographed image with the position of the display device based on the position information; a generation unit that generates a virtual image of the object based on the position information of the object; and display control means that controls the display device to display the virtual image of the object corresponding to the position of the display device superimposed on the real space image. (Supplementary Note 2) The information display system according to Supplementary Note 1, wherein the generation unit compresses the position information of the object to generate the virtual image. (Supplementary Note 3) The information display system according to Supplementary Note 2, wherein the generation unit generates the virtual image by simplifying the shape of the object. (Supplementary Note 4) The information display system according to Supplementary Note 2, wherein the generation unit generates the virtual image only for the object in the captured image that satisfies a preset criterion. (Supplementary Note 5) The information display system according to Supplementary Note 2, wherein the captured image captured by the imaging device is point cloud data including three-dimensional coordinates of the object, and the generation unit generates the virtual image by meshing, polygonizing, or bounding boxing the point cloud data of the object. (Supplementary Note 6) The information display system according to Supplementary Note 2, wherein the captured image captured by the imaging device is point cloud data including three-dimensional coordinates of the object, and the generation unit measures the size of the object based on position information of the object and generates the virtual image including the measurement values.(Supplementary Note 7) The information display system according to Supplementary Note 1, wherein the acquisition unit acquires, from the display device, identification information of the image capturing device acquired by the display device along with position information of the display device, and acquires position information of the image capturing device that is pre-associated with the identification information of the image capturing device. (Supplementary Note 8) An information display method comprising: acquiring position information of a display device that displays a real space image, an image capturing device that captures an image of a predetermined object, and the object in an image captured by the image capturing device, corresponding a position of the object in the image captured by the image capturing device based on the position information, generating a virtual image of the object based on the position information of the object, and controlling the display device to display the virtual image of the object corresponding to the position of the display device so as to be superimposed on the real space image. (Supplementary Note 9) A program for causing a computer to execute processing of acquiring positional information of a display device that displays a real space image, an imaging device that images a predetermined object, and the object in an image captured by the imaging device, corresponding the position of the object in the image based on the positional information to the position of the display device, generating a virtual image of the object based on the positional information of the object, and controlling the display device to display the virtual image of the object, corresponding to the position of the display device, superimposed on the real space image.

[0057] In addition, the present invention claims the benefit of priority based on the patent application of Japanese Patent Application No. 2022-156277 filed on September 29, 2022 in Japan, and all contents described in the patent application are included in this specification.

[0058] REFERENCE SIGNS LIST 10 Information processing server 11 Position information acquisition unit 12 Alignment function unit 13 Data collection unit 14 Information management unit 15 Mesh processing unit 16 Information storage unit 17 Data delivery unit 20 User terminal 21 Data collection unit 22 Information display unit 23 Information assignment unit 24 Data delivery unit 30 Shooting data 40 Sensor 100 Information display system 101 CPU 102 ROM 103 RAM 104 Program group 105 Storage device 106 Drive device 107 Communication interface 108 Input / output interface 109 Bus 110 Storage medium 111 Communication network 121 Acquisition unit 122 Alignment unit 123 Generation unit 124 Display control unit

Claims

1. A display device for displaying a real - space image, a photographing device for photographing a predetermined object, and an acquisition unit for acquiring position information of the object in the photographed image captured by the photographing device, An alignment unit that aligns the position of the object in the photographed image with the position of the display device based on the position information, A generation unit that generates a virtual image of the object based on the position information of the object, Display control means for controlling the display device to superimpose and display the virtual image of the object corresponding to the position of the display device on the real - space image, An information display system comprising the above.

2. The information display system according to claim 1, wherein the generation unit compresses the position information of the object to generate the virtual image. An information display system.

3. The information display system according to claim 2, wherein the generation unit generates the virtual image with the shape of the object simplified. An information display system.

4. The information display system according to claim 2, wherein the generation unit generates the virtual image only for the object that satisfies a preset criterion in the photographed image. An information display system.

5. The information display system according to claim 2, wherein the photographed image captured by the photographing device is point - cloud data including the three - dimensional coordinates of the object, and the generation unit generates the virtual image by meshing, polygonizing, or bounding - boxing the point - cloud data of the object. An information display system.

6. The information display system according to claim 2, wherein the photographed image captured by the photographing device is point - cloud data including the three - dimensional coordinates of the object, and the generation unit measures the size of the object based on the position information of the object and generates the virtual image including the measured value. An information display system.

7. The information display system according to claim 1, wherein the acquisition unit acquires, from the display device, the position information of the display device together with the identification information of the photographing device acquired by the display device, and acquires the position information of the photographing device that is pre - associated with the identification information of the photographing device. An information display system.

8. A display device for displaying a real - space image, a photographing device for photographing a predetermined object, and acquires the position information of the object in the photographed image captured by the photographing device, Based on the position information, associate the position of the object in the captured image with the position of the display device. Generate a virtual image of the object based on the position information of the object. Control the display device to display the virtual image of the object corresponding to the position of the display device by superimposing it on the real-space image. Information display method.

9. A program for causing a computer to execute a process, the program including: acquiring position information of a display device that displays a real-space image, an imaging device that captures a predetermined object, and the object in a captured image captured by the imaging device; Based on the position information, associate the position of the object in the captured image with the position of the display device. Generate a virtual image of the object based on the position information of the object. Control the display device to display the virtual image of the object corresponding to the position of the display device by superimposing it on the real-space image. A program for causing a computer to execute the process.