Information display system, information display method, program

Abstract

An information display system 100 according to the present disclosure is provided with: an acquisition unit 121 for acquiring position information of a display device for displaying a real-space image, an image-capturing device for capturing an image of a predetermined object, and an object in a captured image captured by the image-capturing device; a position alignment unit 122 for aligning the position of the object in the captured image with the position of the display device on the basis of the position information; a generation unit 123 for generating a virtual image of the object on the basis of the position information of the object; and a display control means 124 for controlling the display device so as to display the virtual image of the object aligned with the position of the display device in a superimposed manner on the real-space image.

Description

Technical Field

[0001] The present disclosure relates to an information display system, an information display method, and a program.

Background Art

[0002] Based on the Aviation Law, a restricted surface is set around an airport for the safe takeoff and landing of aircraft. Therefore, in a certain space around the airport, the installation of buildings and trees that exceed the height above the restricted surface is prohibited, and a monitoring operation for objects exceeding the restricted surface is required.

[0003] Here, as an example of a system used for the monitoring operation, Patent Document 1 describes a system using a head-mounted display. Specifically, in Patent Document 1, first, 3D structure data representing a monitoring target on a virtual 3D space is stored. Then, image data is collected from a camera installed in the area of the monitoring target, detection result data detecting an abnormality at the monitoring location is obtained from such image data, and positioning result data including position data and azimuth data of the head-mounted display is obtained. Thereafter, based on the 3D structure data, the detection result data, and the positioning result data, 3D integrated data integrating all the positional relationships is generated, and a virtual image is displayed on the head-mounted display during the monitoring operation.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the system described in Patent Document 1 above only displays the detection results of anomalies in the monitored object, making it difficult for the monitor to recognize the position of the monitored object in the real-world image. This results in the problem that the efficiency of on-site monitoring work cannot be improved.

[0006] Therefore, the purpose of this disclosure is to provide an information display system that can solve the aforementioned problem of being unable to improve the efficiency of on-site monitoring operations. [Means for solving the problem]

[0007] An information display system, which is one form of this disclosure, A display device that displays a real-world spatial image, a shooting device that photographs a predetermined object, and an acquisition unit that acquires positional information of the object in the image captured by the shooting device. A positioning unit that associates the position of the object in the captured image with the position of the display device based on the position information, A generation unit that generates a virtual image of the object based on the position information of the object, The display device includes a display control means that controls the display device to display the virtual image of the object corresponding to the position of the display device superimposed on the real-world image, Equipped with, This is the structure it takes.

[0008] Furthermore, one form of this disclosure is the information display method, A display device that displays a real-world spatial image, a shooting device that photographs a predetermined object, and the acquisition of positional information of the object in the image captured by the shooting device, Based on the position information, the position of the object in the captured image is associated with the position of the display device. Based on the location information of the object, a virtual image of the object is generated. The display device is controlled to display the virtual image of the object, corresponding to the position of the display device, superimposed on the real-world image. This is the structure it takes.

[0009] Furthermore, one form of this disclosure is a program, A display device that displays a real-world spatial image, a shooting device that photographs a predetermined object, and the acquisition of positional information of the object in the image captured by the shooting device, Based on the position information, the position of the object in the captured image is associated with the position of the display device. Based on the location information of the object, a virtual image of the object is generated. The display device is controlled to display the virtual image of the object, corresponding to the position of the display device, superimposed on the real-world image. To have the computer perform the process. This is the structure it takes. [Effects of the Invention]

[0010] This disclosure, structured as described above, can improve the efficiency of on-site monitoring operations. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows a schematic diagram of the information display system in Embodiment 1 of this disclosure. [Figure 2] Figure 1 shows an example of a display image on the user terminal disclosed. [Figure 3] This figure shows an example of alignment between the user terminal and the imaging device disclosed in Figure 1. [Figure 4] Figure 1 is a block diagram showing the overall configuration of the information display system. [Figure 5] This block diagram shows the configuration of the user terminal disclosed in Figure 1. [Figure 6] This is a block diagram showing the configuration of the information processing server disclosed in Figure 1. [Figure 7] Figure 1 is a flowchart showing the operation of the information processing server. [Figure 8]It is a flowchart showing the operation of the user terminal disclosed in FIG. 1. [Figure 9] It is a diagram showing another example of the display image on the user terminal disclosed in FIG. 1. [Figure 10] It is a diagram showing another example of the display image on the user terminal disclosed in FIG. 1. [Figure 11] It is a block diagram showing the hardware configuration of the information display system in Embodiment 2 of the present disclosure. [Figure 12] It is a block diagram showing the configuration of the information display system in Embodiment 2 of the present disclosure.

Mode for Carrying Out the Invention

[0012] <Embodiment 1> The first embodiment of the present disclosure will be described with reference to FIGS. 1 to 10. First, the outline of the information display system in this embodiment will be described.

[0013] [Outline] As shown in FIG. 1, the information processing system in this embodiment includes a user terminal 20 which is a display device used by user P, a sensor 40 which is a photographing device for photographing a predetermined object, and an information processing server 10 which processes display images. And the information processing system 10 in this embodiment is used for user P to monitor buildings and trees protruding from restricted surfaces around the airport. For example, as shown in FIG. 1, user P wears a user terminal 20 composed of a head-mounted display and views the surrounding area of the airport which is the place to be monitored through such a user terminal 20. Then, a real space image as shown in the range of the reference sign Rd shown in FIG. 1 is displayed on the display unit A of the user terminal 20 which is a head-mounted display. Note that the real space image displayed on the user terminal 20 may be an image obtained by viewing the real space through the display unit A, or an image obtained by converting the real space into digital data and displaying it on the display unit A.

[0014] The sensor 40 then acquires image data 30, which is an image taken in the area around the airport, as shown in Figure 1 under the code Rc. The image data 30 includes three-dimensional point cloud data, which is positional information indicating the three-dimensional coordinates of trees T1 and T2, which are the objects to be monitored, located within the shooting range, as will be described later. This image data 30 is transmitted to the information processing server 10 via the user terminal 20 or directly from the shooting device 40.

[0015] The information processing server 10 generates virtual images V1 and V2 corresponding to each object from the three-dimensional point cloud data contained in the image data 30 captured by the sensor 40. The information processing server 10 also acquires the position information of the user terminal 20 and the sensor 40 and performs alignment between them. In other words, the information processing server 10 aligns the position of the image data 30 captured by the sensor 40 with the position of the user terminal 20 based on the difference in position between the user terminal 20 and the sensor 40. Then, the information processing server 10 transmits the virtual images V1 and V2 based on the aligned image data 30 to the user terminal 20 and controls it to overlay the virtual images V1 and V2 onto the real-world image displayed on the user terminal 20.

[0016] As a result, the display unit A of the user terminal 20 will display virtual images V1 and V2 aligned to the real-world image within the range indicated by code Rd, as shown in Figure 2. The configurations and operations will be explained in detail below with reference to Figures 3 to 10.

[0017] [composition] Figure 4 is a system configuration diagram of the entire information processing system. User terminal 20 acquires image data 30 captured by sensor 40 and exchanges data with other user terminals 20 in real time via information processing server 10.

[0018] Here, sensor 40 is a 3D sensor such as a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a ToF camera (Time of Flight Camera), or a stereo camera. However, sensor 40 is not limited to 3D sensors; it may also be a hyperspectral camera, an RGB camera, or other types of sensors.

[0019] The user terminal 20 is a head-mounted display or other MR glasses (Mixed Reality Glass). Therefore, the user terminal 20 is configured to display a real-world image and to overlay virtual image data onto this real-world image. The user terminal 20 is not limited to MR glasses; it may be other glasses-type devices such as VR glasses (Virtual Reality Glass), or any information processing terminal with a display unit, such as a personal computer, tablet, or smartphone. Furthermore, the user terminal 20 may be carried by the user, or it may be a system that exists on-site and exchanges information with the user (for example, a system that can present information to the user using a projector and acquire information from the user using sensors or voice).

[0020] Furthermore, the user terminal 20 may not be equipped with a sensor 40 and may not directly acquire the captured data 30, instead acquiring data only from the information processing server 10. In other words, the user terminal 20 may be connected to a sensor 40 that captures the captured data 30, or it may display the data on the head-mounted display described above. Below, we will mainly describe the configuration of the user terminal 20 connected to the sensor 40.

[0021] Figure 5 is a system configuration diagram of the user terminal 20. The user terminal 20 is an information processing terminal equipped with a computing unit, a storage device, and a display unit. As shown in Figure 5, the user terminal 20 includes a data collection unit 21, an information display unit 22, an information assignment unit 23, and a data distribution unit 24. The functions of the data collection unit 21, the information display unit 22, the information assignment unit 23, and the data distribution unit 24 can be realized by the computing unit executing a program for realizing each function stored in the storage device.

[0022] The data collection unit 21 acquires data such as the image data 30 acquired by the sensor 40 described above, and virtual images stored in the information processing server 10. The data collection unit 21 also acquires the location information of the user terminal 20 and the location information of the sensor 40. For example, the data collection unit 21 acquires location information including the position of the user terminal 20 and the shooting direction from the GPS (Global Positioning System) device or compass sensor equipped on the user terminal 20. The data collection unit 21 also acquires location information including the position of the sensor 40 and the shooting direction from the GPS device or compass sensor equipped on the sensor 40 via the user terminal 20. As shown by the reference numeral Y1 in Figure 3, the data collection unit 21 may also use a camera mounted on the user terminal 20 to photograph the QR code 41 displayed on the sensor 40 and acquire the identification information of the sensor 40 contained in the QR code 41. In this case, the identification information of the sensor 40 is assumed to be pre-associated with location information including the position of the sensor 40 and the shooting direction. The QR code 41 itself may also contain the location information of the sensor 40.

[0023] The information display unit 22 controls the display unit A to display the information acquired by the data collection unit 21. For example, the information display unit 22 overlays data such as virtual images transmitted from the information processing server 10, as described later, onto the real-world image that is transparently displayed on the display unit A or converted into digital data and displayed. For example, virtual images are information generated based on three-dimensional point cloud data, such as mesh data or bounding boxes that simplify the shapes of objects such as trees T1 and T2 that exist in the real world, as described later, and in particular, information generated by compressing three-dimensional point cloud data. The information display unit 22 can also use external data to represent space or objects on the display unit A, or display the user's position and orientation in space in an easy-to-understand way using avatars or viewpoints, but the display method is not limited to these. In addition to displaying point cloud data and meshed data, the information display unit 22 may also overlay them to make them easier to see, switch between them, adjust the degree of overlap, or select which to display.

[0024] The information assignment unit 23 updates and edits data acquired by the data collection unit 21, generates information from other functions of the user terminal 20 (such as camera, GPS, and IMU sensor), and generates information independently by the user P. For example, as mentioned above, it generates information such as QR codes captured by the camera, i.e., identification information for the sensor 40, and information on the positional relationship between the location information of the sensor 40 and the location information of the user terminal 20. It may also perform preprocessing such as noise reduction, outlier removal, and correction of the captured data 30.

[0025] The data distribution unit 24 transmits the aforementioned shooting data 30, location information, and generated information to the information processing server 10 for storage. The distribution of shooting data, etc., may include all data or only differential data each time it is acquired, and throttling or timing adjustments may be performed to consider the transmission load.

[0026] Figure 6 is a system configuration diagram of the information processing server 10. The information processing server 10 is composed of one or more information processing devices equipped with a computing unit and a storage device. As shown in Figure 6, the information processing server 10 includes a location information acquisition unit 11, a position alignment function unit 12, a data collection unit 13, an information management unit 14, and a data distribution unit 17. The information management unit 14 further includes a mesh processing unit 15. The functions of the location information acquisition unit 11, the position alignment function unit 12, the data collection unit 13, the information management unit 14, the mesh processing unit 15, and the data distribution unit 17 can be realized by the computing unit executing a program for realizing each function stored in the storage device. In addition, the information management unit 14 of the information processing server 10 includes an information holding unit 16. The information holding unit 16 is composed of a storage device. The following describes each configuration in detail.

[0027] First, the data collection unit 13 collects data transmitted from the user terminal 20. For example, the data collection unit 13 collects location information of the user terminal 20 and the sensor 40, and collects three-dimensional point cloud data of the captured data 30 as location information of objects within the captured data 30 captured by the sensor 40. The collected information is acquired by the location information acquisition unit 11 and used by the alignment function unit 12 or passed to the mesh processing unit 15.

[0028] The location information acquisition unit 11 (acquisition unit) acquires location information of the user terminal 20, the sensor 40, and objects in the captured data 30. For example, the location information acquisition unit 11 acquires location information from data obtained from the user terminal 20 and the information attached to it. As an example, the location information acquisition unit 11 acquires location information including the position of the user terminal 20 and the sensor 40 and the shooting direction from information obtained from the GPS and compass sensors of the user terminal 20 and the sensor 40. In addition, the location information acquisition unit 11 acquires location information consisting of the three-dimensional coordinates of objects such as trees that appear in the captured data 30 from the point cloud data of the captured data 30. Furthermore, if the location information acquisition unit 11 obtains identification information of the sensor 40 from the QR code 41 of the sensor 40, it acquires the location information of the sensor 40 that is pre-associated and stored with such identification information.

[0029] The alignment function unit 12 (alignment unit) aligns the user terminal 20 and the sensor 40 based on the position information acquired as described above. Specifically, the alignment function unit 12 aligns the user terminal 20 to the position of the sensor 40 based on the difference in position between the user terminal 20 and the sensor 40. In other words, it aligns the three-dimensional coordinates of the image data 30 acquired by the sensor 40 to the position and shooting direction of the user terminal 20. Note that the user terminal 20 performing the alignment is not limited to the user terminal 20 connected to the sensor 40, but also includes other user terminals 20 not connected to the sensor 40. By collecting image data 30 acquired from multiple sensors 40 of the same or different types and aligning them to each user terminal 20, the image data 30 from those sensors 40 can be aligned without contradiction.

[0030] The mesh processing unit 15 (generation unit) of the information management unit 14 generates compressed data by meshing the three-dimensional point cloud data 30 captured by the sensor 40 and stores it in the information holding unit 16. Then, the data distribution unit 17 transmits the stored compressed data to each user terminal 20 in response to requests from each user terminal 20.

[0031] Specifically, the mesh processing unit 15 performs meshing, polygonization, conversion to CAD data, and generation of bounding box virtual images of three-dimensional point cloud data of objects such as trees T1 and T2 in the captured data 30. The objects to be meshed and the granularity are determined according to instructions from the user terminal 20 and other information. Furthermore, it is also possible to control the granularity and frame rate of the captured data 30 from the sensor 40 based on instructions from the user terminal 20 and other information.

[0032] Furthermore, the mesh processing unit 15 can select or change the objects to be compressed, such as meshed, within the captured data 30. For example, the mesh processing unit 15 may automatically detect objects specified by the user terminal 20, such as "trees," from the captured data 30, and generate a virtual image by meshing such objects. Also, if the user terminal 20 sets a range or area for detecting objects, the mesh processing unit 15 may generate a virtual image of objects limited to that range or area. Furthermore, if the user terminal 20 specifies something other than objects, such as "utility poles or buildings," the mesh processing unit 15 may exclude these and mesh other objects (such as trees). Also, the mesh processing unit 15 may measure the size of objects such as trees from the three-dimensional point cloud data and generate a virtual image that includes the measured values.

[0033] Furthermore, the mesh processing unit 15 may determine the target object based on prediction and estimation. For example, as described above, it may measure the size of the object and generate a virtual image of the part of the object that is predicted to exceed the limit surface in the future. The virtual image may be generated in which the predicted part and the part that actually exceeds the limit surface are distinguished by color and shape. It may also cooperate with an external system (CAD system, construction drawing management system, drawing management system, GIS system, etc.) to perform meshing processing, etc. Furthermore, the target of meshing may not be limited by distinguishing between target and non-target objects, and the presence or absence of meshing and the granularity of meshing (precision such as the number of polygons and the number of calculations) may be determined based on processing capacity, number of point clouds, density, accuracy, error, etc. Furthermore, the target of meshing and the granularity of meshing may be determined by instructions from the user or by cooperation with external data. In addition, a temporal history may be maintained, and point cloud processing and meshing processing may be performed based on that history and temporal changes. Furthermore, the meshed data and data that are not subject to meshing can be retained and used for determining whether or not there are objects to be meshed, for object detection (such as object search and matching), and for object tracking. These functions can lighten and simplify point cloud processing. The mesh processing unit 15 may be installed in the user terminal 20, and the user terminal 20 may transmit point cloud data, meshes, or both to the information processing server 10.

[0034] The data distribution unit 17 (display control unit) transmits the virtual images generated by meshing and bounding boxing, as described above, to the user terminal 20. At this time, the data distribution unit 17 transmits the virtual images to be displayed on the display unit of the user terminal 20, corresponding to the location of the user terminal 20 as described above. As a result, as shown in Figure 2, the display unit of the user terminal 20 displays meshed and bounding boxed virtual images V1 and V2 superimposed on the locations of trees T1 and T2, which are objects in the real-world image, or displays a virtual image V3 that includes measured height information.

[0035] [Operation] Next, the operation of the information display system described above will be explained with reference to the flowcharts in Figures 7 and 8. First, the operation of the information processing server 10 will be explained with reference to the flowchart in Figure 7.

[0036] The information processing server 10 acquires location information of the user terminal 20 and the sensor 40 (shooting position, shooting direction, shooting settings and status, additional information, etc.) (step S1). Then, the information processing server 10 aligns the sensor 40 with the user terminal 20 (step S2). The information processing server 10 also acquires point cloud data, which is the shooting data 30 (step S3). At this time, the information processing server 10 associates the point cloud data with the position of the user terminal 20 according to the alignment described above. The information processing server 10 may also perform operations such as combining point cloud data captured by the moving sensor 40 or synthesizing (registration) point cloud data captured from multiple sensors 40. When combining several fields of view captured by the 3D sensor, the point cloud data is also aligned using location information or external data. The alignment may be performed after compression processing such as meshing of the three-dimensional point cloud data.

[0037] Next, the information processing server 10 generates a virtual image by meshing the three-dimensional point cloud data so that the parts exceeding the limiting surface are clearly visible (step S4). In addition, the virtual image may be generated to display normal parts as objects, not just parts exceeding the limiting surface or violations / abnormal parts, and the virtual image may be generated according to the purpose requested by the user P. Note that the meshing and other processing may be performed on the user terminal 20 at the site instead of on the information processing server 10. Alternatively, a virtual image may be generated that displays the parts exceeding the limiting surface by placing a box (bounding box) instead of meshing, or a virtual image may be generated with arrows or pin-shaped shapes placed on it. Note that pre-processing and post-processing such as noise reduction and correction may be performed during point cloud processing and meshing processing.

[0038] The information processing server 10 then stores the necessary or all processing data in the information storage unit 16 (step S5) and updates the necessary or all data (step S6), enabling it to always display real-time information on the user terminal 20. At this time, the information processing server 10 can adjust the granularity and accuracy of the virtual images it generates based on instructions from individuals such as user P, coordination with external systems, and judgments regarding the real-time nature of internal processing and resource availability, thereby enabling real-time information display. The information processing server 10 also generates virtual images with additional information, such as measured values ​​of the object's size, as needed (step S7), and transmits these virtual images to each user terminal 20 (step S8).

[0039] Next, the operation of the user terminal 20 will be explained with reference to the flowchart in Figure 8. The user terminal 20 captures the image data 30 using the sensor 40 (step S11) and acquires a meshed or bounding boxed virtual image transmitted from the information processing server 10 (step S12). At this time, location information and other data are also acquired. The user terminal 20 then displays the transparent or digitally converted real-world image on the display unit A and superimposes the acquired virtual image onto this real-world image (step S13). If the user terminal 20 is not connected to the sensor 40, step S11 may be omitted. If the user terminal 20 generates a virtual image, it may display such a virtual image.

[0040] Subsequently, the user terminal 20 may add information such as messages to the captured data 30 or other acquired data as needed (step S14). Then, the user terminal 20 transmits the acquired captured data and location information to the information processing server 10 (step S15).

[0041] As described above, according to this embodiment, three-dimensional point cloud data of trees and buildings around the airport can be acquired using an imaging device such as 3DLIDAR, and a virtual image that reveals parts that protrude from the restricted surface can be superimposed on the real-world image and displayed on a user terminal 20 such as MR glasses. As a result, inspection and monitoring that takes aviation laws into consideration can be made easier, and the efficiency of on-site monitoring work can be improved.

[0042] Here, other examples of the use of the information display system described above will be explained with reference to Figures 9 to 10. Figure 9 shows a case where monitoring tasks such as measuring the distance between utility poles and power lines are performed on a large site.

[0043] As shown in Figure 9, user P wears the user terminal 20 and goes to the site to observe the target utility poles and power lines. The user also takes a photograph of the site with the three-dimensional sensor 40, and uses the three-dimensional point cloud data obtained from the photograph to perform detection of the target utility poles and service lines, as well as measurement of the height and spacing of the utility poles on the information processing server (not shown). The information processing server generates a virtual image consisting of "measured values" and text information of the detected "utility poles" and "service lines," and transmits it to the user terminal 20. As a result, as shown in Figure 9, the display unit A of the user terminal 20 can display a real-world image that is transparent to or digitally converted by the display unit A, while superimposing the virtual images V11 and V12, which consist of measured values ​​and text information of the target objects generated by the information processing server, onto this real-world image.

[0044] Figure 10 also illustrates a scenario where monitoring operations are performed to detect cracks in the ground in vast areas such as airports, highways, and national roads. User P wears a user terminal 20 and goes to the site to observe the target ground. A sensor 40, such as 3DLIDAR, photographs the site, and the three-dimensional point cloud data obtained from the images is used by an information processing server (not shown) to detect cracks in the ground and measure their length. The information processing server generates a virtual image consisting of text information of the "measured value" and transmits it to the user terminal 20. As a result, as shown in Figure 10, the display unit A of the user terminal 20 can display a real-world image that is either transparent to the display unit A or digitally converted, and superimpose a virtual image V21, which contains text information of the measured value indicating the length of the cracks, generated by the information processing server, onto this real-world image. Note that the acquisition of image data by the sensor 40 may be performed, for example, by a camera mounted on a drone or vehicle.

[0045] <Embodiment 2> Next, a second embodiment of the present disclosure will be described with reference to Figures 11 to 12. Figures 11 to 12 are block diagrams showing the configuration of the information display system in Embodiment 2. In this embodiment, the configuration of the information display system described in the above-described embodiment is shown in outline.

[0046] First, with reference to Figure 11, the hardware configuration of the information display system 100 in this embodiment will be described. The information display system 100 is composed of a general information processing device, and as an example, it is equipped with the following hardware configuration. ·CPU(Central Processing Unit)101(Arithmetic unit) ROM (Read Only Memory) 102 (Storage Device) • RAM (Random Access Memory) 103 (Storage Device) • Program group 104 loaded into RAM 103 • Storage device 105 for storing the program group 104 • Drive device 106 for reading and writing to external storage medium 110 of the information processing device. • Communication interface 107 connecting to a communication network 111 outside the information processing device. • Input / output interface 108 for data input and output. • Bus 109 connecting each component

[0047] Figure 11 shows an example of the hardware configuration of the information processing device, which is the information display system 100, and the hardware configuration of the information processing device is not limited to the case described above. For example, the information processing device may consist of only a part of the configuration described above, such as not having a drive device 106. In addition, the information processing device may use a GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof instead of the CPU described above.

[0048] The information display system 100 can be equipped with the acquisition unit 121, alignment unit 122, generation unit 123, and display control unit 124 shown in Figure 12, by having the CPU 101 acquire the program group 104 and execute it. The program group 104 is, for example, stored in advance in a storage device 105 or ROM 102, and the CPU 101 loads it into RAM 103 and executes it as needed. The program group 104 may also be supplied to the CPU 101 via a communication network 111, or it may be stored in advance in a storage medium 110, and the drive device 106 reads the program and supplies it to the CPU 101. However, the acquisition unit 121, alignment unit 122, generation unit 123, and display control unit 124 described above may be constructed with dedicated electronic circuits to realize such means.

[0049] The acquisition unit 121 acquires positional information of a display device that displays a real-world image, a shooting device that photographs a predetermined object, and the object in the image captured by the shooting device.

[0050] The alignment unit 122, based on position information, associates the position of the object in the captured image with the position of the display device.

[0051] The generation unit 123 generates a virtual image of the object based on the object's position information. For example, the generation unit 123 generates a virtual image with a simplified shape of the object by compressing the object's position information.

[0052] The display control unit 124 controls the display device to overlay a virtual image of the object, corresponding to the position of the display device, onto the real-world image.

[0053] This disclosure, configured as described above, aligns the positional information of the captured data using the imaging device with the position of the user's terminal, and further superimposes a virtual image of the object within the captured data onto the real-world image displayed on the user's terminal. As a result, the user can easily recognize the position and status of the monitored object on the real-world image using the user's terminal. Consequently, the efficiency of on-site monitoring operations can be improved.

[0054] The programs described above can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0055] Although the present disclosure has been described above with reference to the embodiments described above, the present disclosure is not limited to the embodiments described above. Various modifications to the configuration and details of the present disclosure can be made that can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, at least one of the functions of the acquisition unit 121, alignment unit 122, generation unit 123, and display control unit 124 described above may be performed on an information processing device installed and connected at any location on the network, that is, it may be performed using so-called cloud computing.

[0056] <Note> Some or all of the above embodiments may also be described as follows. The following outlines the configuration of the information display system, information display method, and program in this disclosure. However, this disclosure is not limited to the following configuration. (Note 1) A display device that displays a real-world spatial image, a shooting device that photographs a predetermined object, and an acquisition unit that acquires positional information of the object in the image captured by the shooting device. A positioning unit that associates the position of the object in the captured image with the position of the display device based on the position information, A generation unit that generates a virtual image of the object based on the position information of the object, The display device includes a display control means that controls the display device to display the virtual image of the object corresponding to the position of the display device superimposed on the real-world image, Equipped with an information display system. (Note 2) The information display system described in Appendix 1, The generation unit compresses the position information of the object to generate the virtual image. Information display system. (Note 3) The information display system described in Appendix 2, The generation unit generates the virtual image, which is a simplified representation of the shape of the object. Information display system. (Note 4) The information display system described in Appendix 2, The generation unit generates virtual images only for objects in the captured image that meet pre-set criteria. Information display system. (Note 5) The information display system described in Appendix 2, The image captured by the aforementioned imaging device is point cloud data including the three-dimensional coordinates of the object. The generation unit generates the virtual image by meshing, polygonizing, or bounding boxing the point cloud data of the object. Information display system. (Note 6) The information display system described in Appendix 2, The image captured by the aforementioned imaging device is point cloud data including the three-dimensional coordinates of the object. The generation unit measures the size of the object based on the object's position information and generates a virtual image that includes the measured value. Information display system. (Note 7) The information display system described in Appendix 1, The acquisition unit acquires, from the display device, the location information of the display device along with the identification information of the imaging device acquired by the display device, and acquires the location information of the imaging device that is pre-associated with the identification information of the imaging device. Information display system. (Note 8) A display device that displays a real-world spatial image, a shooting device that photographs a predetermined object, and the acquisition of positional information of the object in the image captured by the shooting device, Based on the position information, the position of the object in the captured image is associated with the position of the display device. Based on the location information of the object, a virtual image of the object is generated. The display device is controlled to display the virtual image of the object, corresponding to the position of the display device, superimposed on the real-world image. Information display method. (Note 9) A display device that displays a real-world spatial image, a shooting device that photographs a predetermined object, and the acquisition of positional information of the object in the image captured by the shooting device, Based on the position information, the position of the object in the captured image is associated with the position of the display device. Based on the location information of the object, a virtual image of the object is generated. The display device is controlled to display the virtual image of the object, corresponding to the position of the display device, superimposed on the real-world image. A program that causes a computer to perform a process.

[0057] Furthermore, this invention benefits from the priority claim based on the patent application No. 2022-156277 filed in Japan on September 29, 2022, and all contents described in said patent application are incorporated herein by reference. [Explanation of symbols]

[0058] 10. Information Processing Server 11 Location information acquisition section 12 Alignment function unit 13 Data Collection Unit 14 Information Management Department 15 Mesh Processing Unit 16. Information Holding Department 17 Data Distribution Department 20 User terminals 21 Data Collection Unit 22 Information display section 23 Information Provision Department 24 Data Distribution Department 30 Shooting data 40 sensors 100 Information Display System 101 CPU 102 ROM 103 RAM 104 Program Groups 105 Storage device 106 Drive unit 107 Communication Interface 108 Input / Output Interfaces 109 Bus 110 Storage medium 111 Communication Network 121 Acquisition Department 122 Alignment section 123 Generation part 124 Display Control Unit

Claims

1. A display device that displays a real-world spatial image, a shooting device that photographs a predetermined object, and an acquisition unit that acquires positional information of the object in the image captured by the shooting device, and also acquires positional information of the object from the image, which is point cloud data including the three-dimensional coordinates of the object. A positioning unit that associates the position of the object in the captured image with the position of the display device based on the position information, A generation unit generates a virtual image of the object, including a shape indicating the portion of the object that exceeds the limiting surface, based on the point cloud data of the object. The display device includes a display control means that controls the display device to display the virtual image of the object corresponding to the position of the display device superimposed on the real-world image, Equipped with an information display system.

2. An information display system according to claim 1, The generation unit detects the designated object in the captured image and generates a virtual image that includes a shape indicating the portion of the object that exceeds the limiting surface, for that object only. Information display system.

3. An information display system according to claim 1, The generation unit detects the object within a set range in the captured image and generates a virtual image that includes a shape indicating the portion of the object that exceeds the limiting surface. Information display system.

4. An information display system according to claim 1, The generation unit detects the object in the captured image and generates a virtual image that includes a shape indicating the portion of the object that exceeds the limiting surface, with respect to only the other objects excluding the specified object. Information display system.

5. An information display system according to claim 1, The generation unit measures the height of the object based on the point cloud data of the object and generates a virtual image that includes character information representing the measured value. Information display system.

6. The information display system according to claim 5, When the generation unit generates the virtual image of the object predicted from the measured value of the object, it generates the virtual image of the portion of the object where the measured value is expected to exceed the limiting surface. Information display system.

7. An information display system according to claim 1, The acquisition unit acquires, from the display device, the location information of the display device along with the identification information of the imaging device acquired by the display device, and acquires the location information of the imaging device that is pre-associated with the identification information of the imaging device. Information display system.

8. A display device that displays a real-world image, a shooting device that photographs a predetermined object, and the acquisition of positional information of the object in the image captured by the shooting device, as well as the acquisition of positional information of the object from the captured image, which is point cloud data including the three-dimensional coordinates of the object. Based on the position information, the position of the object in the captured image is associated with the position of the display device. Based on the point cloud data of the object, a virtual image of the object is generated, which includes a shape indicating the portion of the object that exceeds the restricted surface. The display device is controlled to display the virtual image of the object, corresponding to the position of the display device, superimposed on the real-world image. Information display method.

9. A display device that displays a real-world image, a shooting device that photographs a predetermined object, and the acquisition of positional information of the object in the image captured by the shooting device, as well as the acquisition of positional information of the object from the captured image, which is point cloud data including the three-dimensional coordinates of the object. Based on the position information, the position of the object in the captured image is associated with the position of the display device. Based on the point cloud data of the object, a virtual image of the object is generated, which includes a shape indicating the portion of the object that exceeds the restricted surface. The display device is controlled to display the virtual image of the object, corresponding to the position of the display device, superimposed on the real-world image. A program that causes a computer to perform a process.

Images