Information processing apparatus, displacement observation system, displacement observation method, and program

Abstract

To provide an improved technology with high convenience while allowing a user to easily and clearly visually recognize observation of displacement of an object.SOLUTION: An information processing apparatus includes: first acquisition means configured to acquire image data from imaging means configured to capture an image of an object; second acquisition means configured to acquire point cloud data indicating a surface shape of the object as a point cloud from measurement means configured to measure distances to a plurality of points on a surface of the object; generation means configured to generate displacement data representing a displacement amount of the object based on a plurality of pieces of the point cloud data acquired at different times; and first display control means configured to cause display means to display a first superimposed image in which a first image object indicating the displacement amount is superimposed on an image of the object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for observing the displacement of an object. [Background technology]

[0002] There are known techniques for measuring and displaying the displacement of an object. For example, Patent Document 1 discloses a 3D laser scanner and a technique for processing measurement data obtained by the 3D laser scanner to display an image showing the displacement of a structure, such as a bridge girder, and a displacement distribution map in a surface model.

[0003] Patent Document 2 discloses a technique for detecting a displacement from an image captured by a camera, and displaying an image indicating a vector of the detected displacement by superimposing it on the captured image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4114792 [Patent Document 2] Patent No. 6970893 Summary of the Invention [Problem to be solved by the invention]

[0005] The invention of Patent Document 1 is limited to simply measuring displacement using a laser, while the invention of Patent Document 2 is merely detecting displacement using a camera alone.

[0006] In view of the above background, the present invention provides an improved technology that allows a user to easily and clearly visually observe a displacement in an object, while being highly convenient. [Means for solving the problem]

[0007] One aspect of the present disclosure provides an information processing device having a first acquisition means for acquiring image data from an imaging means for capturing an image of an object, a second acquisition means for acquiring point cloud data indicating the surface shape of the object as a point cloud from a measurement means for measuring distances to multiple points on the surface of the object, a first generation means for generating displacement data indicating an amount of displacement of the object based on multiple pieces of the point cloud data acquired at different times, and a first display control means for causing a display means to display a first superimposed image in which an image object indicating the amount of displacement is superimposed on the image of the object.

[0008] The image processing device may further include a position alignment unit that aligns the image of the object with the image object in the first superimposed image.

[0009] In this information processing device, the first acquisition means and the second acquisition means may acquire the image data and the point cloud data, respectively, in approximately real time, and the first display control means may display the superimposed image on the display means in approximately real time.

[0010] This information processing device may have a first receiving means for receiving a designation of a reference time and a designated time in the second acquisition means, and the first generating means may generate the displacement data indicating the amount of displacement of the object from the reference time to the designated time.

[0011] The first receiving means may receive the designation for a portion of the plurality of points on a surface of the target object, and the first generating means may generate the displacement data indicating the amount of displacement at the portion of the points.

[0012] The first accepting means may accept designation of a measurement range including coordinates of the partial point, and the first generating means may generate the displacement data indicating a statistical value of the amount of displacement within the measurement range.

[0013] The second acquisition means may acquire the point cloud data at a plurality of times including a first time and a second time, and the information processing device may have a second display control means for causing the display means to display a screen including a first display area for displaying a point cloud image of the object obtained from the point cloud data at at least one of the first time and the second time and a second image object showing a cross-section specified for the point cloud image, and a second display area for displaying a second superimposed image in which the point cloud of the cross-section at the first time and the point cloud of the cross-section at the second time are superimposed.

[0014] The information processing device may include a second accepting means for accepting designation of the cross section in the first display area.

[0015] The second accepting means may accept an instruction to rotate the point cloud image in the first display area, and the second display control means may display the point cloud image rotated in accordance with the instruction.

[0016] The second accepting means may accept designation of the first time and the second time.

[0017] The sensor may further include a warning unit that warns of an abnormality in the object when the amount of displacement exceeds a predetermined threshold value.

[0018] The measurement means may include a LiDAR.

[0019] Another aspect of the present disclosure provides a displacement observation system having an imaging means for capturing an image of an object, a measuring means for measuring distances to multiple points on a surface of the object, a first acquisition means for acquiring image data representing the image from the imaging means, a second acquisition means for acquiring point cloud data representing the surface shape of the object as a point cloud from the measuring means, a first generation means for generating displacement data representing an amount of displacement of the object based on multiple pieces of the point cloud data acquired at different times, and a first display control means for causing a display means to display a first superimposed image in which an image object representing the amount of displacement is superimposed on the image of the object.

[0020] Yet another aspect of the present disclosure provides a displacement observation method comprising the steps of acquiring image data from an imaging means that captures an image of an object, acquiring point cloud data indicating a surface shape of the object as a point cloud from a measurement means that measures distances to a plurality of points on the surface of the object, generating displacement data indicating the displacement of the object based on the point cloud data acquired at different times, and displaying on a display means a first overlay image in which an image object indicating the displacement is superimposed on the image of the object.

[0021] Yet another aspect of the present disclosure provides a program for causing a computer to execute the steps of acquiring image data from an imaging means that captures an image of an object, acquiring point cloud data indicating the surface shape of the object as a point cloud from a measurement means that measures distances to multiple points on the surface of the object, generating displacement data indicating the amount of displacement of the object based on the multiple point cloud data acquired at different times, and displaying on a display means a first overlay image in which an image object indicating the amount of displacement is superimposed on the image of the object. Effect of the Invention

[0022] According to the present invention, it is possible to provide an improved and highly convenient technique that enables a user to easily and clearly visually observe the displacement of an object. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a displacement observation system 1. [Diagram 2] FIG. 2 is a diagram illustrating an example of the functional configuration of the displacement observation system 1. [Diagram 3] FIG. 2 is a diagram illustrating an example of the hardware configuration of a user terminal 10. [Figure 4] 4 is a flowchart illustrating an initial setting method in the displacement observation system 1. [Diagram 5] FIG. 13 is a diagram illustrating an example of an input screen for designated items. [Figure 6] 4 is a sequence chart illustrating a real-time monitoring method in the displacement observation system 1. [Figure 7] FIG. 4 is a diagram illustrating image data of an object. [Figure 8] FIG. 2 is a diagram illustrating an example of a point cloud database in the displacement observation system 1. [Figure 9] FIG. 2 is a diagram illustrating an example of a displacement database in the displacement observation system 1. [Figure 10] FIG. 13 is a diagram illustrating an example of a display of a first superimposed image. [Figure 11] FIG. 4 is a diagram illustrating an example of an outline of a first superimposed image. [Figure 12] 4 is a sequence chart illustrating a free analysis method in the displacement observation system 1. [Figure 13] FIG. 13 is a diagram illustrating a cross section setting screen. [Figure 14] FIG. 4 is a diagram illustrating an example of a first display area. [Figure 15] FIG. 4 is a diagram illustrating a second display area. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] 1. Configuration FIG. 1 is a diagram illustrating the system configuration of a displacement observation system 1. In this example, the displacement observation system 1 (or simply the system) is a system for observing the "displacement or amount of displacement" of objects such as buildings, roads (road surfaces), slopes, and the ground. The displacement observation system 1 is used, for example, to observe displacement in areas where workers cannot or have difficulty entering. The displacement observation system 1 has, for example, a LiDAR (Light Detection And Ranging) device capable of laser surveying, which is installed at a high point on the site with a single pipe, and a camera for taking images. In this example, "displacement" refers to, for example, a deformation of an object (so-called subsidence, uplift, landslide, horizontal movement, etc.) that may affect surrounding facilities or the ground, etc., due to construction work, etc. In other words, one of the purposes of the displacement observation system 1 is to easily and clearly present the displacement of the object and the range in which the displacement has occurred to the user using the above-mentioned observation technology, and to detect the presence or absence of an abnormality (danger) based on the displacement of the object.

[0025] Here, the problems of the above-mentioned construction work and the like that have been encountered in the past will be described with a more specific construction example. For example, consider a case where a nearby road is deformed, such as subsidence, by a steel pipe pile installation work (construction work in which steel pipe piles are inserted into the ground and installed). In this example, the construction workers who manage the work have difficulty checking the road deformation during the day when vehicles are passing by, so for example, they close the road at night and the workers enter the road to check the deformation. However, such a method involves the risk of danger to the workers due to nighttime work, such as the risk of accidents such as tripping and falling. There is also a problem that it is not possible to constantly monitor the occurrence of accidents and disasters due to deformation of the road. The displacement observation system 1 according to this embodiment addresses such problems.

[0026] In this example, the displacement observation system 1 includes a user terminal 10, a user terminal 20, a camera 310, and a LiDAR device 320. The user terminal 10 (an example of an information processing device) is a terminal device operated by a user U1 who is a worker at the site. The user U1 (or simply a user) is an example of a user who observes the displacement of an object. The user U1 operates the user terminal 10 in an office installed at the site to monitor the displacement of the object.

[0027] In this example, the user terminal 20 (an example of an information processing device) is a terminal device used by an administrator (an example of a user). The administrator is a user who manages the entire work process at the site. For example, the administrator analyzes the displacement of an object based on past data accumulated in a database. Unlike users who constantly monitor the displacement of an object, such as field workers, the administrator is distinguished as a user who evaluates the danger of an object from past data and predicts future displacement. In this example, the user terminal 10 and the user terminal 20 can exchange various data via Wi-Fi 51 and a network 52 (or the cloud).

[0028] In this example, the camera 310 (an example of a photographing means) photographs an image of an object and generates image data. In this example, the image data is data of an image (for example, a 2D image) of the object captured at a certain angle of view. In this example, the LiDAR device 320 (an example of a measuring means) measures the distance to a plurality of points on the surface of the object and generates point cloud data showing the object (surface shape) as a point cloud. In this example, the point cloud data is data including distance (coordinate or position) information from the LiDAR device 320 to the object. In addition, in this example, the point cloud data is data including a point cloud image that expresses the surface shape of the object with points. The camera 310 and the LiDAR device 320 are connected to the user terminal 10 via a repeater and transmit the various data described above. In addition, at least some of these devices are powered by a generator or the like. Alternatively, they may be powered via a repeater or the like using PoW (Power over Ethernet).

[0029] In order to photograph or measure any position on-site, the camera 310 and the LiDAR device 320 are installed, for example, at a high place a little away from the target object. In this example, the installation location is assumed to be on top of a structure constructed using single pipes or the like. In addition, when installing, it is necessary to select a place with a solid foundation, and it is preferable to set up the structure constructed using single pipes or the like so that it does not move from its position.

[0030] In this example, the warning light 330 (one example of the warning means 33) warns of an abnormality in the object when the amount of displacement in the object exceeds a predetermined threshold. The warning light 330 includes, for example, a sound generating device such as a patrol light (registered trademark), a siren, and an alert (notification) function to a pre-registered smartphone. By using these devices and terminals, the displacement observation system 1 can notify the user U1 of an abnormality in the object. Note that multiple smartphones, tablets, or portable terminals such as iPads (registered trademarks) can be registered in advance.

[0031] 2 is a diagram illustrating an example of the functional configuration of the displacement observation system 1. In this embodiment, the user terminal 10 has a first acquisition means 11, a second acquisition means 12, a generation means 13, a first display control means 14, an alignment means 15, a second display control means 16, a first reception means 17, a second reception means 18, a storage means 191, a control means 192, and a display means 193. In this example, the storage means 191 stores various data including, for example, a database. In this example, the control means 192 performs various controls. In this example, the display means 193 displays various information.

[0032] In this example, the first acquisition means 11 acquires image data from an imaging means 31 that captures an image of an object. In this example, the second acquisition means 12 acquires point cloud data that indicates the surface shape of the object as a point cloud from a measurement means 32 that measures distances to multiple points on the surface of the object.

[0033] Moreover, the first acquisition means 11 and the second acquisition means 12 acquire image data and point cloud data in approximately real time, respectively. In this example, approximately real time refers to a state in which there is almost no time difference between the time when the image data is acquired and the time when the image data is generated (i.e., the image is displayed on a display or the like). Almost no time difference refers to a state in which a change in an object in an image displayed by image data has a delay of, for example, only a few seconds at most from a change in the object in reality. Therefore, by constantly synchronizing the user terminal 10 (or the user terminal 20) with the camera 310 and the LiDAR device 320, the user terminal 10 can always update (acquire) the latest data. Furthermore, in this example, the first acquisition means 11 and the second acquisition means 12 can acquire image data and point cloud data at a plurality of times (each of which is an example of different times) including the first time and the second time. The different times are, for example, a predetermined time (for example, 12 o'clock every day), a predetermined time interval (for example, every 12 hours), or a random time.

[0034] In this example, the generating means 13 generates displacement data representing the amount of displacement of an object based on point cloud data acquired at different times. In this example, the displacement data represents the amount of displacement of a specified point (or measurement point) between the start time and end time of measurement in the coordinates of the specified point.

[0035] In this example, the first display control means 14 causes the display means 193 to display a first superimposed image in which a first image object indicating a displacement amount is superimposed on an image of the object. In this example, the first image object is a mark indicating a position and a displacement amount at the coordinates of a specified point observed at a different time. In this example, the mark includes a symbol such as "●" and color information in the case of a position, and includes a displacement direction and a numerical value (including units such as cm or mm) in the case of a displacement amount. In this example, the first superimposed image includes an image of the object and the first image object, and includes an image in which each element is superimposed based on the respective correspondence relationships. In this example, superimposition refers to a process of superimposing multiple types of images (or objects) according to a predetermined correspondence relationship and display format. In this example, the correspondence relationship includes a correspondence relationship between the image of the object and the position of the specified point in the first image object.

[0036] In this example, the alignment means 15 aligns the image of the object with the first image object in the first superimposed image. In this example, the alignment includes adjusting the orientation, scale, and positional relationship between the image of the object and the first image object.

[0037] In this example, the second display control means 16 causes the display means 193 to display a screen including a first display area displaying a point cloud image of an object obtained from point cloud data at least one of the first time and the second time and a second image object showing a cross section in a direction specified for the point cloud image. In this example, the point cloud image is an image displayed according to point cloud data, and is an image in which the surface shape of the object is expressed by points. In this example, the second image object is an object (a landmark or a figure) on the display that visualizes a virtual cross section (or simply a surface) cutting the object in a certain direction in the point cloud image of the object. In this example, the first display area represents, for example, a division (section) on the display screen provided to display an image (data) on the display means 193 according to various purposes.

[0038] Further, the second display control means 16 causes the display means 193 to display a screen including a second display area displaying a second superimposed image in which the point cloud of the cross section at the first time and the point cloud of the cross section at the second time are superimposed. In this example, the second superimposed image is an image including the point cloud of the cross section at the first time and the point cloud of the cross section at the second time, and visualizing the displacement or the amount of displacement of the cross section in the object. In this example, the second display area, like the first display area, represents a division (section) on the display screen for each purpose. Note that it is desirable to display the point cloud of the cross section at the first time and the point cloud of the cross section at the second time in different colors.

[0039] In this example, the first receiving means 17 receives the designation of the reference time and the designated time in the second acquiring means 12. In this example, the generating means 13 generates displacement data indicating the amount of displacement of the object from the reference time to the designated time. The first receiving means 17 receives the designation of some of the points among the multiple points on the surface of the object. In this example, the generating means 13 generates displacement data indicating the amount of displacement at the coordinates of the designated some of the points. Hereinafter, the point designated by the coordinates is referred to as the "designated point". The first receiving means 17 receives the designation of a measurement range including the coordinates of the designated point. In this example, the measurement range refers to, for example, a region formed by a set of the designated point and a set of multiple points selected in order of proximity to the designated point up to a designated number, or a designated area. In this example, the range of the region formed by the set of the above-mentioned multiple points is referred to as the measurement range. In this example, the generation means 13 generates displacement data indicating a statistical value of the amount of displacement in the measurement range. In this example, the statistical value is, for example, any of the average / weighted average value, median value, maximum value, and mode (most frequent value).

[0040] In this example, the second receiving means 18 receives a designation of a cross-sectional direction in the first display region. The second receiving means 18 receives an instruction to rotate the point cloud image in the first display region. In this example, the second display control means 16 can display the point cloud image rotated in accordance with the instruction. The second receiving means 18 receives a designation of a first time and a second time.

[0041] FIG. 3 is a diagram illustrating a hardware configuration of the user terminal 10. In this example, the user terminal 10 is a computer or information processing device having a CPU (Central Processing Unit) 101, a memory 102, a storage 103, a communication IF 104, an input device 105, and a display device 106, and includes, for example, a smartphone, a tablet, or a personal computer. The CPU 101 is a processor that performs various calculations according to a program. The memory 102 is a main storage device that functions as a work area when the CPU 101 executes a program, and includes, for example, a RAM (Random Access Memory). The storage 103 is an auxiliary storage device that stores various data and programs, and includes, for example, an SSD (Solid State Drive) or an HDD (Hard Disc Drive). The communication IF 104 is a device that communicates with other devices according to a predetermined communication standard, and includes, for example, a NIC (Network Interface Card). Or, for example, in the case of wireless communication, includes a wireless chip. The input device 105 is a device for inputting information to the user terminal 10, and includes, for example, a touch screen, a keyboard, a mouse, or a pointing device. The display device 106 is a device for displaying information, and includes, for example, an organic EL display or a liquid crystal display.

[0042] In this example, the programs stored in the storage 103 include a program (hereinafter referred to as a "server program") for making the computer function as a server in the displacement observation system 1. When the CPU 101 is executing the server program, the CPU 101, the memory 102, the storage 103, the communication IF 104, the input device 105, and the display device 106 are examples of functions for operating the user terminal 10. The CPU 101 is an example of the generation means 13, the first display control means 14, the positioning means 15, the second display control means 16, and the control means 192. At least one of the memory 102 and the storage 103 is an example of the storage means 191. The communication IF 104 is an example of the first acquisition means 11 and the second acquisition means 12. The input device 105 is an example of the first reception means 17 and the second reception means 18. The display device 106 is an example of the display means 193.

[0043] The user terminal 20 is a computer having the same hardware configuration as the above-mentioned user terminal 10, and includes, for example, a smartphone, a tablet, or a personal computer (not shown). In this example, the user terminal 20 has the same server program (or a different client program, etc.) as the above-mentioned user terminal 10. The configuration of the displacement observation system 1 has been described above. Next, the operation of the displacement observation system 1 will be described.

[0044] 2. Operation Here, the operation of the displacement observation system 1 will be explained. More specific operations regarding the processing of the various devices (subjects) included in the displacement observation system 1 will be explained in the following two sections. In this example, the operation of the displacement observation system 1 is broadly divided into Section 2-1 "Real-time monitoring" and Section 2-2 "Free analysis". Both operations are included in the embodiments of the displacement observation system 1. First, the initial setting method for real-time monitoring will be explained.

[0045] FIG. 4 is a flowchart illustrating an initial setting method in the displacement observation system 1. In this example, for example, when a worker (monitor) at the site performs monitoring, it is necessary to install each device for the object to be observed and to construct a system network for the displacement observation system 1. In step S1, the camera 310 and the LiDAR device 320 are set up at the site, and the shooting and measurement positions are determined. In this example, the camera 310 and the LiDAR device 320 are usually installed at a high position using a single pipe or the like, and the object is observed from above, for example. In this example, when the object is a road, for example, each device is placed at a high position constructed with a single pipe or the like on a place with solid ground.

[0046] In step S2, the displacement observation system 1 synchronizes each device. Here, "synchronizing" means that the user terminal 10 starts acquiring image data and point cloud data from the camera 310 and the LiDAR device 320 in approximately real time. This allows the user terminal 10 to always acquire the latest data. At this time, the displacement observation system may automatically detect and record the positions of the objects photographed or measured by the camera 310 and the LiDAR device 320, or may synchronize the camera 310 and the LiDAR device 320 to pan, tilt, or roll.

[0047] In step S3, the displacement observation system 1 acquires various data. In this example, the user terminal 10 acquires image data and point cloud data from the camera 310 and the LiDAR device 320, respectively. In this example, the acquisition of various data is, for example, performed on a trial basis, and the user, a worker, checks at this point whether there are any abnormalities in the data acquired from each device, whether the acquired data includes an object, etc. Alternatively, the user may check whether the positions of the objects in each data in the acquired image data and point cloud data correspond (match) with each other in the image data and the point cloud data.

[0048] In step S4, the displacement observation system 1 accepts designated items (or simply designated items) from the user. In this example, the user inputs the designated items via the user terminal 10. Here, the designated items will be explained below.

[0049] FIG. 5 is a diagram illustrating an input screen for the designated items. In this example, FIG. 5 shows an example of a display screen displayed by the user terminal 10. In this example, the user inputs the designated items in the field F11 (an example of the first receiving means 17) according to the display of the display screen D1. In this example, the field F11 receives the reference time (or the designated time), the designation of the coordinates of the point on the surface of the object, and the designation of the measurement range. In this example, the reference time is a time that is a starting point (reference) for observing the displacement amount of the object. In this example, if it is the initial setting stage, the date and time when the system is started may be set as the reference time. This allows the user terminal 10 to calculate the displacement amount of the object from the reference time to the current time (an example of the designated time). Note that the user terminal 10 may receive the designated time instead of the current time (not shown). This allows the user terminal 10 to calculate the displacement amount of the object from the reference time to the designated time.

[0050] In this example, the initial value (an example of a point) is the coordinate of a designated point, which is a part of a point designated on the surface of an object that the user desires to monitor. In this example, the coordinate includes coordinates (for example, X coordinate, Y coordinate, and Z coordinate) from the LiDAR device 320 to the designated point. In this example, the user may directly input the coordinate of the designated point of the object, or may specify the coordinate of the measurement range from an image / point cloud image of the object, or may calculate the coordinate of the designated point or the measurement range of the object using a feature point whose three-dimensional coordinate value on the image or point cloud image is known. In addition, if the three-dimensional coordinate value of the location where the LiDAR device 320 is installed is known, the coordinate of the object may be calculated from the absolute coordinate value. In addition, the coordinate of the designated point or the measurement range may be singular / plural. In this example, the current value is the coordinate of the designated point or the point of the measurement range at the current time (or the specified time). The user terminal 10 can update the coordinate acquired substantially in real time as the current value.

[0051] In this example, the number of measurements (an example of the measurement range) represents the number of multiple points on the point cloud data including the coordinates of the specified point as an initial value. In this example, the numerical value (e.g., 500) input as the number of measurements represents the number of points determined according to the distance from the specified point as an initial value. For example, 500 points are determined as the measurement range in order of the distance from the specified point as an initial value. This allows the displacement observation system 1 to observe the displacement not only of the specified point as an initial value, but also of multiple points around it. Note that the displacement observation system 1 can generate displacement data by using the specified point as an initial value and a statistical value (average value, median value, maximum value, mode value, etc.) of the amount of displacement at multiple points specified as the measurement range (described later). The user terminal 10 can accept multiple specified points.

[0052] Returning to Fig. 4, in step S5, the user terminal 10 records the various specifications received from the user in the database. In this example, the recorded specification items are used when generating displacement data. Next, the actual monitoring method will be described.

[0053] 2-1. Real-time monitoring method 6 is a sequence chart illustrating a real-time monitoring method in the displacement observation system 1. In this example, real-time monitoring includes a method for monitoring the displacement of an object in approximately real time at a site where construction work or the like is actually performed. For example, a worker at the site can observe the displacement of the object using the user terminal 10 and monitor for the presence or absence of an abnormality. Also, instead of the user terminal 10, the presence or absence of an abnormality can be monitored using a portable terminal such as a smartphone that has been registered in advance as a warning light 330.

[0054] In step S101, the camera 310 captures an image of an object and generates image data. The LiDAR device 320 measures distances to a plurality of points on the surface of the object and generates point cloud data that represents the object as a point cloud.

[0055] In step S102, the user terminal 10 acquires various data. In this example, the user terminal 10 acquires image data from the camera 310 and point cloud data from the LiDAR device 320. In this example, the user terminal 10 can, for example, synchronize with the camera 310 and the LiDAR device 320 in real time (or approximately real time) and always acquire the latest image data and point cloud data. Alternatively, the user terminal 10 may acquire various data at a predetermined time (such as midnight) or at a predetermined time interval (every hour). Note that the predetermined time or the predetermined time interval are both examples of different times. First, the image data acquired from the camera 310 will be described.

[0056] 7 is a diagram showing a schematic diagram of image data of an object. In this example, image I1 represents an example of a photograph taken by camera 310 or image data obtained by digitizing the photograph. In this example, road R1 is the object. In this example, camera 310 is installed at a high position assembled from single pipes or the like. Therefore, the viewpoint of image I1 is set, for example, at a position looking down on road R1 from above.

[0057] In this example, the single tube T1 represents a support for holding the camera 310 (and the LiDAR device 320) at an elevated position. In this example, the single tubes T1 are combined to construct the installation location of the camera 310. Note that the image I1 depicts a state in which a part of the single tube T1 captured by the camera 310 is reflected in the photograph (image).

[0058] In this example, bridge B1 is a structure that serves as a reference for associating image data and point cloud data. In this example, the displacement observation system 1 may, for example, identify the positional relationship between image I1 and the point cloud image based on the image of the object (bridge B1) and the point cloud image. Note that image I1 in FIG. 7 is merely an illustration (diagram) that imitates a real landscape. Therefore, the three-dimensionality (planarity) of the object, the distance between the objects, and the perspective are depicted in a simplified manner, and can be ignored. From the above, the user terminal 10 can acquire image data representing the appearance of the object (for example, road R1) from the camera 310.

[0059] Returning to FIG. 6, in step S103, the user terminal 10 records the various acquired data in a database. In this example, the user terminal 10 can record image data and point cloud data in a database for each time of shooting and measurement. In this example, the recording of point cloud data acquired from the LiDAR device 320 will be described.

[0060] FIG. 8 is a diagram illustrating a point cloud database in the displacement observation system 1. In this example, the point cloud database 1001 is a database for recording and managing point cloud data acquired from the LiDAR device 320 in the user terminal 10. In this example, the point cloud database 1001 records a timestamp, a coordinate ID, and coordinates. In this example, the timestamp is a character string indicating the date, time, and the like when the LiDAR device 320 measured the target object. In this example, the coordinate ID is ID information assigned to uniquely identify a point (coordinate). In this example, the coordinate ID is assigned, for example, P1, P2, . . . Pn in order of distance from the LiDAR device 320. In this example, the coordinate is position information for identifying the position of a point from a starting point (origin), and includes, for example, an X coordinate, a Y coordinate, and a Z coordinate. Note that the origin may be, for example, the position of the LiDAR device 320.

[0061] Returning to Fig. 6, in step S104, the user terminal 10 outputs the various acquired data to the user terminal 20. In this example, the user terminal 20 can acquire image data from the camera 310 and point cloud data from the LiDAR device 320 via the user terminal 10 (Wi-Fi 51 and network 52). Note that the user terminal 20 may be installed in multiple locations, such as a site office and a client's office.

[0062] In step S105, the user terminal 20 records the acquired various data in a database, similar to step S103. In this example, the user terminal 20 records and manages the various data in a database having the same (or equivalent) configuration as the above-mentioned database (not shown). The various data recorded in this example is used in Section 2-2 "Free Analysis" described later.

[0063] The above-described processes in steps S101 to S105 are processes for acquiring image data and point cloud data from each device in the system. These processes are appropriately performed at a predetermined number of times, time, and / or frequency. Alternatively, the processes are performed continuously in approximately real time. Next, the generation of displacement data will be described.

[0064] In step S106, the user terminal 10 generates displacement data. In this example, the displacement data is generated based on a plurality of point cloud data acquired at different times. In this example, the different times refer to, for example, two times from a pre-specified starting time (e.g., a reference time) to an end time (e.g., a current time or a specified time). Therefore, in this example, the displacement data is defined as the amount of displacement of the object at these two times. Here, the generation of the displacement data will be described.

[0065] FIG. 9 is a diagram illustrating a displacement database in the displacement observation system 1. In this example, the displacement database 2001 is a database for recording and managing displacement data representing the amount of displacement of an object based on point cloud data acquired at different times. In this example, the displacement database 2001 records a coordinate ID, coordinates, and a displacement amount. In this example, for a point uniquely distinguished by a "coordinate ID", for example, "coordinates" at time A (reference time) and time B (current time) are recorded and compared. In this example, a displacement (an example of displacement data) is, for example, a vector corresponding to the difference in coordinates at time A and time B for a certain measurement point. The magnitude (i.e., length) of this vector is called the displacement amount. For example, the displacement at the point with coordinate ID [P1] indicates that it has displaced upward by a displacement amount corresponding to length 2 in the Z-axis direction (positive numbers indicate upward displacement, negative numbers indicate downward displacement) between time A and time B. The user terminal 10 calculates the amount of displacement for a specified number of points (e.g., 500 points) in the vicinity of the specified point in order of distance from the specified point, or within the range of the multiple points. Furthermore, the user terminal 10 statistically processes the amounts of displacement for these points, for example, calculates an average value. This average value of the displacement amounts is the displacement data for the specified point. In this way, the displacement observation system 1 can generate and record displacement data.

[0066] Returning to FIG. 6, in step S107, the user terminal 10 superimposes a first image object indicating the amount of displacement on the image of the object. In this example, the user terminal 10 identifies pre-specified coordinates and a reference time from a database, and acquires displacement data of each specified point at the current time. After that, the user terminal 10 superimposes the acquired displacement data as a first image object at a corresponding position in the image of the object. This allows the user terminal 10 to generate a first superimposed image. The first image object will be described later.

[0067] In step S108, the user terminal 10 displays the first superimposed image on the display means 193. Here, the display of the first superimposed image will be described.

[0068] FIG. 10 is a diagram illustrating the display of the first superimposed image. In this example, FIG. 10 shows an example of a display screen displayed by the user terminal 10. In this example, a field F21 on the display screen D2 shows the display of the first superimposed image. In this example, in the image of the field F21, a plurality of marks (six of each) of "●" and "[displacement direction (axial direction): numerical value]" which are examples of the first image object are arranged on the road R1 which is the object. In this example, a point P1 (an example of the first image object) is a mark which indicates the position and displacement of a coordinate which is specified in advance in an image I1 showing the object. In this example, [Z:2] of the point P1 includes the displacement direction of the specified point and the displacement amount expressed in a predetermined unit (for example, cm / mm, etc.). The displacement direction in each axis is expressed by a positive number or a negative number.

[0069] In this example, field F22 is a menu-selectable field, and includes UI objects (icons in this example) corresponding to items such as initial values, measurement points, settings, and alerts. In this example, the user can check initial values, add measurement points (coordinates), and specify various settings and alerts.

[0070] In this example, the field F23 (one example of the alignment means 15) is a field that displays a UI object (in this example, an operation button) for performing alignment between the image of the object and the first image object. For example, the operation button includes a cross key. In this example, the user can perform alignment by operating the cross key for either the image of the object or the first image object.

[0071] In this example, field F24 (an example of the alignment means 15) is a field where a switch button is provided for switching between a first image object indicating the amount of displacement and a point cloud image. When the switch button is pressed, the image superimposed on the 2D image of the object alternates between the first image object and the point cloud image. In this example, when performing alignment, the user can use the point cloud image in the point cloud data to perform alignment. As a result, the user can easily and clearly visually recognize the displacement of the object.

[0072] FIG. 11 is a diagram illustrating an outline of the first superimposed image. In this example, FIG. 11 shows an example of a conceptual diagram of the first superimposed image. Here, the first superimposed image is explained by dividing it into two layers (a hierarchical structure consisting of virtual planes) for simplicity. In this example, layer L1 is a layer assigned to image data including a road R1 that is an object. In this example, layer L2 is a layer assigned to a first image object including marks "●" and "[number]" at the coordinates of a specified point. The positions of these layers correspond to each other and are superimposed to form the first superimposed image. In addition, such a hierarchical structure is merely one example of a method for displaying the first superimposed image, and the first superimposed image may be generated and displayed by any method.

[0073] The user terminal 10 may display the mark of a specified point whose displacement amount exceeds the threshold in an appearance (for example, at least one of color, shape, size, and time change (for example, blinking)) different from that of a specified point whose displacement amount does not exceed the threshold. For example, the user terminal 10 displays the mark of a specified point whose displacement amount exceeds the threshold in red (may be blue, yellow, or red depending on the displacement amount from the set threshold), and displays the mark of a specified point whose displacement amount does not exceed the threshold in black.

[0074] When there is a designated point whose displacement amount exceeds the threshold, the user terminal 10 notifies the user that there is a designated point whose displacement amount exceeds the threshold. That is, the user terminal 10 drives the warning light 330. The threshold for driving the warning light 330 may be the same as or different from the threshold for changing the appearance of the mark of the designated point. For example, when the displacement amount is at a slightly dangerous level (e.g., equal to or greater than the first threshold and less than the second threshold), the user terminal 10 only changes the appearance of the mark and does not drive the warning light 330. When the displacement amount is at a very dangerous level (e.g., equal to or greater than the second threshold), the user terminal 10 drives the warning light 330 in addition to changing the appearance of the mark.

[0075] As a result, even if a deformation occurs in an object that cannot be detected by the naked eye at a construction site or the like, the user can visually confirm quantitatively at which point and to what extent the displacement has occurred using this observation system. Furthermore, the user can perform real-time monitoring using the above-mentioned method. Up to this point, the real-time monitoring method in the displacement observation system 1 has been explained. Next, the free analysis method will be explained.

[0076] 2-2.Free analysis method 12 is a sequence chart illustrating a free analysis method in the displacement observation system 1. Here, a case will be described in which a manager (an example of a user) who manages on-site construction and the like performs analysis (free analysis) of displacement data. In this example, the manager does not constantly monitor the object for abnormalities like on-site workers, but analyzes the deformation of the object based on past data (image data, point cloud data, and displacement data) accumulated in a database. In this example, the manager mainly operates the user terminal 20 to perform the analysis.

[0077] In steps S201 to S205, the displacement observation system 1 performs the same processes as steps S101 to S105 in Fig. 6 described above. By repeating these processes, various data can be accumulated in the user terminal 20. Alternatively, various data may be downloaded in bulk every time a free analysis is performed in the user terminal 20. Next, a method in which the administrator performs analysis based on the accumulated data will be described.

[0078] In step S206, the user terminal 20 accepts a cross section designation from the user. In this example, the user can designate a cross section direction, etc. for the point cloud image. Here, the designation of the cross section will be described.

[0079] FIG. 13 is a diagram illustrating a cross-section setting screen. In this example, FIG. 13 shows an example of a display screen displayed by the user terminal 20. In this example, the user inputs a cross-section direction and a time in a field F30 (an example of the second receiving means 18) according to the display of the display screen D3. In this example, the user directly inputs a cross-section start point coordinate (P) and a cross-section end point coordinate (Q) as the cross-section direction. A plane obtained by connecting the two input coordinates with a straight line and cutting in a certain direction (for example, a vertical direction, a gravity direction, or a predetermined direction) is defined as a cross-section. Alternatively, the user may select a field F31 (an example of the second receiving means 18) and select a cross-section direction from a point cloud image of the object by a predetermined method (not shown). In this example, the user specifies the number of cross-sections, the interval between the cross-sections, and the depth of the cross-sections. In this example, the number of cross-sections is the number of cross-sections including the specified cross-section. The interval between the cross-sections is the interval (distance) between each of the multiple cross-sections. In this example, the depth of the cross-section indicates the depth of the cross-section from the earth's surface (not shown). The depth of the cross section represents the depth when the surface of the object represented by the point cloud image is defined as the ground surface, or may be an index (degree) that represents how far the specified point is from the line that connects the cross section start point coordinate (P) and the cross section end point coordinate (Q) as the ground surface.

[0080] In this example, fields F32 and F33 accept the designation of the analysis period from the user. In this example, field F32 accepts the designation of a first time, which is the starting time (reference time), and field F33 accepts the designation of a second time, which is the ending time (designated time). In this way, the user can analyze the displacement of the cross section during the designated period.

[0081] Returning to FIG. 12, in step S207, the user terminal 20 displays a second image object showing the point cloud image of the object and the cross section specified for the point cloud image. More specifically, the user terminal 20 refers to the database, reads the point cloud data at the specified first time and second time, and displays it as a point cloud image. Based on the cross section specification, the user terminal 20 also displays an object on the display (an example of a second image object) visualized as a virtual cross section in the read point cloud image. All of these are displayed in the first display area. Here, the image displayed in the first display area will be described.

[0082] FIG. 14 is a diagram illustrating an example of the first display area. In this example, FIG. 14 shows an example of a display screen displayed by the user terminal 20. In this example, the field F1 of the display screen D4 is an example of the first display area. In this example, the road R1 (R2) represents a point cloud image of an object. In this example, the point cloud image is obtained from point cloud data at least one of the first time and the second time. Alternatively, the road R1 (R2) in the field F1 may be an overlap of point cloud images at two times (the first time and the second time). Note that the point cloud image displayed on the display in the field F1 is, for example, a two-dimensional image of a point cloud in a virtual space viewed from a certain viewpoint (for example, a virtual camera) in the virtual space.

[0083] Here, the position of the point cloud in the virtual space is represented by coordinates in the space defined by the X-axis, Y-axis, and Z-axis with respect to a determined reference point. In this example, the three-dimensional grid Cb1 represents the outer frame of the space including the reference point (origin), the X-axis, the Y-axis, and the Z-axis. In addition, in this example, the surface X1 is an example of a second image object showing a cross section specified for the point cloud image. In this example, the surface X1 is, for example, a surface that passes through two coordinates (cross section start point coordinate P and cross section end point coordinate Q) input in advance and has a pre-specified orientation. In this example, the surface X1 is depicted as a cross section that cuts a surface including points P and Q. Therefore, the user may perform an operation to change the direction of the surface X1 by specifying and rotating the orientation of the surface (for example, a direction perpendicular to the road) after specifying two points, points P and Q. In this example, the origin, the X-axis, the Y-axis, the Z-axis, and the cubic grid Cb1 are displayed in the field F1 for the convenience of explaining the surface X1, but they do not have to be displayed in the first place. In this example, the cursor represents an operation UI for selecting any point of the object in the field F1.

[0084] On this screen, when the user drags and drops an element (for example, an edge or a vertex) of the surface X1, the user terminal 20 may change or update various conditions such as the cross-sectional depth of the surface X1 in response to the operation.

[0085] In this example, the field F41 of the display screen D4 is a field for displaying information of the field F1 and accepting operations by the user. In this example, the field F41 displays information of the first time and the second time. In this example, the field F41 accepts operations (instructions) of setting, cross section, right rotation, and left rotation from the user. In this example, the user can specify the time and the cross section. In this example, the user can rotate the point cloud image (road R1) in the first display area. Regarding the rotation of the point cloud image, the field F41 may accept instructions of forward rotation and backward rotation from the user. In this example, the field F41 displays the "Cross Section No.", "Position", "Time", "Depth (or Z-axis coordinate)", and "Difference" for the point on the cross section pointed to by the cursor in the first display area (field F1). In this example, "Section No.", "Position", "Time", and "Depth" are information for identifying which position (coordinate) on which section the point indicated by the cursor is at (or how deep it is), and whether the point is at the first time or the second time (note that X-axis, Y-axis, and Z-axis coordinates may also be displayed). In this example, "Difference" represents the difference between the "depth (or Z-axis coordinate)" at the first time and the second time, in other words, the amount of displacement. Note that in this example, depth is defined as the degree of depth of the line connecting points P and Q relative to the earth's surface.

[0086] Returning to Fig. 12, in step S208, the user terminal 20 displays a second superimposed image in which the point cloud of the cross section at the first time and the point cloud of the cross section at the second time are superimposed. More specifically, the user terminal 20 can display on the display means 193 a screen including a second display area for displaying the second superimposed image. Here, the display screen of the user terminal 20 will be described.

[0087] FIG. 15 is a diagram illustrating an example of the second display area. In this example, FIG. 15 shows an example of a display screen displayed by the user terminal 20. In this example, the field F2 of the display screen D5 is an example of the second display area. In this example, the road R1 represents the point cloud of the plane X1 at the first time, and in this example, the road R2 represents the point cloud of the plane X1 at the second time. That is, in this example, the field F2 represents a second superimposed image in which the point cloud of the cross section at the first time and the point cloud of the cross section at the second time are superimposed. In this example, the plane X1 is depicted as a cross section cutting a plane including points P and Q representing the earth's surface (depth reference) in the same manner as the field F1. In this example, the depth represents how far the specified point is from the straight line (reference) connecting the points P and Q. In this example, the plane X1 and the points P and Q forming this cross section are displayed in the field F2 for the sake of convenience of explanation only, but they may not be displayed in the first place. In this example, the analysis line represents a line perpendicular to the reference line PQ representing depth, and is a landmark (object) for analyzing the displacement of an object point in field F2. In this example, the analysis line includes points on roads R1 and R2. In this example, the displacement to be analyzed is the displacement at a point belonging to the analysis line. Note that in this example, the "cursor" in FIG. 14 and the "analysis line" in FIG. 15 described above may be linked. In other words, when the cursor points to a specific point including "depth (or Z-axis coordinate): 5" in the point cloud image at "first time", the analysis line includes multiple points on the line to which the specific point belongs.

[0088] In this example, the field F51 of the display screen D5 is a field for displaying information of the field F2. In this example, the field F51 displays information of the first time and the second time. In this example, the field F51 displays "Section No.", "Position", "Depth (Depth 1 and Depth 2)", and "Difference" for points on the cross section included in the analysis line in the second display area (field F2). In this example, the depth 1 and depth 2 represent, for example, the depth (or Z-axis coordinate) of a specified point of the object at the first time and the second time. In this example, the "Difference" represents the difference between the "depth (or Z-axis coordinate)" at the first time and the second time, that is, the amount of displacement. Note that multiple analysis lines can be selected according to the user's request. In this example, the field F51 displays all the displacement amounts at the selected points (up to the nth point).

[0089] It should be noted that the display screen D4 in FIG. 14 and the display screen D5 in FIG. 15 do not have to be separate, but may be the exact same display screen. In other words, various display formats may be adopted according to the convenience of the user. As described above, a user who is a manager in charge of on-site construction work etc. can analyze deformation of an object based on past data (image data, point cloud data, and displacement data) stored in the database. The user can also perform free analysis using the above-mentioned method. The free analysis method in the displacement observation system 1 has been described so far.

[0090] 3. Variations The present invention is not limited to the above-described embodiment, and various modifications are possible. Some modifications will be described below. Two or more of the following items may be combined and applied.

[0091] (1) Displacement Observation System 1 The hardware configuration, network configuration, and functional configuration in the displacement observation system 1 are not limited to those exemplified in the embodiment. The displacement observation system 1 may have any hardware configuration, network configuration, and functional configuration as long as the required functions can be realized. For example, a plurality of devices may physically cooperate to function as the displacement observation system 1. Note that the subject, configuration, and system structure shown in FIG. 1 are merely an example and merely show an overview of the system. Therefore, for example, the user terminal 10, the user terminal 20, the camera 310, and the LiDAR device 320 may all be connected to a computer network such as the Internet. In addition, the displacement observation system 1 may include a warning light 330. Note that in the displacement observation system 1, the user terminal 10 and the user terminal 20 are actually the same terminal device, so they may have exactly the same (or equivalent) functions. Alternatively, the user terminal 10 and the user terminal 20 may be made to function as one information processing device.

[0092] (2) User terminal 10 The form of the functional configuration in the user terminal 10 is not limited to that exemplified in the embodiment. The user terminal 10 may have any form of functional configuration as long as the required functions can be realized. In addition, the correspondence between the functional elements and the hardware is not limited to that exemplified in the embodiment. For example, in the embodiment, at least a part of the functions described as being implemented in the user terminal 10 may be implemented in another device or system, and conversely, at least a part of the functions described as being implemented in another device or system may be implemented in the user terminal 10. In this example, the user terminal 10 may have at least a part of the functions of the user terminal 20. In this example, the user terminal 10 may have at least a part of the functions of the camera 310, the LiDAR device 320, and the warning light 330. For example, both the real-time monitoring function and the free analysis function may be implemented in the user terminal 10.

[0093] (3) User terminal 20 The form of the functional configuration in the user terminal 20 is not limited to that exemplified in the embodiment. The user terminal 20 may have any form of functional configuration as long as the required functions can be realized. Furthermore, the correspondence between the functional elements and the hardware is not limited to that exemplified in the embodiment. For example, in the embodiment, at least a part of the functions described as being implemented in the user terminal 20 may be implemented in another device or system, and conversely, at least a part of the functions described as being implemented in another device or system may be implemented in the user terminal 20.

[0094] (4) Camera 310 The camera 310 is not limited to the one exemplified in the embodiment. In this example, at least a part of the functions of the camera 310 may be controlled by the user terminal 10. The image data in the camera 310 is not limited to the one exemplified in the embodiment. In this example, the resolution of the image in the image data may be changed according to the user terminal 10. In addition, the camera 310 may be any type, and may be an RGB camera, an infrared camera, a hyperspectral camera, a stereo camera, or the like. In this example, for example, the image of the object may be an image by a stereo camera. In this example, the displacement observation system 1 may superimpose an image object indicating the amount of displacement on an image including the object and having distortion correction applied by the stereo camera or a parallax image including brightness according to the distance from the camera 310 by matching processing.

[0095] (5)LiDAR device 320 The LiDAR device 320 is not limited to the one exemplified in the embodiment. In this example, at least a part of the functions of the LiDAR device 320 may be controlled by the user terminal 10. The LiDAR device 320 is not limited to a LiDAR device, and may be any device that can acquire three-dimensional point cloud data, such as a high-performance scanner. The point cloud data in the LiDAR device 320 may include color information of an object that is predetermined according to the intensity of detected light.

[0096] (6) Warning Light 330 - Other The warning light 330 is not limited to those exemplified in the embodiment. In this example, at least a part of the functions of the warning light 330 may be controlled by the user terminal 10. In this example, when the alert is controlled by the user terminal 10, in addition to the notification function of the terminal, a warning may be issued to the user by linking with an external device. The repeater, the generator, Wi-Fi 51, and the network 52 are not limited to those exemplified in the embodiment. In this example, the repeater may include an existing Wi-Fi router, etc. In this example, the generator may include a device that is permanently installed at the site and is capable of supplying power, etc.

[0097] (7) Initial setup method The flowchart shown in Fig. 4 merely shows one example of the operation, and the operation of the displacement observation system 1 is not limited to this. Some of the operations shown in the figure may be omitted, the order may be changed, or new operations may be added. In this example, the acceptance of the specified items in step S4 may be any information. In this example, the user terminal 10 may accept the above-mentioned specifications for each of the multiple objects regarding the specification of time, point, and measurement range.

[0098] (8) Real-time monitoring method The sequence chart shown in FIG. 6 merely shows one example of the operation, and the operation of the displacement observation system 1 is not limited to this. Some of the operations shown in the figure may be omitted, the order may be changed, or new operations may be added. In this example, the user may set a threshold value in advance for the amount of displacement of the object. In this example, when the displacement data is generated in step S106, if the amount of displacement of the specified point exceeds the predetermined threshold, the warning light 330 (or the user terminal 10) warns of an abnormality in the object. Note that the real-time monitoring function may be omitted.

[0099] (9) Free analysis method The sequence chart shown in FIG. 12 merely shows one example of the operation, and the operation of the displacement observation system 1 is not limited to this. Some of the operations shown in the figure may be omitted, the order may be changed, or new operations may be added. In this example, in step S207, the user terminal 20 may output a graph showing the amount of displacement of the object over time in addition to (instead of) various displays. In this example, the graph includes a graph showing the amount of displacement of the object at a plurality of specified times and cross sections. Note that the free analysis function may be omitted. In addition, in step S206, the designation accepted by the user terminal 20 (second accepting means 18) may be any designation related to a cross section, in addition to the above-mentioned embodiments.

[0100] (10) Database The database of the displacement observation system 1 shown in Fig. 8 and Fig. 9 is not limited to the one exemplified in the embodiment. In this example, any data may be registered in the database, and may be, for example, image data, point cloud data, and displacement data, as well as alert history data (date and time of alert occurrence, etc.) obtained by real-time monitoring, or analysis data (graphs, etc.) obtained by free analysis. The layout of the database is not limited to the one shown in the drawings, and data may be managed in any layout. The information displayed by the user terminal 10 and the user terminal 20 may be any information as long as it is data registered in the database.

[0101] (11) Display screen and input method The display screens shown in Figures 5, 10, 13, 14, and 15 are not limited to those exemplified in the embodiment. In this example, the layout of the display field may be defined for the display screens of the user terminal 10 and the user terminal 20 according to the UI of each display screen. In addition, the layout or configuration of the display screen may be set to an optimal display screen according to the UI of a terminal such as a smartphone. Similarly, the input method may also be designed to be optimal for the UI of the user terminal 10 and the user terminal 20.

[0102] (12) AI With regard to the configurations and operations exemplified in the embodiments, AI and machine learning functions may be implemented in the displacement observation system 1. In this example, with regard to the implementation of AI in the displacement observation system 1, for example, a function for analyzing data registered in a database may be implemented. Alternatively, a function for automatically performing self-alignment of an image and a point cloud image may be implemented.

[0103] (13) Blockchain Regarding the configuration and operation exemplified in the embodiment, blockchain technology may be applied to the displacement observation system 1. Regarding the application of blockchain technology to the displacement observation system 1 in this example, for example, data registered in a database may be recorded in a blockchain network. This makes it possible to protect various types of data in a state where they cannot be deleted or rewritten. In addition, any type of data may be recorded in the blockchain network.

[0104] (14)Other The various programs executed by the CPU 101 etc. may be provided by downloading via a network such as the Internet, or may be provided in a state recorded on a computer-readable non-transitory recording medium such as a DVD-ROM. Each processor may be, for example, an MPU (Micro Processing Unit) or a GPU (Graphics Processing Unit) instead of a CPU. [Explanation of symbols]

[0105] 1...displacement observation system, 10...user terminal, 20...user terminal, 310...camera, 320...LiDAR device, 330...warning light, U1...user, 11...first acquisition means, 12...second acquisition means, 13...generation means, 14...first display control means, 15...alignment means, 16...second display control means, 17...first reception means, 18...second reception means, 191...storage means (DB), 192...control means, 193...display means, 31... Imaging means, 32...measuring means, 33...warning means, 51...Wi-Fi, 52...network, 101...CPU, 102...memory, 103...storage, 104...communication IF, 105...input device, 106...display device, 1001...point cloud database, 2001...displacement database, D...display screen, F...field, I...image, R...road, T...single pipe, B...bridge, L...layer, X...surface (cross section), Cb...cubic grid, P / Q...point

Claims

1. A first acquisition means for acquiring image data from an image capturing means for capturing an image of an object; a second acquisition means for acquiring point cloud data representing a surface shape of the object as a point cloud from a measurement means for measuring distances to a plurality of points on the surface of the object; a first generating means for generating displacement data representing a displacement amount of the object based on a plurality of point cloud data acquired at different times; a first display control means for causing a display means to display a first superimposed image in which an image object indicating the amount of displacement is superimposed on an image of the object; An information processing device having the above configuration.

2. A positioning unit is provided for positioning the image of the object and the image object in the first superimposed image. The information processing device according to claim 1 .

3. the first acquisition means and the second acquisition means acquire the image data and the point cloud data, respectively, in substantially real time; The first display control means causes the display means to display the superimposed image in substantially real time.

3. The information processing device according to claim 1 or 2.

4. The second acquisition means has a first reception means for receiving a reference time and a specified time, The first generating means generates the displacement data indicating the amount of displacement of the object from the reference time to the specified time. The information processing device according to claim 3 .

5. the first receiving means receives the designation for a portion of the plurality of points on a surface of the object; The first generating means generates the displacement data indicating the amount of displacement at the certain point. The information processing device according to claim 1 .

6. The first reception means receives a designation of a measurement range including coordinates of the part of points; The first generating means generates the displacement data indicating a statistical value of the amount of displacement in the measurement range. The information processing device according to claim 5 .

7. the second acquisition means acquires the point cloud data at a plurality of times including a first time and a second time; a second display control means for causing the display means to display a screen including a first display area for displaying a point cloud image of the object obtained from the point cloud data at least one of the first time and the second time and a second image object showing a cross section specified for the point cloud image, and a second display area for displaying a second superimposed image in which the point cloud of the cross section at the first time and the point cloud of the cross section at the second time are superimposed. The information processing apparatus according to claim 1 ,

8. a second receiving means for receiving a designation of the cross section in the first display area; The information processing device according to claim 7.

9. the second receiving means receives an instruction to rotate the point cloud image in the first display area; The second display control means displays the point cloud image rotated in accordance with the instruction. The information processing device according to claim 8.

10. The second accepting means accepts designation of the first time and the second time. The information processing device according to claim 8.

11. A warning means for warning of an abnormality in the object when the amount of displacement exceeds a predetermined threshold value is provided. The information processing device according to claim 1 .

12. The measuring means includes LiDAR. The information processing device according to claim 1 .

13. An image capturing means for capturing an image of an object; A measuring means for measuring distances to a plurality of points on a surface of the object; a first acquisition means for acquiring image data representing the image from the photographing means; a second acquisition means for acquiring point cloud data representing a surface shape of the object as a point cloud from the measurement means; a first generating means for generating displacement data representing a displacement amount of the object based on a plurality of point cloud data acquired at different times; a first display control means for causing a display means to display a first superimposed image in which an image object indicating the amount of displacement is superimposed on an image of the object; A displacement observation system having a

14. acquiring image data from an image capture means for capturing an image of an object; acquiring point cloud data representing a surface shape of the object as a point cloud from a measurement means that measures distances to a plurality of points on the surface of the object; generating displacement data representing a displacement of the object based on the point cloud data acquired at different times; displaying, on a display means, a first superimposed image in which an image object indicating the displacement is superimposed on an image of the object; A displacement observation method having the following structure.

15. On the computer, acquiring image data from an image capture means for capturing an image of an object; acquiring point cloud data representing a surface shape of the object as a point cloud from a measurement means that measures distances to a plurality of points on the surface of the object; generating displacement data representing a displacement amount of the object based on a plurality of point cloud data acquired at different times; displaying, on a display means, a first superimposed image in which an image object indicating the amount of displacement is superimposed on an image of the object; A program for executing.

Images