Information processing methods, programs, information processing terminals, and information processing systems

Abstract

The aim is to reduce the burden of progress management. [Solution] An information processing method is provided, in which a computer performs data acquisition processing and AR processing, wherein the data acquisition processing acquires design data of an object and measurement data of the completed form of the object, and the AR processing superimposes an image based on the design data and an image based on the measurement data onto an image based on on-site imaging data of the completed form.

Description

Technical Field

[0001] The present invention relates to an information processing method, a program, an information processing terminal, and an information processing system.

Background Art

[0002] As an image processing technology used for the work of construction contractors, a technology has been proposed that processes construction design drawing images and has a function of overlapping and displaying object images such as various figures and characters (see Patent Document 1). As described above, although the utilization of image processing technology has been promoted in the construction field, on the other hand, when managing the finished shape of earthwork such as embankment and cutting at a construction site or a civil engineering site, for example, a surveying instrument such as a total station (hereinafter also referred to as "TS") is still used to measure the dimensions of the finished shape and confirm the difference from the design value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional finished shape management, the comparison between the data of the finished shape measured by a TS or the like and the design data is generally performed by comparing numerical values on a form, and it is not easy to intuitively grasp the difference between the shape of the object constructed at the site and the design shape. For example, in earthwork such as embankment and cutting, the three-dimensional shape of the object must be confirmed on a two-dimensional form or drawing, and a great deal of labor is required to spatially understand the relationship between the design data and the measured value of the finished shape.

[0005] According to the present invention, it is an object to suppress the burden of finished shape management.

Means for Solving the Problems

[0006] According to the present invention, an information processing method is provided, in which a computer performs data acquisition processing and AR processing, wherein in the data acquisition processing, design data of an object and measurement data of the completed form of the object are acquired, and in the AR processing, an image based on the design data and an image based on the measurement data are superimposed on an image based on on-site imaging data of the completed form.

[0007] According to the present invention, by superimposing images based on design data and images based on measurement data onto images based on on-site imaging data in AR processing, progress management can be effectively supported. In other words, according to the present invention, on-site workers can intuitively confirm the relationship between the completed state of an object and its design shape by superimposing it onto on-site video footage. Compared to the conventional method of making judgments based solely on numerical values ​​on forms, the accuracy and efficiency of progress management can be greatly improved, thereby reducing the burden of progress management. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing a user U using the computer of the embodiment (e.g., portable information processing terminal 1), an artificial satellite G, a base station K, and an object (embankment Ebk1) at the site X (real space). [Figure 2] Figure 2A shows the rear side of the information processing terminal 1 (the terminal body 2 to which the location acquisition terminal 3 is attached), and Figure 2B shows its front side. [Figure 3] Figure 3 is an explanatory diagram showing the hardware configuration of information processing terminal 1. [Figure 4] Figure 4 is a functional block diagram illustrating the functions of the control unit 12 of the terminal body 2 of the information processing terminal 1. [Figure 5] Figure 5 is a flowchart showing an example of the control flow of the information processing terminal 1 according to the embodiment. [Figure 6] Figure 6 is a schematic diagram illustrating the contents of the AR process S4 in the embodiment. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other. Furthermore, each feature constitutes an independent invention.

[0010] 1. Description of the Configuration of the Embodiment 1-1. Overall Structure As shown in Figure 1, in this embodiment, when a user U is holding a portable information processing terminal 1 and viewing an object (e.g., an embankment Ebk1) at a construction site X, for example, a civil engineering site, an image based on design data and an image based on measurement data are superimposed on the image, thereby effectively supporting progress management.

[0011] Site X is a location where progress management is carried out, and is a place where various earthworks such as embankments, cuts, slopes, roadbeds, and structural foundations are (or have been) constructed. The object in question is, for example, a construction object in an earthwork such as an embankment Ebk1. However, the object in question is not limited to embankments, but may also be various structures such as cuts, slopes, roadbeds, retaining walls, and utility piping.

[0012] Here, the objects can be classified based on their construction methods. For example, the objects may be earthwork structures whose main construction content is the loading, unloading, or shaping of soil (e.g., embankments, cuts, slopes, roadbeds, etc.), or they may be structural structures built from concrete, steel, or a combination thereof (e.g., retaining walls, structural foundations, bridge abutments, bridge piers, culverts, equipment piping, etc.). Furthermore, the object in question may be a linear structure in which construction management is carried out continuously along its extension (e.g., roads, embankments, waterways, tunnels, etc.), or a planar structure in which construction management is carried out over an area that extends over a surface (e.g., developed land, reclaimed land, large-scale foundation ground, etc.). Furthermore, the object in question may be an above-ground structure constructed on the ground, or an underground structure buried underground (for example, an underground pipe, a foundation pile, etc.). Thus, the types of objects subject to this regulation are not particularly limited, and any structure or construction work that may be subject to progress management is included.

[0013] The configuration of an information processing system having an information processing terminal 1 is described below. In one embodiment, the information processing system consists of an information processing terminal 1, an artificial satellite G, and a base station K. The information processing terminal 1 is a portable terminal. In this embodiment, the location acquisition terminal 3 of the information processing terminal 1 is capable of communicating with at least one of the artificial satellite G and the base station K. Through this communication, the information processing terminal 1 can acquire its location information. The location acquisition terminal 3 may be configured to communicate with both the artificial satellite G and the base station K, or it may be configured to communicate with only one of the artificial satellite G and the base station K.

[0014] 1-2. Hardware Configuration 1-2-1. Exterior Configuration Figures 2A and 2B illustrate an example of an information processing terminal 1 (a terminal body 2 with a location acquisition terminal 3 attached). The information processing terminal 1 comprises a terminal body 2 and a location acquisition terminal 3, which is an example of a location acquisition unit. In this embodiment, a smartphone is used as the terminal body 2, as an example of a mobile terminal that can be carried in the field. The location acquisition terminal 3 is attached to the surface of the terminal body 2. In this embodiment, as an example, the location acquisition terminal 3 can be attached to the back of the terminal body 2. Preferably, the location acquisition terminal 3 is configured to be detachably connected to the terminal body 2, for example. The planar dimensions of the location acquisition terminal 3 are defined so that the part other than the antenna 3b is substantially contained within the back of the terminal body 2.

[0015] In the embodiment, the information processing terminal 1 is configured such that the terminal body 2 responsible for functions such as information processing and the position acquisition terminal 3 responsible for the function of acquiring position information are separate (independent) entities. For example, by means of a simple connection operation of attaching the position acquisition terminal 3 to a commonly used smartphone (terminal body 2), the terminal can be enhanced in functionality, and the convenience and versatility are improved. However, the present invention is not limited to this. The information processing terminal 1 may be an integrated terminal of the terminal body 2 and the position acquisition terminal 3.

[0016] In the position acquisition terminal 3 of the embodiment, various circuit configurations are housed in the housing 3a. In the embodiment, as shown in FIG. 2A, an antenna 3b is connected above the housing 3a, a battery 3c is provided inside the housing 3a, and these are connected to the main circuit 3d that realizes functions such as the position acquisition function.

[0017] Note that the form of the information processing terminal 1 is not limited to a smartphone, and other handheld terminals such as tablets may also be used. That is, in the embodiment, the information processing terminal 1 is a handheld terminal (for example, a smartphone, a tablet, etc.) held and used by the user U, and an aspect of confirming AR display via the display screen of the terminal (so-called video see-through method) is adopted.

[0018] 1-2-2. Functional Blocks of Hardware Configuration As shown in FIG. 3, the terminal body 2 includes a communication unit 10, a storage unit 11, a control unit 12, a display screen 13, a camera 14, and a LiDAR scanner 15, and these components are electrically connected via a communication bus 16 inside the terminal body 2.

[0019] The control unit 12 is configured to perform processing and control related to information processing of the information processing terminal 1. The control unit 12 can be configured as, for example, a central processing unit (CPU), and in this embodiment, the control unit 12 is an example of a processor capable of executing computer programs related to each step of the flowchart described later. The control unit 12 realizes various functions related to the information processing terminal 1 by, for example, reading computer programs stored in the storage unit 11.

[0020] In this embodiment, the display screen 13 is a touch panel. The touch panel has an input unit and a display unit integrated into one unit. The display screen 13 is an input device that accepts user U's operation input and also functions as a display device that displays various screens.

[0021] As shown in Figure 3, the location acquisition terminal 3 can communicate with the storage unit 11 and other components within the terminal body 2 via the communication unit 10 and the communication bus 16.

[0022] 1-3. Software Configuration As shown in the functional blocks in Figure 4, the control unit 12 of the terminal body 2 comprises a data acquisition unit 12a, an imaging unit 12b, a positioning unit 12c, and an AR processing unit 12d. The AR processing unit 12d includes a display object generation unit 12d1, a positioning unit 12d2, and an overlay display unit 12d3. These functional units are realized, for example, by reading and executing a computer program stored in the storage unit 11.

[0023] 1-3-1. Data acquisition unit 12a The data acquisition unit 12a is configured to acquire design data (e.g., basic design data DT1a) of the target object (e.g., embankment Ebk1) and measurement data (e.g., completed shape measurement data DT1b) of the completed shape of the target object. In this embodiment, the design data and measurement data are included in the construction management data DT1. The construction management data DT1 may be data that is installed on the completed shape management total station TS1 and transferred from the completed shape management total station TS1 to the information processing terminal 1, or it may be data created on another PC or the like. That is, the construction management data DT1 can be created by integrating the measurement data acquired by the measuring instrument or measuring system and the design data on a terminal other than the information processing terminal 1 (e.g., a desktop PC, a notebook PC, or a server). In this case, on another terminal, for example, measurement data files output from measuring instruments and design data files (e.g., CAD data, BIM / CIM model data, or data files conforming to the Construction Management Data Exchange Standard) are read, and construction management data DT1 is generated through predetermined data processing (e.g., unification of coordinate systems, conversion of data formats, mapping of measurement data and design data, formatting into a data structure conforming to the Construction Management Data Exchange Standard). The created construction management data DT1 may be transferred to the information processing terminal 1 via communication (e.g., wireless LAN, mobile network, etc.) or recording medium (e.g., USB memory, SD card, etc.). It is also possible to combine this with an embodiment in which the construction management data DT1 created in this way is uploaded to a cloud server, as shown in Modification Example 1 described later, and converted into AR display data on the cloud server side.

[0024] Construction management data DT1 includes basic design data DT1a and completed construction measurement data DT1b. Basic design data DT1a is data that defines the design shape, dimensions, location, etc. of the object, and includes data such as the design cross-sectional shape, design height, design width, and slope gradient. In this embodiment, completed construction measurement data DT1b is described as the measured value of the completed object measured by a total station TS1, but it may also be the measured value of the completed object measured by measuring instruments such as a tape measure or GNSS, or a combination of these measured values ​​may be used.

[0025] Furthermore, the measuring instruments or systems used to acquire measurement data are not limited to the total station TS1, measuring tape, and GNSS described above. The measuring instruments or systems include at least one selected from the group consisting of a total station, measuring tape, GNSS, laser scanner, LiDAR, 3D measuring device, and image analysis system. In other words, the measuring instruments or systems used to acquire measurement data may be a combination of these group.

[0026] A 3D measuring device is a general term for a device capable of non-contact measurement of the three-dimensional shape of an object, and includes, for example, laser scanners, LiDAR (Light Detection and Ranging), TOF (Time of Flight) cameras, structured light scanners, and UAV-mounted surveying devices. Furthermore, a 3D measuring device may be a device that acquires the three-dimensional coordinates (point cloud data) of the object's surface by irradiating it with laser light or patterned light and detecting the reflected light, or it may be a device that acquires three-dimensional coordinates by combining multiple sensors.

[0027] An image analysis system is a general term for a system that calculates the three-dimensional shape or dimensions of an object based on the analysis of image data, and includes systems using technologies such as photogrammetry, SfM (Structure from Motion), MVS (Multi-View Stereo), Gaussian splatting, and NeRF (Neural Radiance Fields). An image analysis system may also be a system that generates point cloud data or a three-dimensional model of an object based on multiple image data captured by a single imaging device or multiple imaging devices. An image analysis system may consist of a combination of dedicated hardware and software, or it may consist of software that runs on a general-purpose computer.

[0028] Thus, the measurement data can be any data obtained by measuring the finished shape of the object by some means, and the type of measurement means is not particularly limited. In addition to the various devices described above, the measuring equipment may also include, for example, contact-type or non-contact-type measurement means such as infrared measuring devices or ultrasonic measuring devices, either as auxiliary or independent.

[0029] Furthermore, the construction management data DT1 may be data that conforms to the Construction Management Data Exchange Standard (Data Exchange Format for TS Progress Management) established by the Ministry of Land, Infrastructure, Transport and Tourism. In this case, the construction management data DT1 may include various elements such as structure information, coordinate reference system set, construction control point set, road structure information, completed cross-section set, measurement point set, measurement equipment installation set, and coordinate points.

[0030] 1-3-2. Imaging unit 12b The imaging unit 12b is configured to acquire on-site imaging data via the camera 14 of the information processing terminal 1. The on-site imaging data is image data of the site X captured by the camera 14, and the image 12b1 based on the on-site imaging data is displayed on the display screen 13. The on-site imaging data may be a still image, but it is preferable that it be a video (live view video). That is, it is preferable that the displayed image is updated in real time when the user U points the camera 14 in any direction.

[0031] 1-3-3. Positioning unit 12c The positioning unit 12c is configured to acquire location data (terminal location data Dt) of the portable information processing terminal 1 in real space through the location acquisition terminal 3. The terminal location data Dt may include, for example, latitude, longitude, and altitude.

[0032] Terminal location data Dt can be obtained by communication with satellite G, communication with base station K, or location acquisition using SLAM. Furthermore, two or more of these acquisition methods may be used in any combination.

[0033] (1) Position acquisition through communication with artificial satellites When the position acquisition terminal 3 communicates with the artificial satellite G to acquire terminal position data Dt, the communication method can be, for example, GNSS (Global Navigation Satellite System), RTK (Real Time Kinematic), CLAS (Centimeter-Level Augmentation Service), or SLAS (Submeter-Level Augmentation Service). These communication methods may be used individually or in combination.

[0034] (2) Location acquisition through communication with base stations If a base station K capable of communicating with the location acquisition terminal 3 is installed at site X, the location acquisition terminal 3 may be configured to acquire terminal location data Dt via communication with this base station K. The base station K is configured to acquire base station K coordinates corresponding to the location of the base station. Communication methods such as GNSS, RTK, CLAS, or SLAS may be used to acquire the base station K coordinates. The location acquisition terminal 3 can acquire terminal location data Dt based on the base station K coordinates and distance information based on the distance between base station K and the location acquisition terminal 3.

[0035] (3) Position acquisition by SLAM To obtain terminal location data Dt, coordinates using SLAM (Simultaneous Localization and Mapping) may be employed. SLAM using camera 14 can be employed.

[0036] 1-3-4. AR Processing Unit 12d The AR processing unit 12d is configured to perform Augmented Reality (AR) display on the display screen 13. The AR processing unit 12d includes a display object generation unit 12d1, a positioning unit 12d2, and a superimposed display unit 12d3.

[0037] 1-3-4-1.Display object generation section 12d1 The display generation unit 12d1 generates (A) an image based on design data (for example, an object Ob1 which is a 3D object image representing the design shape of the object) and (B) an image based on measurement data (for example, a dimension object that shows the measured values ​​of the completed work). The images based on (A) design data and (B) measurement data will be described in detail below.

[0038] (A) Image based on design data The image based on the design data includes at least one of the following: (a1) an image showing the design shape, (a2) a geometric shape image, or (a3) ​​an image showing the design values. The image based on the design data is an image that visualizes the three-dimensional shape, cross-sectional shape, etc., of the object in question.

[0039] (a1) Images showing the design shape may include, in one example of the embodiment, images such as 3D object images and 2D drawing images as described below. A 3D object image is an object that has the three-dimensional shape of an object, generated based on design data. Two-dimensional drawing images are images of two-dimensional drawings such as cross-sectional views and plan views based on design data.

[0040] Here, the image showing the design shape may include images showing control points, control lines, control planes, or tolerance ranges included in the design data. Control points, control lines, and control planes are briefly explained below. Control points are points that serve as reference points for measurement and management in construction progress management. Examples include measurement points on the road centerline, slope crest points, slope toe points, and reference points on the construction base surface. Control points are defined by coordinate values ​​included in the design data. Control lines are lines that serve as the basis for measurement and management in construction progress management. Examples include road centerlines (alignments), slope shoulder lines, slope toe lines, lines indicating planned elevations, contour lines of management sections, and boundaries of construction areas. Control lines are defined as line segments or curves connecting multiple coordinate points included in the design data. In AR display, control lines are superimposed as line segments or curves on images based on on-site imaging data, assisting the user U in visually recognizing the design shape of the object. A control surface is a surface that serves as the standard for measurement and management in construction progress management, and includes, for example, the design surface of a slope, the design surface of a roadbed, the design surface of the top of an embankment, and the cut surface. A control surface is a plane or curved surface defined by multiple coordinate points included in the design data. In AR display, the control surface is displayed superimposed on the image based on on-site imaging data as an object such as a semi-transparent polygon mesh or wireframe. By displaying the control surface, user U can spatially understand the positional relationship between the completed surface of the object and the design surface.

[0041] (a2) Geometric shape images are images that include points, lines, or surfaces, and for example, images that represent various reference lines (control lines) such as design reference points (control points) and centerlines, various reference surfaces (control surfaces), dimension lines, and various cross-sectional shapes. Furthermore, geometric shape images can also include images based on one-dimensional data, such as those where the position is defined by the distance from the center line to the base point. Furthermore, while geometric shape images may overlap in some aspects with images representing design shapes, they are a distinct type of image from the perspective of their display format.

[0042] (a3) The image showing the design values ​​is an image of text objects that visualize dimensional data such as design height, design width, slope gradient, and slope length. The image showing the design values ​​may also include dimension objects (e.g., dimension lines such as arrows) to indicate this dimensional data. Here, the image showing the design values ​​may include images showing control points, control lines, control planes, or tolerance ranges included in the design data.

[0043] (B) Images based on measurement data Images based on measurement data include images showing the actual measured values ​​of the completed work. These images showing the measured values ​​can include, for example, dimension objects indicating dimensions, text objects indicating numerical values, etc.

[0044] (b1) Displaying the difference Images showing measured values ​​may include images that display the difference between the design value and the measured value. Specifically, the difference can be displayed numerically, with dimension lines, or as a heatmap. In the case of a heatmap display, the color changes according to the size of the difference.

[0045] (b2) Indication of deviation from the acceptable range Images showing measured values ​​may include images that highlight areas where the measured values ​​deviate from the acceptable range using color coding. This allows user U to quickly determine whether the completed work is within the acceptable range.

[0046] (b3) Visualization of time series data Measurement data may include time-series data acquired through multiple measurements. In this case, the image showing the measured values ​​may include an image that visualizes the changes in the completed work over time based on the time-series data. Here, time-series data is a collection of multiple measurement data acquired for the same object or the same measurement point at different measurement times, and each measurement data is associated with the measurement date and time.

[0047] Examples of image formats that visualize changes in construction over time include, for example, any one or a combination thereof of the following: Firstly, as an example of time-selection type visualization, there is a display switching method using a time slider. When user U selects a measurement time by operating the slider on the display screen 13 or by selecting a date and time, an image based on the actual measured value of the completed work at the selected measurement time is displayed in AR. Secondly, as an example of continuous playback visualization, this is an animation display method that visualizes the changes in the completed work over time in a video-like manner by continuously switching and displaying images of the completed work at multiple measurement points in chronological order. Thirdly, as an example of difference-enhanced visualization, there is a difference color mapping method that overlays the amount of change in the completed work between a reference point (e.g., the initial measurement point or the previous measurement point) and a comparison point onto an image based on on-site imaging data by changing the intensity or hue of the color. Fourthly, as another example of difference-enhanced visualization, there is a method for displaying changes in cross-sectional shape over time, in which the cross-sectional shape of the completed work in the control section is overlaid and displayed for each measurement point.

[0048] By visualizing these changes over time, user U can intuitively grasp the progress of embankment construction (e.g., the increase in embankment height for each layer), ground displacement (e.g., changes in settlement over time), slope deformation (e.g., displacement of the slope surface), or seasonal variations (e.g., changes in frost heave) as temporal changes on-site. Furthermore, the visualization of time-series data contributes to the early detection of anomalies in construction progress management and quality control.

[0049] The data formats for images based on design data and measurement data generated by the display object generation unit 12d1 are not particularly limited. For example, the data formats for 3D object images can be OBJ, FBX, glTF (GL Transmission Format), USDZ, or a combination thereof. Furthermore, metadata related to images based on design data and measurement data (e.g., coordinate information, display attributes, measurement values, etc.) may be described in general-purpose data formats such as JSON or XML.

[0050] 1-3-4-2. Alignment section 12d2 The alignment unit 12d2 uses terminal position data Dt to align the image to be superimposed in AR processing S4 (object Ob1, image of the target object, and dimension object, etc.) with the image 12b1 based on the on-site imaging data. Specifically, the alignment unit 12d2 calculates the display position and display orientation of the image to be superimposed based on the terminal position data Dt and the position information contained in the design data or measurement data, and places it in an appropriate position on the image 12b1 based on the on-site imaging data.

[0051] The alignment unit 12d2 can reflect changes in the position of the information processing terminal 1 by updating the alignment in response to updates to the terminal position data Dt. This update may be performed continuously and sequentially, or it may be performed when a position change event is detected. In other words, the alignment update method can be classified into a continuous update method, which sequentially updates the alignment in synchronization with the acquisition cycle of terminal position data Dt, and an event-driven update method, which updates the alignment when the terminal's position change exceeds a predetermined threshold or when a predetermined event is detected. The continuous update method is suitable for situations where the user U is moving and checking the AR display in real time, while the event-driven update method is suitable for situations where the positional accuracy of the AR display is maintained while suppressing the processing load or power consumption of the terminal. Furthermore, an adaptive update method may be adopted that switches between the continuous update method and the event-driven update method depending on the situation. For example, in an adaptive update method, the continuous update method may be adopted when the terminal's movement speed is above a predetermined value, and the system may switch to the event-driven update method when the terminal's movement speed is below a predetermined value.

[0052] It should be noted that alignment is not limited to automatic alignment based on terminal position data Dt as described above. In this embodiment, manual alignment is also possible based on user U's input. In manual alignment, user U can manually adjust the display position, display angle, and display scale of the image to be superimposed by performing touch operations or gesture operations on the display screen 13.

[0053] Specifically, in the manual alignment process, user U can perform the following operations, for example. The types of operation input in the manual alignment process can be classified into touch gesture operations (e.g., drag operations, rotation operations, pinch-in operations, and pinch-out operations) in which user U performs touch operations on the display screen 13, GUI operations that operate on user interface elements such as buttons or sliders displayed on the display screen 13, and non-contact operations that do not involve touching the display screen 13 (e.g., voice input, air gesture operations, eye-tracking input, etc.).

[0054] Manual alignment processing may be used alone as an alternative to automatic alignment based on terminal position data Dt, or it may be used in combination with automatic alignment as a correction. For example, a hybrid approach can be adopted in which rough alignment is performed by automatic alignment, and then fine adjustments are made by manual operation. This makes it possible to perform construction progress management using AR display even in environments where positioning means such as GNSS cannot be used (for example, in tunnels, inside buildings, mountainous areas, and other environments where satellite signals are difficult to receive), and it also functions as a means of correction when positioning accuracy is insufficient.

[0055] 1-3-4-3. Superimposed display section 12d3 The superimposed display unit 12d3 superimposes images based on design data and measurement data generated by the display object generation unit 12d1 onto the image 12b1 based on on-site imaging data. As a result, the user U can see through the display screen 13 that objects based on design data and dimensional objects based on measurement data appear to exist within the actual video of the site X.

[0056] 2. Explanation of Information Processing Methods The information processing method according to the embodiment will be explained using Figure 5. Figure 5 is an example of the processing flow of the information processing method executed on the information processing terminal 1.

[0057] The processing flow of the information processing method according to the embodiment shown in Figure 5 comprises data acquisition processing (S1), image acquisition processing (S2), positioning processing (S3), and AR processing (S4). AR processing (S4) includes alignment processing (S4a), display object generation processing (S4b), and superimposed display processing (S4c).

[0058] Note that the execution order of each process is not limited to the order shown in Figure 5; some processes may be executed in parallel, and the order may be changed.

[0059] 2-1. Data acquisition process (S1) In the data acquisition process (S1), the data acquisition unit 12a acquires the design data of the object (basic design data DT1a) and the measurement data of the completed state of the object (completed state measurement data DT1b). In this embodiment, the data acquisition unit 12a acquires the basic design data DT1a and the completed state measurement data DT1b included in the construction management data DT1. The construction management data DT1 may be acquired, for example, from a total station TS1 for completed state management, or from another terminal such as a PC.

[0060] The data acquisition unit 12a is not limited to acquiring design data and measurement data from an external device or system. For example, it may read data pre-stored in the memory unit 11 of the terminal body 2, download data from a project folder on the cloud, or extract data from 3D model data such as a BIM / CIM model.

[0061] Furthermore, the data acquisition unit 12a may perform consistency checks and preprocessing of the acquired data. For example, it may perform operations such as unifying coordinate units (m / mm), unifying elevation standards, imputing missing values, removing outliers, thinning point clouds, and interpolating cross-sectional data.

[0062] 2-2. Imaging process (S2) In the imaging process (S2), the imaging unit 12b acquires on-site imaging data via the camera 14 of the information processing terminal 1. It is preferable that the acquisition of on-site imaging data is performed continuously, for example, while the AR display function is enabled on the information processing terminal 1.

[0063] 2-3. Positioning processing (S3) In positioning processing (S3), the positioning unit 12c acquires terminal location data Dt of the information processing terminal 1 in real space via the position acquisition terminal 3. The method for acquiring terminal location data Dt can be any of the above-mentioned methods: (1) communication with an artificial satellite, (2) communication with a base station, or (3) SLAM.

[0064] 2-4. AR processing (S4) AR processing (S4) includes alignment processing (S4a), display object generation processing (S4b), and superimposed display processing (S4c).

[0065] 2-4-1. Alignment process (S4a) In the alignment process (S4a), the alignment unit 12d2 uses terminal position data Dt to align the image to be superimposed with the image 12b1 based on the on-site imaging data. Specifically, based on the terminal position data Dt and the position information included in the design data and measurement data, it calculates where the image to be superimposed should be placed within the field of view of the camera 14.

[0066] 2-4-2. Display object generation process (S4b) In the display object generation process (S4b), the display object generation unit 12d1 generates images based on design data (object Ob1, etc.) and images based on measurement data (dimension object, etc.).

[0067] Figure 6 is a schematic diagram illustrating the contents of AR processing S4. As shown in Figure 6, on the display screen 13, a 3D object Ob1 based on design data (basic design data) is superimposed on an image 12b1 based on on-site imaging data. The 3D object Ob1 is an object that visualizes the design shape of the target object (embankment Ebk1) in three dimensions. Furthermore, images based on design data or measurement data (e.g., dimension objects) are also superimposed.

[0068] Figure 6 shows image V1 of a 3D object (in this case, an embankment) based on design data (basic design data), viewed from a desired angular direction. Here, image V2 in Figure 6 is an enlarged view of the area around the dashed line region P1 schematically shown in image V1 in Figure 6. The dashed line region P1 corresponds to the top of the embankment. Note that image V2 is slightly shifted from image V1 in terms of the direction in which the 3D object in image V1 is viewed. As an example, image V2 schematically shows the position coordinates PN1, which are located approximately 50 mm above the top of the 3D object related to the embankment.

[0069] The position coordinate PN1 is a point representing a spatial location identified based on coordinate information contained in the design data or measurement data. The position coordinate PN1 may be, for example, a location calculated based on the coordinates of control points included in the design data, a location identified based on the measured coordinates of measurement points included in the measurement data, or a location calculated based on both the design data and the measurement data (for example, a location that reflects the difference between the design location and the measured location).

[0070] In AR display, the position coordinate PN1 is superimposed as a point marker (e.g., a dot, pin, cross mark, or other point-like display element) on the image based on on-site imaging data. This allows user U to spatially perceive a predetermined location on or near an object through the AR display. In the example in image V2, it is shown that the position coordinate PN1 is located approximately 50 mm above the top of the 3D object Ob1. This allows user U to intuitively understand the spatial relationship of that position to the design shape (for example, the direction and magnitude of the deviation from the design reference position).

[0071] While the display of point markers such as position coordinate PN1 is particularly useful when observing a specific area of ​​an object in detail, as shown in image V2 of Figure 6, in situations where a wide area of ​​an object is being observed, as shown in image V3, the point marker may not be included in the field of view depending on the field of view and display magnification on the display screen. Furthermore, although position coordinate PN1 has been described as a single point, it goes without saying that multiple position coordinates can be displayed simultaneously in AR. For example, by simultaneously displaying multiple point markers corresponding to multiple control points or multiple measurement points on an object, user U can grasp the positional relationship between multiple locations on the object and the design shape at a glance.

[0072] 2-4-3. Overlay display processing (S4c) In the superimposed display process (S4c), the superimposed display unit 12d3 superimposes images based on design data and measurement data generated in the display object generation process (S4b) onto the image 12b1 based on on-site imaging data. In other words, the superimposed display unit 12d3 synthesizes the image 12b1 based on on-site imaging data acquired by the camera 14 and the display objects generated in the display object generation process (S4b) and whose placement is determined in the alignment process (S4a) using a rendering engine (e.g., GPU) and outputs it to the display screen 13. As a result, the user U can see through the display screen 13 that objects based on design data and dimensional information based on measurement data appear to exist within the actual video of the site X, enabling effective progress management.

[0073] In the superimposed display processing (S4c), the superimposed display unit 12d3 displays an example of an image, which is image V3 shown in Figure 6. Image V3 in Figure 6 schematically shows how a 3D object is superimposed (AR display) on an image 12b1 based on on-site imaging data for the dashed line region P2 schematically shown in image V1 in Figure 6. The dashed line region P2 corresponds to the slope area of ​​the embankment. In the superimposed display of this embodiment, as shown in image V3, a 3D object Ob1 (3D object image) related to the design data and an image showing the design values ​​related to the design data (numerical values ​​such as 1.50m in the figure and dimension lines showing those values) are superimposed on the image 12b1 based on the imaging data, and actual measured values ​​related to measurement data (numerical values ​​such as 1.56m in the figure and dimension lines showing those values) are also superimposed.

[0074] Furthermore, the display methods for the displayed objects are not limited, and various display methods can be adopted, such as wireframe displays that represent the arrangement and shape of the displayed objects using lines or frames, semi-transparent displays, shaded displays, and contour-enhanced displays. These display methods can be classified based on their display characteristics. For example, they can be classified into transparent display methods that superimpose the displayed object while ensuring the visibility of images based on on-site imaging data (e.g., wireframe displays, semi-transparent displays, etc.) and shape-enhanced display methods that enhance the three-dimensional shape recognition of the displayed object (e.g., shaded displays, contour-enhanced displays, etc.). The transparent display mode is suitable when you want to check the design shape without obstructing the on-site video, while the shape-enhancing display mode is suitable when you want to clearly recognize the irregularities and edges of the design shape. Furthermore, a composite display mode combining a transparent display mode and a shape-enhancing display mode (for example, a combination of semi-transparent display and contour-enhancing display) may be adopted. User U may be configured to switch display modes depending on the object being checked and the on-site situation.

[0075] Furthermore, the AR processing unit 12d may perform display quality improvement processing to improve the visibility and naturalness of the superimposed display. The display quality improvement processing includes at least one of the following: processing to improve the spatial consistency between the image based on the on-site imaging data and the superimposed display object, and processing to improve the temporal stability of the superimposed display. An example of processing to improve spatial consistency is the occlusion representation processing described later. An example of processing to improve temporal stability is the filter processing for jitter reduction described later. The superimposed display unit 12d3 may perform occlusion processing. For example, it may generate a depth map or occlusion mesh of the on-site object from depth information obtained by the LiDAR scanner 15 or depth camera, and hide the display of the part of the object that is behind the on-site object. This allows the design shape to blend naturally into the on-site image, reducing the cognitive load on the user U. The superimposed display unit 12d3 may perform filtering to reduce flickering and jitter in the superimposed display. For example, the conversion parameters obtained in the alignment process (S4a) may be smoothed (low-pass filter, Kalman filter) and corrected to change continuously between frames. In addition, when the terminal moves, the effect of drawing delay can be suppressed by pre-calculating the projection of the display object based on the predicted orientation.

[0076] The superimposed display unit 12d3 may accept a display mode switch. That is, in addition to the mode that simultaneously displays the design shape, design value, and measured value as shown in image V3 of Figure 6, it may be possible to switch to a mode that displays at least one of these. This makes it possible to selectively display the necessary information, improving convenience.

[0077] 2-5. Supplementary explanation (Image generation using various elements of construction management data) Here, we will explain the relationship between the various elements included in the construction management data DT1 and the image generation in AR processing S4, assuming that the construction management data DT1 conforms to the construction management data exchange standard set by the Ministry of Land, Infrastructure, Transport and Tourism.

[0078] The structure information defines the type of object (e.g., embankment, cut, slope, etc.), structural form, and construction section, and can be used, for example, to identify the type of shape model of the object when the display object generation unit 12d1 generates an image showing the design shape (e.g., a 3D object image).

[0079] The coordinate reference system set is information that defines the coordinate system (e.g., geodetic coordinate system, plane rectangular coordinate system, etc.) that serves as the basis for the coordinate values ​​included in the construction management data DT1. For example, it can be used by the display object generation unit 12d1 or the alignment unit 12d2 to convert the coordinate values ​​of the design data and measurement data to the same coordinate system as the terminal position data Dt. This enables coordinate transformation to superimpose the image showing the design shape and the image showing the measured values ​​onto the correct position on the image based on the on-site imaging data.

[0080] The construction control point set is information that defines the coordinate values ​​of control points installed at site X. For example, it can be used by the alignment unit 12d2 to establish a correspondence between the design coordinate system and the site coordinate system during the alignment process. It can also be used by the display object generation unit 12d1 to generate geometric shape images (for example, images showing control points) that represent the positions of control points in an image showing the design shape.

[0081] Road structure information is information that defines the alignment information of the road (e.g., horizontal alignment, vertical alignment, cross-sectional shape, etc.) and the design dimensions of each cross-section (e.g., road width, slope gradient, slope length, etc.). For example, when the display object generation unit 12d1 generates an image showing the design shape (e.g., a 3D object image), it can be used to construct the three-dimensional shape of the object in the design. In addition, the design dimensions included in the road structure information can also be used to generate an image showing the design values.

[0082] The completed cross-section set is information that defines the measurement results of the completed cross-sectional shape at each control section. For example, when the display generation unit 12d1 generates an image showing the measured values, it can be used to display the measured cross-sectional shape of the completed structure in a way that allows comparison with the designed cross-sectional shape. For example, an image is generated that displays the difference between the measured cross-sectional shape included in the completed cross-section set and the designed cross-sectional shape included in the road structure information using numerical values, dimension lines, or a heat map.

[0083] The measurement point set is information that defines the coordinate values ​​and measured values ​​of each measurement point used for measuring the completed work. For example, it can be used when the display generation unit 12d1 generates an image showing the measured values ​​to display the measured values ​​at each measurement point at the corresponding position in space.

[0084] The measuring instrument installation set contains information that specifies the installation position and conditions of the measuring instruments (e.g., total station) used for measurement. For example, it can be used as a reference when the alignment unit 12d2 converts the coordinate values ​​of the measurement points to the field coordinate system.

[0085] Coordinate points are coordinate values ​​that define the positions of various elements included in the construction management data DT1. For example, they can be used to identify the spatial placement of displayed objects in the generation of images showing the design shape, images showing design values, and images showing measured values.

[0086] It should be noted that not all of the above elements are required to be used in image generation during AR processing S4. For example, at least some of the elements may be selectively used depending on the type of object, the measurement method, and the content to be displayed.

[0087] 3. Operation and Effects of the Embodiments (1) According to the embodiment, in AR processing, images based on design data and images based on measurement data are superimposed on images based on on-site imaging data, thereby effectively supporting progress management. That is, on-site workers can intuitively confirm the design shape of the object and the measured values ​​of the completed work by simply looking at the terminal screen, overlaid on the on-site video. In conventional paper-based progress management, it was necessary to confirm the difference between the design value and the measured value using only numerical values, but in the embodiment, by visualizing these differences as an AR display on the on-site video, spatial understanding becomes easier, and the accuracy and efficiency of progress management can be greatly improved.

[0088] (2) In the embodiment, the image based on the design data includes at least one of an image showing the design shape (3D object image or 2D drawing image) and a geometric shape image (point, line, or surface), thereby enabling the visualization of the design shape of the object in an appropriate representational manner according to the user's verification purpose and on-site conditions. For example, a 3D object image is effective for understanding the overall shape of the object, while a geometric shape image is effective for checking specific cross-sections or reference lines, thus improving the visibility of information in progress management.

[0089] (3) In the embodiment, design data and measurement data are acquired from construction management data, and the measuring instrument or measurement system used to acquire the measurement data includes at least one selected from the group consisting of a total station, measuring tape, GNSS, laser scanner, LiDAR, 3D measuring device, and image analysis system. This makes it possible to select the optimal measurement means according to the measurement environment at the site and the type of object, and to flexibly acquire the measurement data necessary for AR display at various construction sites.

[0090] (4) In this embodiment, the construction management data includes data created by integrating measurement data acquired by measuring instruments or a measurement system with design data on a terminal separate from the computer. This makes it possible to create construction management data by integrating and formatting the measurement data acquired by measuring instruments or a measurement system with design data on a terminal with high processing power, for example, and then perform AR display based on this construction management data. In other words, the process from acquiring measurement data to creating construction management data and the process of AR display based on construction management data can be divided and executed on terminals suitable for each, thereby increasing compatibility with the actual construction management workflow and making it possible to provide AR display after sufficient quality checks and preprocessing of the data, thereby improving the reliability of progress management.

[0091] (5) Images based on design data include images showing the design shape or design values, and images based on measurement data include images showing the measured values ​​of the completed work, so that the design shape or dimensions and the measured values ​​of the completed work can be compared and confirmed on the same screen. This clarifies the relationship between the design data and the measurement data, making it possible to quickly and intuitively determine on site whether the completed work has been constructed according to the design.

[0092] (6) In the embodiment, the image showing the design shape or the image showing the design values ​​includes images showing control points, control lines, control surfaces, or tolerance ranges included in the design data, thereby making the design control elements that serve as the basis for construction management visible on the on-site video. This makes it possible for workers to spatially grasp the reference positions, reference lines, reference surfaces, and tolerance ranges that should be focused on in the measurement and evaluation of the construction, thereby improving the accuracy of measurement work and pass / fail judgments.

[0093] (7) In this embodiment, the image showing the measured values ​​includes an image that displays the difference between the design value and the measured value using numerical values, dimension lines, or a heat map, thereby enabling the degree of deviation between the completed work and the design to be visualized in a multifaceted way. Numerical display contributes to quantitative understanding, dimension line display contributes to understanding the spatial relationship of the difference, and heat map display contributes to intuitive understanding of the distribution trend of the difference. As a result, on-site workers can select the optimal display format according to the object to be checked and purpose to check the difference, improving the quality of completed work management.

[0094] (8) In this embodiment, the image showing the measured values ​​includes an image that displays, in a color-coded manner, areas where the measured values ​​deviate from the acceptable range, making it possible to grasp at a glance whether the finished product is within the acceptable range. This makes it easy to identify unacceptable areas and allows for immediate on-site determination of whether rework is necessary, thereby contributing to ensuring construction quality and reducing rework.

[0095] (9) In this embodiment, the measurement data includes time-series data acquired through multiple measurements, and the image showing the measured values ​​includes an image that visualizes the changes in the completed work over time based on the time-series data, making it possible to check the progress of construction and the displacement of the object over time along the time axis. This allows for intuitive on-site management of the progress of each stage of embankment construction, monitoring of the progression of ground subsidence, and early detection of slope deformation, thereby improving the accuracy of quality control and safety management.

[0096] (10) In this embodiment, in AR processing, an image is generated using at least some of the information included in the construction management data, such as the structure information, coordinate reference system set, construction control point set, road structure information, completed cross-section set, measurement point set, measurement equipment installation set, and coordinate points. This makes it possible to generate an image for AR display using existing construction management data that conforms to the construction management data exchange standard set by the Ministry of Land, Infrastructure, Transport and Tourism. This makes it possible to realize completed work management by AR display while minimizing the creation of new data formats and the conversion of existing data, and improves compatibility with existing construction management workflows.

[0097] (11) In this embodiment, the position data of the terminal body is acquired in the positioning process, and the position of the image to be superimposed is aligned using the position data in the alignment process of the AR process, thereby enabling the image based on the design data and the image based on the measurement data to be displayed in AR to be placed in the spatially correct position on the image based on the on-site imaging data. This ensures the positional accuracy of the AR display and enables the user to accurately recognize the spatial correspondence between the design shape and the completed product.

[0098] (12) In this embodiment, GNSS, RTK, CLAS, or SLAS are used for communication with the satellite, enabling high-precision acquisition of the terminal's location at an outdoor construction site. In particular, when using RTK or CLAS, centimeter-level positioning accuracy can be obtained, enabling AR display with the high positional accuracy required for construction progress management.

[0099] (13) In this embodiment, by acquiring location data using SLAM in the positioning process, it is possible to acquire terminal location data and perform AR display even in environments where it is difficult to receive signals from artificial satellites (e.g., inside tunnels, inside buildings, mountainous areas, etc.). This expands the environments in which construction progress management using AR display can be applied.

[0100] (14) In this embodiment, by acquiring location data by communicating with a base station, it becomes possible to acquire location data via a base station even in environments where communication with artificial satellites is unavailable or where positioning accuracy is insufficient. This ensures redundancy of the means for acquiring location data, and enables stable construction progress management using AR display in a variety of field environments.

[0101] (15) In this embodiment, the base station is configured to acquire base station K coordinates, and by acquiring position data based on the base station K coordinates and distance information, the position of the terminal can be calculated with high accuracy based on the known position of the base station. By combining the base station K coordinates and distance information between the terminal, high-precision alignment based on relative position measurement becomes possible.

[0102] (16) In this embodiment, by using GNSS, RTK, CLAS, or SLAS to acquire base station coordinates, the position of the base station itself can be acquired with high accuracy, and the accuracy of the terminal position data based on this is also improved.

[0103] (17) In this embodiment, a portable information processing terminal includes a terminal body and a location acquisition terminal. The terminal acquires design data and measurement data from construction management data stored in a total station for progress management. The terminal body's display screen overlays a 3D object image or a 2D drawing image with an image showing actual measured values, and the terminal's position changes are reflected by updating the alignment based on the position data. This makes it possible to utilize the construction management data acquired by the total station for progress management as an AR display on-site. Furthermore, because the user can check the AR display in real time while moving around the site by updating the alignment, the work efficiency of progress management is greatly improved. In addition, by configuring the terminal body and the location acquisition terminal as separate components, high-precision AR display with location information can be achieved simply by attaching the location acquisition terminal to a general-purpose smartphone, which reduces the cost of introducing the system and increases convenience.

[0104] (18) In this embodiment, the AR processing includes manual alignment processing, and the images to be superimposed are aligned based on user input, so that even in environments where automatic alignment is difficult (for example, inside tunnels or buildings where satellite signals are difficult to receive), the user can manually adjust the alignment and perform AR display. It also functions as a means of correcting automatic alignment, and the positional accuracy of the AR display can be improved by making fine adjustments by manual operation when the positioning accuracy is insufficient.

[0105] 4. Variations 4-1. Variation 1: Processing via a cloud server Prior to the data acquisition process (S1), a data conversion process may be performed on the cloud server (not shown) to convert the construction management data into data for AR display. In other words, a series of flows can be adopted in which user U uploads construction management data to the cloud server, the cloud server performs data conversion for AR display, and the information processing terminal 1 acquires the AR display data, including design data and measurement data, from the cloud server.

[0106] The cloud server is a server computer configured to communicate with the information processing terminal 1 via a network (e.g., the Internet or an intranet), and includes a processor and storage device. The cloud server may consist of a single server or a distributed processing environment consisting of multiple servers. Communication between the information processing terminal 1 and the cloud server is performed via communication protocols such as HTTP, HTTPS, WebSocket, or MQTT. The information processing terminal 1 may also communicate with the cloud server via a mobile network (e.g., 4G, 5G, or LPWA) or wireless LAN (Wi-Fi).

[0107] In the data conversion process, the cloud server analyzes the basic design data DT1a and the completed measurement data DT1b included in the construction management data and converts them into a data format suitable for AR display. Specifically, the data conversion process may include at least some of the following processes. Firstly, the process involves generating a 3D model based on the basic design data DT1a, which generates 3D mesh data or 3D polygon data of the object from the design data. Secondly, the coordinate transformation process for the completed measurement data DT1b involves converting the coordinate system of the measurement data to the coordinate system used for AR display. Thirdly, the process involves generating rendering data for the displayed object, setting drawing attributes such as texture, color information, and transparency. Fourthly, data optimization is performed, including reducing the number of polygons (decimation) and generating LOD (Level of Detail) data, taking into account the rendering load on the information processing terminal 1.

[0108] The cloud server generates AR display data, stores the generated AR display data in a storage device, and transmits the AR display data to the information processing terminal 1 in response to an acquisition request from the information processing terminal 1. The transmission of AR display data may be a batch transmission, or it may be a selective transmission of only the data within the required range based on the location information of the information processing terminal 1 (so-called streaming method or tile method). In the tile method, the three-dimensional space of the object is divided into predetermined areas, and data from only the required areas is sequentially acquired based on the current location of the information processing terminal 1, thereby achieving both a reduction in communication volume and real-time display. In other words, the method of delivering AR display data from the cloud server to the information processing terminal 1 can be classified into a batch delivery method, which transmits the entire AR display data at once, and a sequential delivery method, which transmits parts of the AR display data sequentially according to the state of the information processing terminal 1 (e.g., location information, display range, or communication bandwidth). The streaming method and tile method described above are specific examples of the sequential delivery method. The batch delivery method is suitable when the scale of the object is relatively small and the amount of data is limited, or when the communication environment is stable, while the sequential delivery method is suitable when the scale of the object is large and the amount of data is enormous, or when the communication environment is unstable. Furthermore, in the sequential delivery method, a configuration may be adopted in which the data acquisition range is dynamically updated in accordance with the change in the position of the information processing terminal 1, thereby making it possible to suppress display delays caused by the movement of the terminal.

[0109] Furthermore, the cloud server may be configured to regenerate AR display data when construction management data is updated and to send a push notification or automatically update the data to the information processing terminal 1. This configuration reduces the processing load on the information processing terminal 1 while enabling the cloud server to efficiently handle large amounts of data and complex data conversion processes, and also makes it possible to continuously provide AR displays based on the latest construction management data.

[0110] In this modified version 1, the processing load on the information processing terminal is reduced, and efficient data conversion is possible even for large volumes of construction management data. Furthermore, by performing re-conversion on the cloud server when the construction management data is updated, it is possible to always provide AR displays based on the latest data.

[0111] 4-2. Variation 2: Simultaneous AR sharing display by multiple users In the above embodiment, a configuration in which AR display is performed on a single information processing terminal 1 has been described, but the invention is not limited to this. In Modification 2, a configuration can be adopted in which multiple information processing terminals share the AR display for the same object in real time.

[0112] Specifically, multiple information processing terminals are configured to communicate with each other via a network (e.g., the internet, an intranet, or a local network). Each information processing terminal transmits information related to its AR display (e.g., on-site image data, terminal location data, images based on design data and measurement data being displayed, and display mode setting information) to other information processing terminals. This makes it possible, for example, for an information processing terminal of a worker on-site and an information processing terminal of a manager in a remote location to discuss while simultaneously viewing the AR display of the same object.

[0113] The following methods can be used to share AR displays: Firstly, a screen sharing method in which data (screen capture or video stream) of the AR display screen displayed on the display screen of one information processing terminal is transmitted to the other information processing terminal via a communication protocol (e.g., WebRTC, RTSP, or HLS), and the other information processing terminal displays the AR display screen. Secondly, a data synchronization method in which design data, measurement data, terminal position data, and parameters necessary for generating the display object are synchronized among multiple information processing terminals, and each information processing terminal independently performs AR rendering according to its respective viewpoint. In the data synchronization method, since each information processing terminal can perform AR display from an independent viewpoint, it becomes possible to operate in a way that allows workers on site and managers in remote locations to view the object from different angles.

[0114] Furthermore, the system may be configured to share annotations (e.g., markers indicating points of concern, text memos, arrows, etc.) among multiple information processing terminals. When a user on one information processing terminal adds an annotation to the AR display screen, that annotation is also displayed on the AR display screen of the other information processing terminal. This makes it possible to spatially share information such as points to check on the completed work and correction instructions between the site and a remote location, improving the efficiency of progress management during remote site visits and remote supervision.

[0115] Furthermore, communication between multiple information processing terminals may be conducted via a cloud server or directly using a peer-to-peer method. Additionally, the system may also provide voice call and text chat functions for use during communication.

[0116] 4-3. Modification 3: Automated judgment processing using AI / machine learning In the above embodiment, a configuration was described in which images based on design data and measurement data are displayed in AR, allowing the user to visually confirm the completed work. However, the invention is not limited to this configuration. In Modification 3, a configuration can be adopted in which an automatic judgment process is further performed using an AI / machine learning model to determine whether the completed work is satisfactory, to automatically detect abnormal areas, or to predict construction quality.

[0117] The automatic judgment process is performed, for example, in the control unit 12 of the information processing terminal 1 or on a cloud server. In the automatic judgment process, the measured values ​​of the completed work included in the measurement data and the design values ​​and tolerance values ​​included in the design data are input to a trained AI model or machine learning model, and a judgment result is generated based on the output of the model.

[0118] The following are some examples of specific forms of the automated judgment process. Firstly, the system automatically determines whether the measured values ​​at each measurement point or each control cross-section are within the acceptable range. The AI ​​model then displays the result (pass, fail, or require attention, etc.) on the AR display screen. Secondly, the system automatically detects abnormal areas. An AI model, trained on past construction and measurement data, automatically detects areas that deviate from the normal construction pattern and highlights those areas on the AR display screen. Thirdly, regarding the prediction of construction quality, an AI model analyzes trends in changes over time based on time-series data, predicts areas that are highly likely to deviate from acceptable limits in the future, and displays warnings on the AR display screen.

[0119] As the AI ​​model or machine learning model, various algorithms such as neural networks (CNN, RNN, transformer, etc.), support vector machines, random forests, and gradient boosting can be employed. Furthermore, as training data for the AI ​​model, design data, measurement data, pass / fail judgment results, and construction records from past construction projects can be used. The AI ​​model may be implemented in an edge inference manner, where inference is performed in the local environment of the information processing terminal 1, or in a cloud inference manner, where inference is performed on a cloud server.

[0120] The results of the automated judgment process may be displayed on the AR display screen, or alternatively, output as a judgment result report. For example, a report including the judgment result, the basis for the judgment (e.g., which measured values ​​were compared to which tolerance values ​​for the judgment), and a confidence score may be automatically generated. This allows the user to check the judgment results of the AI ​​model and make their own final pass / fail decision.

[0121] 4-4. Modification 4: Automatic generation process for forms and reports In the above embodiment, a configuration was described in which the completed work is confirmed on-site using AR display. However, in addition to this, a configuration can be adopted in which a document generation process is further executed to automatically generate forms or reports necessary for managing the completed work.

[0122] The report generation process is performed, for example, in the control unit 12 of the information processing terminal 1 or on a cloud server. In the report generation process, a progress management report is automatically generated based on at least a portion of the design data, measurement data, and screen capture data during AR display.

[0123] The automatically generated reports may include, for example, the following: Firstly, a progress management chart, which is a report that lists the design value, measured value, difference, and pass / fail judgment result for each management cross section or each measurement point in a table format. Secondly, a progress unfolded drawing, which is a report that displays the measured value or difference of the progress on a drawing that unfolds the surface of the object, such as a slope, using color coding. Thirdly, a progress cross-sectional drawing, which is a drawing that overlays the design cross-sectional shape and the measured cross-sectional shape of the progress at each management cross section. Fourthly, a progress management photo album, which is a report that organizes screen captures acquired during AR display, along with the date and time of shooting, the location of shooting, and the measurement data.

[0124] The data format for the reports can be, for example, PDF format, spreadsheet software file format (e.g., XLSX format), or CSV format. Furthermore, it is preferable that the reports be generated in a format compliant with the construction progress management format specified by the Ministry of Land, Infrastructure, Transport and Tourism.

[0125] The timing of the report generation process may be, for example, executed as needed based on user input, or it may be executed automatically when measurement data is acquired or updated. The generated report may be stored in the storage unit 11 of the information processing terminal 1, sent to and stored on a cloud server, or sent directly to the ordering party via email or the like.

[0126] This modified version allows for a seamless process from on-site completion verification using AR display to report creation, significantly reducing the time and effort required to create reports in the office after bringing measurement data back to the site. This improves the efficiency of document creation work related to completion management.

[0127] 4-5. Modification 5: Drone Integration In the above embodiment, a configuration was described in which on-site imaging data is acquired by the camera 14 of the information processing terminal 1 held by the user. However, in addition to this, or as an alternative, a configuration using a UAV (Unmanned Aerial Vehicle, so-called drone) can be adopted.

[0128] In the first embodiment of Modification 5, aerial image data captured by a camera mounted on a UAV is used as on-site image data. Specifically, image data captured by the UAV while flying over the object is transmitted in real time to the information processing terminal 1 via wireless communication, and on the display screen 13 of the information processing terminal 1, an image based on the aerial image data is superimposed with an image based on design data and an image based on measurement data. This embodiment makes it possible to display AR images from angles that are difficult for the user to confirm from the ground (for example, from above or to the side of the object), making it easier to confirm the top surface of an embankment or the top of a slope.

[0129] In the second embodiment of Modification 5, measurement data (e.g., point cloud data or a set of images for photogrammetry) acquired by a LiDAR, laser scanner, or camera mounted on a UAV is used as measurement data for the completed structure. Specifically, the three-dimensional shape of the object is measured based on the point cloud data or image data acquired by the UAV while flying over the object, and the obtained measurement data is compared with the design data and displayed in AR. According to this embodiment, it is possible to measure the completed structure of a wide range of objects in a short time and to immediately confirm the measurement results as an AR display.

[0130] In the third embodiment of Modification 5, the AR display is aligned using the UAV's flight position data (for example, position data acquired by a GNSS receiver or RTK receiver mounted on the UAV). That is, the UAV's flight position data is used as data equivalent to the terminal position data Dt in the embodiment, and the image to be superimposed on the aerial image transmitted from the UAV is aligned.

[0131] Furthermore, each of the embodiments in Modification 5 can be used in combination with each other. For example, it is possible to use aerial image data acquired by a UAV as on-site imaging data, while using point cloud data acquired by a LiDAR mounted on the same UAV or another UAV as measurement data. Alternatively, it is also possible to use a combination of data acquired by a UAV and data acquired by a ground-based total station or the like.

[0132] Flight control of the UAV may be performed manually by the user, or it may be performed by autonomous flight along a pre-set flight path. In the case of autonomous flight, the optimal flight path may be automatically planned based on the shape of the object and the measurement area.

[0133] According to this modified version, even in locations that are not easily accessible to people or in vast construction areas, it becomes possible to efficiently measure completed work and verify it using AR display by utilizing UAVs, further improving the scope of application and work efficiency of completed work management.

[0134] 4-6. Variation 6: Other terminal configurations In the above embodiment, a configuration was described in which a handheld terminal (smartphone, tablet, etc.) is used as the information processing terminal 1, but the invention is not limited to this. For example, a head-mounted terminal (e.g., a head-mounted display, smart glasses, etc.) or a vehicle-mounted terminal (e.g., a display terminal installed inside the cabin of heavy machinery, etc.) may be used as the information processing terminal 1. In these terminal configurations as well, it is possible to perform AR display by superimposing images based on design data and images based on measurement data onto images based on on-site imaging data acquired by a camera mounted on the terminal.

[0135] 5. Addendum Various embodiments are illustrated below. The embodiments shown below can be combined with each other. [Note 1] Information processing method, The computer performs the data acquisition process and the AR processing. In the aforementioned data acquisition process, the design data of the object and measurement data of the completed form of the object are acquired. The AR processing method involves superimposing an image based on the design data and an image based on the measurement data onto an image based on the on-site imaging data of the completed work. [Note 2] The information processing method described in Appendix 1, The image based on the aforementioned design data includes at least one of an image showing the design shape and a geometric shape image. The image showing the design shape includes a 3D object image or a 2D drawing image. The aforementioned geometric shape image includes points, lines, or surfaces; an information processing method. [Note 3] The information processing method described in Appendix 1 or Appendix 2, In the aforementioned data acquisition process, the design data and the measurement data are acquired from the construction management data. The aforementioned measurement data includes data measured by measuring instruments or measuring systems. The aforementioned measuring instrument or measuring system includes at least one selected from the group consisting of a total station, a measuring tape, a GNSS, a laser scanner, a LiDAR, a 3D measuring device, and an image analysis system, and is an information processing method. [Note 4] The information processing method described in Appendix 3, The construction management data is generated by an information processing method in which data measured by the measuring instrument or the measuring system and the design data are integrated at a terminal separate from the computer. [Note 5] The information processing method described in Appendix 3 or Appendix 4, The image based on the aforementioned design data includes an image showing the design shape or an image showing the design values. An information processing method comprising an image based on the aforementioned measurement data, which includes an image showing the actual measured values ​​of the completed work included in the completed work measurement data. [Note 6] The information processing method described in Appendix 5, An information processing method wherein the image showing the design shape or the image showing the design value includes an image showing control points, control lines, control surfaces, or tolerance ranges included in the design data. [Note 7] The information processing method described in Appendix 5 or Appendix 6, An information processing method that includes an image showing the measured values, which displays the difference between the design values ​​based on the design data and the measured values ​​using numerical values, dimension lines, or a heat map. [Note 8] An information processing method described in any one of the appendices 5 to 7, An information processing method comprising an image showing the measured values, which includes an image that displays, by color coding, the locations where the measured values ​​deviate from the acceptable range. [Note 9] An information processing method described in any one of the appendices 5 to 8, The measurement data includes time-series data obtained through multiple measurements. An information processing method comprising an image showing the measured values, which includes an image that visualizes the change in the completed form over time based on the time-series data. [Note 10] An information processing method described in any one of the appendices 5 to 9, The AR processing method involves generating an image showing the design shape, an image showing the design values, or an image showing the measured values ​​using at least some of the information included in the construction management data, such as the structure information, coordinate reference system set, construction control point set, road structure information, completed cross-section set, measurement point set, measurement equipment installation set, and coordinate points. [Note 11] An information processing method described in any one of the appendices 1 to 10, The positioning process is then performed by a computer. In the positioning process described above, the location data of the terminal body used to capture the on-site image data is acquired. The aforementioned AR process includes a positioning process, The alignment process is an information processing method that uses the position data to align the image to be superimposed in the AR process with the image based on the on-site imaging data. [Note 12] The information processing method described in Appendix 11, It is possible to acquire the aforementioned position data by communicating with artificial satellites, An information processing method in which GNSS, RTK, CLAS, or SLAS is used for communication with the aforementioned artificial satellite. [Note 13] The information processing method described in Appendix 11 or Appendix 12, The positioning process described above is an information processing method that acquires the position data using SLAM. [Note 14] An information processing method described in any one of the appendices 11 to 13, An information processing method for acquiring location data by communicating with a base station. [Note 15] The information processing method described in Appendix 14, The base station is configured to acquire base station coordinates corresponding to the location of the base station, Based on the base station coordinates and distance information, the position data is acquired. An information processing method wherein the distance information is based on the distance between the base station and the location acquisition terminal attached to the terminal body. [Note 16] The information processing method described in Appendix 15, An information processing method in which GNSS, RTK, CLAS, or SLAS is used to obtain the base station coordinates. [Note 17] An information processing method described in any one of the appendices 1 to 16, The computer includes a portable information processing terminal, The aforementioned information processing terminal includes a terminal body and a location acquisition terminal attached to the terminal body, The aforementioned location acquisition terminal is configured to acquire location data by communicating with an artificial satellite or base station. GNSS, RTK, CLAS, or SLAS are used for communication with the aforementioned satellite or for obtaining base station coordinates corresponding to the location of the base station. In the aforementioned data acquisition process, the basic design data included in the construction management data generated by integrating the construction management data and the design data acquired by the total station for progress management, or the measurement data acquired by the measuring instrument or measuring system, at a terminal separate from the computer, is acquired as the design data, and the progress measurement data included in the construction management data is acquired as the measurement data. The AR processing involves superimposing, on the display screen of the terminal, a 3D object image or 2D drawing image based on the design data and an image showing the measured values ​​of the completed work in the completed work measurement data onto an image based on the on-site image data of the completed work. The aforementioned AR process includes a positioning process, The alignment process includes an information processing method that reflects changes in the position of the information processing terminal by updating the alignment based on the position data. [Note 18] An information processing method described in any one of the appendices 1 to 17, The aforementioned AR process includes a manual alignment process. The alignment process is an information processing method that aligns the position of an image to be superimposed in the AR process on an image based on the on-site imaging data, based on user input. [Note 19] An information processing method described in any one of the appendices 1 to 18, Prior to the aforementioned data acquisition process, a data conversion process is performed on the cloud server to convert the construction management data into data for AR display. The data acquisition process is an information processing method in which the computer acquires the AR display data, including the design data and the measurement data, from the cloud server. [Note 20] A program that executes the information processing method described in any one of the appendices 1 through 19. [Note 21] A portable information processing terminal, It comprises a camera, a display screen, and a control unit. The control unit comprises a data acquisition unit, an imaging unit, and an AR processing unit. The data acquisition unit acquires the design data of the object and the measurement data of the completed form of the object. The imaging unit acquires on-site imaging data of the completed work via the camera. The AR processing unit is an information processing terminal that superimposes an image based on the design data and an image based on the measurement data onto an image based on the on-site imaging data on the display screen. [Note 22] An information processing system including a cloud server and a portable information processing terminal, The aforementioned cloud server performs a data conversion process to convert construction management data into data for AR display. The AR display data includes design data of the object and measurement data of the completed form of the object. The aforementioned information processing terminal comprises a camera, a display screen, and a control unit. The control unit comprises a data acquisition unit, an imaging unit, and an AR processing unit. The data acquisition unit acquires the AR display data from the cloud server, The imaging unit acquires on-site imaging data of the completed work via the camera. The AR processing unit is an information processing system that superimposes an image based on the design data and an image based on the measurement data onto an image based on the on-site imaging data on the display screen. [Explanation of Symbols]

[0136] 1: Information processing terminal 2: The main unit 10: Communications Department 11: Storage section 12: Control Unit 12a: Data acquisition unit 12b: Imaging unit 12b1: Image 12c: Positioning unit 12d: AR Processing Unit 12d1:Display object generation part 12d2: Alignment section 12d3: Overlay display section 13:Display screen 14: Camera 15: LiDAR scanner 16: Communications bus 3: Location acquisition device 3a: Enclosure 3b: Antenna 3c: Battery 3d: Main circuit DT1: Construction Management Data DT1a: Basic design data DT1b: Construction progress measurement data Dt: Terminal location data G:Artificial satellite K:Base station Ob1: Object P1: Dashed line area P2: Dashed line area PN1:Position coordinates TS1: Total Station U: User V1: Image V2: Image V3: Image X: On-site

Claims

1. Information processing method, The computer performs data acquisition processing and AR processing. In the aforementioned data acquisition process, the design data of the object and the measurement data of the completed state of the object are acquired from the construction management data. In the aforementioned AR processing, an image based on the on-site imaging data of the completed work is superimposed with an image based on the design data and an image based on the measurement data. The aforementioned measurement data includes data measured by measuring instruments or measuring systems. The measuring instrument or measuring system includes at least one selected from the group consisting of a total station, a measuring tape, a GNSS, a laser scanner, a LiDAR, a three-dimensional measuring device, and an image analysis system. The image based on the aforementioned design data includes an image showing the design shape or an image showing the design values. An information processing method comprising an image based on the aforementioned measurement data, which includes an image showing the actual measured values ​​of the completed work included in the completed work measurement data.

2. The information processing method according to claim 1, The image based on the aforementioned design data further includes a geometric shape image, The image showing the design shape includes a three-dimensional object image or a two-dimensional drawing image. The aforementioned geometric shape image includes points, lines, or surfaces; an information processing method.

3. The information processing method according to claim 1, The construction management data is generated by an information processing method in which data measured by the measuring instrument or the measuring system and the design data are integrated at a terminal separate from the computer.

4. The information processing method according to claim 1, An information processing method wherein the image showing the design shape or the image showing the design value includes an image showing control points, control lines, control surfaces, or tolerance ranges included in the design data.

5. The information processing method according to claim 1, An information processing method that includes an image showing the measured values, which displays the difference between the design values ​​based on the design data and the measured values ​​using numerical values, dimension lines, or a heat map.

6. The information processing method according to claim 1, An information processing method comprising an image showing the measured values, which includes an image that displays, by color coding, the locations where the measured values ​​deviate from the acceptable range.

7. The information processing method according to claim 1, The measurement data includes time-series data obtained through multiple measurements. An information processing method comprising an image showing the measured values, which includes an image that visualizes the change in the completed form over time based on the time-series data.

8. The information processing method according to claim 1, The AR processing method involves generating an image showing the design shape, an image showing the design values, or an image showing the measured values ​​using at least some of the information included in the construction management data, such as the structure information, coordinate reference system set, construction control point set, road structure information, completed cross-section set, measurement point set, measurement equipment installation set, and coordinate points.

9. The information processing method according to claim 1, The positioning process is then performed by a computer. In the positioning process described above, the location data of the terminal body used to capture the on-site image data is acquired. The aforementioned AR processing includes alignment processing, The alignment process is an information processing method that uses the position data to align the image to be superimposed in the AR process with the image based on the on-site imaging data.

10. The information processing method according to claim 9, It is possible to acquire the aforementioned position data by communicating with artificial satellites, An information processing method in which GNSS, RTK, CLAS, or SLAS is used for communication with the aforementioned artificial satellite.

11. The information processing method according to claim 9, The positioning process described above is an information processing method that acquires the position data using SLAM.

12. The information processing method according to claim 9, An information processing method for acquiring location data by communicating with a base station.

13. The information processing method according to claim 12, The base station is configured to acquire base station coordinates corresponding to the location of the base station, Based on the base station coordinates and distance information, the position data is acquired. An information processing method wherein the distance information is based on the distance between the base station and the location acquisition terminal attached to the terminal body.

14. The information processing method according to claim 13, An information processing method in which GNSS, RTK, CLAS, or SLAS is used to obtain the base station coordinates.

15. An information processing method, The computer performs data acquisition processing and AR processing. In the aforementioned data acquisition process, the design data of the object and measurement data of the completed form of the object are acquired. In the aforementioned AR processing, an image based on the on-site imaging data of the completed work is superimposed with an image based on the design data and an image based on the measurement data. The computer includes a portable information processing terminal, The aforementioned information processing terminal includes a terminal body and a location acquisition terminal attached to the terminal body, The aforementioned location acquisition terminal is configured to acquire location data by communicating with an artificial satellite or base station. GNSS, RTK, CLAS, or SLAS are used for communication with the aforementioned satellite or for obtaining base station coordinates corresponding to the location of the base station. In the aforementioned data acquisition process, the basic design data included in the construction management data generated by integrating the construction management data and the design data acquired by the total station for progress management, or the measurement data acquired by the measuring instrument or measuring system, at a terminal separate from the computer, is acquired as the design data, and the progress measurement data included in the construction management data is acquired as the measurement data. The AR processing involves superimposing, on the display screen of the terminal, a three-dimensional object image or two-dimensional drawing image based on the design data and an image showing the measured values ​​of the completed work in the completed work measurement data onto an image based on the on-site image data of the completed work. The aforementioned AR processing includes alignment processing, The alignment process includes an information processing method that reflects changes in the position of the information processing terminal by updating the alignment based on the position data.

16. The information processing method according to claim 1, The aforementioned AR processing includes manual alignment processing. The alignment process is an information processing method that aligns the position of an image to be superimposed in the AR process on an image based on the on-site imaging data, based on user input.

17. The information processing method according to claim 1, Prior to the aforementioned data acquisition process, a data conversion process is performed on the cloud server to convert the construction management data into data for AR display. The data acquisition process is an information processing method in which the computer acquires the AR display data, including the design data and the measurement data, from the cloud server.

18. A program that performs the information processing method described in any one of claims 1 to 17.

19. A portable information processing terminal, It comprises a camera, a display screen, and a control unit. The control unit includes a data acquisition unit, an imaging unit, and an AR processing unit. The data acquisition unit acquires the design data of the object and the measurement data of the completed state of the object from the construction management data. The imaging unit acquires on-site imaging data of the completed work via the camera. The AR processing unit superimposes the image based on the design data and the image based on the measurement data onto the image based on the on-site imaging data on the display screen. The aforementioned measurement data includes data measured by measuring instruments or measuring systems. The measuring instrument or measuring system includes at least one selected from the group consisting of a total station, a measuring tape, a GNSS, a laser scanner, a LiDAR, a three-dimensional measuring device, and an image analysis system. The image based on the aforementioned design data includes an image showing the design shape or an image showing the design values. An information processing terminal that includes an image based on the aforementioned measurement data, which shows the actual measured values ​​of the completed work included in the completed work measurement data.

20. An information processing system including a cloud server and a portable information processing terminal, The aforementioned cloud server performs a data conversion process to convert construction management data into data for AR display. The AR display data includes design data of the object and measurement data of the completed form of the object. The aforementioned information processing terminal comprises a camera, a display screen, and a control unit. The control unit includes a data acquisition unit, an imaging unit, and an AR processing unit. The data acquisition unit acquires the AR display data from the cloud server. The imaging unit acquires on-site imaging data of the completed work via the camera. The AR processing unit superimposes the image based on the design data and the image based on the measurement data onto the image based on the on-site imaging data on the display screen. The aforementioned construction management data includes the measurement data measured by measuring instruments or measuring systems and the design data, The measuring instrument or measuring system includes at least one selected from the group consisting of a total station, a measuring tape, a GNSS, a laser scanner, a LiDAR, a three-dimensional measuring device, and an image analysis system. The image based on the aforementioned design data includes an image showing the design shape or an image showing the design values. An information processing system in which the image based on the measurement data includes an image showing the actual measured value of the completed work included in the completed work measurement data.

Images