Information processing system, information processing method, and program

Abstract

The present invention relates to an information processing system that is suitable for inspection work of a structure. An information processing device includes a model management device and a terminal device. The model management device stores model data of a structure made up of a plurality of members. The terminal device estimates an own position with respect to the structure in reality, and corrects the own position by using the model data.

Description

Information processing system, information processing method and program

[0001] The present invention relates to an information processing system, an information processing method, and a program suitable for maintenance and management work of structures.

[0002] In the past, in maintenance work (hereinafter simply referred to as inspections) such as the inspection of bridges and other structures, workers would record, refer to, and edit the location and nature of any abnormalities that had occurred in the structure using paper drawings and forms. However, in large-scale structures, especially those with repeated similar local structures such as long-span bridges, it becomes difficult for workers to identify their own location or the location of any abnormalities while referring to paper drawings and forms. This increases the possibility that workers will misidentify their own location or the location of any abnormalities.

[0003] One method for solving this problem is to use a Global Navigation Satellite System (GNSS) to identify the location of the worker or the abnormality.

[0004] For example, Patent Document 1 describes a system used for inspecting structures. In this system, a head-mounted display (HMD) worn by a worker acquires the worker's current position using GNSS, recognizes various instructions given while visually inspecting the structure, converts the current position into a corresponding location on a drawing, and records the various instructions on the drawing.

[0005] Japanese Patent Application Laid-Open No. 2022-30893

[0006] However, GNSS has the problem of being unable to pinpoint locations in places where satellite communication is not possible. For example, this problem is likely to occur inside or behind large structures such as long-span bridges. Therefore, there is a need for a system that can pinpoint the locations of workers and abnormalities with high accuracy, even in places where satellite communication is difficult, to achieve highly efficient and high-quality inspection and maintenance of structures.

[0007] According to one embodiment, the information processing system includes a model management device and a terminal device. The model management device stores model data of a structure consisting of multiple components. The terminal device estimates its own position on the actual structure and corrects the own position using the model data. According to one embodiment, the information processing system further includes an additional data management device, which defines a predetermined area overlaid on the structure as a management section. The model management device stores information on a deformation occurring in the component along with the position of the deformation identified based on the own position. When reading the information on the deformation from the model management device, the terminal device identifies the management section to which the deformation belongs and displays the identified management section. According to one embodiment, the information processing system further includes an additional data management device, which defines a predetermined area overlaid on the structure as an inspection range section. The terminal device reads only the components associated with the inspection range section from the model management device and displays the read components. According to one embodiment, the terminal device photographs the actual structure and estimates the area of ​​the deformation included in the photographed image. According to one embodiment, the terminal device estimates resources required to repair the deformation. According to one embodiment, the terminal device allows a worker to visually recognize the actual structure, invisibly superimposes the model data on the actual structure, accepts a position on the actual structure specified by the worker, and identifies the component corresponding to the specified position. According to one embodiment, the model management device stores information on the deformation occurring in the component in association with the identified component. According to one embodiment, the model management device stores multiple pieces of information regarding the specific deformation acquired at different times. According to one embodiment, the terminal device superimposes the position of the terminal device on the visible model data. According to one embodiment, the terminal device estimates its own position on the actual structure and corrects the own position using model data of the structure.According to one embodiment, the information processing method includes a step of estimating a self-position on a real structure by a terminal device, and a step of correcting the self-position using model data of the structure. According to one embodiment, the program causes a computer to execute the information processing method.

[0008] The present invention can provide an information processing system, an information processing method, and a program suitable for maintenance and management work of structures.

[0009] 1 is a block diagram showing a hardware configuration of an information processing system. FIG. 1 is a block diagram showing a functional configuration of a terminal device. FIG. 1 is a block diagram showing a functional configuration of a model management device. FIG. 2 is a diagram showing an example of the data structure of a BIM / CIM model. FIG. 2 is a block diagram showing a functional configuration of a reference data management device. FIG. 3 is a diagram showing an example of the data structure of reference data. FIG. 3 is a block diagram showing the functional configuration of an additional data management device. FIG. 4 is a diagram showing an example of the data structure of additional data. FIG. 5 is a diagram explaining management classifications. FIG. 6 is a diagram explaining a self-position correction function. FIG. 7 is a diagram explaining a self-position correction function. FIG. 8 is a diagram explaining a self-position correction function. FIG. 9 is a diagram explaining a self-position correction function. FIG. 10 is a diagram explaining a self-position correction function. FIG. 11 is a diagram explaining a self-position display screen. FIG. 12 is a diagram showing an example of a member information display screen. FIG. 13 is a diagram showing an example of a new deformation registration screen. FIG. 14 is a diagram showing an example of a deformation location list display screen. FIG. 15 is a diagram showing an example of a deformation reference / edit / addition screen. FIG. 16 is a block diagram showing the operation of a terminal device. FIG. 17 is a block diagram showing the operation of a terminal device. FIG. 18 is a block diagram showing the operation of a terminal device. FIG. 19 is a block diagram showing the operation of a terminal device. FIG. 19 is a block diagram showing the operation of a terminal device. FIG. 19 is a block diagram showing the operation of a terminal device. FIG. 20 is a block diagram showing a modified example of an information processing system. FIG. 21 is a block diagram showing a modified example of an information processing system.

[0010] 1 is a diagram showing a hardware configuration of an information processing system 1 according to an embodiment of the present invention. The information processing system 1 includes a terminal device 10, a model management device 20, a reference data management device 30, and an additional data management device 40. The terminal device 10 is communicably connected to the model management device 20 and the additional data management device 40. The model management device 20 is communicably connected to the reference data management device 30.

[0011] The terminal device 10, the model management device 20, the reference data management device 30, and the additional data management device 40 are all information processing devices having a processor, a memory, a communication device, an input / output device, etc. The processor reads and executes a program stored in the memory, thereby logically realizing each of the functional units described below.

[0012] The terminal device 10 is an information processing device used by a worker, and is typically a tablet computer, a smartphone, an HMD (e.g., Hololens, etc.), etc. In addition to the above-mentioned processor, memory, etc., the terminal device 10 may include hardware such as a display device (display) that outputs information to the worker, an input device (touch panel, eye-gaze input device, pointing device, etc.) that accepts information input by the worker, a camera for capturing images, a GNSS receiving device, and various sensors for SLAM (Simultaneous Localization and Mapping) (gyro, acceleration sensor, depth sensor, ToF sensor, LiDAR, etc.).

[0013] The model management device 20, the reference data management device 30, and the additional data management device 40 are typically server computers. The model management device 20, the reference data management device 30, and the additional data management device 40 may be realized in a virtual information processing environment such as cloud computing, or may be integrated with the terminal device 10.

[0014] 2 is a block diagram showing the functional configuration of the terminal device 10. The terminal device 10 includes a display unit 101, an input unit 102, an image capturing unit 111, a position estimation unit 112, a model operation unit 113, an additional data operation unit 114, a deformation detection unit 115, and a repair estimate unit 116.

[0015] The display unit 101 performs processing to display an image on a display device (display). For example, the display unit 101 can display a three-dimensional model of a structure, a still image, a video, character data, etc. showing the inspection result of the structure on the display device so that the worker can visually confirm them.

[0016] When the display device is a transmission type, the worker can point the display device toward the real structure and directly see the real structure that is visible behind the display device. This is called optical see-through. In this state, the display unit 101 can project desired information onto the display device, superimposing the information on the real structure.

[0017] If the display device is a non-transmissive type, the display device can display in real time an image of the real structure being captured by the image capturing unit 111 (described later). This is called video see-through. In this state, the display unit 101 can superimpose desired information onto the video see-through image and display it on the display device, thereby superimposing the information on the real structure.

[0018] The input unit 102 accepts information input by a worker. The input unit 102 typically allows input via a touch panel, eye-gaze input, input via a pointing device, input via a keyboard, voice input, etc. However, the input method is not limited to these and any input method can be used.

[0019] For example, when the display unit 101 displays a model of a structure, the input unit 102 can recognize the coordinates (model coordinates) corresponding to a point input by the operator. It can also identify a component present at the coordinates and recognize that the component has been selected.

[0020] In addition, the input unit 102 can accept as input images (still images or videos) captured by the worker using the imaging unit 111 described below, text information entered via a keyboard or voice, and graphic information drawn using a touch panel or pointing device.

[0021] The photographing unit 111 controls the camera to photograph actual structures, deformations, etc., and outputs images (still images or video images).

[0022] The position estimation unit 112 receives positioning signals from GNSS satellites using a GNSS receiving device, and can calculate the coordinates (latitude, longitude, and altitude) of the reception point.

[0023] Additionally, the position estimation unit 112 uses SLAM (Scanning Latency Measurement) to recognize (map) the three-dimensional shape of the real space and estimate the position and orientation of the terminal device 10 (self-position estimation), primarily using camera images. SLAM is a well-known technology, so a detailed description of the SLAM algorithm will be omitted here. The terminal device 10 can also improve the accuracy of SLAM by using information obtained from various sensors, such as a gyro sensor that detects rotation (angular velocity), an acceleration sensor that detects tilt, a depth sensor that detects the distance to an object in the real space, a ToF (Time of Flight) sensor, and a LiDAR (Light Detection and Ranging) sensor.

[0024] By performing self-location estimation using SLAM, the position estimation unit 112 can determine the coordinates of the terminal device 10 even when it cannot receive GNSS signals, for example, when it is inside or behind a large-scale structure such as a long bridge.

[0025] The position estimation unit 112 can also detect the position of the terminal device 10 using known techniques such as self-position estimation using passive RFID.

[0026] Structures can be deformed by temperature changes, wind, waves, ground movement, and other factors. These effects are particularly noticeable in large-scale structures such as long-span bridges. For example, as shown in Figure 10, in a girder bridge, the length of the bridge girders differs between the hot summer months and the cold winter months. In addition, in a suspension bridge, the shape of the bridge girders, such as the height, differs between the hot summer months and the cold winter months.

[0027] In an ideal state where a structure is not deformed, if the self-position estimated by GNSS or SLAM is superimposed on a 3D model of the structure, the self-position on the structure can be accurately plotted. However, this is premised on the fact that the coordinate systems of the self-position and the 3D model of the structure match. However, if the self-position estimated by GNSS or SLAM is simply superimposed on the 3D model of the structure when the structure is deformed, the self-position will be plotted in an incorrect position on the structure. This is because the shape of the actual structure differs from the 3D model.

[0028] To address this problem, in this embodiment, the position estimation unit 112 has a function of correcting its own position. The processing procedure will be described with reference to the flowchart in FIG.

[0029] Step S1: Associating structure model data with latitude and longitude information As a preparation step, any two points on the actual structure are set as measurement points, and the latitude and longitude at the measurement points are measured. It is preferable that the measurement points are in locations that are less susceptible to temperature changes, etc. Figure 11 is a diagram explaining the self-position correction function. The two-point triangle displayed on the support indicates the measurement points.

[0030] Step S2: Measuring self-position At any two measurement points on the real structure (which do not have to be the same as the two measurement points in step S1), the position estimation unit 112 estimates the self-position and calculates the latitude and longitude. Fig. 12 is a diagram explaining the self-position correction function. Two tablet terminals indicate the measurement points.

[0031] Step S3: Superimposed display of structure Using the longitude and latitude information measured in steps S1 and S2, it becomes possible to convert between the coordinate system recognized by the position estimation unit 112 and the coordinate system of the model data. The position estimation unit 112 converts the model data into the coordinate system recognized by the position estimation unit 112. The display unit 101 displays this model data on the screen.

[0032] The display unit 101 also displays the real structure on the display device by optical see-through or video see-through, so that the real structure and the model data of the structure are superimposed on the display device.

[0033] Step S4: Obtaining the correction amount In an ideal state where the structure is not deformed, the actual structure and the model data of the structure will appear to overlap perfectly. On the other hand, when the structure is deformed, the two will be observed to be misaligned. If a misalignment occurs, the system will switch to calibration mode.

[0034] In the calibration mode, the input unit 102 accepts user input such as tapping, dragging, or pinching on the screen. In response to the user input, the model data of the structure is freely moved or rotated on the screen (see FIG. 13 ). When the actual structure and the model data of the structure appear to be perfectly superimposed on each other, the calibration mode is terminated, for example, by an instruction from the operator.

[0035] The position estimation unit 112 compares the display position and angle of the model data when it is displayed in step S3 with the display position and angle of the model data when the calibration mode ends, and calculates the difference as the amount of correction.

[0036] Step S5: Correcting Self-Location The position estimation unit 112 measures the current self-location and adds the correction amount calculated in step S4 to it. Then, the self-location after correction is converted into the coordinate system of the model data.

[0037] This makes it possible to superimpose the self-position estimated by GNSS or SLAM on the correct position on the 3D model of the structure even when the structure is deformed. Furthermore, even when recording the position of a deformation, for example, by correcting the viewpoint position at the time of the deformation discovery using the above-mentioned procedure and then recording it, it becomes possible to record the deformation in association with the correct position on the 3D model of the structure. Even if the shape of the structure changes each time due to temperature changes, etc., inspection information can always be recorded linked to the correct position on the 3D model of the structure.

[0038] The model operation unit 113 accesses the model management device 20 (described later) and performs processing to operate the BIM / CIM model managed by the model management unit 201 (described later). That is, the model operation unit 113 reads the BIM / CIM model and reads, registers, and edits attribute data and reference data. The BIM / CIM model, attribute data, and reference data will be described later.

[0039] The additional data operation unit 114 accesses the additional data management device 40, which will be described later, and reads, registers, and edits the additional data managed by the additional data management unit 401. The additional data will be described later.

[0040] The deformation detection unit 115 analyzes camera images of the actual structure and detects deformations that have occurred in the structure. A specific deformation detection method will be described later.

[0041] The repair estimate unit 116 performs a repair estimate (calculation of necessary materials, man-hours, costs, etc.) for the deformation detected by the deformation detection unit 115. A specific repair estimate method will be described later.

[0042] 3 is a block diagram showing the functional configuration of the model management device 20. The model management device 20 includes a model management unit 201.

[0043] The model management unit 201 stores BIM / CIM models. A BIM / CIM (Building Information Modeling / Construction Information Modeling / Management) model is an information model that combines attribute data and reference data with a three-dimensional model that represents a structure in three-dimensional form. The three-dimensional model of the structure is shared throughout a series of processes in construction and civil engineering projects, such as planning, design, construction, and maintenance, and the information required for each process is managed in association with this three-dimensional model. The information model formed through such efforts is the BIM / CIM model. Utilizing BIM / CIM models is expected to improve quality and productivity throughout the entire project.

[0044] An example of the data structure of a BIM / CIM model handled in this embodiment is shown in Fig. 4. The BIM / CIM model includes model data and attribute data.

[0045] The model data is a three-dimensional model of a structure. This three-dimensional model includes one or more component data. The component data includes the three-dimensional shape, coordinates, and unique key (GUID) of each component.

[0046] The attribute data is additional data associated with each component, and typically the name of the component is registered. In this case, the attribute data includes the name of the component and a unique key (GUID).

[0047] 5 is a block diagram showing the functional configuration of the reference data management device 30. The reference data management device 30 includes a reference data management unit 301.

[0048] The reference data management unit 301 stores reference data. The reference data is also incidental data associated with each component, but differs from attribute data in that it is placed outside the BIM / CIM model.

[0049] 6 shows an example of the data structure of the reference data handled in this embodiment. The reference data management unit 301 can store any text data or media files such as photos, videos, and audio as reference data. The reference data also includes a unique key (GUID) that indicates the component to which the data is linked.

[0050] Typically, the production classification is registered as reference data. The production classification is an identifier that is uniformly assigned to multiple components that form a single semantic entity, mainly during the design and production stages. For example, a bridge (structure) is composed of multiple components, but in reality, all components are not assembled at once; rather, substructures are produced in multiple phases and then assembled. Therefore, a unique production classification is assigned to each component for each production phase, such as "NW21" for all components that make up a substructure produced in one production phase and "NW22" for all components that make up a substructure produced in another production phase. This facilitates the production process. The production classification is registered in the reference data for each component.

[0051] In addition, information regarding deformations that have occurred in components (deformation data) is registered as reference data. The deformation data may include a deformation ID, text data, a media file, and a viewpoint position. The deformation ID is a unique identifier assigned when a new deformation is registered. The text data includes, for example, comments registered by workers. The media file includes, for example, photos and videos taken during inspection, and audio input by workers as comments. The viewpoint position is the position of the terminal device 10 when these data were generated or input.

[0052] In this way, each component of the three-dimensional model, the attribute data, and the reference data are associated with each other via a unique key (GUID).

[0053] 7 is a block diagram showing the functional configuration of the additional data management device 40. The additional data management device 40 includes an additional data management unit 401.

[0054] 8 shows an example of the data structure of additional data handled in this embodiment. The additional data management unit 401 stores additional data. The additional data includes information used mainly for the convenience of inspection work. In this embodiment, the inspection range classification, management classification, and inside / outside classification are registered as additional data.

[0055] Inspection Scope Classification: From the perspective of work efficiency, it is common to narrow down inspection work to similar content and perform it in batches, such as only the deck, only the inside of the box girder, or only the diaphragm. In this case, loading the BIM / CIM data for the entire bridge and then attempting to access the attribute data and reference data for the inspection target area can increase processing load and cause problems such as reduced response. To prevent this, it is sufficient to control the loading of only the BIM / CIM data required for the time being. For example, only the BIM / CIM data for the substructure scheduled for inspection (deck, inside of the box girder, diaphragm, etc.) can be loaded. In addition, BIM / CIM data for parts that are likely to be used simultaneously can be loaded. For example, since deformation may occur across multiple components, it is also possible to load BIM / CIM data for parts adjacent to the planned inspection area. This reduces processing load and improves response. Specifically, a common inspection scope classification is assigned to the zones where inspection work is to be performed together. For example, assign inspection range classifications such as "deck slab" to the deck slab zone, "inside box girder" to the box girder interior zone, and "diaphragm" to the diaphragm zone. By determining the targets that should be inspected together in accordance with the inspection plan and defining that inspection plan as the inspection range classification, it becomes possible to carry out inspection work efficiently.

[0056] In this embodiment, inspection range sections are defined by assigning unique identifiers (names) to predefined spaces (zones). A zone can typically be defined as a three-dimensional object. That is, a zone can be defined by defining the coordinates of the vertices of a three-dimensional shape.

[0057] The BIM / CIM data for which components should be loaded can be determined by determining whether the zone assigned the desired inspection range category overlaps with the coordinates of the component. For example, if you want to inspect only the deck, you can determine whether the zone assigned the inspection range category of "deck" overlaps with the components that make up the structure, and then load only the BIM / CIM data for the components that are determined to overlap.

[0058] Management Classification: To identify inspection locations, it is useful to define a classification specific to the inspection work for a structure. For example, as shown in FIG. 9 , a structure may be composed of multiple box girders, and one manufacturing classification may be defined for each group of box girders. While this manufacturing classification could be used directly for inspection work, some ingenuity is required to identify which box girders are the inspection locations, making it less user-friendly. For inspection work, it is more convenient to define classifications for each individual box girder included in the manufacturing classification. Therefore, this embodiment introduces the concept of a management classification specific to inspection work. For example, by assigning different management classifications to each individual box girder in FIG. 9 , inspection work can be carried out more efficiently. For example, if the manufacturing classification "NW22" is composed of six box girders, the zones containing these box girders can be assigned management classifications such as "NW22-1," "NW22-2," ..., "NW22-6."

[0059] In this embodiment, management divisions are defined by assigning unique identifiers (names) to predefined spaces (zones). In the example of FIG. 9, a structure is divided into multiple zones, each defined as a different management division. A zone can typically be defined as a three-dimensional object. That is, a zone can be defined by defining the coordinates of the vertices of a three-dimensional shape.

[0060] For example, if you want to know which management category a worker's current location falls into, you can determine whether the worker's current location overlaps with several zones assigned with management categories. The management category of the zone that is determined to overlap corresponds to the worker's current location.

[0061] Inside / outside distinction: During inspection work, it is sometimes necessary to identify whether a worker is inside or outside a structure. For example, when registering the inspection results of a box girder, there are cases where it is necessary to manage whether an abnormality was detected on the outside or inside of the box girder. In this regard, BIM / CIM models do not have the data to distinguish between inside and outside. Therefore, a zone for determining inside / outside is defined in the space inside the box girder, and this zone is assigned an inside / outside distinction indicating "inside."

[0062] In this embodiment, an inside / outside distinction is assigned to a predefined space (zone). A zone can typically be defined as a three-dimensional object. That is, a zone can be defined by defining the coordinates of the vertices of a three-dimensional shape.

[0063] For example, if you want to determine whether the location of a defect is inside or outside the box girder, you can determine whether the zone assigned the inside / outside classification overlaps with the position (viewpoint position) of the worker, i.e., the terminal device 10, when the defect was discovered. If it is determined that there is an overlap, the defect was discovered inside the box girder. On the other hand, if it is determined that there is no overlap, the defect was discovered outside the box girder.

[0064] Several use cases of the operation of the information processing system 1 will be described.

[0065] <Display of Current Location of Worker> The terminal device 10 can display on the screen the current location of the worker in the structure. The processing procedure will be described using the flowchart of FIG.

[0066] Step S101: Estimation, correction, and coordinate conversion of self-position By the procedure of steps S1 to S5 described above, the position estimation unit 112 estimates the self-position of the terminal device 10, applies a predetermined correction to it, and then converts it into the coordinate system of the model data of the structure.

[0067] Step S102: Superimposed display The model operation unit 113 accesses the model management unit 201 and acquires model data of the structure. The display unit 101 displays the acquired model data on the screen. At the same time, the self-position converted in S101 is superimposed on the model data of the structure. For example, an icon can be displayed at the self-position as shown in FIG. 14.

[0068] This function allows the terminal device 10 to clearly indicate where a worker is currently standing within a long structure such as a continuous truss bridge, thereby reducing the time and effort that would previously have been spent on identifying the worker's own position and the location of any abnormalities on a paper drawing, and preventing misidentification.

[0069] <Loading only necessary BIM / CIM data> When an operator inspects a specific inspection target in accordance with an inspection plan, the operator can selectively load only the BIM / CIM data related to the inspection target. The processing procedure is shown using the flowchart in Figure 21.

[0070] Step S201: Selection of inspection range section The additional data operation unit 114 accesses the additional data management unit 401 and acquires the registered inspection range sections. The display unit 101 displays a list of the acquired inspection range sections on the screen. For example, inspection range sections such as "deck slab," "inside box girder," and "diaphragm" are displayed on the screen.

[0071] The input unit 102 accepts a user input for selecting an inspection range category.

[0072] Step S202: Reading BIM / CIM data for selected inspection management category The additional data operation unit 114 accesses the additional data management unit 401 and acquires the zone associated with the inspection range category selected in S201.

[0073] The model operation unit 113 accesses the model management unit 201 and extracts BIM / CIM data of components that overlap with the acquired zone.

[0074] The display unit 101 reads the model data of the extracted members and displays them on the screen.

[0075] Step S203: Additional Loading of BIM / CIM Data In addition to the inspection range section selected by the user in step S201, BIM / CIM data for zones that are likely to be used at the same time may be loaded in advance. For example, since a deformation may occur across multiple components, BIM / CIM data for zones adjacent to the inspection range section selected by the user can be loaded.

[0076] In this case, the additional data operation unit 114 accesses the additional data management unit 401 and additionally acquires zones that are likely to be used simultaneously with the inspection range segment selected by the user.

[0077] The model operation unit 113 accesses the model management unit 201 and extracts BIM / CIM data of components that overlap with the additionally acquired zone.

[0078] The display unit 101 reads the model data of the extracted members and displays them on the screen.

[0079] With this function, the terminal device 10 reads only the BIM / CIM data of components related to the zones specified in the inspection plan, thereby reducing the processing load on the system and enabling inspection work to be carried out efficiently.

[0080] <Display of Component Information> When the worker observes the actual structure through the terminal device 10, information on the components that are visible within the worker's field of view is displayed on the screen. The processing procedure will be described using the flowchart of FIG.

[0081] Step S301: Invisible superimposed display of model data The display unit 101 makes the real structure visible on the display device by optical see-through or video see-through.

[0082] The model operation unit 113 accesses the model management unit 201 and acquires model data of the structure. The display unit 101 displays the acquired model data on the display device. At this time, the coordinate system of the model data of the structure is converted into the real coordinate system recognized by the terminal device 10, and the model data is projected onto the display device taking into account the correction amount calculated in step S4 above. As a result, the model data projected on the display device is perfectly superimposed on the real structure visible on the display device by optical see-through or video see-through.

[0083] At this time, the display unit 101 displays the model data of the structure in an invisible state. In other words, the model data is displayed with 0% opacity. That is, the model data is superimposed on the actual structure as an internal process, but this fact is not noticeable to the worker.

[0084] It is preferable that the projection of the model data is invisible (transparent), but it does not necessarily have to be completely transparent. If the worker can see the deformation of the actual structure, it is acceptable to overlay the data in a semi-transparent state with low opacity (about 50% or less), for example.

[0085] Step S302: Identifying the gaze point The display unit 101 extends the normal of the display device in the case of optical see-through, or the optical axis of the camera in the case of video see-through, and identifies the point (gazing point) that collides with the model data that is invisibly superimposed on the real structure.

[0086] Alternatively, when a worker points to a component included in the real structure on the display device by, for example, tapping, using a pointing device, or by eye-gaze input, the input point is projected along the normal line of the display device or the optical axis direction of the camera, and the point of collision (point of gaze) with the model data invisibly superimposed on the real structure is identified. In this case, the worker can select a component included in the model data by directly specifying that exact point while visually recognizing the component of the real structure through optical see-through or video see-through.

[0087] Step S303: Identifying Components The model operation unit 113 accesses the model management unit 201 and extracts components that overlap with the gaze point identified in step S302. If a component cannot be uniquely identified, multiple components existing around the projection point may be listed and the worker may identify the component. If the worker does not identify the component, the component closest to the gaze point may be treated as the identified component.

[0088] Step S304: Acquisition of component information The model operation unit 113 accesses the model management unit 201 and acquires model data and / or reference data of the structure. The display unit 101 displays the acquired information on the display device. An example of a screen is shown in FIG. 15. In this example, a photo taken by a camera is displayed in the center of the screen, and the name of the component, the coordinates of the viewpoint position (the current position of the terminal device 10), etc. are displayed at the top of the screen. This name is determined by the display unit 101 identifying the component reflected at the point of interest, and the model operation unit 113 acquiring the attribute data of that component.

[0089] With this function, the terminal device 10 invisibly overlays the model data of the structure on the real structure, and identifies the component that the worker is looking at while the real structure is directly visible. Therefore, the worker does not need to switch his / her attention between the drawing and the reality, and can refer to the information on the desired component without being aware of the existence of the model.

[0090] <Registering inspection results> Workers inspect structures and register information on any abnormalities they discover. At this time, the worker can specify (tap) the location of the actual structure's abnormality displayed on the screen to specify the component where the abnormality is occurring. In addition, for the convenience of subsequent maintenance work, the management category and inside / outside category of the location where the abnormality was discovered are determined and registered together with the information on the abnormality. The processing procedure is shown using the flowchart in Figure 23.

[0091] Step S401: Invisibly superimposed display of model data As in step S301, model data of the structure is invisibly superimposed and displayed on the actual structure that the worker is viewing through optical see-through or video see-through.

[0092] Step S402: Identifying the Point of Gazing As in step S302, the point of gaze is identified.

[0093] Step S403: Identifying a component As in step S303, a component is identified based on the gaze point.

[0094] Step S404: New registration of abnormality The input unit 102 displays a new registration screen of abnormality on the display device. For example, as shown in Fig. 15, when the "New registration of abnormality" button is pressed after the component is identified based on the gaze point, the new registration screen of abnormality can be displayed.

[0095] An example of a new abnormality registration screen is shown in Figure 16. On this screen, the worker can enter comments such as an outline of the abnormality, its details, the estimated cause, and a proposed countermeasure. The photo / video import function can also be activated to register previously taken photos and videos, or to take new photos and videos.

[0096] The model operation unit 113 assigns a unique identifier (deformation ID). This deformation ID is combined with the input text or media file and the current position (viewpoint position) of the terminal device 10 to create deformation data. This deformation data is associated with the identifier (GUID) of the component identified in step S303 and registered as reference data in the reference data management unit 301. Note that other information may be added to the deformation data. For example, the registration and editing date and time of the deformation data may be linked to the deformation ID and recorded.

[0097] It is possible to register multiple pieces of deformation data for one component (GUID). In other words, multiple different deformation IDs can be associated with one GUID. This is possible by registering new deformations multiple times for one component.

[0098] With this function, the terminal device 10 invisibly overlays the model data of the structure on the actual structure, allowing the user to identify the component where the deformation occurred while the deformation can be directly seen. This not only eliminates the need for the worker to switch between the drawing and reality, but also enables the worker to register the deformation in association with the component without being aware of the existence of the model.

[0099] In addition, associating multiple deformation IDs with a component's GUID can provide additional functionality for evaluating the component's health. That is, the number of deformation IDs associated with a component ID is used as an indicator of the component's health. For example, the more deformation IDs associated with a component ID, the more the component's health can be estimated to be declining. Alternatively, the higher the rate of increase in the number of deformation IDs associated with a component ID, the more the component's health can be estimated to be declining. The terminal device 10 can output an alert when the number of deformation IDs, the rate of increase, etc., exceeds a predetermined threshold. Alternatively, information indicating the number of deformation IDs, the rate of increase, etc., can be displayed on the Deformation Reference / Edit / Add Screen (described later, Figure 18), etc.

[0100] <Referring to, editing, and adding to registered anomalies> Workers can refer to, edit, and add to information about anomalies that have been previously registered. At this time, workers can also view information that is useful for inspection work, such as management classification and inside / outside classification. The processing procedure is shown using the flowchart in Figure 24.

[0101] Step S501: Display a list of registered deformation locations The model operation unit 113 accesses the model management unit 201 and acquires model data and reference data for the structure. The display unit 101 displays the acquired model data on the screen and plots the deformation data included in the reference data on the model data. Specifically, as shown in Figure 17, an icon indicating the position of the component for which deformation data is registered or the viewpoint position included in the deformation data is generated and displayed superimposed on the model data.

[0102] Alternatively, in the procedure shown in step S301, the model data may be superimposed on the actual structure in a transparent or semi-transparent state, and an icon indicating the location of the deformation may be superimposed in a visible state (opacity greater than 50%, preferably 100%).

[0103] Step S502: Selection of registered abnormality location The worker selects an icon indicating a registered abnormality on the display device. For example, the worker selects the icon indicating the abnormality by tapping, using a pointing device, or by eye contact. The camera may be started, and the icon superimposed on the direction the camera is pointed (on the optical axis) may be treated as the selected icon.

[0104] The input unit 102 identifies the icon selected by the operator. The model operation unit 113 accesses the model management unit 201 to acquire the deformation data corresponding to the selected icon. The additional data operation unit 114 accesses the additional data management unit 401 to acquire the management classification and / or the inside / outside classification corresponding to the viewpoint position included in the acquired deformation data.

[0105] Step S503: Referencing / Editing Registered Deformation Locations The display unit 101 displays a screen for referring to, editing, and adding deformations on the display device. Fig. 18 shows an example of the screen for referring to, editing, and adding deformations.

[0106] The initial state is the reference screen, which displays text and media files (photos, videos, etc.) contained in previously registered abnormality data. It also displays the management category and internal / external category to which the abnormality belongs.

[0107] When the edit button is pressed on the view / edit / add screen of the input unit 102, the display unit 101 displays the edit screen. On the edit screen, the input unit 102 accepts operations such as editing text, adding and deleting media files, etc. The edited data is saved and overwrites the existing deformation data.

[0108] When the Add button is pressed on the deformation reference / edit / add screen, the display unit 101 displays the Add screen. On the Add screen, the input unit 102 accepts operations such as input of new text data and registration of media files. The model operation unit 113 adds the input data to the model management unit 201 as new deformation data with the same deformation ID as the existing deformation data. This results in multiple deformation data with the same deformation ID being registered under one component (GUID).

[0109] In this way, by allowing multiple entries of deformation data with the same deformation ID, it becomes possible to track changes in a specific deformation over time. For example, in Figure 18, multiple tabs with different dates and times, such as "2022 / 2 / 4," "2021 / 2 / 10," etc., are displayed at the top. This indicates that for this deformation, there are multiple pieces of deformation data registered at different dates and times, in other words, a history of deformation data. By switching between tabs, workers can go back and check the status of past deformations.

[0110] With this function, the terminal device 10 displays registered deformed areas superimposed on the structure model data or the actual structure. It also displays details of the deformed area selected by the worker along with additional data such as the management classification and inside / outside classification. This allows the worker to easily understand the status of the deformed area in detail. It also allows historical data to be managed for each deformed area. This allows the worker to easily and precisely trace changes in the deformed area over time. Furthermore, the desired deformed area can be easily found by utilizing the management classification and inside / outside classification.

[0111] <Detection and Estimation of Deformation> The position and area of ​​the deformation are recognized from a camera image of the deformation, and the resources required to repair the deformation are estimated. The processing procedure is shown using the flowchart in FIG.

[0112] Step S601: Photographing Deformation The deformation detection unit 115 controls the camera to photograph the deformed area.

[0113] Step S602: Identifying a Member The deformation detection unit 115 identifies a member in which a deformation has occurred. For example, by superimposing the model data on the actual structure in the procedure shown in step S301 and then specifying the member in the procedure shown in step S302, the member in which a deformation has occurred can be identified.

[0114] Step S603: Recognizing Deformation The deformation detection unit 115 estimates the area of ​​the deformation portion included in the captured image. An example of the estimation method is shown below.

[0115] The anomaly detection unit 115 divides the entire image into multiple grids and identifies grids where discoloration suggesting an anomaly is observed. Typically, this can be achieved by matching the color of each grid with a color sample to identify grids where the similarity exceeds a threshold, or by calculating the color difference from surrounding grids to identify grids where the similarity exceeds a threshold. In this case, the anomaly detection unit 115 may also identify the type of anomaly.

[0116] The deformation detection unit 115 calculates the ratio of the number of grids recognized as deformed areas to the total number of grids.

[0117] The deformation detection unit 115 estimates the area of ​​the entire area included in the image based on the camera's angle of view (focal length) and the distance to the subject, and estimates the area of ​​the deformed area by multiplying this by the proportion of the grid that is recognized as the deformed area.

[0118] Step S604: Repair Estimate for Deformation The repair estimate unit 116 estimates the resources (materials, funds, manpower, etc.) required to repair the deformed portion of the area recognized in step S503. For example, a repair estimate can be made by multiplying the area recognized in step S503 by the resources (materials, funds, manpower, etc. required to repair the deformation of a unit area) predetermined for each type of deformation.

[0119] Step S605: Save the estimate The model operation unit 113 associates the area of ​​the deformation and the estimated results of the repair estimate with the component and registers them in the reference data management unit 301. Typically, this information is recorded as part of the deformation data.

[0120] The present invention is not limited to the above-described embodiment, and constituent elements can be replaced, added, deleted, disassembled, or combined as appropriate without going against the spirit of the present invention.

[0121] For example, in the above-described embodiment, an example was shown in which the terminal device 10 includes the display unit 101, input unit 102, photographing unit 111, position estimation unit 112, model operation unit 113, additional data operation unit 114, deformation detection unit 115, and repair estimate unit 116, but these functions can also be installed separately in multiple devices. Several variations are shown below. Note that only the differences from the above-described embodiment will be mainly described here, and explanations of commonalities will be omitted as appropriate.

[0122] <Modification 1> In this example, a first worker inspects a structure on-site to detect any abnormalities, and a second worker makes a repair estimate based on the inspection results in an office, etc. (See FIG. 26.) FIG. 26 is a block diagram showing a modification of the information processing system 1.

[0123] The terminal device 10 used by the first worker is typically a tablet computer, smartphone, HMD, etc. The terminal device 10 includes hardware such as a processor, memory, display device, input device, camera, GNSS receiving device, and various sensors for SLAM. In terms of functionality, the terminal device 10 includes a display unit 101, an input unit 102, an imaging unit 111, a position estimation unit 112, a model operation unit 113, an additional data operation unit 114, and a deformation detection unit 115.

[0124] The second terminal device 50 used by the second worker is typically a tablet computer, a smartphone, a PC (Personal Computer), etc. The second terminal device 50 includes hardware such as a processor, a memory, a display device, an input device, etc. In terms of functionality, the second terminal device 50 includes a repair estimate unit 116.

[0125] The deformation detection unit 115 of the terminal device 10 estimates the area and / or type of the deformation location based on the captured image and stores the estimation result in a storage area (not shown). The storage area may be, for example, an online server, cloud storage, memory in the terminal device 10, or removable storage media.

[0126] The repair estimate unit 116 of the second terminal device 50 obtains the estimated results of the area and / or type of the deformed area from the above-mentioned memory area and estimates the resources (materials, funds, manpower, etc.) required to repair the deformed area.

[0127] <Modification 2> In this example, an image of a real structure is taken by a camera mounted on a mobile object such as a drone or a vehicle, and an operator receives the image at a remote location to perform an inspection (see FIG. 27). FIG. 27 is a block diagram showing a modification of the information processing system 1.

[0128] The terminal device 10 mounted on the mobile object is typically a tablet computer, a smartphone, an HMD, etc. The terminal device 10 includes hardware such as a processor, a memory, a display device, an input device, a camera, a GNSS receiving device, and various sensors for SLAM. In terms of functionality, the terminal device 10 includes an imaging unit 111 and a position estimation unit 112.

[0129] The second terminal device 50 used by the worker is typically a tablet computer, a smartphone, a PC, etc. The second terminal device 50 includes hardware such as a processor, a memory, a display device, an input device, etc. In terms of functionality, it includes a display unit 101, an input unit 102, a model operation unit 113, an additional data operation unit 114, a deformation detection unit 115, and a repair estimate unit 116.

[0130] The image capturing unit 111 of the terminal device 10 captures an image of a real structure. At that time, the position estimating unit 112 acquires position information and attitude information (yaw, pitch, roll) at the time of capturing the image. The terminal device 10 transmits the captured image, position information, and attitude information to the second terminal device 50.

[0131] The second terminal device 50 corrects its own position (steps S1 to S5) using the captured image, position information, and attitude information received from the terminal device 10. This makes it possible to superimpose model data of the structure on the image captured by the moving object. In displaying the current position (steps S101 to S102), the current position of the moving object can be displayed. In displaying component information (steps S301 to S304), component information within the captured image can be obtained by calculating the viewpoint and focus point based on the position information and attitude information of the moving object.

[0132] <Modification 3> In this example, an image of an actual structure is taken by a 360-degree panoramic camera installed on-site, and an operator receives the image at a remote location to carry out an inspection (see FIG. 28). FIG. 28 is a block diagram showing a modification of the information processing system 1.

[0133] The panoramic camera functions as the photographing unit 111. The panoramic camera is set so that photographed images can be acquired on the terminal device 10 side.

[0134] The terminal device 10 is typically a tablet computer, a smartphone, an HMD, a PC, etc. The terminal device 10 includes hardware such as a processor, a memory, a display device, an input device, etc. In terms of functionality, the terminal device 10 includes a display unit 101, an input unit 102, a position estimation unit 112, a model operation unit 113, an additional data operation unit 114, a deformation detection unit 115, and a repair estimate unit 116.

[0135] As a preliminary step, the coordinates of the installation position of the panoramic camera are measured. Also, the direction is measured using a gyrocompass or the like mounted on the panoramic camera. The measured position and direction are recorded in a storage area (not shown). For example, an installation point can be added to the model data.

[0136] The terminal device 10 can realize the same functions as those of the above-described embodiment by using the captured image received from the panoramic camera and the position and angle measured in advance. For example, in displaying component information (steps S301 to S304), the component information in the captured image can be acquired by using the pre-measured position of the panoramic camera as the viewpoint and specifying the point of interest based on a point indicated by the worker by tapping or the like on the captured image displayed on the display device.

[0137] <Modification 4> This is an example in which a first worker records images (photographs or videos) of an actual structure on-site, and a second worker uses the recorded images to carry out an inspection in an office, etc. (See FIG. 27 ). FIG. 27 is a block diagram showing a modification of the information processing system 1.

[0138] The terminal device 10 used by the first worker is typically a tablet computer, a smartphone, an HMD, etc. The terminal device 10 includes hardware such as a processor, a memory, a display device, an input device, a camera, a GNSS receiving device, and various sensors for SLAM. In terms of functionality, the terminal device 10 includes an imaging unit 111 and a position estimation unit 112.

[0139] The second terminal device 50 used by the second worker is typically a tablet computer, a smartphone, a PC, etc. The second terminal device 50 includes hardware such as a processor, a memory, a display device, an input device, etc. In terms of functionality, the second terminal device 50 includes a display unit 101, an input unit 102, a model operation unit 113, an additional data operation unit 114, a deformation detection unit 115, and a repair estimate unit 116.

[0140] The image capturing unit 111 of the terminal device 10 captures an image of a real structure. At that time, the position estimating unit 112 acquires position information and attitude information (yaw, pitch, roll) at the time of capturing the image. The terminal device 10 records the captured image, position information, and attitude information in association with the image (or each frame in the case of a video).

[0141] The second terminal device 50 corrects its own position (steps S1 to S5) using the captured image, position information, and attitude information acquired by the terminal device 10. This makes it possible to superimpose model data of the structure on the image captured by the moving object. In displaying the current position (steps S101 to S102), the current position of the moving object can be displayed. In displaying component information (steps S301 to S304), component information within the captured image can be acquired by calculating the viewpoint and focus point based on the position information and attitude information of the moving object.

[0142] The first worker and the second worker may be the same person. In this case, the functions of the second terminal device 50 may be integrated into the terminal device 10.

[0143] Each processing means constituting the present invention may be configured by hardware, or any process may be realized by having a CPU execute a computer program. Furthermore, the computer program may be stored and supplied to a computer using various types of temporary or non-temporary computer-readable media. A temporary computer-readable medium includes, for example, an electromagnetic signal supplied to a computer by wire or wirelessly.

[0144] REFERENCE SIGNS LIST 1 Information processing system 10 Terminal device 20 Model management device 30 Reference data management device 40 Additional data management device 50 Second terminal device 101 Display unit 102 Input unit 111 Photography unit 112 Position estimation unit 113 Model operation unit 114 Additional data operation unit 115 Deformation detection unit 116 Repair estimate unit

Claims

1. An information processing system including a model management device and a terminal device, wherein the model management device stores model data of a structure consisting of a plurality of components, and the terminal device estimates its own position on the actual structure and corrects the own position using the model data.

2. An information processing system as described in claim 1, further comprising an additional data management device, wherein the additional data management device defines a predetermined area set overlaid on the structure as a management division, the model management device stores information on deformations that have occurred in the components together with the position of the deformation identified based on the self-location, and when the terminal device reads the information on the deformation from the model management device, it identifies the management division to which the deformation belongs and displays the identified management division.

3. An information processing system as described in claim 1, further comprising an additional data management device, wherein the additional data management device defines a predetermined area set overlaid on the structure as an inspection range section, and the terminal device reads out only the components related to the inspection range section from the model management device and displays the read-out components.

4. The information processing system according to claim 1, wherein the terminal device takes an image of the actual structure and estimates the area of ​​the deformation contained in the image.

5. The information processing system according to claim 4, wherein the terminal device estimates resources required to repair the abnormality.

6. The information processing system according to claim 1, wherein the terminal device allows a worker to visually recognize the actual structure, invisibly superimposes the model data on the actual structure, accepts a position on the actual structure specified by the worker, and identifies the component corresponding to the specified position.

7. The information processing system according to claim 6, wherein the model management device stores information about the deformation that has occurred in the component in association with the identified component.

8. The information processing system according to claim 7, wherein the model management device stores a plurality of pieces of information acquired at different times regarding a specific deformation.

9. The information processing system according to claim 1, wherein the terminal device displays the position of the terminal device superimposed on the model data displayed in a visible state.

10. A terminal device that estimates its own position on a real structure and corrects said position using model data of said structure.

11. An information processing method comprising: a step in which a terminal device estimates its own position on a real structure; and a step in which the terminal device corrects the own position using model data of the structure.

12. A program for causing a computer to execute the information processing method according to claim 11.

Images