Information processing system, information processing method, and program

Abstract

To provide an information processing system, an information processing method, and a program suitable for inspection work of a structure.SOLUTION: An information processor includes a model management device and a terminal, the model management device stores a model datum of a structure composed of a plurality of members, and the terminal estimates a self-position on the actual structure and corrects the self-position using the model datum.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program suitable for maintenance and management work of structures. [Background technology]

[0002] In the past, in maintenance work (hereinafter simply referred to as inspections) such as the inspection of bridges and other structures, workers would record, refer to, and edit the location and nature of any abnormalities that had occurred in the structure using paper drawings and forms. However, in large-scale structures, particularly those with repeated similar local structures such as long-span bridges, it becomes difficult for workers to identify their own location or the location of any abnormalities while referring to paper drawings and forms, increasing the possibility of misidentifying the location.

[0003] One method to solve this problem is to use the Global Navigation Satellite System (GNSS) to identify the location of workers and abnormalities.

[0004] For example, Patent Document 1 describes a system used for inspecting structures, in which an HMD (Head Mounted Display) worn by a worker acquires the worker's current position using GNSS, recognizes various instructions given while visually inspecting the structure, converts the current position to a corresponding location on a drawing, and records the various instructions on the drawing. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7000513 Summary of the Invention [Problem to be solved by the invention]

[0006] However, GNSS has the problem of being unable to pinpoint locations in places where satellite communication is not possible. For example, this problem is likely to occur inside or behind large structures such as long-span bridges. Therefore, there is a need for a system that can pinpoint the locations of workers and abnormalities with high accuracy even in places where satellite communication is difficult, thereby enabling highly efficient and high-quality inspection and maintenance of structures. [Means for solving the problem]

[0007] According to one embodiment, the information processing system includes a model management device and a terminal device, wherein the model management device stores model data of a structure consisting of multiple components, and the terminal device estimates its own position on the actual structure and corrects the own position using the model data. According to one embodiment, the information processing system further includes an additional data management device, which defines a predetermined area set overlaid on the structure as a management category, and the model management device stores information on deformations that have occurred in the components together with the position of the deformation identified based on the terminal device's own position, and when the terminal device reads the information on the deformation from the model management device, it identifies the management category to which the deformation belongs and displays the identified management category. According to one embodiment, the information processing system further includes an additional data management device, which defines a predetermined area set overlaid on the structure as an inspection range section, and the terminal device reads out only the components related to the inspection range section from the model management device and displays the read-out components. According to one embodiment, the terminal device captures an image of the actual structure and estimates the area of ​​the deformation included in the captured image. According to one embodiment, the terminal device estimates resources required to repair the abnormality. According to one embodiment, the terminal device allows the worker to visually view the real structure, invisibly overlays the model data on the real structure, accepts the worker's specification of a position on the real structure, and identifies the component corresponding to the specified position. According to one embodiment, the model management device stores information about the deformation that has occurred in the component in association with the identified component. According to one embodiment, the model management device stores a plurality of pieces of information acquired at different times regarding a particular anomaly. According to one embodiment, the terminal device displays the position of the terminal device superimposed on the model data displayed in a visible state. According to one embodiment, the terminal device estimates its own position on a real structure and corrects the own position using model data of the structure. According to one embodiment, an information processing method includes a step in which a terminal device estimates its own position on a real structure, and a step in which the terminal device corrects the own position using model data of the structure. According to one embodiment, a program causes a computer to execute the information processing method. [Effects of the Invention]

[0008] The present invention can provide an information processing system, an information processing method, and a program suitable for maintenance and management work of structures. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a hardware configuration of an information processing system 1. FIG. [Figure 2] 2 is a block diagram showing the functional configuration of the terminal device 10. FIG. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a model management device 20. [Figure 4] FIG. 1 is a diagram illustrating an example of a data structure of a BIM / CIM model. [Figure 5]FIG. 2 is a block diagram showing the functional configuration of a reference data management device 30. [Figure 6] FIG. 2 is a diagram illustrating an example of a data structure of reference data. [Figure 7] FIG. 2 is a block diagram showing the functional configuration of an additional data management device 40. [Figure 8] FIG. 10 is a diagram illustrating an example of a data structure of additional data. [Figure 9] FIG. 10 is a diagram illustrating management classification. [Figure 10] FIG. 10 is a diagram illustrating a self-position correction function. [Figure 11] FIG. 10 is a diagram illustrating a self-position correction function. [Figure 12] FIG. 10 is a diagram illustrating a self-position correction function. [Figure 13] FIG. 10 is a diagram illustrating a self-position correction function. [Figure 14] FIG. 10 is a diagram showing an example of a display screen of the self-position. [Figure 15] FIG. 10 is a diagram showing an example of a display screen for member information. [Figure 16] FIG. 10 is a diagram showing an example of a new registration screen for a defect. [Figure 17] FIG. 10 is a diagram showing an example of a list display screen of deformed portions. [Figure 18] FIG. 10 is a diagram showing an example of a screen for viewing, editing, and adding deformations. [Figure 19] FIG. 2 is a block diagram showing the operation of the terminal device 10. [Figure 20] FIG. 2 is a block diagram showing the operation of the terminal device 10. [Figure 21] FIG. 2 is a block diagram showing the operation of the terminal device 10. [Figure 22] FIG. 2 is a block diagram showing the operation of the terminal device 10. [Figure 23] FIG. 2 is a block diagram showing the operation of the terminal device 10. [Figure 24] FIG. 2 is a block diagram showing the operation of the terminal device 10. [Figure 25] FIG. 2 is a block diagram showing the operation of the terminal device 10. [Figure 26]FIG. 10 is a block diagram showing a modified example of the information processing system 1. [Figure 27] FIG. 10 is a block diagram showing a modified example of the information processing system 1. [Figure 28] FIG. 10 is a block diagram showing a modified example of the information processing system 1. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1 is a diagram showing a hardware configuration of an information processing system 1 according to an embodiment of the present invention. The information processing system 1 includes a terminal device 10, a model management device 20, a reference data management device 30, and an additional data management device 40. The terminal device 10 is communicably connected to the model management device 20 and the additional data management device 40. The model management device 20 is communicably connected to the reference data management device 30.

[0011] The terminal device 10, the model management device 20, the reference data management device 30, and the additional data management device 40 are all information processing devices having a processor, a memory, a communication device, an input / output device, etc. The processor reads and executes a program stored in the memory, thereby logically realizing each of the functional units described below.

[0012] The terminal device 10 is an information processing device used by a worker, and is typically a tablet computer, a smartphone, an HMD (e.g., Hololens, etc.), etc. In addition to the above-mentioned processor, memory, etc., the terminal device 10 may include hardware such as a display device that outputs information to the worker, an input device (touch panel, eye-gaze input device, pointing device, etc.) that accepts information input by the worker, a camera for capturing images, a GNSS receiving device, and various sensors for SLAM (Simultaneous Localization and Mapping) (gyro, acceleration sensor, depth sensor, ToF sensor, LiDAR, etc.).

[0013] The model management device 20, the reference data management device 30, and the additional data management device 40 are typically server computers. The model management device 20, the reference data management device 30, and the additional data management device 40 may be realized in a virtual information processing environment such as cloud computing. Alternatively, they may be integrated with the terminal device 10.

[0014] 2 is a block diagram showing the functional configuration of the terminal device 10. The terminal device 10 includes a display unit 101, an input unit 102, an image capturing unit 111, a position estimation unit 112, a model operation unit 113, an additional data operation unit 114, a deformation detection unit 115, and a repair estimate unit 116.

[0015] The display unit 101 performs processing to display images on a display device (display). For example, the display unit 101 can display a three-dimensional model of a structure, a still image showing the inspection result of the structure, a video, character data, etc. on the display device so that the worker can visually confirm them.

[0016] If the display device is a transmission type, the worker can point the display device at the real structure and directly see the real structure that is visible behind the display device (optical see-through). In this state, the display unit 101 can display desired information on the display device, superimposing the information on the real structure.

[0017] If the display device is a non-transmissive type, the display device can display in real time (video see-through) an image of the real structure being captured by the image capturing unit 111 (described later). In this state, the display unit 101 can superimpose desired information onto the video see-through image and display it on the display device, thereby superimposing the information on the real structure.

[0018] The input unit 102 accepts information input by the worker. Typically, input can be made using a touch panel, eye contact, a pointing device, a keyboard, or voice input. However, the input method is not limited to these and any input method can be used.

[0019] For example, when the display unit 101 displays a model of a structure, the input unit 102 can recognize the coordinates (model coordinates) corresponding to the point specified and input by the worker. It can also identify the component present at the coordinates and recognize that the component has been selected.

[0020] In addition, the input unit 102 can accept as input images (still images or videos) captured by the worker using the photographing unit 111 (described below), text information entered via keyboard or voice, graphic information drawn using a touch panel or pointing device, etc.

[0021] The photographing unit 111 controls the camera to photograph the actual structure, deformation, etc., and outputs the image (still image or video).

[0022] The position estimation unit 112 receives positioning signals from GNSS satellites using a GNSS receiver and can calculate the coordinates (latitude, longitude, and altitude) of the reception point.

[0023] Additionally, the position estimation unit 112 mainly uses camera images to recognize (mapping) the three-dimensional shape of the real space and estimate the position and orientation of the terminal device 10 (self-position estimation) (SLAM). SLAM is a well-known technology, so a description of the specific SLAM algorithm will be omitted in this article. Note that the terminal device 10 can also improve the accuracy of SLAM by using information obtained from various sensors, such as a gyro sensor that detects rotation (angular velocity), an acceleration sensor that detects tilt, a depth sensor that detects the distance to an object in the real space, a ToF (Time of Flight) sensor, and a LiDAR (Light Detection and Ranging).

[0024] By performing self-location estimation using SLAM, the position estimation unit 112 can determine the coordinates of the terminal device 10 even when it cannot receive GNSS signals, for example, when it is inside or behind a large-scale structure such as a long bridge.

[0025] The position estimation unit 112 can also detect the position of the terminal device 10 by using a known technique such as self-position estimation using passive RFID.

[0026] Incidentally, structures can be deformed by the effects of temperature changes, wind, waves, ground movement, and so on. These effects are particularly noticeable in large-scale structures such as long-span bridges. For example, as shown in Figure 10, in a girder bridge, the length of the bridge girders differs between the hot summer months and the cold winter months. In addition, in a suspension bridge, the shape (height) of the bridge girders differs between the hot summer months and the cold winter months.

[0027] In an ideal situation where a structure is not deformed, if the self-position estimated by GNSS or SLAM is superimposed on a 3D model of the structure, the self-position on the structure can be plotted accurately (assuming that the coordinate systems of the self-position and the 3D model of the structure match). However, if the self-position estimated by GNSS or SLAM is simply superimposed on the 3D model of the structure when the structure is deformed, the self-position will be plotted in an incorrect position on the structure. This is because the shape of the actual structure differs from the 3D model.

[0028] To address this problem, in this embodiment, the position estimation unit 112 has a function of correcting its own position. The processing procedure will be described with reference to the flowchart in FIG.

[0029] S1: Linking structure model data with latitude and longitude information As a preparation step, two arbitrary points on the actual structure are selected as measurement points, and the latitude and longitude of the measurement points are measured. It is preferable that the measurement points are in locations that are less susceptible to temperature changes, etc. (See Figure 11. The two-pointed triangle displayed on the support indicates the measurement points.)

[0030] S2: Self-location measurement At any two measurement points (not necessarily S1) on a real structure, the position estimation unit 112 estimates its own position and calculates the latitude and longitude (see FIG. 12. Two tablet terminals indicate the measurement points).

[0031] S3: Overlay display of structures The longitude and latitude information measured in S1 and S2 enables conversion between the coordinate system recognized by the position estimation unit 112 and the coordinate system of the model data. The position estimation unit 112 converts the model data into the coordinate system recognized by the position estimation unit 112. The display unit 101 displays this model data on the screen.

[0032] Furthermore, the display unit 101 displays the real structure on the display device by optical see-through or video see-through, whereby the real structure and the model data of the structure are displayed superimposed on the display device.

[0033] S4: Acquisition of correction amount In an ideal state where the structure is not deformed, the actual structure and the model data of the structure will appear to overlap perfectly. On the other hand, when the structure is deformed, the two will be observed to be misaligned. If a misalignment occurs, the system will switch to calibration mode.

[0034] In the calibration mode, the input unit 102 accepts user inputs such as tapping, dragging, and pinching on the screen. In response to the user input, the model data of the structure is freely moved or rotated on the screen (see FIG. 13). When the actual structure and the model data of the structure appear to be perfectly superimposed on each other, the calibration mode is terminated, for example, by an instruction from the operator.

[0035] The position estimation unit 112 compares the display position and angle of the model data when it is displayed in S3 with the display position and angle of the model data when the calibration mode ends, and calculates the difference as the amount of correction.

[0036] S5: Self-position correction The position estimation unit 112 measures the current self-position, adds the correction amount calculated in S4, and converts the self-position after correction into the coordinate system of the model data.

[0037] This makes it possible to superimpose the self-position estimated by GNSS or SLAM on the correct position on the 3D model of the structure even when the structure is deformed. Furthermore, even when recording the position of a deformation, for example, by correcting the viewpoint position at the time of the deformation using the above-mentioned procedure and then recording it, it becomes possible to record the deformation in association with the correct position on the 3D model of the structure. Even if the shape of the structure changes each time due to temperature changes, etc., inspection information can always be recorded linked to the correct position on the 3D model of the structure.

[0038] The model operation unit 113 accesses the model management device 20 (described later) and performs processing to operate the BIM / CIM model managed by the model management unit 201 (described later). That is, it reads the BIM / CIM model, and reads, registers, and edits attribute data and reference data. The BIM / CIM model, attribute data, and reference data will be described later.

[0039] The additional data operation unit 114 accesses an additional data management device 40 (described later) and reads, registers, and edits the additional data managed by an additional data management unit 401. The additional data will be described later.

[0040] The deformation detection unit 115 analyzes camera images of the actual structure and detects deformations that have occurred in the structure. A specific deformation detection method will be described later.

[0041] The repair estimate unit 116 performs a repair estimate (calculation of necessary materials, man-hours, costs, etc.) for the deformation detected by the deformation detection unit 115. A specific repair estimate method will be described later.

[0042] 3 is a block diagram showing the functional configuration of the model management device 20. The model management device 20 includes a model management unit 201.

[0043] The model management unit 201 stores BIM / CIM models. A BIM / CIM (Building Information Modeling / Construction Information Modeling / Management) model is an information model that combines attribute data and reference data with a 3D model that represents a structure in a three-dimensional solid shape. The 3D model of the structure is shared throughout a series of processes in construction and civil engineering projects, such as planning, design, construction, and maintenance, and the information required for each process is managed in association with this 3D model. The information model formed through such efforts is the BIM / CIM model. Utilizing the BIM / CIM model is expected to improve quality and productivity throughout the entire project.

[0044] An example of the data structure of a BIM / CIM model handled in this embodiment is shown in Fig. 4. The BIM / CIM model includes model data and attribute data.

[0045] Model data is a 3D model of a structure. This 3D model includes one or more component data. The component data includes the 3D shape, coordinates, and unique key (GUID) of the component.

[0046] The attribute data is additional data associated with each component, and typically the name of the component is registered. In this case, the attribute data includes the component name and a unique key (GUID).

[0047] 5 is a block diagram showing the functional configuration of the reference data managing device 30. The reference data managing device 30 includes a reference data managing unit 301.

[0048] The reference data management unit 301 stores reference data. The reference data is also incidental data associated with each component, but differs from attribute data in that it is placed outside the BIM / CIM model.

[0049] 6 shows an example of the data structure of reference data handled in this embodiment. The reference data management unit 301 can store any text data or media files such as photos, videos, and audio as reference data. The reference data also includes a unique key (GUID) that indicates the component to which the data is linked.

[0050] Typically, the production classification is registered as reference data. The production classification is an identifier that is uniformly assigned to multiple components that form a single semantic entity, mainly during the design and production stages. For example, a bridge (structure) is made up of multiple components, but in reality, all components are not assembled at once; rather, substructures are produced in multiple phases and then assembled. Therefore, a unique production classification is assigned to each component for each production phase, such as "NW21" for all components that make up a substructure produced in one production phase and "NW22" for all components that make up a substructure produced in another production phase. This facilitates the production process. The production classification is registered in the reference data for each component.

[0051] In addition, information (deformation data) regarding deformations that have occurred in components is registered as reference data. The deformation data may include a deformation ID, text data, a media file, and a viewpoint position. The deformation ID is a unique identifier assigned when a new deformation is registered. The text data includes, for example, comments registered by workers. The media file includes, for example, photos and videos taken during inspection, and audio input by workers as comments. The viewpoint position is the position of the terminal device 10 when these data were generated or input.

[0052] In this way, each component of the three-dimensional model, the attribute data, and the reference data are related to each other via a unique key (GUID).

[0053] 7 is a block diagram showing the functional configuration of the additional data management device 40. The additional data management device 40 includes an additional data management unit 401.

[0054] Fig. 8 shows an example of the data structure of additional data handled in this embodiment. The additional data management unit 401 stores additional data. The additional data includes information used mainly for the convenience of inspection work. In this embodiment, the inspection range classification, management classification, and inside / outside classification are registered as additional data.

[0055] Inspection range classification: From the perspective of work efficiency, it is common to narrow down inspection work to similar content and perform it in batches, such as only the deck, only the inside of the box girder, or only the diaphragm. In this case, loading the BIM / CIM data for the entire bridge and then attempting to access the attribute data and reference data for the inspection target area can increase processing load and cause problems such as reduced response. To prevent this, it is possible to control the loading of only the BIM / CIM data that is currently required. For example, only the BIM / CIM data for the substructure planned for inspection (deck, inside of the box girder, diaphragm, etc.) can be loaded. In addition, it is also possible to load BIM / CIM data for parts that are likely to be used simultaneously. For example, since deformation may occur across multiple components, it is possible to load BIM / CIM data for parts adjacent to the planned inspection area. This reduces processing load and improves response. Specifically, a common inspection range division is assigned to the zones where inspection work is to be performed together. For example, assign inspection range classifications such as "deck slab" to the deck slab zone, "inside box girder" to the box girder interior zone, and "diaphragm" to the diaphragm zone. By determining the targets that should be inspected together in accordance with the inspection plan and defining that inspection plan as the inspection range classification, it becomes possible to carry out inspection work efficiently.

[0056] In this embodiment, inspection range divisions are defined by assigning unique identifiers (names) to predefined fixed spaces (zones). A zone can typically be defined as a three-dimensional object. That is, a zone can be defined by defining the coordinates of the vertices of a three-dimensional shape.

[0057] The BIM / CIM data of which components should be loaded can be determined by determining whether the zone with the desired inspection range classification overlaps with the coordinates of the component. For example, if you want to inspect only the deck, you can determine whether the zone with the inspection range classification of "deck" overlaps with the components that make up the structure, and then load only the BIM / CIM data of the components that are determined to overlap.

[0058] Management classification: To identify inspection locations, it is useful to define a classification specialized for inspection work for a structure. For example, as shown in Figure 9, a structure may be composed of multiple box girders, and one manufacturing classification may be defined for each group of box girders. While it is possible to directly use this manufacturing classification for inspection work, some ingenuity is required to identify which box girders are the inspection locations, making it less user-friendly. For inspection work, it is more convenient to define classifications for each individual box girder included in the manufacturing classification. Therefore, this embodiment introduces a new concept of management classification specialized for inspection work. For example, by assigning different management classifications to each individual box girder in Figure 9, inspection work can be carried out more efficiently. For example, if the manufacturing classification "NW22" is composed of six box girders, management classifications such as "NW22-1," "NW22-2," ..., "NW22-6" can be assigned to the zones containing these box girders.

[0059] In this embodiment, management divisions are defined by assigning unique identifiers (names) to predefined spaces (zones). In the example of FIG. 9, a structure is divided into multiple zones, each defined as a different management division. A zone can typically be defined as a three-dimensional object. That is, a zone can be defined by defining the coordinates of the vertices of a three-dimensional shape.

[0060] For example, if you want to know which management category a worker's current location falls into, you can determine whether the worker's current location overlaps with several zones assigned with management categories. The management category of the zone that is determined to overlap corresponds to the worker's current location.

[0061] Internal / external classification: During inspection work, there are times when it is necessary to identify whether a worker is inside or outside a structure. For example, when registering the inspection results of a box girder, there are cases where it is necessary to manage whether the abnormality was found on the outside or inside of the box girder. In this regard, BIM / CIM models do not have the data to distinguish between inside and outside. Therefore, a zone for determining inside / outside is defined in the space inside the box girder, and this zone is assigned an inside / outside classification indicating "inside."

[0062] In this embodiment, an inside / outside distinction is assigned to a predefined fixed space (zone). A zone can typically be defined as a three-dimensional object. That is, a zone can be defined by defining the coordinates of the vertices of a three-dimensional shape.

[0063] For example, if you want to identify whether the location of a defect is inside or outside the box girder, you can determine whether the zone assigned the inside / outside classification overlaps with the position (viewpoint position) of the worker, i.e., the terminal device 10, when the defect was discovered. If it is determined that there is an overlap, the defect was discovered inside the box girder. On the other hand, if it is determined that there is no overlap, the defect was discovered outside the box girder.

[0064] Several use cases of the operation of the information processing system 1 will be described.

[0065] <Displaying the worker's current location> The terminal device 10 can display on the screen the current position of the worker in the structure. The processing procedure will be described using the flowchart of FIG.

[0066] S101: Estimation of self-position, correction, and coordinate transformation Through the above-described steps S1 to S5, the position estimation unit 112 estimates the self-position of the terminal device 10, applies a predetermined correction to this, and then converts it into the coordinate system of the model data of the structure.

[0067] S102: Overlay display The model operation unit 113 accesses the model management unit 201 and acquires the model data of the structure. The display unit 101 displays the acquired model data on the screen. At the same time, the self-position converted in S101 is superimposed on the model data of the structure. For example, an icon can be displayed at the self-position as shown in FIG. 14.

[0068] This function allows the terminal device 10 to clearly show where a worker is currently standing within a long structure such as a continuous truss bridge at a glance, thereby reducing the time and effort that would previously have been spent identifying the worker's own position and the location of any abnormalities on a paper drawing, and preventing misidentification.

[0069] <Load only the necessary BIM / CIM data> When a worker inspects a specific inspection target according to an inspection plan, he or she can selectively load only the BIM / CIM data related to the inspection target. The processing procedure is shown using the flowchart in Figure 21.

[0070] S201: Selection of inspection range category The additional data operation unit 114 accesses the additional data management unit 401 and acquires the registered inspection range classifications. The display unit 101 displays a list of the acquired inspection range classifications on the screen. For example, inspection range classifications such as "deck slab," "inside box girder," and "diaphragm" are displayed on the screen.

[0071] The input unit 102 accepts a user input for selecting an inspection range category.

[0072] S202: Loading BIM / CIM data for selected inspection management category The additional data operation unit 114 accesses the additional data management unit 401 and acquires the zone associated with the inspection range segment selected in S201.

[0073] The model operation unit 113 accesses the model management unit 201 and extracts the BIM / CIM data of the components that overlap with the acquired zone.

[0074] The display unit 101 reads the model data of the extracted members and displays them on the screen.

[0075] S203: Additional loading of BIM / CIM data In addition to the inspection range section selected by the user in S201, BIM / CIM data for zones that are likely to be used at the same time may be pre-loaded. For example, since deformation may occur across multiple components, BIM / CIM data for zones adjacent to the inspection range section selected by the user can be loaded.

[0076] In this case, the additional data operation unit 114 accesses the additional data management unit 401 and additionally acquires zones that are likely to be used simultaneously with the inspection range segment selected by the user.

[0077] The model operation unit 113 accesses the model management unit 201 and extracts the BIM / CIM data of the members that overlap the additionally acquired zone.

[0078] The display unit 101 reads the model data of the extracted members and displays them on the screen.

[0079] With this function, the terminal device 10 reads only the BIM / CIM data of components related to the zones defined in the inspection plan, thereby reducing the processing load on the system and enabling inspection work to be carried out efficiently.

[0080] <Displaying component information> When the worker observes the actual structure through the terminal device 10, information on the members that are visible within the field of view is displayed on the screen. The processing procedure will be explained using the flowchart of FIG.

[0081] S301: Overlay display of model data in an invisible state The display unit 101 makes the real structure visible on the display device by optical see-through or video see-through.

[0082] The model operation unit 113 accesses the model management unit 201 and acquires model data of the structure. The display unit 101 displays the acquired model data on the display device. At this time, the coordinate system of the model data of the structure is converted into the real coordinate system recognized by the terminal device 10, and the model data is projected onto the display device taking into account the correction amount calculated in step S4 above. As a result, the model data projected on the display device is perfectly superimposed on the real structure visible on the display device by optical see-through or video see-through.

[0083] At this time, the display unit 101 displays the model data of the structure in an invisible state. In other words, the model data is displayed with 0% opacity. That is, the model data is superimposed on the actual structure as an internal process, but this fact is not noticeable to the worker.

[0084] It is preferable that the projection of the model data is invisible (transparent), but it does not necessarily have to be completely transparent. As long as the worker can see the deformation of the actual structure, it is acceptable to overlay the data in a semi-transparent state with low opacity (around 50% or less), for example.

[0085] S302: Identifying the gaze point The display unit 101 extends the normal of the display device in the case of optical see-through, or the optical axis of the camera in the case of video see-through, and identifies the point (point of gaze) that collides with the model data that is invisibly superimposed on the real structure.

[0086] Alternatively, when a worker points to a component included in the real structure on the display device by, for example, tapping, using a pointing device, or by eye-gaze input, the input point is projected along the normal line of the display device or the optical axis direction of the camera, and the point of collision (point of gaze) with the model data invisibly superimposed on the real structure is identified. In this case, the worker can select a component included in the model data by directly specifying that exact point while visually recognizing the component of the real structure through optical see-through or video see-through.

[0087] S303: Identification of components The model operation unit 113 accesses the model management unit 201 and extracts components that overlap with the gaze point identified in S302. If a component cannot be uniquely identified, multiple components existing around the projection point may be listed and the worker may identify the component. If the worker does not identify the component, the component closest to the gaze point may be treated as the identified component.

[0088] S304: Acquisition of component information The model operation unit 113 accesses the model management unit 201 and acquires model data and / or reference data of the structure. The display unit 101 displays the acquired information on the display device. An example of a screen is shown in FIG. 15. In this example, a photo taken by a camera is displayed in the center of the screen, and the names of components, coordinates of the viewpoint position (current position of the terminal device 10), etc. are displayed at the top of the screen. This name is determined by the display unit 101 identifying the component reflected at the point of interest, and the model operation unit 113 acquiring attribute data of the component.

[0089] With this function, the terminal device 10 invisibly overlays the model data of the structure on the real structure, and identifies the component that the worker is looking at while the real structure is directly visible. Therefore, the worker does not need to switch his / her attention between the drawing and the reality, and can refer to the information on the desired component without being aware of the existence of the model.

[0090] <Registering inspection results> Workers inspect structures and register information about any abnormalities they find. At this time, workers can specify (tap) the location of the abnormality in the actual structure displayed on the screen to specify the component where the abnormality is occurring. In addition, to facilitate future maintenance work, the management category and inside / outside category of the location where the abnormality was found are determined and registered together with the information about the abnormality. The processing procedure is shown using the flowchart in Figure 23.

[0091] S401: Overlay display of model data in an invisible state As in S301, the model data of the structure is invisibly superimposed on the actual structure that the worker is viewing through an optical see-through or video see-through.

[0092] S402: Identifying the point of gaze As in S302, the gaze point is identified.

[0093] S403: Identification of components As in S303, the component is identified based on the gaze point.

[0094] S404: New registration of abnormality The input unit 102 displays a new deformation registration screen on the display device. For example, as shown in Fig. 15, when the "New Deformation Registration" button is pressed after a component has been identified based on the gaze point, the new deformation registration screen can be displayed.

[0095] Figure 16 shows an example of the new abnormality registration screen. On this screen, workers can enter comments such as an outline of the abnormality, its details, the estimated cause, and countermeasures (proposed). They can also activate the photo / video import function to register previously taken photos and videos, or take new photos and videos.

[0096] The model operation unit 113 assigns a unique identifier (deformation ID). Deformation data is created by combining this deformation ID with the input text or media file and the current position (viewpoint position) of the terminal device 10. This deformation data is associated with the identifier (GUID) of the component identified in S303 and registered as reference data in the reference data management unit 301. Note that other information may be added to the deformation data. For example, the registration and editing date and time of the deformation data may be linked to the deformation ID and recorded.

[0097] It is possible to register multiple pieces of deformation data for one component (GUID). In other words, multiple different deformation IDs can be associated with one GUID. This is possible by registering new deformations multiple times for one component.

[0098] With this function, the terminal device 10 invisibly overlays the model data of the structure on the actual structure, allowing the user to identify the component where the deformation occurred while the deformation can be directly seen. This not only eliminates the need for the worker to switch between the drawing and reality, but also enables the worker to register the deformation in association with the component without being aware of the existence of the model.

[0099] In addition, by associating multiple deformation IDs with a component GUID, it is possible to provide additional functionality for evaluating the component's health. That is, the number of deformation IDs associated with a component ID is used as an indicator of the component's health. For example, the more deformation IDs associated with a component ID, the more the component's health can be estimated to be deteriorating. Alternatively, the higher the rate of increase in the number of deformation IDs associated with a component ID, the more the component's health can be estimated to be deteriorating. The terminal device 10 can output an alert when the number of deformation IDs, the rate of increase, etc., exceeds a predetermined threshold. Alternatively, information indicating the number of deformation IDs, the rate of increase, etc., can be displayed on the Deformation Reference, Edit, and Add screen (described later, Figure 18), etc.

[0100] <View, edit, and add registered changes> Workers can refer to, edit, and add information about abnormalities that have been registered in the past. At this time, workers can also view information that is useful for inspection work, such as management classification and inside / outside classification. The processing procedure is shown using the flowchart in Figure 24.

[0101] S501: Display a list of registered abnormalities The model operation unit 113 accesses the model management unit 201 and acquires the model data and reference data of the structure. The display unit 101 displays the acquired model data on the screen and plots the deformation data included in the reference data on the model data. Specifically, as shown in Fig. 17, an icon indicating the position of the component for which deformation data is registered or the viewpoint position included in the deformation data is generated and displayed superimposed on the model data.

[0102] Alternatively, in the procedure shown in S301, the model data may be superimposed on the actual structure in a transparent or semi-transparent state, and an icon indicating the location of the deformation may be superimposed in a visible state (opacity greater than 50%, preferably 100%).

[0103] S502:Selection of registered deformation points The worker selects an icon indicating a registered abnormality on the display device. For example, the worker selects the icon indicating the abnormality by tapping, using a pointing device, or by using eye gaze input. The camera can also be started, and the icon superimposed on the direction the camera is pointed (on the optical axis) can be treated as the selected icon.

[0104] The input unit 102 identifies the icon selected by the worker. The model operation unit 113 accesses the model management unit 201 and acquires the deformation data corresponding to the selected icon. The additional data operation unit 114 accesses the additional data management unit 401 and acquires the management classification and / or the inside / outside classification corresponding to the viewpoint position included in the acquired deformation data.

[0105] S503: View and edit registered abnormalities The display unit 101 displays a screen for referring to, editing, and adding to the abnormality on the display device. Fig. 18 shows an example of the screen for referring to, editing, and adding to the abnormality.

[0106] The initial state is the reference screen, which displays text and media files (photos, videos, etc.) contained in previously registered abnormality data. It also displays the management category and internal / external category to which the abnormality belongs.

[0107] When the edit button is pressed on the deformation reference / edit / add screen of the input unit 102, the display unit 101 displays the edit screen. On the edit screen, the input unit 102 accepts operations such as editing text, adding and deleting media files, etc. The edited data is saved and overwrites the existing deformation data.

[0108] When the Add button is pressed on the deformation reference / edit / add screen, the display unit 101 displays the Add screen. On the Add screen, the input unit 102 accepts operations such as input of new text data and registration of media files. The model operation unit 113 adds the input data to the model management unit 201 as new deformation data with the same deformation ID as the existing deformation data. This results in multiple deformation data with the same deformation ID being registered under one component (GUID).

[0109] In this way, by allowing multiple entries of deformation data with the same deformation ID, it becomes possible to track changes in a specific deformation over time. For example, in Figure 18, multiple tabs with different dates and times, such as "2022 / 2 / 4" and "2021 / 2 / 10," are displayed at the top. This indicates that for this deformation, there are multiple pieces of deformation data registered at different dates and times, in other words, a history of the deformation data. By switching between tabs, workers can look back and check the status of past deformations.

[0110] With this function, the terminal device 10 displays registered deformed areas superimposed on the model data of the structure or the actual structure. It also displays details of the deformed area selected by the worker along with additional data such as the management classification and inside / outside classification. This allows the worker to easily understand the status of the deformed area in detail. It also allows historical data to be managed for each deformed area. This allows the worker to easily and precisely trace changes in the deformed area over time. Furthermore, the desired deformed area can be easily found by utilizing the management classification and inside / outside classification.

[0111] <Detection and estimation of abnormalities> The location and area of ​​the deformation are recognized from the camera image taken of the deformation, and the resources required to repair the deformation are estimated. The processing procedure is shown using the flowchart in Figure 25.

[0112] S601: Photographing the deformation The deformation detection unit 115 controls the camera to take a photograph of the deformation area.

[0113] S602: Identification of components The deformation detection unit 115 identifies the member in which the deformation has occurred. For example, the model data is superimposed on the actual structure in the procedure shown in S301, and then the member in which the deformation has occurred can be identified by specifying the member in the procedure shown in S302.

[0114] S603: Deformation Recognition The deformation detection unit 115 estimates the area of ​​the deformation portion included in the captured image. An example of the estimation method is shown below.

[0115] The anomaly detection unit 115 divides the entire image into multiple grids and identifies grids where discoloration that suggests an anomaly is observed. Typically, this can be done by matching the color of each grid with a color sample to identify grids where the similarity exceeds a threshold, or by calculating the color difference from surrounding grids to identify grids where the similarity exceeds a threshold. In this case, the anomaly detection unit 115 may also identify the type of anomaly.

[0116] The deformation detection unit 115 calculates the ratio of the number of grids recognized as deformed locations to the total number of grids.

[0117] The deformation detection unit 115 estimates the area of ​​the entire area included in the image based on the camera's angle of view (focal length) and the distance to the subject, and estimates the area of ​​the deformed area by multiplying this by the proportion of the grid that is recognized as the deformed area.

[0118] S604: Repair estimate for deformation The repair estimate unit 116 estimates the resources (materials, funds, manpower, etc.) required to repair the deformed portion of the area recognized in S503. For example, a repair estimate can be made by multiplying the area recognized in S503 by the resources (materials, funds, manpower, etc. required to repair the deformed portion of the unit area) predetermined for each type of deformation.

[0119] S605: Save Quote The model operation unit 113 associates the area of ​​the deformation and the estimated results of the repair estimate with the component and registers them in the reference data management unit 301. Typically, this information is recorded as part of the deformation data.

[0120] The present invention is not limited to the above-described embodiment, and constituent elements can be replaced, added, deleted, disassembled, or combined as appropriate as long as it does not go against the spirit of the present invention.

[0121] For example, in the above-described embodiment, an example was shown in which the terminal device 10 includes the display unit 101, input unit 102, photographing unit 111, position estimation unit 112, model operation unit 113, additional data operation unit 114, deformation detection unit 115, and repair estimate unit 116, but these functions can also be installed separately in multiple devices. Several variations are shown below. Note that only the differences from the above-described embodiment will be mainly described here, and explanations of commonalities will be omitted as appropriate.

[0122] <Variation 1> In this example, a first worker inspects a structure on-site to detect any abnormalities, and a second worker performs a repair estimate based on the inspection results in an office or other location (see Figure 26).

[0123] The terminal device 10 used by the first worker is typically a tablet computer, smartphone, HMD, etc. The terminal device 10 includes hardware such as a processor, memory, display device, input device, camera, GNSS receiver, and various sensors for SLAM. In terms of functionality, the terminal device 10 includes a display unit 101, an input unit 102, an image capture unit 111, a position estimation unit 112, a model operation unit 113, an additional data operation unit 114, and a deformation detection unit 115.

[0124] The second terminal device 50 used by the second worker is typically a tablet computer, a smartphone, a PC (Personal Computer), etc. The second terminal device 50 includes hardware such as a processor, a memory, a display device, an input device, etc. In terms of functionality, it includes a repair estimate unit 116.

[0125] The deformation detection unit 115 of the terminal device 10 estimates the area and / or type of the deformation location based on the captured image and stores the estimation result in a storage area (not shown). The storage area may be, for example, an online server, cloud storage, memory in the terminal device 10, or removable storage media.

[0126] The repair estimate unit 116 of the second terminal device 50 obtains the estimated results of the area and / or type of the deformed area from the above-mentioned memory area, and estimates the resources (materials, funds, manpower, etc.) required to repair the deformed area.

[0127] <Variation 2> In this example, an image of a real structure is taken by a camera mounted on a mobile object such as a drone or vehicle, and an inspector receives the image at a remote location to carry out an inspection (see Figure 27).

[0128] The terminal device 10 mounted on the moving object is typically a tablet computer, a smartphone, an HMD, etc. The terminal device 10 includes hardware such as a processor, a memory, a display device, an input device, a camera, a GNSS receiver, and various sensors for SLAM. In terms of functionality, the terminal device 10 includes an image capture unit 111 and a position estimation unit 112.

[0129] The second terminal device 50 used by the worker is typically a tablet computer, a smartphone, a PC, etc. The second terminal device 50 includes hardware such as a processor, a memory, a display device, and an input device. In terms of functionality, it includes a display unit 101, an input unit 102, a model operation unit 113, an additional data operation unit 114, a deformation detection unit 115, and a repair estimate unit 116.

[0130] The image capturing unit 111 of the terminal device 10 captures an image of a real structure. At that time, the position estimation unit 112 acquires position information and attitude information (yaw, pitch, roll) at the time of capturing the image. The terminal device 10 transmits the captured image, position information, and attitude information to the second terminal device 50.

[0131] The second terminal device 50 corrects its own position (S1-S5) using the captured image, position information, and attitude information received from the terminal device 10. This makes it possible to superimpose model data of the structure on the image captured by the moving object. In displaying the current position (S101-102), the current position of the moving object can be displayed. In displaying component information (S301-S304), component information in the captured image can be obtained by calculating the viewpoint and focus point based on the position information and attitude information of the moving object.

[0132] <Variation 3> In this example, an image of the actual structure is captured by a 360-degree panoramic camera installed on-site, and an inspector receives the image at a remote location to carry out an inspection (see Figure 28).

[0133] The panoramic camera functions as the photographing unit 111. The panoramic camera is set so that photographed images can be acquired on the terminal device 10 side.

[0134] The terminal device 10 is typically a tablet computer, smartphone, HMD, PC, etc. The terminal device 10 includes hardware such as a processor, memory, display device, and input device. In terms of functionality, the terminal device 10 includes a display unit 101, an input unit 102, a position estimation unit 112, a model operation unit 113, an additional data operation unit 114, a deformation detection unit 115, and a repair estimate unit 116.

[0135] As a preliminary step, the coordinates of the installation position of the panoramic camera are measured. Also, the direction is measured using a gyrocompass or the like mounted on the panoramic camera. The measured position and direction are recorded in a storage area (not shown). For example, an installation point can be added to the model data.

[0136] The terminal device 10 can realize the same functions as those of the above-described embodiment by using the captured image received from the panoramic camera and the position and angle measured in advance. For example, in displaying component information (S301-S304), the position of the panoramic camera measured in advance is used as the viewpoint, and the worker can acquire component information in the captured image by specifying the point of interest based on the point indicated by tapping or the like on the captured image displayed on the display device.

[0137] <Variation 4> In this example, a first worker records images (photos or videos) of the actual structure on-site, and a second worker uses the recorded images to inspect the structure in an office or other location (Figure 27).

[0138] The terminal device 10 used by the first worker is typically a tablet computer, a smartphone, an HMD, etc. The terminal device 10 includes hardware such as a processor, a memory, a display device, an input device, a camera, a GNSS receiver, and various sensors for SLAM. In terms of functionality, the terminal device 10 includes an imaging unit 111 and a position estimation unit 112.

[0139] The second terminal device 50 used by the second worker is typically a tablet computer, a smartphone, a PC, etc. The second terminal device 50 includes hardware such as a processor, a memory, a display device, and an input device. In terms of functionality, it includes a display unit 101, an input unit 102, a model operation unit 113, an additional data operation unit 114, a deformation detection unit 115, and a repair estimate unit 116.

[0140] The image capturing unit 111 of the terminal device 10 captures an image of a real structure. At that time, the position estimation unit 112 acquires position information and attitude information (yaw, pitch, roll) at the time of capturing the image. The terminal device 10 records the captured image, position information, and attitude information in association with the image (or each frame in the case of a video).

[0141] The second terminal device 50 corrects its own position (S1-S5) using the captured image, position information, and attitude information acquired by the terminal device 10. This makes it possible to superimpose model data of the structure on the image captured by the moving object. In displaying the current position (S101-102), the current position of the moving object can be displayed. In displaying component information (S301-S304), component information in the captured image can be obtained by calculating the viewpoint and focus point based on the position information and attitude information of the moving object.

[0142] The first worker and the second worker may be the same person. In this case, the functions of the second terminal device 50 may be integrated into the terminal device 10.

[0143] Each processing means constituting the present invention may be configured by hardware, or any process may be realized by having a CPU execute a computer program. Furthermore, the computer program may be stored and supplied to a computer using various types of temporary or non-temporary computer-readable media. Temporary computer-readable media include, for example, electromagnetic signals supplied to a computer via wire or wirelessly. [Explanation of symbols]

[0144] 1. Information Processing Systems 10 Terminal Equipment 20 Model Management Device 30 Reference Data Management Device 40 Additional data management device 50 Second terminal device 101 Display section 102 Input section 111 Filming Department 112 Position estimation part 113 Model Operation Unit 114 Additional data operation section 115 Deformation detection unit 116 Repair Estimate Department

Claims

1. A model management device and a terminal device, The model management device It stores model data of a structure consisting of multiple components, The terminal device The self-position on the actual structure is estimated, and the self-position is corrected using the model data. Information processing system.

2. Further comprising an additional data management device; the additional data management device defines a predetermined area set over the structure as a management section, the model management device stores information about the deformation that has occurred in the member together with the position of the deformation that is identified based on the self-location; When reading out the information on the abnormality from the model management device, the terminal device identifies the management category to which the abnormality belongs and displays the identified management category.

2. The information processing system according to claim 1.

3. Further comprising an additional data management device; the additional data management device defines a predetermined area set overlaid on the structure as an inspection range section, The terminal device reads out only the components related to the inspection range section from the model management device and displays the read out components.

2. The information processing system according to claim 1.

4. The terminal device The actual structure is photographed, and the area of ​​the deformation included in the photographed image is estimated.

2. The information processing system according to claim 1.

5. The terminal device Estimate the resources required to repair the above-mentioned defects 5. The information processing system according to claim 4.

6. The terminal device allowing a worker to visually recognize the actual structure; Invisibly superimposing the model data on the actual structure; accepting a designation of a position on the actual structure by the worker; Identifying the member corresponding to the specified position 2. The information processing system according to claim 1.

7. The model management device stores information about the deformation that has occurred in the component in association with the identified component.

7. The information processing system according to claim 6.

8. The model management device stores a plurality of pieces of information acquired at different times regarding a specific deformation.

8. The information processing system according to claim 7.

9. The terminal device The position of the terminal device is superimposed on the model data displayed in a visible state.

2. The information processing system according to claim 1.

10. Estimate the self-position on the actual structure, and correct the self-position using model data of the structure. Terminal device.

11. The terminal device a step of estimating a self-position on a real structure; and correcting the self-position using model data of the structure. Information processing methods.

12. A program for causing a computer to execute the information processing method according to claim 11.

Images