Structure management support device and structure management support system

Abstract

To provide a device and a system for supporting the management of a structure that allow a user to have an objective and ex-post facto knowledge of a virtual field of view according to three-dimensional information when the user manages a structure by using three-dimensional data.SOLUTION: A structure management supporting device 10 includes: display means 13 for displaying three-dimensional data of a structure; storage means 14; vision control means 21 for displaying three-dimensional data in display means 13 by a first-person vision picture VF according to the three-dimensional positional information of a user; and viewpoint history recording means 22 for recording the temporal change of the first-person vision picture VF in the storage means 14.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a structure management support device and a structure management support system. [Background technology]

[0002] In the past, it was common knowledge that the person in charge of maintenance work for social infrastructure would go to the target site. However, when it comes to inspection work, the number of experienced inspection engineers for social infrastructure scattered throughout Japan is limited, and in many cases inspection engineers with low technical skills have to be in charge. For example, even if an experienced inspection engineer uses drawings, photographs, or receives verbal guidance, oversights during inspections and inconsistencies in inspection results are inevitable, and there are limits to improving the quality of inspections.

[0003] Meanwhile, systems have been developed that can display a virtual three-dimensional space in the viewing area of ​​a head mounted display (HMD) or the like, and accurately reproduce environments and situations in special spaces such as high places and closed spaces, and dangerous places that are difficult to experience in reality (see, for example, Patent Document 1). Such technologies are increasingly being applied in the construction and building industries and the like for the purpose of providing simulated experiences for safety education, inspection work, and the like.

[0004] For example, it would be possible to display a so-called first-person view 3D image (such as VR video) on the HMD of an experienced inspection technician, as if the technician were viewing it, and record video of the inspection's field of view (the field of view of the HMD (ideally the camera's angle of view)). This would then be able to be played back (shared) by a less experienced inspection technician at a later time, making it possible for multiple parties involved (such as an experienced inspection technician and an inexperienced inspection technician) to simulate an inspection (simulated experience) and pass on (develop) inspection skills without having to go (gather) at the site.

[0005] In addition, technology has been developed that displays 3D images based on point cloud data within facilities on a PC and makes them visible at the site and in the office, allowing the virtual positions of inspection points or inspection technicians to be grasped on the 3D images (3D data) and enabling inspection and maintenance work to be performed efficiently (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-061186 A [Patent Document 2] Re-tabled publication 2019 / 107420 Summary of the Invention [Problem to be solved by the invention]

[0007] However, even when using three-dimensional images (three-dimensional data) to share a virtual field of view and provide a simulated experience or share information, it is currently difficult for users to objectively and retrospectively grasp what each user focused on in the virtual field of view (how they moved their gaze and where they focused on it).

[0008] For example, in inspection and maintenance work, what is more important than the field of view (what can be seen) is where within that field of view the attention (point of gaze) is. However, even if the field of view of a skilled inspection technician was reproduced (shared), there was a problem in that it was not possible to objectively and retrospectively confirm changes over time in the point of gaze (where the attention was) at that time.

[0009] Even if it is possible to grasp (recognize) the virtual position of an inspection point or an inspection engineer on a 3D image (3D data), it is only possible to recognize the fact that the inspection point was approached, and it is not possible to objectively grasp how the inspection point was actually gazed upon. Even when multiple inspection engineers share the inspection of multiple inspection points on 3D data (virtual inspection), they are unable to grasp each other's gaze points, which can lead to problems such as overlapping inspections or missed inspections due to misunderstandings and misconceptions about the inspection target and range between multiple users (stakeholders).

[0010] In view of the above-mentioned circumstances, the present invention provides a structure management support device and a structure management support system that enable objective and retrospective understanding of the point of gaze in a virtual field of view according to a user's three-dimensional position information when managing structures using three-dimensional shape data. [Means for solving the problem]

[0011] The present invention includes a display means capable of displaying three-dimensional shape data of a structure; A storage means; A computer; The computer includes: a visual field control means for controlling the display means to display the three-dimensional shape data as an image corresponding to the three-dimensional position information of the user (hereinafter referred to as a "first-person visual field image"), and a visual field history recording means for storing, in the storage means, time-varying information indicating the time-varying change of the first-person visual field image; and an electronic signature assigning means for assigning an electronic signature to the first-person field of view image and / or the time-varying information. The present invention relates to a structure management support device characterized by the above.

[0012] The present invention also provides a system for managing a structure comprising a plurality of the above-mentioned structure management support devices, a server in which the three-dimensional shape data is stored, a communication line connecting the plurality of structure management support devices and the server, a surveying means for surveying an arbitrary target area of ​​the three-dimensional shape data to generate survey information, an estimation means for estimating a work-related quantity related to the target area by referring to the surveying information, and another electronic signature providing means for providing electronic signatures by a plurality of users to the work-related quantity; and Multiple structure management support device The present invention relates to a structure management support system characterized in that the three-dimensional shape data is made shareable. Effect of the Invention

[0013] According to the present invention, when managing structures using three-dimensional shape data, it is possible to provide a structure management support device and a structure management support system that enable objective and retrospective understanding of the point of gaze in a virtual field of view based on a user's three-dimensional position information. [Brief description of the drawings]

[0014] [Figure 1] 1A is an external view showing an overview of a structure management support device according to an embodiment of the present invention, and FIG. 1B is a block diagram showing the same. [Diagram 2] 1 is a block diagram showing an overview of some functions of a structure management support device according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing an example of a display of the structure management support device according to the embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing an example of a display of the structure management support device according to the embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing an example of a display of the structure management support device according to the embodiment of the present invention. [Figure 6] FIG. 2 is a diagram showing an example of a display of the structure management support device according to the embodiment of the present invention. [Figure 7] 1A is a schematic diagram showing the overall configuration of a structure management support system according to an embodiment of the present invention, and FIG. 1B is a schematic diagram showing a mode of use. [Figure 8] 1 is a block diagram showing a portion of the functions of a structure management support system according to an embodiment of the present invention. [Figure 9] 1 is a schematic diagram showing an example of a display of a structure management support system according to an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram showing an example of a display of a structure management support system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0016] <First embodiment / structure management support device> FIG. 1 is a diagram showing the overall configuration of a structure management support device 10 which is an example of a first embodiment of the present invention, where FIG. 1(A) is a schematic diagram of the exterior and FIG. 1(B) is a block diagram showing an outline of the hardware configuration.

[0017] 1, the structure management support device 10 of this embodiment includes, for example, a computer (information processing device) 19 having a control means 11 and a storage means 14 (141), and an HMD (Head Mounted Display) which is a display means 13. The information processing device 19 is, for example, a PC (personal computer) in which a program (application program) according to this embodiment is installed or incorporated. The HMD 13 also includes a storage means 142. That is, the storage means 14 includes the storage means 141 of the information processing device 19 and the storage means 142 of the HMD 13.

[0018] However, the structure management support device 10 of this embodiment is not limited to this configuration, and may be a portable information processing terminal in which the control means 11, display means 13, storage means 14, etc. are integrally configured. Specifically, the device may be a notebook PC, tablet terminal, smartphone, mobile phone, personal information terminal (PDA), handheld terminal, GPS terminal, digital camera, etc. in which the program (application program) according to this embodiment is installed or incorporated (any device that has the display means 13 does not have to be an HMD). Also, the HMD 13 may be configured to integrally include the control means 11 and / or storage means 14.

[0019] Referring to the same figure (B), the structure management support device 10 of this embodiment has a control means 11, an input means 12 (121, 122, 123), a display means 13, a memory means 14 (141, 142), a communication means 16 (161, 162, 163), an audio output means 17, various sensors 18, and a business support means 20.

[0020] The control means 11 is, for example, included in an information processing device (PC) 19, and has an arithmetic processing device (CPU; central processing unit, MPU; microprocessor unit, etc.), and provides various functions including those described below by executing an operating system (OS; Operating System), various application programs, and browser software stored (installed) in the storage means 14. The information processing device 19 is, for example, a notebook computer or a desktop computer, but the information processing device 19 may also be a wearable computer.

[0021] The display means 13 is a means for displaying an image that is visually recognized by each user (e.g., an inspection engineer, etc.) of the structure management support device 10, and is preferably capable of displaying at least three-dimensional shape data of (actual) structures in the real environment and is portable. As an example, the case of the HMD 13 will be described as above. The HMD 13 may be a glasses type or a hat type. The display method of the HMD 13 may be a non-transparent type, a video transmission type, an optical transmission type, or other type. The projection method may be any of a virtual image projection method, a retinal projection method, and other types. The HMD 13 has a display 131 (such as a liquid crystal display or an organic EL display) placed in front of the user's eyes, a communication means 162, a storage means 142, etc.

[0022] In addition to the 3D shape data of the structure, the HMD 13 can display images (2D images and / or 3D images) as the execution results of various applications, icons, buttons, and other images (including text). In this specification, "image" includes still images and moving images unless otherwise specified.

[0023] In addition, the three-dimensional shape data in this embodiment refers to data having at least three-dimensional position information (and numerical values). Examples of three-dimensional shape data include data obtained by measuring (scanning) a structure with a three-dimensional scanning means (e.g., a three-dimensional laser scanner), data obtained by processing the scan data, three-dimensional point cloud data that is a collection of points each having three-dimensional position information (relative X, Y, Z information from the scanner), polygonal surface data (three-dimensional image, polygon data) used to express a solid shape in three-dimensional computer graphics, three-dimensional CAD data, and data that can be used in BIM (Building Information Modeling).

[0024] The HMD 13 of this embodiment has various sensors 18, such as a gyro sensor, an acceleration sensor, a depth sensor, and an infrared sensor, as well as an imaging means (camera) 15, and is configured to be able to detect the tilt and position information of the XYZ three axes in a virtual three-dimensional space based on three-dimensional shape data (detect forward / backward / left / right movement), and as an example, a six degrees of freedom (6DoF) method is adopted.

[0025] In this example, the storage means 14 is an internal memory (main memory (RAM; Random Access Memory), storage, ROM (Read Only Memory), etc.), but may include an external storage device. In this example, the storage means 14 includes a storage means 141 on the information processing terminal (PC) side and a storage means 142 on the HMD 13 side. In this case, for example, the storage means 141 on the PC side stores and saves an OS, firmware, browser software, various application programs, various drivers, as well as various data (image data, text data, and other data that can be processed by various application programs, documents), three-dimensional shape data to be displayed on the HMD 13, and the like. The storage means 142 on the HMD 13 side stores and saves at least the three-dimensional shape data to be displayed and / or displayed on the HMD 13. The storage means 142 may be an auxiliary storage medium, such as a USB (Universal Serial Bus) memory or a card-type memory (memory card).

[0026] The input means 12 is a means capable of inputting various information handled by the structure management support device 10, and includes, as an example here, an operation means (e.g., an input pointer) 121 configured separately from either the PC 19 or the HMD 13, an HMD side operation means (operation button) 122 integrally provided in the HMD 13, and a voice input means (microphone) 123 provided in the HMD 13. Here, a physical hand controller attached to the HMD 13 will be described as an example of the input means (input pointer) 12. However, the input means 12 of this embodiment is not limited to this, and also includes a hand tracking function (of the HMD 13) that recognizes the user's hands and fingers using a sensor in the HMD 13 and performs image processing to enable various operations without using a physical hand controller.

[0027] The input pointer 121 is configured to be able to point to a predetermined portion in the three-dimensional shape data, for example, in the same way that at least a laser pointer points to a predetermined position in real space. For example, the HMD 13 detects the movement of the input pointer 121 with a sensor, and the PC 19 displays a cursor (pointer) at a predetermined portion in the three-dimensional shape data displayed on the display 131 based on the detected amount and direction of movement. In addition, it is desirable that the input pointer 121 is provided with a physical operation button and / or a jog dial for instructing a desired operation (performing a desired function).

[0028] The input pointer 121 also has a free-draw function, which allows you to input any line drawing (handwritten characters, etc.) by overlaying it on the displayed 3D shape data. This makes it possible to replace the task of writing notes directly on bridge piers with chalk, for example, during inspection work in real space.

[0029] As an example, the HMD 13 of this embodiment employs an inside-out method of acquiring a positional relationship with an external (real) space by the imaging means 15 and the sensor 18 mounted on the HMD 13. In this case, for example, a method of calculating one's own position after storing the shape of the external space by the imaging means (stereo camera) 15 and the depth sensor (sensor 18), or a method of calculating the position from the time difference of infrared rays incident from the outside, etc. are employed. In addition, without being limited to this, the HMD 13 may employ an outside-in method of acquiring the position and angle of the HMD 13 and the hand controller which is the input means 12 by the imaging means (for example, a sensor including a camera) arranged outside the HMD 13.

[0030] The communication means 16 includes, for example, a communication means 161 on the PC 19 side, a communication means 162 on the HMD 13 side, and a communication means 163 of the input pointer 121, and can perform data communication and the like with each other via a network (wired or wireless communication line). Wireless data communication is performed by short-distance wireless communication such as Wi-Fi (registered trademark), Bluetooth (registered trademark), NFC (Near Field Communication), etc., or various wireless communication methods via a mobile communication network such as LTE (Long Term Evolution), 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), FDMA, TDMA, CDMA, W-CDMA, and PHS (Personal Handyphone System).

[0031] The audio output means 17 is provided in the HMD 13, and is configured to be capable of outputting audio transmitted by another user via the audio input means 123 and audio from the PC 19 (such as an alarm sound or a guide voice).

[0032] Other than the above, the HMD 13 has the same configuration as conventionally known devices. For example, the HMD 13 has a display control means and an input / output interface, receives three-dimensional shape data stored in the storage means 141 of the information processing device 19, and displays it on the display 131. In addition, the HMD 13 transmits and receives various information to and from the information processing device 19 and the input means 12.

[0033] In addition to the above, the information processing device 19 also has conventionally known components such as a display means and an input means (such as a keyboard).

[0034] The information processing device 19 may be a terminal (such as a tablet terminal) that the user can carry around, and the HMD 13 may be configured to communicate with the tablet terminal via wired or wireless communication. The information processing device 19 may also be a small computer that is attached to (built into) the HMD 13.

[0035] The task support means 20 will be described with reference to Fig. 2. The figure is a block diagram for explaining the function of the task support means 20. The task support means 20 is an application program installed in the storage means 14 of the information processing device 19. That is, the task support program is installed in the information processing device 19, and the task support function of the structure management support device 10 is realized by cooperating with each piece of hardware such as the information processing device 19, the HMD 13, and the input means 12 described above. The task support means 20 has a field of view control means 21, a viewpoint history recording means 22, an information providing means 24, and the like.

[0036] The visual field control means 21 causes the display means 13 (here, the display 131 of the HMD 13) to display the three-dimensional shape data as an image corresponding to the user's three-dimensional position information (hereinafter, referred to as a "first-person visual field image VF").

[0037] Here, "user's three-dimensional position information" refers to information including position information within a virtual three-dimensional space based on three-dimensional shape data of the display means 13 carried by the user, i.e., the HMD 13 worn by the user, and the attitude of the display 131 (the direction in which the display 131 is facing (range of the angle of view)).

[0038] Moreover, the "first person view image VF" refers to an image from the user's own line of sight (a so-called first person view (FPV) image), and refers to an image within the field of view (FOV) of the HMD 13. Hereinafter, the field of view range of the HMD 13 will be referred to as the "virtual field of view range 301."

[0039] The field of view control means 21 displays a first-person field of view image VF on the HMD 13 (display 131) according to the user's three-dimensional position information, allowing the user to virtually enter a virtual three-dimensional space and view the three-dimensional shape data of the structure in question, as if he or she were viewing the target structure on site.

[0040] The visual field control means 21 performs user management control to individually associate a first-person field of view image VF displayed by the user on the HMD 13 with the user for each user. The user of the structure management support device 10 of this embodiment registers the use of the device (at least the HMD 13) in advance (at the first time of use) by authentication means such as a known login (sign-in) using a user ID that identifies the individual user. Then, the visual field control means 21 performs user management control when the user starts using the HMD 13 (for example, after the structure management support device 10 (and further the HMD 13) is turned on). That is, the authentication means determines that the user is a legitimate user, and when the user is proven to be a legitimate user, the user can use the structure management support device 10 and the HMD 13. Then, the first-person field of view image VF displayed on the HMD 13 used after being authenticated by a certain user ID corresponds to the actual visual field of the user corresponding to the user ID. In addition, the user management control also makes it possible to associate the user's role (for example, a work role such as a person who performs a task, a supervisor, or a person in charge) with the user ID.

[0041] The visual field control means 21 also performs gaze range display control. Here, the gaze range display control refers to control including display of a virtual gaze range (virtual gaze range 302) when the first-person view image VF is displayed, and change of the range. The virtual gaze range 302 refers to a range in the virtual visual field range 301 (the range in which the first-person view image VF is displayed) that is assumed to be actually observed by the user, and a range that the user can actually gaze at. That is, it can also be expressed as a range in which an object can be correctly recognized and grasped in accordance with the purpose of the present structure management support device 10, and can be set to a range narrower than the range of an actual human viewing angle (or a viewing angle set in the HMD 13 in the system), and can also vary depending on the distance to the object. When a person focuses both eyes and looks at a certain point (gazing point) in a real environment, a certain range of area centered on the gaze point can be said to be a range in which the person pays attention to and gazes at the gaze point with the intention of seeing, that is, the gaze range. This gaze range is smaller than the range that can be seen around the gaze point without moving the head and eyes (the range of a real human's visual field (visual field)). In order to reproduce this gaze range in the first-person view image VF displayed by the HMD 13, the visual field control means 21 displays an image showing a virtual gaze range 302 in the first-person view image VF.

[0042] The gaze range display control will be described with reference to Fig. 3. Fig. 3(A) is a schematic diagram of the display area of ​​the display 131 of the HMD 13. Fig. 3(B) to Fig. 3(F) are schematic diagrams showing an example of a first-person view image VF viewed by the user on the HMD 13, Fig. 3(B) is a schematic diagram showing an example of the positional relationship between the overall image of a structure represented by three-dimensional shape data D and the user's HMD 13 from a bird's-eye view (from a third-person perspective), and Fig. 3(C) to Fig. 3(F) are display examples of the first-person view image VF of the HMD 13.

[0043] As shown in FIG. 1A, in this embodiment, the entire display area of ​​the display 131 is the virtual visual field range 301. The virtual gaze range 302 is, for example, a region having a certain area (for example, an area in a range of a horizontal visual field angle of X degrees and a vertical visual field angle of Y degrees, or an area of ​​a predetermined ratio to the entire virtual visual field range 301) always centered on the center of the virtual visual field range 301, specifically, a point with a horizontal visual field angle of 0 degrees and a vertical visual field angle of 0 degrees (hereinafter referred to as a "virtual gaze point VP"). The virtual gaze point VP is, for example, a point on the optical axis of the imaging means 15, but the virtual gaze point VP may be, for example, a point determined by the direction (position) of the head (neck) of the user wearing the HMD 13. As already described, the image displayed in the virtual visual field range 301 is the first-person view image VF. The image displayed within the virtual gaze range 302 of the first-person view image VF is hereinafter referred to as a "first-person gaze image VA". The virtual gaze range 302 and the virtual gaze point VP are displayed as objects (images) such as circles or squares on the three-dimensional image.

[0044] The gaze range display control includes control for setting the virtual gaze range 302 to, for example, a certain default range and displaying it on the HMD 13 (the display 131), and further changing the virtual gaze range 302. The control for changing the virtual gaze range 302 will be described in detail later.

[0045] Referring to FIG. 1B, the three-dimensional shape data D of the structure is, as an example, a building having ten windows W. The numbers in the windows W are added for convenience of explanation, and will be described below as windows W(1) to W(10). The three-dimensional shape data D of the structure is shown here as a two-dimensional line drawing for simplicity. When the virtual gaze point VP of the HMD 13 is present in the upper right corner of the window W(5) of this three-dimensional shape data D as shown in the relative positional relationship in FIG. 1B, the user visually recognizes the first-person view image VF as shown in FIG. 1C. The view control means 21 displays the virtual gaze range 302 as a predetermined range centered on the virtual gaze point VP. In this case, the user is gazing at the virtual gaze range 302 (the first-person gaze image VA inside it) in the upper right corner of the window W(5).

[0046] Fig. 2(D) shows the state in which the user has moved the HMD 13, i.e., moved the viewpoint (line of sight). As the HMD 13 moves, the first-person field of view image VF changes, and the virtual gaze point VP and virtual gaze range 302 also move. In this case, the user has moved his or her line of sight and is now gazing at approximately the center of window W(4). Figs. 2(E) and 2(F) show the state in which the user has further moved the HMD 13, i.e., moved the viewpoint (line of sight) from the lower left of window W(3) to approximately the outer lower right of window W(10).

[0047] In this way, in the structure management support device 10 of this embodiment, the first-person field of view image VF, the first-person gaze image VA, and the virtual gaze point VP correspond to events that can be seen within the user's field of view in the real environment, events that are seen in the gaze range, and the gaze point, respectively.

[0048] Furthermore, as shown in Figures (C) to (F), when the virtual field of view range 301 changes over time in response to the user's movement and the direction and tilt of the head (HMD13), the first-person field of view image VF, first-person gaze image VA, and virtual gaze point VP also change over time.

[0049] The viewpoint history recording means 22 stores (records) the time-varying change of the first-person view image VF, more specifically, the time-varying change of the first-person gaze image VA, in the storage means 14 so that the time-varying change of the first-person view image VF (at least the first-person gaze image VA) can be reproduced (confirmed) at any time or can be played back later. The time-varying change of the first-person view image VF can be played back (reproduced) based on the time-varying change information of the first-person gaze image VA and the three-dimensional shape data. The time-varying change of the first-person gaze image VA is hereinafter referred to as "viewpoint history TR". The viewpoint history TR may be, for example, a direct recording (save) of the time-varying change of the first-person gaze image VA as a video, or may include the display of the virtual gaze range 302 and / or the virtual gaze point VP in the video.

[0050] Furthermore, a method of recording information that "it was viewed" on the three-dimensional shape data side as the first-person gaze image VA included in the virtual gaze range 302 is also included as a type of gaze history record. For example, a method of recording information that the three-dimensional shape data viewed by the user is viewed by switching the display mode is also included in the gaze history record. Specifically, for example, in the case of point cloud data, the display mode is changed by coloring (discoloring) the points of the viewed part with a predetermined color (a color different from the original color). The points (point cloud) colored (discolored) in this way are also included in the gaze history TR. In this case, there are cases where another three-dimensional shape data (e.g., data of pillars, reinforcing bars, etc.) is superimposed behind the three-dimensional shape data in which the information that "it was viewed" is recorded (e.g., data of the exterior wall surface of a building). The three-dimensional shape data superimposed behind the three-dimensional shape data in which the information that "it was viewed" is recorded is not viewed even if it is included in the virtual gaze range 302 partitioned as a two-dimensional plane, and is not included in the gaze history TR.

[0051] 3(G) is a diagram showing an example of a recorded gaze history TR, and is a schematic diagram showing (reproducing) a certain user's gaze history TR from a bird's-eye view (from a third person's point of view) based on the three-dimensional shape data D of the structure. In this way, in this embodiment, since the user's gaze history TR can be recorded at any time, it is possible to objectively and retrospectively grasp not only changes in the user's simple movement of the virtual visual field range 301 (movement of the HMD 13) but also which part of the virtual visual field the user was particularly paying attention to.

[0052] The viewpoint history TR may be a record of only the movement information of the virtual gaze range 302 and / or the virtual gaze point VP in three-dimensional coordinates, and may not include the first-person gaze image VA. For example, when reproducing the recorded viewpoint history, it may be generated and displayed as a first-person gaze image VA based on the movement information of the virtual gaze range 302 and / or the virtual gaze point VP in three-dimensional coordinates. Similarly, the range of the display area of ​​the display 131 may be generated and displayed as a first-person view image VF. Also, the first-person view image VF may be separately stored as the viewpoint history TR. Also, when reproducing the viewpoint history, it may be possible to switch between the first-person gaze image VA and the first-person view image VF and reproduce them. For example, in the case of a simulation of an inspection work, when it is sufficient to grasp only an outline such as a movement route, a wide-angle first-person view image VF may be easier to recognize than the first-person gaze image VA.

[0053] In the user management control of the field of view control means 21, the first-person field of view image VF (or the first-person gaze image VA that enables reproduction (playback) of the first-person field of view image VF, and the movement information of the three-dimensional coordinates of the virtual gaze range 302 and the virtual gaze point VP) that the user has displayed on the HMD 13 is associated with the user by a user ID or the like. In other words, the gaze history TR is also associated with the user by a user ID or the like.

[0054] The viewpoint history recording means 22, for example, temporarily records the viewpoint history TR in the storage means 141 of the HMD 13 in order to transmit the viewpoint history TR to the information processing device 19 at periodic timing. Alternatively, the viewpoint history recording means 22 may be capable of saving (accumulating) the viewpoint history TR in the storage means 142 of the HMD 13 so that the viewpoint history TR can be transmitted (or manually copied, moved, etc.) to the information processing device 19 at any timing (as needed) by the user.

[0055] The gaze history recording means 22 starts and ends recording of the gaze history TR at any timing, for example, by the user operating the input means 12 (for example, by appropriate operation of the input pointer 121, the HMD side operation button 122, etc., or by voice input by the user via the voice input means (microphone) 123). The gaze history recording means 22 may be configured to continuously record the gaze history TR (the circle marks of the virtual gaze range 302 are continuously recorded in a line) during the period from the start to the end of recording of the gaze history TR (recording period), or, for example, when the virtual gaze point VP does not move for a certain time (for example, several seconds) during the recording period, it may be determined that there was a gaze corresponding to an inspection, and the virtual gaze range 302 at that position may be recorded in the gaze history TR. This makes it possible to exclude recording of unnecessary gaze movements that do not correspond to gaze (inspection, etc.). In addition, the gaze history recording means 22 may record the gaze history TR every time the user operates the input means 12 (for example, recording when a button is pressed and not recording when released). The user operates the input means 12 at any position that he or she wishes to record. This allows the user to leave a point-by-point gaze history TR, which also makes it possible to eliminate the recording of unnecessary gaze movements.

[0056] The task support means 20 also includes an electronic signature providing means 23. The electronic signature providing means 23 is a means for providing an electronic signature, which is an electronic certificate, to, for example, the first-person view image VF and / or information on changes over time (for example, the viewpoint history TR). The electronic signature providing means 23 also attaches an electronic certificate issued by a certification authority to certify that the person who signed the electronic signature is the person himself / herself. The electronic certificate is issued by a certification authority (CA) and certifies the identity of the person by a cryptographic technique called a public key infrastructure (PKI). In other words, the electronic signature providing means 23 uses the electronic certificate to provide an electronic signature and identify the user of the HMD 13. This proves that the viewpoint history TR recorded by the HMD 13 is displayed as the first-person gaze image VA or first-person view image VF of the user of the HMD 13, and that the viewpoint history TR has not been tampered with since the electronic signature was provided.

[0057] It is preferable that the electronic signature adding means 23 can further add a time stamp. This makes it possible to prove that the time of the electronic signature is correct and without error, separate from the time set by the HMD 13, the PC 19, etc. That is, by having the electronic signature adding means 23 add, for example, the electronic signature and time stamp of the user to the viewpoint history TR by the user's operation, it becomes possible to prove that the viewpoint history TR is displayed as the first-person gaze image VA or the first-person field of view image VF of the user of the HMD 13, that the viewpoint history TR has not been tampered with since the time of the electronic signature, that the viewpoint history TR exists at the time of the time stamp, and that the viewpoint history TR has not been tampered with after the time stamp.

[0058] As already mentioned, in the user management control of the field of view control means 21, the viewpoint history TR displayed by the user on the HMD 13 is individually associated with the user by means of a user ID or the like, and the user's role (for example, a work role such as a task performer, supervisor, or person in charge) is also associated with each user by means of, for example, a user ID.

[0059] In other words, by managing users and making it possible to assign electronic signatures (with electronic certificates) and time stamps to the viewpoint history TR, a specific user with a certain business role can be linked one-to-one with the viewpoint history TR of that user, and the identity and non-tampering of the history can be verified. This makes it possible, for example, for the viewpoint history TR displayed on the HMD 13 by the user, who is the inspection manager, to be one that was made by the inspection manager and has not been tampered with. This also makes it possible for the viewpoint history TR to be a record of an inspection (inspection deliverable) that shows that an inspection equivalent to a real inspection was performed in a virtual three-dimensional space, specifically on three-dimensional shape data obtained by scanning an actual structure.

[0060] The electronic signature assignment means 23 assigns an electronic signature and / or timestamp to the viewpoint history TR, for example, by the user operating the input means 12 (for example, by appropriate operation of the input pointer 121, HMD side operation button 122, etc., or by the user inputting voice via the voice input means (microphone) 123, etc.).

[0061] For example, it is more preferable that the electronic signature assignment means 23 prompts the user to perform an operation when the recording of the viewpoint history TR is completed (upon instruction to terminate), by displaying a menu on the display 131 of the HMD 13 or outputting it as audio, the menu prompting the user to select whether or not to assign an electronic signature and / or a timestamp.

[0062] The digital signature providing means 23 can provide digital authentication to both the viewpoint history TR and the first-person view image VF.

[0063] The information providing means 24 stores information related to the structure in the storage means 14 (141, 142) in association with the three-dimensional shape data in response to information providing operations (mark providing operations and information input operations) by the user.

[0064] In detail, the information providing means 24 displays an object image (hereinafter referred to as "mark image M") superimposed on the first-person view image being displayed on the display 131 of the HMD 13 in response to a marking operation by the user (see FIG. 3(F)). Specifically, the marking operation is, for example, an operation of the input pointer 121 by the user, and displays the mark image M superimposed on the first-person view image pointed to by the input pointer 121, and obtains the three-dimensional coordinates of the mark image M in the virtual three-dimensional space, and stores (image data of) the mark image M, an identification number (mark identification number MID) unique to the mark image M, and three-dimensional position information of the mark image M in the virtual three-dimensional space in the storage means 14 (for example, registers in a database, etc.) as one information group (data unit) using the mark identification number MID as a key.

[0065] The marking operation may also be an appropriate operation of the HMD side operation button 122, or voice input by the user via a voice input means (microphone) 123, and may add a mark image M to the three-dimensional coordinates of the user's point of gaze, for example, when these operations are performed.

[0066] Information related to a structure (hereinafter referred to as "structure-related information") is, for example, captured image data (still images, moving images) such as photographs and videos, character data (text data, document data created with word processing software, tabular data created with spreadsheet software, etc.), voice data, electronic signatures, and other data related to a structure. Specific examples of structure-related information include inspection records, work instructions, manuals, various documents shared among multiple parties, electronic seals that replace stamps, IDs (signas, marks) for identifying users, and data (handwritten characters) entered by the free-draw function using the input pointer 121.

[0067] Note that, as an example, the mark image M shows a predetermined object image (such as a pin mark or a circle mark), but for example, when the cursor of the input pointer 121 is moved to the mark image M, a list or thumbnails of the structure-related information associated with the mark image M may be displayed. Also, the structure-related information associated with the mark image M may be directly displayed as a thumbnail (or list) as the mark image M (without displaying a predetermined object image).

[0068] The information adding means 24 also accepts an information input operation by the user for the already added mark image M. The information input operation is, for example, an operation by the input means 12 (the input pointer 121, the HMD side operation button 122, the voice input means (microphone) 123) pointing to the mark image M, and the operation method is the same as the mark adding operation. When the information input operation is performed by the user, the information adding means 24 accepts the input of the structure-related information, and additionally registers the input structure-related information using the mark identification number MID as a key to an information group (data unit) corresponding to the mark identification number MID. In other words, the mark image M of this embodiment has position coordinates (three-dimensional coordinates) in a virtual three-dimensional space, and can be said to be an information holding means (folder) capable of holding the structure-related information. In addition, it can be said that the structure-related information associated with (held in) the mark image M also has position coordinates (three-dimensional coordinates) in the virtual three-dimensional space. Note that the electronic signature adding means 23 can also add electronic certification to the structure-related information.

[0069] The structure management support device 10 can transmit various information including three-dimensional shape data (including associated structure-related information) and viewpoint history TR stored in the memory means 14, for example, via the communication means 16, or copy and output various information via an external auxiliary memory to an external device (such as a separate information processing device or printer).

[0070] Furthermore, the structure management support device 10 can transmit and receive data units to and from an external device. The external device is another structure management support device 10 or another device, and the data unit is various information including three-dimensional shape data (including associated structure-related information) and viewpoint history TR created by another structure management support device 10, or various information including three-dimensional shape data (including associated structure-related information) created by another device. This allows the structure management support device 10 to import data units from, for example, an external device via the communication means 16 and display them on the display means 13.

[0071] <Displaying and changing the virtual gaze range> The virtual gaze range 302 will be further described with reference to Figures 4 and 5. The visual field control means 21 sets the virtual gaze range 302 as an initial value of an appropriate range according to the usage environment (usage situation) of the structure management support device 10, the usage state, the purpose of use, the ability of the user, or the operation of the user, and can also change it.

[0072] FIG. 4 is an example of a display on the display 131 of the HMD 13. As an example, FIG. 4(A) and FIG. 4(B) show a case where the three-dimensional shape data of the structure shown in FIG. 3 is viewed. The virtual gaze range 302 is set as an initial value at the start of use as a region having a predetermined ratio (e.g., 20%) of the entire virtual visual field range 301 as an area ratio (FIG. 4(A)). In addition, since the three-dimensional shape data is data having numerical values, it may be set as an area of ​​a predetermined numerical range at a predetermined distance from the user's position (position in the virtual three-dimensional space) (e.g., an area within a circle with a radius of 2 m 10 m away from the user), or may be set as an area of ​​an angle range narrower than the general visual angle of a human (or a visual angle set in the system of the HMD 13). The initial value of the virtual gaze range 302 is selected appropriately according to various factors that affect vision and visibility, such as the usage state (usage environment) and purpose of use of the structure management support device 10.

[0073] The usage state (usage environment) is, for example, the distance to the structure image X when using the HMD 13 (how close to get), the moving speed in the virtual three-dimensional space of the user wearing the HMD 13 (for example, when performing an inspection, whether to virtually move on foot or by work vehicle), etc. Furthermore, the purpose of use is, for example, the purpose of inspection (close inspection, regular inspection, inspection when a problem occurs, etc.), the purpose of a simulated experience, etc., and the ability of the user is, for example, in the case of inspection, the level of proficiency in inspection, physical ability such as eyesight, etc.

[0074] The visual field control means 21 automatically changes the initial value of the set virtual gaze range 302 in response to the above factors, or accepts manual changes by the user. For example, Fig. 4(B) shows a state in which the virtual gaze range 302 is expanded in the same virtual visual field range 301 as Fig. 4(A).

[0075] As described above, the virtual gaze range 302 can be controlled as a fixed value within an angle range narrower than the viewing angle (controlling so that the size (occupied area) relative to the virtual visual field range 301 (or the display 131, the first-person visual field image VF) does not change), but on the other hand, the virtual gaze range 302 may be controlled so that the size (occupied area) relative to the virtual visual field range 301 (or the display 131, the first-person visual field image VF) changes according to the distance from the user to the object. For example, the user's eyes lose their recognition ability when looking at a distant object, and when the object becomes even more distant, they are unable to recognize it at all. To explain this in terms of an image on the HMD 13, a distant object may occupy too small an area on the image, resulting in a state where the object is in the field of view but cannot be gazed upon (zero virtual gaze range). On the other hand, the user's eyes improve their recognition ability when looking at a close object, so they can gaze upon a wide range. An example reflecting this state is shown in FIG. 1C. In Fig. 1C, the area of ​​the virtual gaze range 302 in the virtual visual field range 301 becomes larger as a result of being closer to the three-dimensional shape data D of the structure than in the state of Fig. 1A. Since it is expected that the user's visual recognition ability will improve, the state is shown in which the size of the virtual gaze range 302 is relatively enlarged compared to the size of the object (the range (area) of the object that can be viewed within the virtual gaze range 302 becomes smaller).

[0076] For this reason, the virtual gaze range 302 may be automatically changeable according to the distance to the target area of ​​the three-dimensional shape data D, for example, so that when the user approaches the object, the virtual gaze range 302 expands in a direction approaching the viewing angle (the area occupied on the virtual viewing range 301 expands), and when the user moves away from the object, the virtual gaze range 302 contracts in a direction smaller than the viewing angle (the area occupied on the virtual viewing range 301 contracts). The three-dimensional shape data D also has length information, and the distance from the user to the target area of ​​the three-dimensional shape data D can be grasped based on the three-dimensional position information of the user in the virtual three-dimensional space.

[0077] In the inspection of structures in the real environment, for example, a close-up inspection (close-up visual inspection) where palpation or tapping inspection can be performed may be required. When the structure management support device 10 of the present embodiment is used as an alternative to such an application, the virtual gaze range 302 is set according to the conditions of the close-up inspection (close-up visual inspection). Specifically, the view control means 21 expands the virtual gaze range 302 when the distance to the target area of ​​the three-dimensional shape data D becomes 1 m or less. Furthermore, the view control means 21 may also display the virtual gaze range 302 based on a predetermined condition, rather than displaying it from the beginning of the display on the display 131. In other words, during close-up inspection, the virtual gaze range 302 and / or the virtual gaze point VP may be displayed when the distance to the target area becomes 1 m or less. If the viewpoint history is recorded in this state (the viewpoint history has been recorded), the first-person gaze image VA is displayed in a state where the conditions for close-up inspection are met. In other words, when inspection using the structure management support device 10 is used in place of inspection of the real environment, the record of the viewpoint history TR by the viewpoint history recording means 22 can be used as a deliverable (evidence) of the completion of the inspection.

[0078] As an alternative to inspection work by actual palpation, a glove-type 3D image-compatible device may be used to utilize tactile feedback such as bumps and recesses in a virtual 3D space.

[0079] 13E and 13F are diagrams showing an example of changing the virtual gaze range 302 according to the distance to the target structures X1 and X2 on the three-dimensional shape data D. For example, the user's eyesight in a real environment becomes more difficult to see the farther away the object is. Therefore, as shown in 13E, the distance to the target object (e.g., structure X1) may be measured based on the three-dimensional shape data D, and the virtual gaze range 302 may be displayed if the structure X1 included in the virtual gaze range 302 is at a distance where it can be seen (for example, seen for the purpose of inspection) in the real environment. On the other hand, 13F shows an example in which the virtual gaze point VP is located at the three-dimensional shape data D of the structure X2, which can be seen (its existence can be recognized) in the real environment but is too far away to be inspected. In this case, inspection is not possible in the real environment, so the virtual gaze range 302 may be hidden as shown in 13F. In this example, the structure X1 is close to the user, but is outside the virtual gaze range 302. As shown in FIG. 1(G), the virtual gaze range 302 is displayed when the structure X2 approaches to a predetermined distance. In this case, the distance at which the virtual gaze range 302 can be displayed is appropriately determined based on the distance at which gaze is generally possible or on conditions set in the real environment such as close inspection. The virtual gaze range 302 may be changed manually by the user according to his / her own eyesight. In the examples of FIG. 1(E) to FIG. 1(G), the size of the virtual gaze range 302 is not changed, but it may be changed. For example, the virtual gaze range 302 may be gradually reduced during the transition from FIG. 1(E) to FIG. 1(F), and gradually expanded during the transition from FIG. 1(F) to FIG. 1(G).

[0080] Also, the display control of the virtual gaze range 302 may take into consideration the time of gaze. For example, if the virtual gaze point VP moves instantly even if the object is close, it may not be regarded as gaze (inspection). Therefore, the virtual gaze range 302 may be displayed when the virtual gaze point VP does not move for a predetermined time (such as several seconds).

[0081] The shape of the virtual gaze range 302 is not limited to a circle, but may be a polygon (such as a square or a horizontally elongated rectangle).

[0082] 5 is a schematic diagram for further explaining the first-person field of view image VF and the first-person gaze image VA (virtual gaze range 302). Depending on the shape of the structure X, there may be a part in the virtual gaze range 302 that does not directly face the display 131 of the HMD 13, or the user may gaze at a position that does not directly face the display 131.

[0083] For example, Fig. 1A, Fig. 1C, and Fig. 1E are diagrams showing the relationship between the virtual position of the user in the virtual three-dimensional space and the position of the three-dimensional shape data showing the structure when gazing at, for example, a wall surface of the three-dimensional shape data D showing the structure, Fig. 1A and Fig. 1C are conceptual diagrams looking down from above (from a third person's point of view), and Fig. 1E is a conceptual diagram seen from the side. Fig. 1B is a schematic diagram showing the virtual field of view 301 and the virtual gaze range 302 corresponding to Fig. 1A, Fig. 1D is a schematic diagram showing the virtual field of view 301 and the virtual gaze range 302 corresponding to Fig. 1C, and Fig. 1F is a schematic diagram showing the virtual field of view 301 and the virtual gaze range 302 corresponding to Fig. 1E. Here, the virtual gaze range 302 is shown in a rectangular shape.

[0084] In this case, the structure has a corner formed by walls a and b, and the user gazes at the structure from near the corner. In detail, wall a in front of the user's left eye in FIG. 1A faces the user almost directly, but wall b in front of the user's right eye does not face the user, but is inclined at a certain angle (so that the outer angle of the corner is angle α) with respect to the plane extending from wall a, and wall b moves away from the user as it moves away horizontally from the virtual gaze point VP (as the horizontal viewing angle increases).

[0085] In principle, the visual field control means 21 sets the virtual gaze range 302 as a region (indicated by a dashed line in FIG. 1A) that is symmetrical in the left and right and up and down directions with the virtual gaze point VP as the center. However, when the virtual gaze range 302 includes a structure including a region that is (almost) directly facing the user (directly facing region SC: directly facing such as wall a) and a region that is not (non-directly facing region SN: not directly facing such as wall b), it is preferable that the visual field control means 21 deforms the virtual gaze range 302 for the non-directly facing region SN (deforms the virtual gaze range 302 so that a part of it becomes impossible (difficult) to gaze upon in accordance with the real environment).

[0086] In the same figure (B), since wall b does not face the user directly, the field of view control means 21 narrows the vertical field of view angle of virtual gaze range 302 on the wall b side and changes the virtual gaze range 302 to a trapezoidal shape that is asymmetrical with the wall a side.

[0087] 1C and 1D show cases where the angle of the corner formed by walls a and b is more acute than that in 1A and 1B (external angle β>α). When the external angle of the corner becomes larger in the real environment, the visible range (recognizable range) of wall b that is not directly facing the user becomes smaller. For this reason, the visual field control means 21 deforms the virtual gaze range 302 for the non-directly facing area SN. When deforming in this way, it is advisable to determine whether or not to deform the virtual gaze range 302, or the degree of deformation, based on the degree of non-direction of the non-directly facing area SN relative to the directly facing area SC (here, the angles α and β of the external angles of the corner). For example, when the degree of non-direction, i.e., the exterior angle, is equal to or less than a predetermined threshold Dt (when Dt≧α), the amount of deformation of the virtual gaze range 302 is small (Fig. 1B), and when the exterior angle is greater than the predetermined threshold Dt (Dt<β), the amount of deformation of the virtual gaze range 302 is large (Fig. 1D). In Fig. 1D, the visual field control means 21 changes the virtual gaze range 302 from the dashed line to the solid line, and the size of the virtual gaze range 302 only on the wall a side is changed to 1 / 2 of its initial value.

[0088] Note that the shape of the non-target area SN is just one example, and is not limited to shapes having corners in the vertical direction as shown in Figures (A) and (C), but is similar to shapes having corners in the horizontal direction, for example, as shown in Figures (E) and (F).

[0089] Furthermore, as shown in FIG. 5(G) and FIG. 5(H), when gazing at a structure having a non-facing area SN, the virtual gaze range 302 may be changed according to the distance from the user. FIG. 5(G) and FIG. 5(H) are schematic diagrams corresponding to FIG. 5(A) and FIG. 5(B), respectively. As shown by the dashed line in FIG. 5(G), the visual field control means 21 sets the virtual gaze range 302 in a rectangular area shown by the dashed line in principle. However, for example, when comparing point P1 on wall a and point P2 on wall b, the distance from the user to point P2 is greater than the distance from the user to point P1. In other words, although point P2 is within the virtual gaze range 302, it may actually exceed the distance that can be gazed at (recognized). Therefore, when a non-facing area SN exists, a predetermined distance Lx facing the user may be set as a threshold value, and the virtual gaze range 302 may be deformed so that the distance Lx is the maximum. In this case, the wall b side is made visible (recognizable) up to a distance Lx, and the visual field control means 21 changes the virtual gaze range 302 from the dashed line to the solid line.

[0090] The viewpoint history TR may also be configured to record the distance between the user and the structure being inspected during inspection (a distance indicating how close the user was to the structure when inspecting it, for example, the distance Lx shown in Figure 5 (G)).

[0091] FIG. 6 is another example of a case where a structure with a complex shape is viewed. FIG. 6 is an example of a wall with the same shape of a wave shape being viewed from different positions (positions in virtual three-dimensional space). Depending on the user's positions (positions in virtual three-dimensional space) P1 and P2, the front facing area (front facing) SC for the user is different, and the state of the non-front facing area (non-front facing) SN is also different. In such a case, the visual field control means 21 may make changes such as hiding or filling in areas that are estimated to be invisible to the user even within the virtual gaze range 302, depending on the front facing area (front facing) SC based on the user's position and the degree of non-front facing of the non-front facing area (non-front facing) SN relative to the front facing area SC.

[0092] For example, in Fig. 1A, the face (HMD 13) is turned in the direction of the arrow from position P3, and the user is viewing the surface S3 so as to face it almost directly. In this case, at least the surfaces S2, S4, and S6 are in the non-facing area SN and are in the shadows of the surfaces S1, S3, and S5, respectively, making it difficult for the user to view them. In such a case, the visual field control means 21 hides the parts of the virtual gaze area 302 that correspond to the surfaces S2, S4, and S6.

[0093] In addition, in FIG. 2B, the face (HMD 13) is turned in the direction of the arrow from position P4, and the user is viewing the surface S9 so as to face it almost directly. In this case, at least the surfaces S10, S12, and S14 are in the non-facing area SN and are in the shadows of the surfaces S9, S11, and S13, respectively, making it difficult for the user to view them. In such a case, the visual field control means 21 hides the parts of the virtual gaze area 302 that correspond to the surfaces S10, S12, and S14.

[0094] In this way, when the structure image X (its three-dimensional shape data) displayed in the virtual gaze range 302 includes a non-directive area SN for the user, the gaze management means 21 may determine the area where the user cannot or will have difficulty gazing based on an appropriate threshold value, and may change the virtual gaze range 302 to an arbitrary shape so as to exclude areas beyond the threshold value from the virtual gaze range 302.

[0095] Of course, when the user moves around so that the non-frontal area is located in the frontal area, the line-of-sight management means 21 appropriately changes (transforms) the virtual gaze range 302 according to the user's position. For example, the hidden part in the example of FIG. 6 is redisplayed.

[0096] In this way, the visual field control means 21 performs gaze control to switch between displaying and hiding the virtual gaze range 302 (virtual gaze point VP). In addition, not limited to the above example, when visibility (or inspection requiring reliable visibility) is impossible or difficult in the real environment, the visual field control means 21 may perform gaze control to not display the virtual gaze range 302 (virtual gaze point VP). This gaze control includes control to switch the display mode of the three-dimensional shape data as the first-person gaze image VA inside the virtual attention range 302 while displaying it. The control to switch the display mode of the three-dimensional shape data is, for example, control to change the coloring of the points of the point cloud data. For example, when conditions in the real environment make visibility (or inspection requiring reliable visibility) impossible or difficult, the field of view control means 21 displays the virtual attention range 302 set as an initial value, for example, while coloring the visible point cloud data in the first-person gaze image VA in red, for example, and coloring the point cloud data in an area of ​​the virtual attention range 302 that is difficult to see and requires a change to the virtual gaze range 302 in white.

[0097] Furthermore, in the real environment, in the case of conditions where visual confirmation (or inspection requiring reliable visual confirmation) is not possible, the viewpoint history recording means 22 may be configured not to record the viewpoint history TR.

[0098] Furthermore, in the real-world environment, when conditions are such that visual confirmation (or inspection requiring reliable visual confirmation) is not possible, the digital signature granting means 23 may be configured not to grant a digital signature.

[0099] Also, the display / non-display of the virtual gaze range 302 may be switched by an appropriate operation of the user. In other words, the virtual gaze range 302 may not be displayed with a predetermined initial value, but may be displayed by the visual field control means 21 in response to an arbitrary operation of the user.

[0100] <An example of operation and use of the structure management support device 10> The following describes an example of the operation and use of the structure management support device 10. The structure management support device 10 can be used, for example, when inspecting a structure in a virtual three-dimensional space.

[0101] First, before starting to use the structure management support device 10, the inspection manager, who is the user, inputs a user ID assigned individually to the inspection manager into the input means 12 and is authenticated for use of the structure management support device 10 (at least the HMD 13). The field of view control means 21 determines that the user is a legitimate user based on the input user ID, and if it is proven that the user is a legitimate user, permits the inspection manager to use the structure management support device 10 and the HMD 13. For example, the user's role (e.g., inspection work manager) is also associated with the user ID. As a result, the first-person field of view image VF that the inspection manager displays on the HMD 13 is associated as an image unique to the inspection manager.

[0102] The inspection manager selects the three-dimensional shape data D corresponding to the structure to be inspected and displays it on the display 131 of the HMD 13. Then, the inspection manager moves to a position where the desired inspection point of the structure can be viewed, and starts the inspection. The visual field control means 21 displays the virtual gaze range 302 as a predetermined range centered on the virtual gaze point VP, so the inspection manager moves appropriately so that the desired location to be inspected is included in the virtual gaze range 302 (FIGS. 3(D) to 3(F), and 4). For example, when the inspection is in an inappropriate situation, i.e., when the inspection manager's position in the virtual three-dimensional space is a position that cannot be viewed in relation to the structure image X), the visual field control means 21 does not display the virtual gaze range 302.

[0103] When starting an inspection, the inspection manager starts recording a viewpoint history TR, for example, by operating the input means 12. The viewpoint history recording means 22 starts recording, for example, a first-person gaze image VA based on the operation of the input means 12 (operation to start recording a viewpoint history). The inspection manager inspects each location while moving appropriately so that the planned desired locations are displayed within the virtual gaze range 302. Furthermore, when necessary, for example, at locations where close inspection is mandatory, the inspection manager moves closer to the structure image X. In response to the movement of the inspection manager, the field of view control means 21 appropriately switches between displaying and hiding the virtual gaze range 302, changes the shape of the range, and so on.

[0104] The person in charge of inspection can add structure-related information to a desired position in the three-dimensional shape data D as necessary. For example, a certain inspection location is specified by the input means 12 (e.g., the input pointer 121) and a marking operation is performed. In response to the user's marking operation, the information adding means 24 displays a mark image M superimposed on the first-person field of view image being displayed on the display 131 of the HMD 13 (see FIG. 3(F)). Also, the information adding means 24 acquires the three-dimensional coordinates of the mark image M in the virtual three-dimensional space, and stores the mark image M (image data of the mark image M), the mark identification number MID, and the three-dimensional position information of the mark image M in the virtual three-dimensional space in the storage means 14 as one information group (data unit) using the mark identification number MID as a key.

[0105] When the mark image M is displayed, the inspection manager performs an information input operation for the mark image M. When the information input operation is performed, the information providing means 24 accepts the input of structure-related information. The inspection manager inputs information about the inspection location, for example, via the voice input means 123. The information providing means 24 additionally registers the input structure-related information using the mark identification number MID as a key to an information group (data unit) corresponding to the mark identification number MID.

[0106] When the inspection manager finishes the scheduled inspection, he / she ends the recording of the viewpoint history TR by operating the input means 12. In response to the operation of the input means 12 (operation to end the viewpoint history recording), the viewpoint history recording means 22 associates the viewpoint history TR from the start to the end as one data unit with the user ID of the inspection manager and stores it in a predetermined storage means 14 (the storage means 14 of the PC 19 or the HMD 13) (see FIG. 3(F)).

[0107] When the viewpoint history recording is completed (upon instruction to complete), the electronic signature assignment means 23 prompts the user to perform an operation by displaying a menu on the display 131 of the HMD 13 or outputting it as audio, which asks whether or not to assign an electronic signature and / or a timestamp.

[0108] The inspection manager applies an electronic signature and assigns a time stamp. The first-person view image displayed on the HMD 13, which is used after being authenticated by the inspection manager's user ID, corresponds to the actual view of the user corresponding to the user ID. The viewpoint history TR is displayed by the inspection manager and has not been tampered with since the electronic signature and the assignment of the time stamp. The viewpoint history TR and the structure-related information (hereinafter referred to as management information) assigned by the inspection manager via the mark image M are stored in the storage means 14 in association with the three-dimensional shape data used during the inspection. That is, since both the viewpoint history TR and the mark image M have three-dimensional coordinates, they are associated with the corresponding coordinates of the three-dimensional shape data based on the coordinates.

[0109] The management information can be treated as a record of the inspection performed on the three-dimensional shape data by the inspection manager (inspection deliverable), and can be used as a substitute for visual inspection on site. In other words, the inspection manager can virtually inspect the structure at any place and time without going to the site. In addition, since the virtual gaze range 302 can be changed according to the situation, inspection work on the three-dimensional shape data in the virtual three-dimensional space can be performed in accordance with actual inspection. In addition, the management information can be associated with the three-dimensional shape data and saved and accumulated as electronic information, and can be transmitted to the PC 19 or a separate external device, etc., so that it can be displayed on a display means, output as a printout, and shared with third parties (other related parties, etc.).

[0110] For example, an inexperienced inspection engineer can simulate inspection locations (for example, simulate the order of movement through inspection locations) by playing back the viewpoint history TR, which includes information related to the structure, on the HMD 13 or other devices (such as the PC 19 or external devices). In particular, by playing back the viewpoint history TR (first-person gaze video), it is possible to understand the points that the inspection manager gazed at at each inspection location, making it possible to transfer skills without actually accompanying the person in charge. As a result, this can contribute to improving inspection techniques, preventing oversight of inspection locations, and preventing variation in inspection results.

[0111] <Second embodiment / Structure management support system> The structure management support system 50 of this embodiment will be described with reference to Fig. 7 to Fig. 10. Fig. 7 is an overall view of the structure management support system 50, in which Fig. 7(A) is a schematic diagram showing the system configuration, and Fig. 7(B) is a schematic diagram showing how the system is used.

[0112] 7(A), a structure management support system 50 includes a plurality of structure management support devices 10 of the first embodiment, a server 51 in which the above-described three-dimensional shape data is stored, and a communication line 52 connecting the plurality of structure management support devices 10 and the server 51. The three-dimensional shape data can be shared among the plurality of structure management support devices 10 connected to the communication line 52. The shareable three-dimensional shape data includes management information (viewpoint history TR and / or structure-related information) recorded by a certain structure management support device 10. The server 51 holds not only the three-dimensional shape data but also management information associated with the data. The server 51 may be a cloud server.

[0113] The communication line 52 is any communication network that allows mutual communication between the devices, and may be a wired communication network, a wireless communication network, or both. The communication line 52 may be configured to be one communication network, or may be configured to be a plurality of communication networks. For example, the communication line 52 may include any communication network or communication path of any communication standard, such as the Internet, a public telephone line network, a wide area communication network for wireless mobile devices such as a so-called 3G line, 4G line, or 5G line, a wireless communication network that performs communication in accordance with a WAN (Wide Area Network) or LAN (Local Area Network) standard, a short-distance wireless communication channel such as Bluetooth (registered trademark), Wi-Fi (registered trademark), or NFC (Near Field Communication), a communication path of infrared communication, a wired communication network in accordance with a standard such as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus), etc.

[0114] As shown in FIG. 1B, in the structure management support system 50, one 3D shape data and the management information (viewpoint history TR and / or structure-related information) associated therewith are shared among multiple users. Specifically, for example, a user (inspector) A uses a structure management support device 10A. Another user (inspector) B uses another structure management test device 10B. Here, the structure management support devices 10A and 10B used by each user A and B each include an HMD 13, an input means 12, and an information processing device 19 (see FIG. 1). When the information processing device 19 is, for example, a PC or the like, it may be included in each of the structure management support devices 10A and 10B, but one information processing device 19 may be shared among multiple users, and the HMD 13A and the input means 12A of the structure management support device 10A and the HMD 13B and the input means 12B of the structure management support device 10B may each be connected to the shared information processing device 19 via a communication line 52. Moreover, the functions of the shared information processing device 19 may be realized by the server 51 .

[0115] As an example, user (inspector) A obtains 3D shape data of a desired structure from server 51 at any place and time, displays it on his / her HMD 13A, and performs inspection work. During the inspection, inspector A records his / her own viewpoint history TR, and also adds necessary structure-related information, associates it with the 3D shape data as management information, and stores it, for example temporarily, in storage means 14 of structure management support device 10A. Structure management support device 10A transmits the management information stored in structure management support device 10A to server 51 at any time or based on the operation of inspector A, etc.

[0116] Meanwhile, another user (inspector) B obtains, for example, pre-prepared three-dimensional shape data (including associated management information generated and saved by inspector A, if any) from the server 51 at any place and time, and displays it on his / her own HMD 13B. Furthermore, yet another user C also obtains the same three-dimensional shape data from the server 51 at any place and time, and displays it on the display means 13C of his / her own terminal. In this way, multiple parties involved in the management work (construction, repair, etc.) of a certain structure can share the three-dimensional shape data (including management information) for a certain (single) structure, without each party gathering at the site.

[0117] 8 is a block diagram showing an example of a system configuration of a structure management support system 50. The same components as those in the first embodiment are indicated by the same reference numerals, and the description thereof will be omitted.

[0118] The business support means 20 of the structure management support system 50 further includes, for example, a surveying means 55, an estimation means 56, and another electronic signature providing means 57.

[0119] The surveying means 55 is a means for surveying an arbitrary target area of ​​the three-dimensional shape data to generate surveying information. The target area is an area where some work is performed on the structure, for example, an area that needs repair or restoration, such as an area discovered by an inspector during an inspection of the structure. The target area is set on the three-dimensional shape data by one or more related parties (users of the structure management support device 10 in this embodiment). The surveying means 55 surveys the shape of the target area thus identified. Specifically, for example, when concrete damage or cracks occur in a part of a structure and the damage or restoration is to be performed, the target area is a predetermined area including the damage or cracks in the concrete, and the surveying means 55 surveys the amount of concrete chipping for the repair or restoration based on the shape of the set target area.

[0120] The user first specifies the survey range on the three-dimensional shape data using the input means 12 (e.g., input pointer 121). The three-dimensional shape data retains a real-world scale, and the survey means 55 can measure the length, area, and volume of the specified survey range, as well as the work range and deformed areas within the survey range. The numerical values ​​of the measurement results by the survey means 55 are visualized by the PC 19 so as to be superimposed on the three-dimensional shape data, and can be viewed on the HMD 13 (display 131). In addition, the measurement results by the survey means 55 can be added to the three-dimensional shape data as structure-related information and recorded, and can be shared with related parties (other users).

[0121] The estimation means 56 is a means for estimating work-related quantities related to the target area by referring to the survey information generated by the survey means 55. Work-related quantities are, for example, quantities such as the number of man-hours for work (e.g., repair work), the types and amounts of materials related to the work, and costs related to the work (material costs and labor costs).

[0122] The server 51 holds a database 60 of basic quantities or desired calculation formulas of various data used by the surveying means 55 and the estimating means 56. The database 60 is, for example, a material unit price database, a material specific gravity database, a calculation formula database, etc. In the second embodiment, a business support program functioning as the business support means 20 is held, for example, in the storage means of the server 51. By executing the business support program, necessary three-dimensional shape data (including management information) and various information (data) necessary for each process of the surveying means 55 and the estimating means 56 are transmitted and received between the structure management support devices 10A-10C of each of the users A-C, and are displayed on the display means 13 (HMDs 13A, 13B, display means 13C) of each of the users A-C.

[0123] The other electronic signature granting means 57 grants electronic signatures by multiple users to the work-related quantities estimated by the estimation means 56. This configuration is similar to the electronic signature granting means 23 of the first embodiment, so a detailed description will be omitted, but while the electronic signature granting means 23 of the first embodiment is mainly used as evidence of the inspection work, the other electronic signature granting means 57 of the second embodiment is used, for example, to confirm intentions between multiple users, and thus the method of use is different. The electronic signature granting means 23 of the first embodiment may be used commonly as the electronic signature granting means of this embodiment.

[0124] The surveying means 55, the estimation means 56, and the other digital signature providing means 57 function, for example, as a consensus building support means 58. That is, for example, when repairing a structure, multiple parties involved (for example, designers, inspectors, repairers, etc.) become users of this structure management support system 50 and share the 3D shape data (including management information, the same applies below) corresponding to the same structure.

[0125] Here, as shown in FIG. 7B, user A and user B carry (wear) at least display means (here, HMD) 13A, 13B individually, and although they share the same three-dimensional shape data of the structure, they individually view the first-person view image according to their respective positions (three-dimensional coordinate positions in the virtual three-dimensional space). Due to user management at the start of use of the structure management support device 10, the initial positions (positions at the time of authentication, user initial positions) in the virtual three-dimensional space of each user A and B authenticated by the view control means 21 are set in the shared three-dimensional shape data. For example, this user initial position is a relative position from a display start reference point (reference point for initial display of three-dimensional shape data) that is formally determined when displaying the shared three-dimensional shape data. When each user A and B moves in the virtual three-dimensional space, the position is specified as a relative position from the user initial position (display start reference point of three-dimensional shape data). For example, the display start reference point for multiple users is common (for example, a display start reference point provided by this structure management support system 50), and the positions of multiple users are controlled in the same coordinate system. Furthermore, the position of each user may be stored as appropriate, for example separately from the viewpoint history TR, and the next user's initial position may be set to the recorded user's position. For example, if the system is shut down in the middle of an inspection operation, the user's position at the time of termination can be stored, and the stored user's position can be selected as the user's initial position the next time the system is started.

[0126] 9 and 10 are examples of first-person view images VF displayed on the HMDs 13A and 13B of users A and B. FIG. 9(A) is an example of a first-person view image VFa of user A, and FIG. 9(B) is an example of a first-person view image VF of user B. For ease of explanation, FIG. 9 is a schematic diagram that simplifies the image display. FIG. 10 is an example of a more specific display of the first-person view image VF, where FIG. 9(A) is an example of the first-person view image VFa of user A, and FIG. 9(B) is an example of the first-person view image VF of user B. FIG. 10(C) is an example of a display showing the overlap of multiple virtual gaze ranges.

[0127] 9(A), three-dimensional shape data D of objects, for example, buildings CN1 and CN2, is displayed as a first-person view image VFa of user A in the virtual field of view 301 of user A. Also, as shown in FIG. 9(B), three-dimensional shape data D of the same buildings CN1 and CN2 viewed from a different position is displayed as a first-person view image VFb of user B in the virtual field of view 301 of user B.

[0128] As shown in Fig. 9 and Fig. 10, a mark image M is displayed superimposed on three-dimensional shape data D. Due to the presence of mark image M, user A and user B can respectively recognize that appropriate structure-related information has been added (stored) at the display position. As shown in Fig. 10, it is preferable that the display mode (color, shape, displayed letters / numbers, etc.) of mark image M be different depending on, for example, the type of information to be added (stored) and the user who added mark image M.

[0129] Furthermore, in this example, when the virtual visual field range 301 of user A includes the virtual gaze range 302B of other user B as shown in Figures 9 and 10, the virtual gaze range 302B of other user B is displayed as a circle in the first-person view image VFa, and user A can recognize that other user B is virtually gazing at the position of the circle.

[0130] Similarly, in the first-person view image VFb of user B, the virtual gaze range 302A of user A is displayed as a circle, and user B can recognize that user A is virtually gazing at the position of the circle. When multiple people use the device, it is preferable to make the display mode (color, shape, etc.) of the virtual gaze range 302 corresponding to each user different.

[0131] For example, as shown in Figure 9 (B), if user A's virtual gaze range 302A is located in a position that is shaded by structure CN2 from user B's position, the virtual gaze range 302A will not be displayed in the portion shaded by structure CN2.

[0132] Also, as shown in Fig. 10B, when the virtual gaze range 302A of user A and the virtual gaze range 302A of user B are close to each other or overlap, the gaze points of both users virtually coincide, that is, they recognize the same object. In other words, for example, a certain user A can move his / her virtual gaze range 302A so as to coincide with the virtual gaze range 302B of another user B, thereby overlapping the virtual gaze range 302 of user B, and it can be said that both users recognize the same object. This overlap is displayed in the first-person view images VFa and VFb of both users.

[0133] The structure management support system 50 further has a user position display means 59. The user position display means 59 is a means for displaying an image (character image, hereinafter referred to as "avatar") indicating the position of each user (users A and B) based on the user initial position set by the field of view control means 21 within the three-dimensional shape data shared by multiple authenticated users (users A and B in this case).

[0134] As shown in Figures 9(A) and 10(A), avatar Ab of a certain user (user B) becomes visible when it enters the virtual visual field range 301 of another user (user A). On the other hand, as shown in Figures 9(B) and 10(B), avatar Aa of user A is not within the virtual visual field range 301 of user B, and is therefore not visible to user B.

[0135] Each user A, B knows each other's position based on the display of the avatar Aa, Ab. If the other person's avatar is not included in the user's own first-person view image VF, a guide display 303 suggesting the position (direction) of the other person's avatar may be displayed on the display 131 (FIGS. 9(B) and 10(B)). The guide display 303 may be, for example, an arrow indicating the direction in which the other person's avatar is located, an overhead view that includes both the user's and the other person's positions (a third-person perspective display (and a first-person view image VFc of yet another user C) or a map).

[0136] Also, when necessary information such as the current virtual gaze range 302 of other users or the user's past virtual gaze range 302 (viewpoint history TR) is not included in the user's first-person view image VF, a guide display 303 may be displayed. Also, in addition to or instead of the guide display 303, a voice guide (for example, a voice output such as "XX is nearby") may be displayed.

[0137] In this embodiment, the same three-dimensional shape data is shared in the first-person field of view images VF of the different display means 13 (HMD 13) in this manner, but the virtual gaze ranges 302 can be made to overlap (match) with each other, thereby avoiding recognition errors caused by viewing different parts. When the virtual gaze ranges 302 of multiple users overlap with each other, the display mode (e.g., color, etc.) of the three-dimensional shape data of at least one of the users (e.g., the user who encouraged the other user to gaze at a certain part) may be changed.

[0138] In addition, the structure management support device 10 is configured to be able to communicate with each other by voice. By voice communication, the users A and B can communicate with each other by moving in the virtual three-dimensional space or by overlapping the virtual gaze range 302.

[0139] Furthermore, if at least one of users A and B (for example, user B) records a viewpoint history TR, user A will be able to view the viewpoint history TR if (a part of) that viewpoint history TR, i.e., virtual gaze range 302, is included in the virtual field of view range 301 of the other user, user A (Figures 9(A) and 10(A)).

[0140] Furthermore, the structure-related information added to the three-dimensional shape data by each of the users A and B is displayed as a corresponding mark image M when the position of the information in the virtual three-dimensional space is included in each other's virtual visual field range 301 (FIGS. 9 and 10). Then, the users A and B can confirm (visually recognize, listen to) the structure-related information associated with the mark image M by pointing to the desired mark image M by the input means 12 (FIGS. 9(A) and 10(A)). Note that the information adding means 24 may manage the password of the structure-related information when the structure-related information is associated with each of the mark images M (when the structure-related information is stored in the folder corresponding to the mark image M). This allows management so that the structure-related information is displayed only to specific users.

[0141] The visual field control means 21 may also be capable of displaying the first-person view image VF of the other user B on the display 131 of one user A. For example, in response to an operation of the input means 12 of user A, the first-person view image VFb of user B may be displayed in part of the first-person view image VFa of user A (FIGS. 9(A), 10(A)). Alternatively, the entire first-person view image VFa of user A may be switched to the first-person view image VFb of user B and displayed. The first-person view image VF of the other user may be displayed simply or in a reduced size. Furthermore, for example, an overhead view including the positions of both users A and B (and the first-person view image VFc of another user C) may be displayed in part or all of the first-person view image VFa of user A.

[0142] Next, a description will be given of the consensus formation support means 58. For example, when performing work such as repair, maintenance, or new addition to the structure image X, a consensus is required between the parties involved (user A, user B) regarding the target area and the estimate of the work-related quantity.

[0143] In the past, the parties involved (people in charge, inspectors, repairers, designers, etc.) rarely gathered at the site, and photos taken at the site and CAD drawings were compared at a later date at a location other than the site (each business office, etc.), and a contract was made after reaching an agreement during the ordering process. However, photos and CAD drawings that are mainly taken to cut out only the target part of the structure may not be able to faithfully reproduce the actual structure, and the target parts are usually dealt with in large numbers at once. Even if multiple parties look at the same data such as photos and CAD drawings, they are not necessarily looking at the same place, so there are many cases where oversights and misunderstandings occur. In this way, there are cases where oversights and misunderstandings occur when reaching an agreement based on traditional photos and CAD drawings, and there was a problem of frequent reworking of repair methods and construction methods.

[0144] In this embodiment, the consensus building support means 58 enables users A and B to build a consensus on the three-dimensional shape data they share.

[0145] Specifically, first, as shown in Fig. 9(B) and Fig. 10(B), user A and user B identify a target part 502 that requires repair or maintenance, for example. That is, in order to grasp the same target part 502, user A and user B use voice communication or the like to move their own virtual gaze ranges 302A and 302B so as to overlap (align) with the target part 502. When the virtual gaze ranges 302 of both users overlap, their recognition of the target part 502 coincides.

[0146] 10C is a diagram showing an example of overlap of the virtual gaze ranges 302, in which the virtual gaze range 302A specified based on the virtual gaze point VPa of the user A is shown by a rectangle, the virtual gaze range 302B specified based on the virtual gaze point VPb of the user B is shown by a rectangle, and the overlapping range OL of the (plural) virtual gaze ranges 302A and 302B of both is shown by a dashed rectangle. When the overlap of the plural virtual gaze ranges 302 is set as a condition for a certain process, for example, it is determined that the virtual gaze ranges 302 overlap when 50% or more (preferably 70% to 80% or more) of the size of the virtual gaze ranges 302 overlap (when the overlapping range OL is 50% or more of the size of the virtual gaze ranges 302).

[0147] Next, using voice communication or the like, a target area 505 for repair or maintenance is specified based on the target part 502. For example, the target area 505 is displayed as shown by the solid lines in Fig. 9(B) and Fig. 10(B) by the operation of the input means 12 by one user A. The other user B can also visually confirm the target area 505, and therefore their recognition of the target area 505 is consistent. In this state, the appropriateness of the target area 505 is examined, and the target area 505 is decided based on the agreement of both parties.

[0148] When the target area 505 is determined, the surveying means 55 surveys the target area 505 based on the operation of the input means 12 by the user A and / or the user B. For example, in the case of re-pouring concrete, the amount of concrete chipping is measured.

[0149] The survey results by the surveying means 55 are displayed on the displays 131 of the users A and B, and are examined by both of them. After the two of them reach an agreement on the survey values, the estimation means 56 refers to the various DBs 60 stored in the server 51 and estimates the amount of work in the target area 505 based on the operation of the input means 12 by the users A and / or B. The estimation results are displayed on the displays 131 of the users A and B, and are examined by both of them. When the two of them reach an agreement on the estimate, the two of them move on to the final agreement on the entire work based on the operation of the input means 12 by the users A and / or B. Specifically, the agreement formation support means 58 displays the agreement (meeting book) 510 on the displays 131 of the users A and B. The users A and B operate the input means 12 so that they each add their own electronic signature to a specified portion of the agreement 510. The electronic signature adding means 23 adds the electronic signatures of the users A and B to the work-related quantities based on the operation, and adds a time stamp as necessary. The agreement (meeting book) 510 can be added to the three-dimensional shape data as structure-related information in association with the mark image M. That is, it can be stored in an information storage means (folder) associated with the coordinates of the three-dimensional shape data.

[0150] In addition, the electronic signature assignment means 23 may assign the electronic signature not only at the time of final agreement formation, but also based on the operation of the input means 12 of users A and B at at least one of the timings of identifying the target portion 502, identifying the target area 505, and surveying by the surveying means 55.

[0151] Furthermore, the provision of electronic signatures is premised on the presence of input means 12 for all the people involved (e.g., users A and B). It is particularly preferable to provide electronic signatures on the condition that the virtual gaze ranges 302 of all the people involved overlap.

[0152] Also, for example, the electronic signature attaching means 23 may be configured to attach an electronic signature by a predetermined operation of the input pointer 121, and it may be determined that an agreement has been reached when all the parties concerned point the cursor of the input pointer 121 to the target site 502 or the target area 505. In the above embodiment, the user's gaze point corresponds to the virtual gaze point VP, and in principle the point of interest (cursor position) of the input pointer 121 and the virtual gaze point VP are different positions, but in the case of reaching an agreement, for the convenience of operation between users in particular, the point of interest (cursor position) of the input pointer 121 may be made to correspond to the user's viewpoint (virtual gaze point VP), and when the points of the input pointer 121 match, it may be determined that the gaze points of the real users match, and the electronic signature attaching means 23 may continue to attach an electronic signature in response to the operation of the input pointer 121.

[0153] With this configuration, the parties involved can reach a consensus without gathering at the site, and even if there are many target areas 505, it is possible to prevent discrepancies in understanding among the parties involved and significantly reduce rework.

[0154] In addition, the meeting log and the viewpoint history TR can be combined and added to the 3D shape data as structure-related information. The movement history of the virtual gaze range 302 of each person involved when creating the meeting log (when reaching an agreement) can be recorded, which can be useful for reviewing the consensus at a later date.

[0155] The digital signature providing means 23 may be configured to provide a digital signature and a time stamp for the contract following the consensus building.

[0156] In consensus building, if the virtual gaze ranges 302 of the people involved do not overlap in a predetermined step (at least one step such as identifying the target part 502 and the target area 505, surveying, and estimating), the system may be set up so that the system cannot proceed to the next step or an alert is output. By building a consensus at each step as needed, the level of satisfaction of the people involved can be increased.

[0157] The digital signature providing means 23 may be configured to provide digital signatures between a plurality of users to any information based on the shared three-dimensional shape data, not limited to the work-related quantity.

[0158] The structure management support system 50 may also be used for virtual inspection work by multiple people. For example, when multiple people share the work of performing virtual inspection work on the structure image X, the virtual gaze range 302 is displayed differently for each user (such as displaying an object image of a different color or shape), and the viewpoint history TR is recorded. This allows the inspection end point to be marked with a specific mark, making it possible to avoid omissions or duplication of inspection work.

[0159] It can also be used for the transfer (education) of inspection techniques, inspection simulations, etc. By multiple users sharing the same 3D shape data at the same time (in different locations), it is possible to achieve better mutual understanding in technology transfer and simulations.

[0160] It also makes up for differences in inspection techniques, reduces variation in inspection quality, and avoids fatal mistakes.

[0161] According to the structure management support device 10 and the structure management support system 50, a plurality of related parties can independently share the same three-dimensional shape data (while each viewing the first-person field of view video VF). This also allows each party to record the history of the movement of the virtual gaze range 302 (viewpoint history TR), making it possible to clarify the gaze point of each person on the three-dimensional shape data. This makes it possible to avoid discrepancies in the recognition (on the drawing) between the related parties in work such as repairs, for example. In addition, by sharing the three-dimensional shape data at any location at the same time, it is possible to check whether the gaze ranges of each person match, so that it is possible to accurately reach an agreement on the work range and amount of work without gathering at the site. In addition, since there is no need to go to the site, the travel cost for going to the site is reduced, and lane restrictions at the site are also unnecessary.

[0162] In addition, since users can move freely within the virtual three-dimensional space, they can see areas that cannot be seen in the real environment (for example, it is possible to see both the front and back of an overpass at the same time), which can sometimes improve inspection accuracy compared to the real environment.

[0163] Furthermore, since there is no need to actually enter dangerous areas, this has the advantage of avoiding danger to the inspection worker (user) and ensuring safety.

[0164] It is inevitable that there will be errors in the actual (constructed) structure compared to the design drawing. In this embodiment, inspections etc. are performed based on 3D shape data obtained by scanning the actual (constructed) structure, and surveying repair locations etc. can be performed to estimate quantities related to the work. In other words, compared to management of structures based on design drawings (prediction of inspection locations, surveying and estimating repair locations etc.), it can be said that this allows management of structures that is more in line with reality.

[0165] In the second embodiment, the explanation was centered on users A and B of the HMD 13, but in inspection work and consensus building, the explanation may also include, for example, a structure management support device 10C composed of a tablet terminal or a notebook PC and their display means (display) 13, and its user C.

[0166] The structure management support device 10 and the structure management support system 50 of this embodiment are not limited to the configurations described above. Examples of configurations that can be applied (replaced or added) to the structure management support device 10 and the structure management support system 50 of this embodiment, including configurations that overlap with those described above, are listed below.

[0167] The gaze history recorded by the gaze history recording means 22 may include only the first-person gaze image VA and may not include the display of the virtual gaze range 302 and the virtual gaze point VP.

[0168] The MHD 13 can capture first-person view images and / or virtual gaze images (screenshots, videos) using an imaging means (camera), for example. The captured images can be associated with a mark image M as structure-related information and added to the three-dimensional shape data (associated with the corresponding coordinates of the three-dimensional shape data based on the three-dimensional coordinates of the mark image M).

[0169] When used for close inspection or the like, if a predetermined condition for close inspection (such as the distance from the structure image X) is not met, the virtual gaze range 302 may not be displayed.

[0170] Structure-related information may include an electronic whiteboard.

[0171] For the purpose of reconfirming an inspection task, etc., while playing back a user's viewpoint history TR (first-person view video VF), it is possible to move the virtual gaze range 302, i.e., to display a first-person view video VF based on a new virtual gaze range 302 different from the first-person view video VF being played back, and also to record the viewpoint history TR and attach an electronic signature. For example, if an inspection is found to have been missed, the virtual gaze range 302 can be moved again to pick up the remaining inspection.

[0172] The display mode of the three-dimensional shape data may be configured to be switchable. For example, in the case of point cloud data, the density (size of the point cloud) may be changeable (switchable). Also, the display of the point cloud data and 3DCG (polygon data) may be switchable, or both may be displayed in a transparent superimposed manner (layer display). Alternatively, the three-dimensional shape data and the design / structure data of the object (CAD drawings, etc.) may be switchable, or both may be displayed in a transparent superimposed manner (layer display).

[0173] In addition, the 3D shape data before and after repairs may be compared, and, for example, differences may be extracted. This allows the results of repairs and repaired parts to be recorded (saved), delivered, etc.

[0174] The survey by the surveying means 55 is not limited to the repair target area 505. For example, it can also be used for simulations of what would happen if repairs were to be carried out.

[0175] The virtual gaze range 302 may be changed according to regulations. For example, in the case of a regular inspection that is performed frequently (such as once a month), a change in the virtual gaze range 302 that does not satisfy the conditions for close inspection may be permitted, but in the case of a large-scale inspection that is performed infrequently once every few years, a change in the virtual gaze range 302 that does not satisfy the conditions for close inspection may not be permitted.

[0176] The virtual gaze point VP may be determined based on the state (such as the direction of gaze, pupil movement, blinking, etc.) of the user's eyeball captured by an imaging means included in the HMD 13. In other words, the actual gaze point of the user may be set as the virtual gaze point VP, or the virtual gaze point VP may be determined based on the gaze point.

[0177] The input means 12 may be equipped with an eye tracker capable of measuring the actual gaze points of the left and right eyes of the user of the HMD 13 and the movement of the eyes relative to the head, and may be configured to display a cursor (pointer) on the display means 13 (display 131) using an eye tracking (gaze measurement) technique that visualizes the movement of the user's actual gaze point (movement of the line of sight).

[0178] In the above embodiment, the virtual gaze range 302 is an area having a certain area at the center of the virtual visual field range 301, but the pointing point (position of the cursor or pointer) of the input means 12 (input pointer 121) may be the virtual gaze point VP, and a predetermined range (e.g., a two-dimensional circular area) centered on the virtual gaze point VP may be the virtual gaze range 302. In this case, the gaze history TR records the movement of the pointing point (cursor or pointer) by the input means 12 (input pointer 121) (movement of the virtual gaze point VP) or the movement of the virtual gaze range 302 based thereon.

[0179] Furthermore, the viewpoint position determined by eye tracking may be set as a virtual gaze point VP, and a predetermined range (e.g., a two-dimensional circular area) centered on the virtual gaze point VP may be set as the virtual gaze range 302. In this case, for example, even if the user does not move his / her head (HMD 13), the virtual gaze range 302 moves following the movement of the eyeball.

[0180] The various sensors of the HMD 13 may include a biometric information acquisition sensor capable of acquiring biometric information of the user and a posture sensor. The HMD 13 may also have a self-status acquisition means such as a GPS (unit), an electronic compass, or a SLAM using a monocular camera. The self-status acquisition means may acquire three-dimensional position information and posture information of the HMD 13 in a virtual three-dimensional space, and may be configured to transmit the information to the PC 19 at a predetermined interval, for example.

[0181] The self-state acquisition means of the HMD 13 may be a means for detecting the attitude and position of the aircraft by detecting SLAM using a stereo camera, or infrared light, laser, or other reflected light from an object. The self-state acquisition means may also include an absolute position information acquisition module that can detect the direction, attitude, and position (absolute position) of the aircraft by, for example, radio wave intensity from a wireless base station for a mobile phone or an access point for Wi-Fi (registered trademark), or detect the direction, attitude, and position (absolute position) of the aircraft by a signal from a satellite, such as the Global Navigation Satellite System.

[0182] The functions realized by each component (means) of the structure management support device 10 and the structure management support system 100 are just examples, and it is sufficient if the functions of the above-mentioned embodiment can be realized in terms of hardware or software as a whole of the structure management support device 10 and the structure management support system 100. As an example, some of the functions described as functions realized by the field of view control means 21 may be realized by the viewpoint history recording means 22.

[0183] Furthermore, the structure management support device, information management system, information management method, and information management program of the present invention are not limited to the above-mentioned embodiments, and various modifications can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0184] 10 Structure management support equipment 11 Control Means 12 Input Methods 13 Display means 14 Memory means 16 Means of communication 17 Audio output means 18 Various sensors 19 Information processing equipment 20 Business support means 21 Field of view control means 22 Viewpoint history recording means 23 Electronic signature adding means 24 Means of providing information 50 Structure Management Support System 51 Server 52 Communication lines 55 Surveying means 56 Estimation Methods 56 Surveying instruments 57 Electronic signature adding means 58 Consensus building support means 59 User position display means 60 Databases 100 Structure Management Support System 121 Input Pointer 122 Side operation button 123 Voice input means 131 Display 14, 141, 142,143 Memory means 16, 161, 162, 163 Means of communication 301 Virtual Field of View 302 Virtual Gaze Range 303 Guide Display 502 Target area 505 Target Area Aa Avatar Ab Avatar VA first person gaze video VF first person visual field VP Virtual gaze point X Structure Image

Claims

1. A display means capable of displaying three-dimensional shape data of a structure; A storage means; A computer; The computer includes: a visual field control means for controlling the display means to display the three-dimensional shape data as an image corresponding to three-dimensional position information of a user (hereinafter referred to as a "first-person visual field image"); a viewpoint history recording means for storing, in the storage means, time-varying information indicating a time-varying change of the first-person view image; and an electronic signature assigning means for assigning an electronic signature to the first-person field of view image and / or the time-varying information. A structure management support device characterized by:

2. The visual field control means performs user management for associating the first-person visual field image with each user individually.

2. The structure management support device according to claim 1 .

3. The visual field control means displays a gaze range, which means a range in the first-person visual field image that the user can gaze at.

3. The structure management support device according to claim 1 or 2.

4. An information providing means for storing information related to the structure in the storage means in association with the three-dimensional shape data.

4. The structure management support device according to claim 1, wherein the structure management support device is a device for supporting a structure management system.

5. Having an input means capable of inputting the information, 5. A structure management support device according to claim 1,

6. A structure management support device comprising: a plurality of structure management support devices according to any one of claims 1 to 5; A server in which the three-dimensional shape data is stored; A communication line connecting a plurality of structure management support devices and the server; a surveying means for surveying an arbitrary target area of ​​the three-dimensional shape data to generate survey information; an estimation means for estimating a work-related quantity relating to the target area by referring to the survey information; and another electronic signature providing means for providing electronic signatures by a plurality of users to the work-related quantity; The three-dimensional shape data can be shared among the plurality of structure management support devices. A structure management support system comprising:

Images