Information processing device, information processing method, and program

Abstract

To provide technology that enables efficient and effective management of, for example, the results of on-site inspections related to the maintenance and management of buildings. [Solution] The UI control unit 51 displays a 3D model of the building to be inspected on the inspector T's inspector terminal 2, accepts any point in the three-dimensional space of the 3D model as a defective location through an operation specified by the inspector T, and executes control to activate a UI that accepts defect details information of the defective location through an operation input by the inspector T. In addition, the defect location management unit 52 manages the correspondence between the location of any point of the defective location accepted in the UI and the defect details information.

Description

Technical Field

[0006] ,

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] As a construction-related information management system using Building Information Modeling (BIM), the technology of Patent Document 1 below is known. [[ID=1十三]]

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] BIM is a tool that can not only be used as a three-dimensional design tool but also manage a series of processes from the planning and design stages of a building to the maintenance management stages such as construction and maintenance in a unified manner. However, for example, in the maintenance management of a building, in the conventional technologies including the technology of Patent Document 1, there has been a situation where the results of defective parts and the like inspected by an inspector on-site cannot be efficiently and effectively managed.

[0005] The present invention has been made in view of such a situation, and an object thereof is to provide a technology capable of efficiently and effectively managing, for example, inspection results on-site related to the maintenance management of a building.

Means for Solving the Problems

[0006] To achieve the above object, an information processing apparatus according to an aspect of the present invention is In an information processing apparatus that manages the content of defects in a building, A UI control means that controls a UI (User Interface) that displays a 3D model of the building to be inspected on the inspector's terminal, accepts a point in the 3D space of the 3D model as a defective area through an operation specified by the inspector, and accepts information about the defective area through an operation entered by the inspector. A defect location management means that manages the location of any point in the defect location received in the UI in association with the defect content information, It is equipped with.

[0007] Each of the information processing method and program according to one aspect of the present invention corresponds to each of the method and program corresponding to the information processing apparatus according to one aspect of the present invention. [Effects of the Invention]

[0008] According to the present invention, it becomes possible to efficiently and effectively manage, for example, the results of on-site inspections related to the maintenance and management of buildings. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an overview of the service that can be realized by an information processing system to which a server according to one embodiment of the information processing device of the present invention is applied. [Figure 2] This figure shows an example of the configuration of an information processing system to which a server according to one embodiment of the information processing device of the present invention is applied. [Figure 3] Figure 2 is a block diagram showing an example of the server hardware configuration in the information processing system. [Figure 4] This is a functional block diagram showing an example of the functional configuration of the server in Figure 3 that constitutes the information processing system in Figure 2. [Figure 5] This flowchart shows an example of the input process by inspectors when discovering building defects. [Figure 6]This figure shows an example of a UI screen output from a server having the functional configuration of Figure 4 and displayed, for example, on an inspector's terminal, and is an example of a screen where the selection of the building defect input page is possible. [Figure 7] This figure shows an example of a UI screen output from a server having the functional configuration of Figure 4 and displayed, for example, on an inspector's terminal, where, for example, the user can select the floor number, direction, and classification keyword of the object where the defect is located. [Figure 8] This figure shows an example of a UI screen output from a server having the functional configuration of Figure 4 and displayed, for example, on an inspector's terminal, where the user can select the object in which the defect is located from the displayed candidate list. [Figure 9] This is a magnified view of the object shown in Figure 8. [Figure 10] This figure shows an example of a UI screen output from a server with the functional configuration of Figure 4 and displayed, for example, on an inspector's terminal, and is an example of a screen when an object where a defect is located is selected. [Figure 11] This is a magnified view of the object shown in Figure 10. [Figure 12] This figure shows an example of a UI screen output from a server with the functional configuration of Figure 4 and displayed, for example, on an inspector's terminal, and is an example of a screen after the display of objects has been adjusted. [Figure 13] This figure shows an example of a UI screen output from a server having the functional configuration shown in Figure 4, which is displayed, for example, on an inspector's terminal, and is an example of a screen after the faulty locations have been plotted. [Figure 14] This figure shows an example of a UI screen output from a server with the functional configuration of Figure 4, which is displayed, for example, on an inspector's terminal, specifically an example of the screen just before plotting the faulty locations on the ceiling of an object. [Figure 15] This figure shows an example of a UI screen output from a server with the functional configuration of Figure 4, which is displayed, for example, on an inspector's terminal, specifically an example of the screen just before plotting the faulty areas on the floor of the object. [Figure 16]This is an example of a UI screen output from a server having the functional configuration of FIG. 4 and displayed, for example, on an inspector terminal, showing an example of a screen for inputting the details of a defective part and the like, and an example of a screen when the confirmation button is operated. [Figure 17] This is a diagram showing an example of a UI screen output from a server having the functional configuration of FIG. 4 and displayed, for example, on a viewer terminal. [Figure 18] This is a list diagram of icon variations for the second embodiment of the present invention shown in FIGS. 1 to 4. [Figure 19] This is a diagram showing a comparison screen between the first embodiment and the second embodiment of the present invention shown in FIGS. 1 to 4. [Figure 20] This is an ortho-image mapping concept diagram for the second embodiment of the present invention shown in FIGS. 1 to 4. [Figure 21] This is a diagram showing an image generation flow by SfM for the second embodiment of the present invention shown in FIGS. 1 to 4. [Figure 22] This is a diagram showing an example of a viewing screen for the second embodiment of the present invention shown in FIGS. 1 to 4.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0011] First, referring to FIG. 1, an overview of a service (hereinafter referred to as "this service") that can be realized by an information processing system (see FIG. 2 described later) to which a server according to an embodiment of the information processing apparatus of the present invention is applied will be described. FIG. 1 is a diagram showing an overview of this service that can be realized by an information processing system to which a server according to an embodiment of the information processing apparatus of the present invention is applied.

[0012] This service is a service that enables efficient and effective management of, for example, on-site inspection results related to the maintenance and management of buildings. Specifically, the service shown in Figure 1 allows inspector T to input inspection results, such as identifying defects, obtained during on-site inspections, and manage this inputted inspection result information. Furthermore, the service can output the inspection result information upon request from viewer E.

[0013] For example, in the example shown in Figure 1, the service is provided by the service provider ST to the inspector T and the viewer E, etc. Therefore, in the example service shown in Figure 1, a server 1 managed by the service provider ST, an inspector terminal 2 managed by the inspector T, and a viewer terminal 3 managed by the viewer E are used. The service shown in the example in Figure 1 is connected to a maintenance management system (other system 4 in Figure 2) that utilizes a building information model (BIM model (3D model)) constructed with BIM (Building Information Modeling). This system allows for the input and output of inspection results information, such as defects, using the building information model.

[0014] In step S1 of the example of this service shown in Figure 1, Server 1 enables a UI (User Interface) on Inspector Terminal 2 that can display a 3D model of the object to be inspected on the Inspector Terminal 2 screen, and accept any point from Inspector Terminal 2 as the location of a defect. Specifically, the inspector terminal 2 displays a 3D model of the building to be inspected, and the UI is configured so that inspector T can specify any point in the three-dimensional space within the 3D model as the location of a defect, and accept this operation. Furthermore, the UI will be designed to allow inspector T to input information about the nature of the defects in the affected areas. Server 1 enables a user interface (UI) to allow inspector T to input inspection result information, such as the location of defects obtained during on-site inspections.

[0015] Next, in step S2, server 1 manages the arbitrary points received as faulty locations from inspector terminal 2 by associating them with information about the fault. In other words, Server 1 can manage and associate any point with inspection result information. By having Server 1 function in this way, it becomes possible to manage, for example, the results of on-site inspections related to the maintenance of buildings more efficiently and effectively than before.

[0016] Next, in step S3, server 1 activates a UI on viewer terminal 3 that displays, for example, a predetermined symbol at the location of the defect as an arbitrary point, or displays defect details information near the predetermined symbol. This allows server 1 to output the location of the defect and inspection results information in response to a request from viewer E. In other words, by providing this service, viewer E can obtain information on the location of defects and inspection results related to defects in buildings in a more efficient and effective manner than before.

[0017] Although not specifically shown in Figure 1, the aforementioned 3D model can be composed of multiple objects. Furthermore, the aforementioned UI can accept input from inspector T through an operation that allows them to select an object from among multiple objects that make up the 3D model, where any point of the defect is located. Furthermore, the aforementioned UI can accept the operation specified by inspector T, allowing them to plot any point in the 3D space within the selected object as the location of the defect. Furthermore, the aforementioned UI can display the defect input page on the inspector terminal 2, and after inspector T inputs the defect details in text, it can accept that text as at least part of the defect information. Furthermore, this service can also accept images such as photographs (photo data) of the defective area as information about the defect.

[0018] Next, with reference to Figure 2, we will describe the configuration of an information processing system to which an information processing system that realizes the provision of the above-mentioned service, namely an information processing system to which a server according to one embodiment of the information processing device of the present invention is applied. Figure 2 shows an example of the configuration of an information processing system to which a server according to one embodiment of the information processing device of the present invention is applied.

[0019] The information processing system shown in Figure 2 is configured to include a server 1, an inspector terminal 2, and a viewer terminal 3. Server 1, inspector terminal 2, and viewer terminal 3 are interconnected via a network such as the Internet.

[0020] Server 1 is an information processing device managed by the service provider ST (see Figure 1) of this service. Server 1 performs various processes necessary to realize this service while communicating with the inspector terminal 2 and the viewer terminal 3 as needed. Server 1, in this context, performs various processes necessary to realize this service, while communicating as needed with maintenance and management systems (other systems 4) such as the BIM maintenance system mentioned above.

[0021] The inspector terminal 2 is an information processing device operated by inspector T (see Figure 1) who inspects the building to be inspected, and consists of a smartphone, tablet, personal computer, etc. Inspector terminal 2 is assumed to be managed by n inspectors T (where n is an integer greater than or equal to 1), and each terminal is assigned a designation 2-1 through 2-n. When individual distinction is not necessary, it is simply referred to as inspector terminal 2.

[0022] Viewer terminal 3 is an information processing device operated by viewer E (see Figure 1) who wishes to view the inspection details information, etc., related to the aforementioned building, and is composed of a smartphone, tablet, personal computer, etc. Here, each viewer terminal 3 is managed by m viewers E (where m is an integer greater than or equal to 1), and is assigned the designations 3-1 through 3-m. When individual distinction is not necessary, it is simply referred to as viewer terminal 3.

[0023] Here, inspector T can view the inspection details and other information related to the building mentioned above by operating inspector terminal 2. In other words, server 1 can make the UI described above function for inspector terminal 2 as well as for viewer terminal 3.

[0024] Figure 3 is a block diagram showing an example of the server hardware configuration in the information processing system shown in Figure 2.

[0025] Server 1 is comprised of a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an input unit 16, an output unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0026] The CPU 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 18 into the RAM 13. RAM13 also stores data and other information necessary for the CPU11 to perform various processes.

[0027] The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14. The input / output interface 15 is connected to an input unit 16, an output unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0028] The input unit 16 is composed of, for example, a keyboard, and accepts input of various types of information. The output unit 17 consists of a display such as an LCD and a speaker, and outputs various information as images and sounds. The memory unit 18 is composed of DRAM (Dynamic Random Access Memory) and stores various types of data. The communication unit 19 communicates with other devices (for example, the inspector terminal 2 and the viewer terminal 3 in Figure 2) via a network NW, including the Internet.

[0029] A removable media 30, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately mounted in the drive 20. Programs read from the removable media 30 by the drive 20 are installed in the storage unit 18 as needed. Furthermore, the removable media 30 can store various types of data stored in the storage unit 18, just as the storage unit 18 does.

[0030] Although not shown in the diagram, the inspector terminal 2 and viewer terminal 3 in Figure 2 (and other systems 4 as well) can have a configuration that is basically the same as the hardware configuration shown in Figure 3. Therefore, the explanation of the hardware configuration of inspector terminal 2 and viewer terminal 3 is omitted.

[0031] Through the cooperation of various hardware and software components that make up the information processing system in Figure 2, including Server 1 in Figure 3, various processes for providing the service in Figure 1 can be executed.

[0032] Figure 4 is a functional block diagram showing an example of the functional configuration of the server in Figure 3 within the information processing system shown in Figure 2.

[0033] As shown in Figure 4, the CPU 11 of server 1 functions as follows: UI control unit 51, fault location management unit 52, and fault content viewing control unit 53. Furthermore, one area of ​​the storage unit 18 of server 1 is provided with inspection target information DB71, UI information DB72, and fault location management information DB73. The Inspection Target Information DB71 stores various BIM-related information, including a 3D model of the building to be inspected (described later). (Here, it is assumed that various information is stored in the Inspection Target Information DB71, but this is not limited to this; the 3D model and other information may be provided as needed through other systems 4.)

[0034] The UI control unit 51 performs control to enable a UI (User Interface) that accepts at least an arbitrary point as a faulty location and information about the nature of the fault at that location. Specifically, the UI control unit 51 displays a 3D model of the building to be inspected (for example, a model composed of multiple objects as shown in Figure 14) on the inspector terminal 2, and performs control to enable a UI that can be operated by the inspector T, specifying an arbitrary point in the three-dimensional space within the 3D model as the location of the defect. Furthermore, the UI control unit 51 performs control to enable the UI to function, which can also receive information about the nature of the defect in the defective area through an operation input by the inspector T. The arbitrary point (for example, the defect location P described later) and defect details received by the UI control unit 51 are stored in the defect location management information DB 73 by the defect location management unit 52, which will be described later. Information related to the UI is stored in the UI Information DB 72 and managed as appropriate by the UI Control Unit 51.

[0035] The UI can accept an operation in which inspector T selects an object from among the multiple objects that make up the aforementioned 3D model, in which any point of the defect is located. Furthermore, the UI can also accept an operation in which inspector T specifies the location of a defect by plotting an arbitrary point in the 3D space within the selected object. Furthermore, the UI allows for adjustment of the display by appropriately using sliders to horizontally cut objects and left / right rotation buttons for the model display. Furthermore, the UI can display a defect details input page (see Figure 16), which will be described later, and after inspector T enters the defect details in text, it can accept that text as at least part of the defect details information. Furthermore, the UI can accept, in addition to the text mentioned above, photos (images) taken by inspector T as part of the defect information on the defect input page. Videos are also acceptable, not just photos.

[0036] The defect location management unit 52 associates any point of the defect location received in the UI with defect content information, and executes control to store and manage this in the defect location management information DB 73. As described above, the UI control unit 51 and the fault location management unit 52 function to enable efficient and effective management of, for example, on-site inspection results related to the maintenance of buildings.

[0037] The malfunction details viewing control unit 53 executes control to display malfunction details information on the viewer terminal 3. Specifically, the defect details viewing control unit 53 displays the aforementioned 3D model on the viewer terminal 3, and executes control to display a predetermined symbol at any point (defect location) of the defect within the 3D model. It also executes control to display defect details information in the vicinity of the displayed predetermined symbol. Furthermore, the UI described above, controlled by the UI control unit 51, also functions for displaying 3D models, predetermined symbols, and information about defects.

[0038] With the further functioning of the defect details viewing control unit 53, the server 1 can output the location of the defect and inspection result information related to the defect in response to a request from viewer E. In other words, viewer E can obtain information on the location of defects in the building and inspection results via viewer terminal 3. This acquisition is, of course, more efficient and effective than before.

[0039] Referring to Figure 5, we will explain the input process by inspector T when a building defect is discovered. Figure 5 is a flowchart illustrating an example of the input process by an inspector when a building defect is discovered.

[0040] Inspector T operates inspector terminal 2 to input various information after performing the prescribed inspection on-site, or during the inspection. At this time, the inspector terminal 2 will display a predetermined screen as the aforementioned UI functions (the specific screen display will be described later, referring to Figure 6 and subsequent figures).

[0041] In step S11 shown in Figure 5, select, for example, the "Building Defect Input Page" on the top page (you can also select the daily report input page or the exterior defect input page, which will be described later. Here, we will explain the case where you select the building defect input page). Next, in step S12, select the "floor number" (optional), "direction" (optional), and "classification keyword" of the object where the defect is located. Next, in step S13, select the object in which the defect is located from the displayed list of candidates. Next, in step S14, adjust the display by using the slider to cut the selected object horizontally and the left / right rotation buttons for the model display as appropriate. Next, in step S15, the specific location of the defect is plotted by tapping the model. Next, in step S16, you will proceed to the page for entering the details of the problem and enter the required information. Finally, in step S17, the input content is reviewed and submitted.

[0042] The following describes specific examples of the input process by inspector T when a building defect is discovered, and explains the functions of the UI control unit 51 and the defect location management unit 52, with reference to Figures 6 to 16. Figure 6 shows an example of a UI screen output from a server having the functional configuration shown in Figure 4 and displayed, for example, on an inspector's terminal. This figure shows an example of a screen where, for example, the building defect input page can be selected.

[0043] When the UI control unit 51 is activated, the inspector terminal 2 displays screen HG as shown in Figure 6. Specifically, the main UI display area UH, which is the main display area of ​​the UI, displays "Please select an input page," and here the inspector can select one of the buttons "Daily report input page," "Building defect input page," or "Exterior defect input page." For example, when the "Building Defect Input Page" button is selected, a screen HG as shown in Figure 7 is displayed on the inspector terminal 2.

[0044] Figure 7 shows an example of a UI screen output from a server having the functional configuration shown in Figure 4 and displayed, for example, on an inspector's terminal. This example shows a screen where, for example, the floor number, direction, and classification keyword of the object where the defect is located can be selected.

[0045] In Figure 7, the main UI display area UH shows a 3D model M1 representing the entire building. Note that the 3D model M1 is composed of multiple objects. The main UI display area UH also includes various buttons as shown in Figure 7. Although not explained here, a "Go to Information Viewing Page" button is also provided for viewing information. When the "floor number" of the object where the defect is located is selected, for example, "1FL" (1st floor), a screen HG as shown in Figure 8 is displayed on the inspector terminal 2.

[0046] Figure 8 shows an example of a UI screen output from a server having the functional configuration shown in Figure 4 and displayed, for example, on an inspector's terminal. This example shows a screen where the object in which the defect is located can be selected from the displayed candidate list. Figure 9 is a magnified view of the object shown in Figure 8.

[0047] In Figures 8 and 9, the main UI display area UH shows a 3D model M2 of a building that clearly indicates the specified "1FL". The main UI display area UH also includes various buttons, as shown in Figure 8. For example, if there is a defect in object OB1 shown in Figure 9, then in the main UI display area UH in Figure 8, "Entrance Hall" will be selected for the "Room (Part) Name" item.

[0048] Regarding the selection of "Entrance Hall," the inspector terminal 2 displays screen HG (main UI display area UH) as shown in Figure 10. Figure 10 shows an example of a UI screen output from a server having the functional configuration shown in Figure 4 and displayed, for example, on an inspector's terminal. It is an example of a screen displayed when an object containing a defect is selected. Figure 11 is a magnified view of the object shown in Figure 10. Figures 10 and 11 show the 3D model M3 relating to object OB1 (entrance hall).

[0049] Figure 12 shows an example of a UI screen output from a server having the functional configuration shown in Figure 4 and displayed, for example, on an inspector's terminal, and is an example of a screen after the display of objects has been adjusted. Regarding object OB1 (entrance hall), if you use the slider function described above to cut object OB1 horizontally, for example at the floor level, the 3D model M4 with the display adjusted will be displayed as shown in Figure 12. When you press the "Rotate Left" button, object OB1, which is horizontally cut at the floor level, rotates to the left, and when you press the "Rotate Right" button, it rotates to the right.

[0050] Figure 13 shows an example of a UI screen output from a server having the functional configuration shown in Figure 4 and displayed, for example, on an inspector's terminal, and is an example of a screen after the faulty locations have been plotted. By enlarging the 3D model M5 in Figure 13, it becomes easier to input the location of the defect. Inspector T can plot the location of the defect at the floor F by, for example, tapping on the enlarged 3D model M5. Here, the UI control unit 51 receives location information of the defective area, and as a result, the defective location P can be plotted on the 3D model M5. Alternatively, instead of plotting the defect location P by tapping, the defect location can also be identified by inputting details of the defect (see, for example, Figure 16).

[0051] Figure 14 shows an example of a UI screen output from a server having the functional configuration shown in Figure 4 and displayed, for example, on an inspector's terminal. It is an example of the screen just before plotting the faulty locations on the ceiling of an object. If object OB1 is displayed as shown in Figure 14 without using the slider function (without cutting horizontally), the defect location P (see Figure 13) can be plotted at the ceiling position of object OB1. In addition to the ceiling position, the plots in Figure 14 can also be placed on the side walls.

[0052] Figure 15 shows an example of a UI screen output from a server having the functional configuration shown in Figure 4 and displayed, for example, on an inspector's terminal. It is an example of the screen just before plotting the faulty areas on the floor of the object. By using the slider function to cut horizontally, object OB1 can be displayed as shown in Figure 15, and by tapping on object OB1 at, for example, the floor F position, the defect location P (see Figure 13) can be plotted. Figure 15 shows how plotting onto floor F is possible by cutting object OB1 to the very bottom edge.

[0053] Figure 16 shows examples of UI screens output from a server having the functional configuration of Figure 4 and displayed, for example, on an inspector's terminal. These examples include a screen for inputting details of the defect location and a screen displayed when the confirmation button is pressed. The aforementioned "Building Defect Input Page" also allows input from screen HG as shown in Figure 16. In other words, for example, in the case of the "○○○ Terrace Inspection Sheet" shown on the left side of Figure 16, input is possible using "Inspection Item (Required) (Automatic input based on Figures 7 to 13, and manual input via pull-down menu)", "Defect Location (Optional) (Manual input via keyboard)", "Defect Status (Required) (Manual input via keyboard)", "Judgment (Required) (Manual input via radio buttons)", and "Defect Photo (Optional) (Addition of photo)" (this corresponds to step S16 above). Once inspector T has finished entering the data and pressed the "Confirm" button, the screen changes to HG shown on the right side of Figure 16, and it becomes possible to "modify" or "submit" the "○○○ Terrace Inspection Sheet" (corresponding to step S17 above). The entered information (such as arbitrary points and defect details) is matched as described above by the defect location management unit 52 and then stored in the defect location management information DB 73.

[0054] Figure 17 shows an example of a UI screen output from a server with the functional configuration shown in Figure 4 and displayed, for example, on a viewer's terminal. Viewer E can check the location and status of defects via the viewer terminal 3 using functions such as the defect content viewing control unit 53. Specifically, the location and status of defects can be checked on the screen using a 3D model as shown in Figure 17, for example, a predetermined pin-shaped symbol, defect content information, etc. Note that while Figure 17 shows the screen displaying the bug history, it is also possible to check only "Bug 12," for example, or to view it as an incomplete list. The presence of a padlock symbol near the aforementioned pin-shaped symbol indicates that the information is locked.

[0055] As explained above with reference to Figures 1 through 17, this service allows inspector T to manage, for example, the results of on-site inspections related to the maintenance of buildings more efficiently and effectively than conventional, for example, analog management methods. Furthermore, this service allows viewer E to acquire information on defective areas and inspection results related to buildings more efficiently and effectively than conventional methods.

[0056] Next, referring to Figure 18, we will explain the variations in icons indicating the location of defects, which are implemented by the defect content viewing control unit 53 in Figure 4. Figure 18 is a diagram showing the icon variations for the second embodiment shown in Figures 1 to 4.

[0057] As shown in Figure 18, in this second embodiment, a predetermined symbol indicating a defective area is used, which is identifiable by at least one of different shapes or different colors depending on the room's constituent parts. Specifically, as shown in Figure 18(a), there are three patterns for identification based on the room's constituent parts. Firstly, for example, the floor icon has a circular shape, is blue in color, and has a downward-pointing arrow. Secondly, for example, the ceiling icon has a circular shape, is orange in color, and has an upward-pointing arrow. Thirdly, for example, a wall icon has a rectangular shape and is green in color. These icons may be identifiable by shape alone (for example, all the same color but different shapes), by color alone (for example, all the same shape but different colors), or by both shape and color.

[0058] Furthermore, as shown in Figure 18(a), for example, a wall icon can be provided with a directional indicator showing six directions (up, down, east, west, north, south) depending on the orientation of the wall. Specifically, for example, icon 81 indicating downward (floor) has a blue circle and a downward arrow. Icon 82 indicating upward (ceiling) has an orange circle and an upward arrow. Icon 83 indicating east (east wall) has a green square and a rightward arrow. Icon 84 indicating west (west wall) has a green square and a leftward arrow. Icon 85 indicating south (south wall) has a green square and a downward arrow. Icon 86 indicating north (north wall) has a green square and an upward arrow.

[0059] Furthermore, as shown in Figure 18(c), identification is also possible by combining it with a body part. Specifically, for example, even for the same room component (such as a wall or ceiling), the icon representation can be changed depending on the type of detailed component installed in that component, such as a window, door, or equipment. For example, for a wall icon, if a window is installed, it can be a nested icon with a green square and a small light blue square superimposed on top, and if a door is installed, it can be a nested icon with a green square and a small brown square superimposed on top. Also, for example, for a ceiling icon, if equipment such as an air conditioner is installed, it can be a nested icon with an orange circle and a small gray circle superimposed on top. In this way, by nesting elements of different colors and shapes that represent detailed parts within a basic room component icon (for example, a green square or an orange circle), it becomes possible to visually represent further detailed classifications within the same room component. Furthermore, by combining this with the orientation information (6 directions) shown in Figure 18(a) above, it is possible to simultaneously represent three pieces of information—room configuration, orientation, and detailed parts—with a single icon, such as "a window installed on an east-facing wall" or "air conditioning equipment installed on the ceiling."

[0060] As described above, the defect content viewing control unit 53 in Figure 4 displays icons on the viewer terminal 3 that have at least one of different shapes or colors for each room component (e.g., floor, ceiling, wall), and further represents orientation information (e.g., 6 directions) with arrows or cones, and in addition represents detailed parts (e.g., windows, doors, equipment, etc.) in a nested structure, so that the viewer E can instantly and visually identify the location and orientation of the defect. This results in a significant improvement in visibility compared to simple pin displays.

[0061] Next, a comparison between the first embodiment and the second embodiment will be described with reference to Figure 19. Figure 19 is a diagram showing a comparison screen of the second embodiment shown in Figures 1 to 4 with that of the first embodiment.

[0062] As shown on the left side of Figure 19, in this first embodiment, only a simple pin display was used. This allowed for efficient input and viewing of management and inspection results by associating the location of defects with defect content information on the BIM 3D model. However, because only a simple pin display was used, although it was simple and highly useful, it was unclear which part (floor, wall, ceiling) the defect was in, and it was difficult to identify which direction the wall was facing. Furthermore, all the icons were the same and indistinguishable, making it difficult to grasp the details of the location. As a result, viewer E, who was operating viewer terminal 3 in Figure 2, had to spend time checking details, and inspector T, who was operating inspector terminal 2 in Figure 2, also had to spend time explaining, which led to a decrease in the efficiency of maintenance work.

[0063] In contrast, as shown on the right side of Figure 19, in this second embodiment, the malfunction content viewing control unit 53 in Figure 4 causes an identifiable icon to be displayed on the viewer terminal 3. Specifically, for example, an icon with a blue circle and a downward-pointing arrow is used for the floor. For example, an icon with an orange circle and an upward-pointing arrow is used for the ceiling. For example, an icon with a green square and a directional arrow is used for the wall. Furthermore, by using icons such as a green square with a small light blue square superimposed on it for walls with windows, a green square with a small brown square superimposed on it for walls with doors, and an orange circle with a small gray circle superimposed on it for ceilings with equipment, it becomes possible to identify even the finer details.

[0064] This makes it possible to identify floors (e.g., blue, circle, down arrow), ceilings (e.g., orange, circle, up arrow), walls (e.g., green, square), etc., and for example, the six directions (up, down, east, west, north, south) are clearly indicated with arrows. Furthermore, detailed parts such as windows, doors, and equipment can also be identified through a nested structure, making it possible to distinguish them at a glance by shape, color, orientation, and detailed parts, and making it easy to identify defective areas. As a result, viewer E, operating viewer terminal 3 in Figure 2, can instantly grasp the details, the burden of explanation for inspector T, operating inspector terminal 2 in Figure 2, is reduced, and a significant improvement in the efficiency of the maintenance work in Figure 1 is achieved.

[0065] Next, with reference to Figure 20, we will explain the orthomosaic image mapping method implemented by the defect content viewing control unit 53 in Figure 4. Figure 20 is a conceptual diagram of orthomosaic mapping for the embodiment shown in Figures 1 to 4.

[0066] As shown in Figure 20, in this embodiment, an orthomosaic image 87 generated from a photograph of an actual room can be mapped and displayed on the surface of an object that constitutes a 3D model (for example, the architectural information model used in the service in Figure 1). There are two methods for mapping orthomosaic images 87: Method 1 and Method 2.

[0067] Method 1 is a method of directly mapping the room volume object OB1. As shown in the upper left of Figure 20, in the state before mapping, the room volume object OB1 consists only of a single-colored surface. This object OB1 is the same as the object OB1 shown in Figures 7 to 15. In contrast, as shown in the upper right of Figure 20, after mapping, the defect content viewing control unit 53 in Figure 4 maps the orthomosaic image 87 to each surface and displays it on the viewer terminal 3 in Figure 2. This orthomosaic image 87 is generated from an actual photograph of the room.

[0068] Method 2 involves mapping to a plate-like object 88 created on the interface. As shown in Figure 20, plate-like objects 88 are placed on each boundary surface. For example, plate-like objects 88 are placed on the floor surface and the wall surface. Next, for example, as shown in Figure 20, the defect content viewing control unit 53 in Figure 4 maps the orthomosaic image 87 onto the plate-shaped object 88 and displays it on the viewer terminal 3 in Figure 2. With this method, the original room volume object OB1 does not need to be changed.

[0069] In this way, by mapping the orthomosaic image 87, the actual room photograph is displayed as a texture, making it possible to visually understand the surrounding environment of the problem area, clarifying the location information ("approximately where"), and making it easier to compare with memories from the inspection. Furthermore, Method 1 allows the use of existing objects, while Method 2 allows for flexible updates. Furthermore, to adjust the display of object OB1, the object can be horizontally cut using the slider function shown in Figure 12. This model cutting function makes it possible to properly visualize the mapping state of the orthomosaic image 87 onto the floor and ceiling surfaces.

[0070] Furthermore, by utilizing the mapped orthomosaic images 87, it becomes possible to more efficiently display icons for defective areas. Specifically, for example, when inspector T operates inspector terminal 2 in Figure 2 to plot a defect location at an arbitrary point on the 3D model, the UI control unit 51 in Figure 4 can extract the image region of the orthomosaic image 87 corresponding to the plotted location and perform AI-based image recognition processing on that image region. This image recognition process allows for the automatic identification of the type of object (e.g., window, door, air conditioning equipment, lighting equipment, flooring, wall materials, ceiling materials, etc.) present in the plotted location. Based on the type of object identified, the UI control unit 51 in Figure 4 can automatically select and display an appropriate icon from the icons shown in Figure 18 (for example, if a window is identified, an icon with a green rectangle for walls and a small light blue rectangle superimposed on it; if air conditioning equipment is identified, an icon with an orange circle for ceilings and a small gray circle superimposed on it). This eliminates the need for inspector T to manually select the type of detailed part from a pull-down menu, thereby improving the efficiency of the input process. Furthermore, the defect location management unit 52 in Figure 4 can automatically store the type of object identified by the AI ​​as part of the defect details information in the defect location management information DB 73 in Figure 4. Furthermore, existing machine learning models (such as convolutional neural networks) can be used for image recognition processing, and by pre-training them with image data of various parts and equipment of buildings as training data, highly accurate discrimination becomes possible.

[0071] Next, with reference to Figure 21, we will describe the orthomosaic image generation flow performed by Server 1 or other system 4 in Figure 2. Figure 21 is a diagram showing the image generation flow by SfM for the embodiment shown in Figures 1 to 4.

[0072] In step S21, inspector T, who operates the inspector terminal 2 in Figure 2, uses the camera function of the inspector terminal 2 to take photographs 91 from multiple locations in the room, for example. The number of photos taken will be multiple (for example, 10 to 30). The subject of the photos will be the entire room, including the floor, walls, and ceiling. The captured photograph 91 is sent to server 1 via the network NW shown in Figure 2.

[0073] Next, in step S22, a three-dimensional point cloud model 90 is generated from multiple photographs 91 by an SfM processing engine 89 (not shown) in either the server 1 shown in Figure 2 or another system 4 connected to the server 1 shown in Figure 2. The SfM processing engine 89 performs feature point extraction, camera position estimation, and 3D reconstruction. Structure from Motion (SfM) is a technique that simultaneously estimates the camera's position and orientation, as well as its 3D structure, from a series of 2D images taken from multiple viewpoints. It does not require special equipment and can generate 3D models from photographs taken with a standard camera. Furthermore, it is also possible to generate a 3D point cloud model using LiDAR (Light Detection and Ranging) instead of SfM, or in combination with SfM. LiDAR is a technology that measures the distance to an object by irradiating it with laser light, and by using the LiDAR sensor installed in some smartphones such as the iPhone® Pro, it is possible to generate a 3D point cloud model with higher accuracy and speed. Furthermore, this SfM processing engine 89 (or LiDAR processing engine) may be implemented as software executed by the CPU 11 of server 1 in Figure 3, or as a dedicated processing unit included in another system 4.

[0074] Next, in step S23, orthomosaic images 87 corresponding to each room component (floor, walls, ceiling) are extracted from the generated 3D point cloud model 90. The images to be extracted are orthophotos of the floor, each wall, and the ceiling. The generated orthomosaic image 87 is stored in the inspection target information DB 71 of the storage unit 18 in Figure 4, and is mapped to a 3D model in the manner shown in Figure 20.

[0075] Next, referring to Figure 22, we will explain the improved viewing screen that is displayed on the viewer terminal 3 in Figure 2 by the malfunction content viewing control unit 53 in Figure 4. Figure 22 shows an example of the browsing screen for the second embodiment shown in Figures 1 to 4.

[0076] As shown in Figure 22, the improved viewing screen displays both a 3D model display area and an information panel. The 3D model display area displays a 3D model of a room based on the architectural information model used in this service as shown in Figure 1, and the orthomosaic image 87 extracted in step S23 of Figure 21 is mapped and displayed on the surface of the 3D model. Furthermore, the defect information viewing control unit 53 shown in Figure 4 displays icons indicating the defect location on the 3D model. Specifically, a blue circular icon is displayed for the floor, an orange circular icon for the ceiling, and a green square icon for the walls.

[0077] The information panel displays the defect history list stored in the defect location management information DB73 shown in Figure 4. Each item in the defect history list displays, for example, the defect ID, date and time, inspection area, defect location, defect status, judgment, and photographic data, along with an icon as shown in Figure 18. For example, for defect 10, along with a floor icon, the inspection area is displayed as "Floor (downward)", the defect location as "Near the center of the entrance", the defect status as "Flooring material is lifting", and the judgment as "×Requires repair". For defect 11, the following is displayed: ceiling icon, inspection area is "ceiling (upward)", defect location is "around lighting fixture", defect status is "water leak traces", and judgment is "× requires repair". For defect 12, the following is displayed: wall icon, inspection area is "wall (eastward)", defect location is "upper part of east wall", defect status is "crack found", and judgment is "× requires repair".

[0078] Thus, compared to the first embodiment, the second embodiment achieves the following through the defect content viewing control unit 53 shown in Figure 4. Firstly, the components of a room (floor, ceiling, and walls) can be instantly identified by their shape and color. Secondly, orientation information (6 directions) is clearly indicated by arrows, making it immediately obvious which side has a defect. Thirdly, detailed parts such as windows, doors, and equipment can be identified through the nested structure. Fourth, mapping of orthomosaic images 87 makes it possible to visually understand the surrounding environment. Fifth, AI image recognition on orthomosaic images 87 enables automatic identification of detailed body parts and automatic selection of appropriate icons. Sixth, the icons in the bug history list will also be linked, making the correspondence clearer.

[0079] As described above, the first embodiment only provided a simple pin display, but in the second embodiment (Figures 18 to 22), the defect content viewing control unit 53 and UI control unit 51 in Figure 4 realize identification functions based on the shape, color, orientation, and detailed parts of icons, as well as orthomosaic image mapping and AI-based image recognition functions. As a result, viewer E operating the viewer terminal 3 in Figure 2 can instantly grasp the part, location, detailed parts, and surrounding environment of the defect, significantly improving the efficiency of information transmission in the building maintenance work shown in Figure 1.

[0080] Although one embodiment of the present invention has been described above, the present invention is not limited to the embodiments described above, and any modifications, improvements, etc. that can achieve the objectives of the present invention are considered to be included in the present invention.

[0081] In the embodiment described above, Server 1 listed Inspector Terminal 2 and Viewer Terminal 3 as terminals that enable the UI to function, but it is not limited to these, and other systems 4 may also be included to enable the UI to function. The UI could be an application that functions on the inspector terminal 2 or the viewer terminal 3, or it could be a UI that functions on a website managed by a server on another system 4, for example.

[0082] For example, the system configuration shown in Figure 2 and the hardware configuration of Server 1 shown in Figure 3 are merely illustrative examples for achieving the objectives of the present invention and are not particularly limited.

[0083] Furthermore, the functional block diagram shown in Figure 4 is merely illustrative and not particularly limiting. In other words, it is sufficient that the information processing system in Figure 2 has the functionality to execute the various processes described above as a whole, and the functional blocks and databases used to realize this functionality are not particularly limited to the example in Figure 4.

[0084] Furthermore, the location of the functional blocks and database is not limited to Figure 4, but can be any location. For example, at least a portion of the functional blocks and database located on server 1 may be provided on inspector terminal 2, viewer terminal 3, or other information processing device (not shown).

[0085] Furthermore, the series of processes described above can be executed by hardware or by software. Furthermore, a single functional block may consist of hardware alone, software alone, or a combination of both.

[0086] When a series of processes are executed by software, the programs that make up that software are installed on a computer or other device from a network or storage medium. The computer may be a computer that is built into dedicated hardware. Furthermore, a computer can be any computer capable of performing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0087] Such recording media containing programs consist not only of removable media (not shown) distributed separately from the main unit of the device to provide the program to the user, but also of recording media provided to the user in a state where they are pre-installed in the main unit of the device.

[0088] In this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually.

[0089] In summary, the information processing device to which the present invention applies only needs to have the following configuration, and can take various forms. In other words, the information processing device to which the present invention is applied (for example, Server 1 in Figures 1 to 4) is: In an information processing device for managing the details of defects in buildings, A UI control means (e.g., UI control unit 51 in Figure 4) executes control to enable a UI (User Interface) (e.g., UI shown in Figures 6 to 16) that displays a 3D model of the building to be inspected (e.g., 3D model M1 in Figure 7) on the inspector's terminal (e.g., inspector terminal 2 in Figures 1 and 2), and accepts an arbitrary point (e.g., defect location P in Figure 13) within the three-dimensional space of the 3D model (e.g., the three-dimensional space in 3D model M3 in Figure 10) as a defect location (e.g., the entrance hall where defect location P in Figure 13 is plotted) through an operation specified by the inspector, and accepts defect information (e.g., defect information entered in Figure 16) through an operation entered by the inspector. A defect location management means (for example, the defect location management unit 52 in Figure 4, etc.) manages the location of any point of the defect location received in the UI and the defect content information in association with each other, Having that will suffice.

[0090] Such an information processing device (for example, Server 1 in Figures 1 to 4) makes it possible to efficiently and effectively manage, for example, the results of on-site inspections related to the maintenance and management of buildings.

[0091] In an information processing device to which the present invention is applied (for example, Server 1 in Figures 1 to 4), The aforementioned 3D model is composed of multiple objects (for example, objects OB1 to OB3 in Figure 14), The aforementioned UI is The system accepts an operation in which the inspector selects an object (for example, object OB1 in Figure 11) from among the multiple objects that constitute the 3D model, in which the arbitrary point of the defect is located. The inspector specifies the location of the defect by plotting an arbitrary point in the three-dimensional space within the selected object (for example, by tapping the 3D model M5 in Figure 13). The system displays a page for inputting details of defects (for example, the building defect input page shown in Figure 16), and after the inspector inputs the details of the defects in text (for example, manual input as shown in Figure 16), the system accepts this text as at least part of the defect information. It is possible.

[0092] Furthermore, in an information processing device to which the present invention is applied (for example, Server 1 in Figures 1 to 4), The aforementioned UI further, After the inspector inputs an image of the defective area (for example, the photograph in Figure 16), the system accepts the image as at least part of the defect information. It is possible.

[0093] Furthermore, the information processing device to which the present invention is applied (for example, Server 1 in Figures 1 to 4, etc.) A defect content viewing control means (for example, defect content viewing control 53 in Figure 4) executes control to display the 3D model on a viewing terminal (for example, viewer terminal 3 in Figures 1 and 2), to display a predetermined symbol (for example, a pin-shaped one as shown in Figure 17) at an arbitrary point on the defect location within the 3D model, and to display defect content information (for example, the photo below 202410311556 near the pin-shaped predetermined symbol in Figure 17, or defect content information displayed as defect history on the right side of Figure 17) near the predetermined symbol. It can provide even more.

[0094] Furthermore, in an information processing device to which the present invention is applied (for example, Server 1 in Figures 2 to 4), The aforementioned defect content viewing control means (for example, the defect content viewing control unit 53 in Figure 4) Control can be performed to display the predetermined symbols (for example, icons 81 to 86 in Figure 18) in a way that allows them to be identified by at least one of different shapes (for example, circles and squares in Figure 18) or different colors (for example, blue, orange, and green in Figure 18) depending on the room component (for example, the floor, wall, and ceiling in Figure 18) where the defective area is located.

[0095] In this way, the components and orientation of the room can be visually identified instantly, making it easier to pinpoint faulty areas. Viewer E, operating the viewer terminal 3 in Figure 2, can instantly grasp the details, reducing the burden on inspector T, who is operating the inspector terminal 2 in Figure 2, and enabling a significant increase in the efficiency of the maintenance work shown in Figure 1. [Explanation of Symbols]

[0096] 1...Server, 2...Inspector terminal, 3...Viewer terminal, 11...CPU, 12...ROM, 13...RAM, 14...Bus, 15...Input / Output interface, 16...Input unit, 17...Output unit, 18...Storage unit, 19...Communication unit, 20...Drive, 30...Removable media, 51...UI control unit, 52...Defect location management unit, 53...Defect details viewing control unit, 71...Inspection target information DB, 72...UI Information DB, 73...Defect Location Management Information DB, 81...Icon (downward-facing, floor), 82...Icon (upward-facing, ceiling), 83...Icon (eastward-facing, east wall), 84...Icon (westward-facing, west wall), 85...Icon (southward-facing, south wall), 86...Icon (northward-facing, north wall), 87...Orthomosaic image, 88...Plate-shaped object, 89...SfM processing engine, 90...3D point cloud model, 91...Photograph

Claims

1. In an information processing device for managing the details of defects in buildings, A UI control means that controls a User Interface (UI) that displays a 3D model of the building to be inspected on the inspector's terminal, accepts a point in the three-dimensional space of the 3D model as a defective area through an operation specified by the inspector, and accepts information about the defective area through an operation entered by the inspector. A defect location management means that manages the location of any point of the defect location received in the UI in association with the defect content information, An information processing device equipped with the following features.

2. The aforementioned 3D model is composed of multiple objects, The aforementioned UI is The inspector selects an object from among the multiple objects that constitute the 3D model in which the arbitrary point of the defect is located, thereby receiving the object. The inspection operator specifies the location of the defect by plotting an arbitrary point in the three-dimensional space within the selected object, The system displays a page for inputting the details of a defect, and after the inspector enters the details of the defect in text, it accepts that text as at least part of the defect information. The information processing apparatus according to claim 1.

3. The aforementioned UI further, After the inspector inputs an image relating to the aforementioned defect, the system accepts the image as at least part of the defect information. The information processing apparatus according to claim 2.

4. A defect content viewing control means that performs control to display the 3D model on a viewing terminal, to display a predetermined symbol at an arbitrary point on the defect location within the 3D model, and to display defect content information near the predetermined symbol. The information processing apparatus according to claim 1, further comprising:

5. The aforementioned defect content viewing control means is The predetermined symbol is assigned according to the room configuration where the defective area is located. Controls that make them identifiable by at least one of a different shape or a different color. Execute The information processing apparatus according to claim 4.

6. In an information processing method executed by an information processing device that manages the details of defects in a building, A UI control step is performed to activate a UI (User Interface) that displays a 3D model of the building to be inspected on the inspector's terminal, accepts the inspector's operation to specify an arbitrary point in the three-dimensional space of the 3D model as a defective area, and accepts the inspector's operation to input information about the defective area. A defect location management step that manages the location of any point of the defect location received in the UI in association with the defect content information, Information processing methods including

7. A computer that manages the details of defects in buildings, A UI control step is performed to activate a UI (User Interface) that displays a 3D model of the building to be inspected on the inspector's terminal, accepts the inspector's operation to specify an arbitrary point in the three-dimensional space of the 3D model as a defective area, and accepts the inspector's operation to input information about the defective area. A defect location management step that manages the location of any point of the defect location received in the UI in association with the defect content information, A program that executes control processes, including those mentioned above.

Images