Information Processing System

Abstract

To provide an information processing system or the like capable of promoting digitization for a work site.SOLUTION: According to an embodiment of the present invention, an information processing system is provided. The information processing system includes a processor. In a site acquisition step, the processor acquires a measurement result which is obtained by a measuring device measuring a site where work is performed regarding a construction. The measuring device has a sensor for measuring an electro magnet wave reaching from an object. In an analysis step, prescribed analysis processing is executed on the basis of the acquired measurement result. In an accumulation step, results of the executed analysis processing are accumulated.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system. [Background technology]

[0002] Patent document 1 discloses a technology in which a person in charge of creating albums at a construction site takes photos of the construction work using a digital camera, transmits the taken construction photo data from a work site terminal to a server at a management and control center, and classifies and organizes the construction photos and enters information to explain the construction photos on the server side and / or on an input request terminal requested by the server, all while looking at the contents written on an explanatory blackboard displayed enlarged on the display unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-65321 A Summary of the Invention [Problem to be solved by the invention]

[0004] At construction sites and other work sites, digitization is desirable to improve work efficiency.

[0005] In view of the above circumstances, the present invention provides an information processing system and the like that can promote the digitization of information related to a work site. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an information processing system. The information processing system includes a processor, and in a site acquisition step, the processor acquires measurement results obtained by a measuring device measuring a site where work related to a structure is being performed, the measuring device having a sensor that measures electromagnetic waves arriving from an object. In a analysis step, the processor executes a predetermined analysis process based on the acquired measurement results. In a storage step, the processor stores the results of the executed analysis process.

[0007] According to this embodiment, it is possible to promote the creation of digital data relating to the work site. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing the overall configuration of a site management support system 1. FIG. [Diagram 2] 2 is a diagram showing a hardware configuration of the support server device 10. FIG. [Diagram 3] FIG. 2 is a diagram showing a hardware configuration of a field terminal 20. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of an imaging device 30. [Diagram 5] FIG. 2 is a diagram showing a hardware configuration of a supervisor terminal 40. [Figure 6] 2 is a diagram illustrating an example of a functional configuration of a control unit of the support server device 10. FIG. [Figure 7] FIG. 2 is a diagram illustrating an example of a functional configuration of a control unit of each device. [Figure 8] FIG. 11 is an activity diagram showing an example of captured image processing. [Figure 9] FIG. 13 is a diagram showing an example of a displayed area image. [Figure 10] FIG. 11 is a diagram illustrating an example of a specific image database. [Figure 11] FIG. 13 is a diagram showing an example of a displayed candidate image. [Figure 12] FIG. 11 is a diagram showing an example of displayed candidate information. [Figure 13] FIG. 13 is a diagram showing an example of displayed specific image data. [Figure 14]FIG. 13 is a diagram showing an example of a displayed area image. [Figure 15] FIG. 13 is a diagram illustrating an example of a flow diagram of an automatic extraction process. [Figure 16] FIG. 13 is a diagram showing an example of a displayed check image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic features shown in the following embodiments can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable recording medium, or may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] In addition, in this embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, in this embodiment, various information is handled, and this information is represented, for example, by physical values ​​of signal values ​​representing voltage and current, high and low signal values ​​as a binary bit collection consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be performed on the circuit in the broad sense.

[0012] In addition, a circuit in the broad sense is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, a memory, etc. In other words, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] 1. Hardware Configuration In this section, the hardware configuration of the construction support system according to this embodiment will be described.

[0014] Fig. 1 is a diagram showing the overall configuration of a site management support system 1. Fig. 1 shows an overview of each device provided in the site management support system 1 and users who use those devices. Each overview will be explained as appropriate with reference to other figures.

[0015] The site management support system 1 is an information processing system that executes processes to support the management of work at a site such as architecture, construction, or construction. The site management support system 1 includes a communication line 2, a selfie stick 3, an external battery 4, an external power source 5, a breaker 6, a support server device 10, a site terminal 20, an imaging device 30, a supervisor terminal 40, an operator terminal 50, and a business server device 60.

[0016] The communication line 2 includes the Internet, etc., and mediates data exchange between devices connected to the communication line. The support server device 10 and the business server device 60 are connected to the communication line 2 by wire, and the on-site terminal 20 and the supervisor terminal 40 are connected wirelessly. In this embodiment, the on-site terminal 20 communicates with the communication line 2 by mobile communication. Also, the on-site terminal 20 wirelessly communicates with the imaging device 30 using two communication methods. In this embodiment, the two communication methods are Wi-Fi communication and BLE (Bluetooth (registered trademark) Low Energy) communication.

[0017] The on-site terminal 20 and the imaging device 30 are terminals used by, for example, a site worker W1, installed at the work site, and execute processing related to on-site imaging. The supervisor terminal 40 is a terminal used by, for example, a work supervisor W2 in charge of the work site, and executes processing for managing the progress of work, etc. The operator terminal 50 is a terminal used by, for example, an operator W3 of the site management support system 1, and accepts various operations related to images captured by the imaging device 30.

[0018] The on-site terminal 20 is connected to the external power source 5 via the external battery 4 by a cable. The imaging device 30 is detachably connected to the external power source 5 via the external battery 4 by a cable and a connector 7. In other words, the external power source 5 also supplies power to the imaging device 30 which is detachably connected. The external battery 4 has a so-called pass-through function that allows it to supply power while charging, and when the breaker 6 is on, it supplies power to the on-site terminal 20 and the imaging device 30 while being charged with power supplied from the external power source 5.

[0019] The imaging device 30 is a digital camera equipped with an image sensor, and captures an image represented by light measured by the image sensor. In this embodiment, the imaging device 30 is a 360-degree camera (also called an omnidirectional camera or spherical camera) capable of capturing images in all directions, including up, down, left, right, front, and back. The imaging device 30 is attached to a selfie stick 3, and the selfie stick 3 can be inserted and fixed into a stand 8 installed at the work site.

[0020] When the site worker W1 removes the connector 7 and walks around the work site holding the selfie stick 3 removed from the stand 8, photographed image data is generated showing images captured by the 360-degree camera with the work site as the shooting area. In the site management support system 1, the images captured by the imaging device 30 are moving images in this embodiment, but may be still images captured continuously as long as images of various places in the work site can be obtained. The imaging device 30 transmits the generated photographed image data to the site terminal 20.

[0021] The on-site terminal 20 is a terminal that serves as a main user interface for the on-site worker W1, and is, for example, a smartphone. The on-site terminal 20 controls the operation of the imaging device 30, for example, by using one of the above two communication methods (BLE communication in this embodiment). The on-site terminal 20 also transfers the captured image data transmitted from the imaging device 30 by one of the above two communication methods (Wi-Fi communication in this embodiment) to the support server device 10 by using another wireless communication (mobile communication in this embodiment).

[0022] The support server device 10 performs image processing using the image of the work site indicated by the photographed image data transmitted from the on-site terminal 20, and generates area image data indicating, for example, an image of the work site as seen from the position where the image capturing device 30 captured the image as an image of the photographed area. The supervisor terminal 40 refers to the generated area image data and displays images of various parts of the work site. The work supervisor W2 grasps the situation of the work site from the displayed image of the work site, and gives instructions on the work to the on-site worker W1 as necessary.

[0023] Depending on the work site, the breaker 6 may be turned off after work is completed for reasons such as power saving. In this case, power is no longer supplied from the external power source 5 after the breaker 6 is turned off. In this manner, the external power source 5 can switch between supplying and not supplying power. The on-site terminal 20 and the imaging device 30 also have built-in batteries and therefore do not immediately stop, but some processes, such as transmitting image data, take time. Therefore, in this embodiment, an external battery 4 is provided to increase the operating time of the on-site terminal 20 and the imaging device 30 after the breaker 6 is turned off.

[0024] The operator terminal 50 accepts an operation by the operator W3 to extract a portion showing a specific location as a specific image from an image of the work site captured by the imaging device 30. In this embodiment, a location to be checked to confirm the progress of the work process is used as the specific location. The support server device 10 extracts the specific image based on the operation of the operator W3 on the operator terminal 50, and outputs specific image data showing the extracted specific image. In this embodiment, the specific image data is a checklist that lists the specific locations. The supervisor terminal 40 displays the output specific image data (checklist), and accepts, for example, an operation to check the work progress by the work supervisor W2.

[0025] 2 is a diagram showing a hardware configuration of the supporting server device 10. The supporting server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a bus 14. The bus 14 electrically connects the various units included in the supporting server device 10.

[0026] (Control unit 11) The control unit 11 is, for example, a central processing unit (CPU) not shown. The control unit 11 is a computer that realizes various functions related to the site management support system 1 by reading out a predetermined program stored in the storage unit 12. That is, information processing by software stored in the storage unit 12 can be specifically realized by the control unit 11, which is an example of hardware, and executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and may be implemented with multiple control units 11 for each function. Also, a combination of these may be used.

[0027] (Storage unit 12) The storage unit 12 stores various information defined by the above description. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the site management support system 1 executed by the control unit 11, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program calculations. The storage unit 12 stores various programs, variables, etc. related to the site management support system 1 executed by the control unit 11.

[0028] (Communications Department 13) The communication unit 13 is configured to be able to transmit various electrical signals from the supporting server device 10 to external components. The communication unit 13 is also configured to be able to receive various electrical signals from the external components to the supporting server device 10. More preferably, the communication unit 13 has a network communication function, and thus may be implemented to be able to communicate various information between the supporting server device 10 and external devices via the communication line 2.

[0029] The business server device 60, like the supporting server device 10, includes hardware such as a control unit 61 (only the control unit 61 is given a different reference number from that of the supporting server device 10).

[0030] Fig. 3 is a diagram showing a hardware configuration of the field terminal 20. The field terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, an internal power supply unit 26, and a bus 27. The bus 27 electrically connects the units included in the field terminal 20. The control unit 21 and the storage unit 22 are similar hardware to the control unit 11 and the storage unit 12 shown in Fig. 2, although their performance differs.

[0031] (Communications Department 23) The communication unit 23 includes a first communication unit 231, a second communication unit 232, and a third communication unit 233, and is an example of a wireless communication unit that performs three types of wireless communication. The first communication unit 231 performs wireless communication by Wi-Fi communication in this embodiment as the first wireless communication. The second communication unit 232 performs wireless communication by BLE in this embodiment as the second wireless communication having a slower communication speed and lower power consumption than the first wireless communication. The third communication unit 233 performs wireless communication by mobile communication in this embodiment as the third wireless communication having a wider communication area than the first wireless communication and the second wireless communication.

[0032] (Input section 24) The input unit 24 includes keys, buttons, a touch screen, a mouse, etc., and receives input from the user. (Output section 25) The output unit 25 has a display (including a touch screen) and a speaker, and displays visual information generated in a manner that is visible to the user, such as a screen, an image, an icon, text, etc., on the display surface, and outputs sound including voice.

[0033] (Internal power supply section 26) The internal power supply unit 26 is a battery built into the device itself, i.e., a repeatedly rechargeable battery, and supplies stored power to each unit of the device. The internal power supply unit 26 is an example of a portable battery that can be carried around together with the device itself. The internal power supply unit 26 is charged by power supplied from the external power supply 5. The internal power supply unit 26 has a pass-through function like the external battery 4, and when the breaker 6 is on, it supplies power to each unit while being charged by power supplied from the external power supply 5.

[0034] 4 is a diagram showing a hardware configuration of the imaging device 30. The imaging device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, an internal power supply unit 36, an imaging unit 37, and a bus 38. The bus 38 electrically connects the various units included in the imaging device 30. The various units from the control unit 31 to the internal power supply unit 36 ​​are similar hardware to the various units from the control unit 21 to the internal power supply unit 26 shown in FIG. 3, although their performance differs.

[0035] However, the communication unit 33 only includes a first communication unit 331 and a second communication unit 332. The first communication unit 331, like the first communication unit 231 of the communication unit 23, performs wireless communication by Wi-Fi communication as the first wireless communication in this embodiment. The second communication unit 332, like the second communication unit 232 of the communication unit 23, performs wireless communication by BLE as the second wireless communication having a slower communication speed and lower power consumption than the first wireless communication in this embodiment. The output unit 35 has a light in addition to a display and the like, and emits light to ensure the amount of light required for shooting. The input unit 34 has a switch for turning on the light.

[0036] (Imaging unit 37) The imaging unit 37 is a sensor that has an optical system including a lens, an image sensor, etc., and measures light incident from the lens to generate captured image data. As described above, in this embodiment, the imaging unit 37 uses a super wide-angle lens and multiple image sensors to generate captured image data captured in all directions, including up and down, left and right, and front and back.

[0037] 5 is a diagram showing the hardware configuration of the supervisor terminal 40. The supervisor terminal 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, an output unit 45, and a bus 46. The bus 46 electrically connects the various units included in the supervisor terminal 40. The various units from the control unit 41 to the output unit 45 are similar hardware to the various units from the control unit 31 to the output unit 35 shown in FIG. 4, although there are differences in performance. The operator terminal 50, like the supervisor terminal 40, includes hardware such as a control unit 51 (only the control unit 51 is given a different reference number from that of the supervisor terminal 40).

[0038] 2. Functional configuration In this section, the functional configuration of the present embodiment will be described. As described above, information processing by software stored in the storage unit of each device is specifically realized by a control unit, which is an example of hardware, so that each functional unit included in the control unit can be executed.

[0039] 6 is a diagram showing an example of a functional configuration of a control unit of the support server device 10. The control unit 11 of the support server device 10 includes a site acquisition unit 101, a task acquisition unit 102, a site analysis unit 103, a result accumulation unit 104, a dashboard unit 105, and a tool collaboration unit 106.

[0040] The site acquisition unit 101 is an example of a site acquisition unit that acquires measurement results obtained by a measuring device measuring a site where work related to a structure is being performed. A structure is constructed using a plurality of materials and supplies, and includes architectural structures, civil engineering structures, ships, and the like. The measuring device is a device having a sensor that measures electromagnetic waves arriving from an object, and in this embodiment, is an imaging device 30 equipped with an image sensor. The imaging device 30 outputs an image measured by the image sensor (hereinafter referred to as a "site image") as a measurement result. The site acquisition unit 101 acquires the output site image as a measurement result.

[0041] The work acquisition unit 102 is an example of a work acquisition unit that acquires work information related to the above-mentioned work. The work information includes on-site drawing information, a schedule, material information, component information, cost information, and environmental load information. This work information is stored in the business server device 60 as information related to the business. The on-site acquisition unit 101 and the work acquisition unit 102 execute processing in a data acquisition phase for acquiring data related to on-site work.

[0042] The site analysis unit 103 is an example of an analysis unit that executes a predetermined analysis process based on the measurement results (site images in this embodiment) acquired by the site acquisition unit 101 and the work information acquired by the work acquisition unit 102. The predetermined analysis processes include a progress determination process, a construction capability determination process, a critical path determination process, etc. The details of each process will be described later.

[0043] The result storage unit 104 is an example of a storage unit that stores the results of the analysis process executed by the site analysis unit 103. The result storage unit 104 stores the results of the analysis process, for example, in association with the site where the site image used in the analysis process was taken and the date and time of taking the site image. The result storage unit 104 stores each analysis result (data A, data B, . . ., data X in the example of FIG. 6) for each type of analysis process. The site analysis unit 103 and the result storage unit 104 execute the process in a database phase in which the acquired data is organized and stored.

[0044] The dashboard unit 105 is an example of a dashboard function that collectively displays the results of the analysis processing accumulated by the result accumulation unit 104, and makes it easy to access each result. The tool linking unit 106 is an example of a linking function that executes processing to link various tools used in business with the accumulated results of the analysis processing. The dashboard unit 105 and the tool linking unit 106 execute processing in the data utilization phase in which the accumulated data is utilized.

[0045] Next, the site image acquired by the site acquisition unit 101 and the work information acquired by the work acquisition unit 102 will be described in detail. As described above, in the site management support system 1, the work supervisor W2 grasps the situation of the site and gives instructions to the site worker W1 on the work, so area image data is repeatedly generated and related information is generated. An example of a series of information processing related to the generation of such information on the work site and an example of a functional configuration for executing the information processing will be described with reference to Fig. 7 and subsequent figures.

[0046] 7 is a diagram showing an example of the functional configuration of the control unit of each device. The control unit 11 of the support server device 10 includes a DB control unit 111, a server display unit 112, an image acquisition unit 113, an image processing unit 114, a data generation unit 115, a data output unit 116, a user reception unit 117, and an association processing unit 118. The control unit 21 of the on-site terminal 20 includes a display control unit 211, an operation reception unit 212, an operation control unit 213, and a transmission control unit 214. The control unit 31 of the imaging device 30 includes a display control unit 311, an operation reception unit 312, a shooting control unit 313, and a transmission control unit 314. The control unit 41 of the supervisor terminal 40 includes a display control unit 411 and an operation reception unit 412. The control unit 51 of the operator terminal 50 includes a display control unit 511 and an operation reception unit 512.

[0047] The DB control unit 111 (DB: Database) of the support server device 10 controls the storage of data in the database and the reading of data from the database. The database here may be not only a database stored in the support server device 10 but also a database stored in an external device. For example, the database stores captured image data indicating an image captured by the imaging device 30, the above-mentioned area image data, and the like.

[0048] The server display unit 112 executes processing for displaying on each terminal a system screen related to the site management support system 1. For example, the server display unit 112 performs processing such as generating and transmitting an HTML (Hyper Text Markup Language) file, which is display data, and displays a web page showing the system screen on the supervisor terminal 40, etc. The server display unit 112 may also perform processing such as generating and transmitting display data for an application for using the site management support system 1.

[0049] The image acquisition unit 113 acquires an image of the shooting area captured by the imaging device 30. The image processing unit 114 executes image processing on the image acquired by the image acquisition unit 113. The image processing unit 114 performs processing related to a technology called Visual SLAM (Simultaneous Localization and Mapping), which simultaneously estimates the self-position and creates an environmental map based on the image of the shooting area, for example.

[0050] The data generation unit 115 generates data based on the result of image processing by the image processing unit 114. The data generation unit 115 generates the above-mentioned area image data based on, for example, the result of Visual SLAM. The data output unit 116 outputs the data generated by the data generation unit 115. The data output unit 116 outputs, for example, the generated area image data to the supervisor terminal 40. The user acceptance unit 117 accepts a specific operation by a user via a terminal used by the user (for example, the operator terminal 50). The association processing unit 118 executes processing for associating images with information. The association of specific operations with images and information will be described in detail later.

[0051] The display control unit 511 of the operator terminal 50 controls the display processing on the display means of the own device. The operation reception unit 512 receives operations from a user (e.g., operator W3). The operations received by the operation reception unit 512 include the specific operations described above. The display control unit 411 of the supervisor terminal 40 controls the display processing on the display means of the own device. The operation reception unit 412 receives operations from a user (e.g., work supervisor W2).

[0052] The display control unit 211 of the on-site terminal 20 controls the display process on the display means of the own device. The operation reception unit 212 receives the operation of a user (for example, a field worker W1). The operation control unit 213 controls the operation of the imaging device 30. The transmission control unit 214 controls the transmission process of the captured image data by the own device and the imaging device 30.

[0053] The display control unit 311 of the imaging device 30 controls the display process on the display means of the device itself. The operation reception unit 312 receives operations from a user (e.g., a field worker W1). The shooting control unit 313 controls the shooting process by the imaging unit 37. The transmission control unit 314 controls the transmission process of the captured image data by the device itself.

[0054] 3. Information Processing In this section, information processing that the program causes the computer to execute in this embodiment will be described. The site management support system 1 executes captured image processing that performs various processes based on images captured in a shooting area of, for example, a work site.

[0055] Fig. 8 is an activity diagram showing an example of captured image processing. The activity shown in Fig. 8 is started when the supporting server device 10 receives captured image data of a work site captured by the imaging device 30. First, in A11, the supporting server device 10 acquires a plurality of images captured by the imaging device 30 through the image acquisition unit 113. Hereinafter, these multiple images are also simply referred to as "captured images".

[0056] In this embodiment, the captured images are each frame indicated by the video data, and are omnidirectional images captured by a 360-degree camera. The captured images are also images captured at each position on the path of movement by the imaging device 30 carried by the site worker W1 while moving around the work site. The work site is an example of an imaging area captured by the imaging device 30. The image acquisition unit 113 is an example of an acquisition unit that acquires the captured images.

[0057] <Generation of area image data> Next, the supporting server device 10 performs processing for generating the above-mentioned area image data. First, in A12, the supporting server device 10 causes the image processing unit 114 to estimate the movement route of the imaging device 30 based on the captured image acquired by the image acquisition unit 113. The image processing unit 114 is an example of an estimation unit. The image processing unit 114 estimates the movement route using VSLAM technology. There are, for example, the following two methods for the VSLAM technology.

[0058] The first method is an indirect position estimation method (indirect method), which estimates the positions of feature points and the position of the camera by associating feature points between multiple images. When using the indirect method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the association of feature points of objects indicated by each of the multiple captured image data obtained.

[0059] The second method is a direct position estimation method (direct method), which minimizes the error in the luminance values ​​of corresponding pixels between multiple images to estimate the depth of each pixel and the position of the imaging device 30. When using the direct method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the error in the values ​​of corresponding pixels indicated by each of the multiple acquired captured image data.

[0060] Next, in A13, the support server device 10 generates the above-mentioned area image data, i.e., data showing the work site as seen from the position where the imaging device 30 took the image, as an imaging area, by the data generating unit 115 based on the captured image acquired in A11 and the moving path estimated in A12. The area image data is output to a terminal such as the supervisor terminal 40. The terminal to which the area image data is output displays the area image shown by the area image data.

[0061] FIG. 9 is a diagram showing an example of a displayed area image. In FIG. 9, a display image A11 and a floor plan C11 are shown. The display image A11 is an image showing a part of the area image A1, which is one of a plurality of images (photographed images) taken of the construction site C1. The floor plan C11 is a plan view showing the structure of the construction site C1. The floor plan C11 is shown superimposed on the display image A11. The display image A11 and the floor plan C11 show position images P1, P2, P3, P4, etc. showing the photographing positions along the estimated moving route.

[0062] 9 is an omnidirectional image captured by the imaging device 30 from the shooting position indicated by the position image P9. The display image A11 is a rectangular image showing a certain angle of view of the area image A1. Note that the area image A1 is distorted because the omnidirectional image is represented as a rectangle, but the display image A11 has the distortion corrected to show the object in the same way as a normal photograph.

[0063] When an operation is performed to move the display image A11 in the up-down, left-right, or diagonal direction, the area image A1 moves in the direction of the operation, and the portion displayed as the display image A11 changes. Also, when an operation is performed to select the area image A1 and the position image shown in the drawing B1, the area image photographed from the photographing position indicated by the selected position image is displayed. In this way, by displaying various parts of the area image at each photographing position indicated by the area image data as the display image, the overall state of the construction site C1 can be confirmed.

[0064] <Extracting specific images> Next, the support server device 10 performs a process for extracting a specific image from the captured image. As described above, the specific image is a portion in which a specific location of the area image is shown. In this embodiment, the specific image is extracted through a manual operation in which the user specifies a specific range of one image included in the captured image, and an automatic process as a preparatory step in which candidate images showing candidates for the specific range are presented to the user.

[0065] First, in A21, the supporting server device 10 extracts candidates of images of a specific range from one area image included in the captured image as candidate images by the image processing unit 114. The image processing unit 114 extracts candidate images based on information on check points determined by a business operator performing on-site work to improve work quality, for example. In this embodiment, the supporting server device 10 stores a specific image database including information on check points.

[0066] Fig. 10 is a diagram showing an example of a specific image database. The specific image database DB1 shown in Fig. 10 stores check location information, history of specific images, the current specific image, history of specified information, current specified information, and specific image data. The check location information stores a name and shooting information. The name stores a character string indicating the name of the check location, such as "entrance", "corridor", "stairs", "kitchen wall", and "kitchen floor". Furthermore, the shooting information stores information indicating a shooting position where the check location can be photographed.

[0067] The shooting information is, for example, angle information indicating the angle of view of the check point as viewed from a height at which the imaging device 30 normally shoots. The range information indicates the range of the angle of view, for example, 0 degrees or more and 30 degrees or less for a ceiling, 45 degrees or more and 135 degrees or less for a wall, and 150 degrees or more and 180 degrees or less for a floor, assuming that the vertically upward direction is 0 degrees and the vertically downward direction is 180 degrees. The shooting information also includes a range of distances suitable for shooting the check point. The distance suitable for shooting is, for example, 0 m (directly above or directly below) for a ceiling or floor, and 1 m to 5 m for a wall, which is a distance at which the check point is captured at an appropriate size. In addition, in the case of a part facing a specific direction such as a wall, the direction in which the part faces is also included in the shooting information.

[0068] The image processing unit 114 first identifies a shooting position from the estimated moving path where the check point can be photographed. For example, the image processing unit 114 identifies a position on the moving path that is a distance from the check point on the drawing that is suitable for shooting, and in the case of a wall, is in the direction in which the check point faces, as the shooting position. Then, the image processing unit 114 extracts, as a candidate image, an image with an angle of view indicated by the shooting information from among the area images photographed from the identified position.

[0069] In addition, in the case of a location having a distinctive appearance such as a door or a staircase, a pattern image showing the location may be used as the photographic information. In this case, the image processing unit 114 uses a well-known image recognition technique to extract an image of a location similar to the pattern image as a candidate image. The server display unit 112 transmits the extracted candidate image to, for example, the operator terminal 50. The operator terminal 50 displays the transmitted candidate image by the display control unit 511 at A22.

[0070] Fig. 11 is a diagram showing an example of displayed candidate images. In Fig. 11, the display control unit 511 displays a candidate image selection screen D1. In the candidate image selection screen D1, a character string "Please select a specific image from the candidate images" is displayed, along with candidate images E11, E12, and E13 (referred to as "candidate image E10" when not distinguishing between them), selection buttons B11, B12, and B13 (referred to as "selection button B10" when not distinguishing between them), and adjustment buttons B21, B22, and B23 (referred to as "adjustment button B20" when not distinguishing between them).

[0071] In this way, the server display unit 112 and the image processing unit 114 function as an example of a first presentation unit that presents candidates for images in a specific range to the user as candidate images (candidate images E1, E2, and E3 in the example of FIG. 11). According to this aspect, it is possible to make it easier to select an image of a specific location compared to a case where candidate images are not presented.

[0072] The selection button B10 is a button for accepting an operation to select a candidate image displayed in association with the selection button as a specific image. The adjustment button B20 is a button for accepting an operation to adjust a range displayed as a candidate image. These operations are accepted, for example, by the operation acceptance unit 512 of the operator terminal 50, and operation data indicating the operation content is transmitted to the support server device 10. The user acceptance unit 117 of the support server device 10 accepts the operation indicated by the transmitted operation data as an operation performed by the user.

[0073] The user accepting unit 117 functions as an example of a designation accepting unit that accepts a first operation by a user to designate a specific range in one image included in the multiple images acquired by the image acquiring unit 113. The user accepting unit 117 accepts, for example, an operation to select a candidate image E10 presented by the server display unit 112 (an operation to press any of the selection buttons B10) as the first operation. The first operation here is an operation to designate the range indicated by the candidate image E10 as the specific range.

[0074] Specifically, for example, when an operation of pressing the adjustment button B20 is performed and then an operation of dragging the candidate image E10 is performed, the user accepting unit 117 accepts the user operation, and the server display unit 112 moves the area image displayed as the candidate image E10 in the dragged direction to display a portion of the area image that has not been displayed until now. In this way, the user accepting unit 117 accepts the operation of spatially fine-tuning the image showing a specific location as a first operation.

[0075] In addition to the above operations, the user accepting unit 117 may accept, as the first operation, an operation for finely adjusting an image showing a specific portion in terms of time. For example, when an operation for dragging the candidate image E1 is performed while a specific button (such as a Ctrl key) is pressed, the user accepting unit 117 accepts the user operation as the first operation, and the server display unit 112 displays the same part of the area image captured before or after the area image displayed as the candidate image E1.

[0076] Furthermore, the user accepting unit 117 may accept, as the first operation, an operation of designating a specific range on a screen displaying the area image data shown in Fig. 9. In this case, first, the data output unit 116 outputs the area image data generated by the data generating unit 115 to, for example, the operator terminal 50. The display control unit 511 of the operator terminal 50 displays the output area image data. The operation accepting unit 512 accepts an operation of designating a specific range from the display image displayed as shown in Fig. 9, and the user accepting unit 117 accepts the operation as the first operation.

[0077] When the operator W3 determines that an appropriate range has been specified as a specific image by the above-mentioned first operation, the operator performs an operation of pressing the selection button B10 as a first operation. At A24, the support server device 10 receives a series of first operations by the user reception unit 117, and the image processing unit 114 extracts, as a specific image, a candidate image selected by the received first operation or an area image of the specified specific range. The DB control unit 111 stores the extracted specific image in the specific image database DB1 as the "current specific image" shown in FIG.

[0078] In this way, the image processing unit 114 functions as an extraction unit that extracts, as a specific image, a portion in which a specific location is captured in the shooting area from one or more images included in the multiple images acquired by the image acquisition unit 113. In this embodiment, the image processing unit 114 extracts, as the specific image, an image in the range specified by the first operation as described above. According to this aspect, it is possible to reflect the user's judgment in the extraction of the image of the specific location.

[0079] When a new photographed image is acquired and new area image data is generated, the DB control unit 111 moves the extracted specific image from "current specific image" to "specific image history." In this way, the specific image extracted by the image processing unit 114 is stored as a history image in the specific image history. The image processing unit 114 may extract candidate images based on the history images thus stored. In this case, the image processing unit 114, for example, specifies the photographing position and the angle of view of each of the stored history images.

[0080] Then, the image processing unit 114 extracts images with a viewing angle within the specified range as candidate images from images taken at shooting positions within the specified range among the newly acquired photographed images. The server display unit 112 presents the candidate images thus extracted to the user. In this manner, the candidate images may be images extracted based on the history of the extracted specific images. According to this embodiment, images close to the actual specific images are presented as candidate images, so that the usefulness of the candidate images can be increased compared to a case where the history of the specific images is not taken into consideration.

[0081] <Matching of specified information> Next, the support server device 10 performs a process for associating the extracted specific image with the predetermined information. The predetermined information is information that is determined in the site management support system 1 as information for associating with the specific image, such as the name of a specific location. In this embodiment, the specific image is associated with the predetermined information through a manual operation in which the user specifies the predetermined information in association with the specific image, and an automatic process prior to that in which candidate information indicating candidates for the predetermined information is presented to the user.

[0082] First, in A31, the support server device 10 determines candidates for predetermined information to be associated with a specific image as candidate information by the association processing unit 118. For example, the association processing unit 118 determines the name of check point information stored in the specific image database DB1 as the candidate information. The server display unit 112 transmits display data for displaying the determined candidate information to, for example, the operator terminal 50. In A32, the operator terminal 50 displays the candidate information indicated by the transmitted display data by the display control unit 511.

[0083] Fig. 12 is a diagram showing an example of displayed candidate information. In Fig. 12, the display control unit 511 displays a candidate information selection screen D2. The candidate information selection screen D2 displays a character string saying "Please select specific information from the candidate information," a specific image F14, a selection button B31, and a candidate information input field J11. Below the candidate information input field J11, a candidate information list J12 is displayed as a pull-down list, and candidate information selected from the candidate information list J12 is displayed in the candidate information input field J11.

[0084] In this way, the server display unit 112 and the association processing unit 118 function as a second presentation unit that presents candidate information indicating candidates for the predetermined information to the user. According to this aspect, it is possible to reduce the effort required for association between a specific image and the predetermined information, compared to a case in which the candidate information is not presented. The operation of selecting candidate information and pressing the selection button B31 is accepted as an operation performed by the user by the user acceptance unit 117 of the support server device 10, similar to the above-mentioned first operation.

[0085] The user accepting unit 117 functions as an example of a correspondence accepting unit that accepts a second operation of the user to associate a specific image extracted by the image processing unit 114 with predetermined information (in the example of FIG. 12, the specific image F14 with the name of the checked part "kitchen wall"). The user accepting unit 117 accepts, as the second operation, an operation to select candidate information presented by the server display unit 112 (in the example of FIG. 12, the candidate information included in the candidate information list J12). This second operation is an operation to associate the candidate information with the specific image as predetermined information.

[0086] When the operator W3 determines that appropriate candidate information to be associated with the specific image has been selected by the above-mentioned second operation, the operator W3 performs an operation of pressing the selection button B31 as the second operation. The operator terminal 50 accepts the second operations as selection operations of the candidate information by the operation acceptance unit 512 at A33. The support server device 10 accepts the series of second operations by the user acceptance unit 117 at A34, and stores the candidate information associated with the specific image by the accepted second operations in the specific image database DB1 as "current specified information" shown in FIG. 10 in association with the specific image by the DB control unit 111.

[0087] Then, the data generating unit 115 reads out the "current specific image" and the "current specified information" from the specific image database DB1, and generates a list of these in association with each other as specific image data. For example, suppose that specific image data is requested from the supervisor terminal 40. In this case, the support server device 10 outputs the generated specific image data to the supervisor terminal 40 by the data output unit 116 at A34. The supervisor terminal 40 displays the output specific image data by the display control unit 411 at A35.

[0088] Fig. 13 is a diagram showing an example of the displayed specific image data. In the example of Fig. 13, the display control unit 411 displays a checklist H1. The checklist H1 displays the names of the check locations "Entrance", "Hallway", "Stairs", "Kitchen Wall", and "Kitchen Floor", the specific images F11, F12, F13, F14, and F15 of each check location (referred to as "specific image F10" when not distinguishing between them), and the check results of each check location. The check results include a symbol indicating whether the work progress is good or bad, and a comment.

[0089] The specific image F10 includes images extracted from images taken on different dates, "mm / d1," "mm / d2," "mm / d3," and "mm / d4." For example, the specific images F11-1, F11-2, F11-3, and F11-4 are all images extracted with the "entrance" as a specific location, and are extracted from images taken on "mm / d1," "mm / d2," "mm / d3," and "mm / d4."

[0090] In this way, the data output unit 116 outputs a list of a plurality of specific images (checklist H1 in the example of FIG. 13) as specific image data. The plurality of specific images (specific images F11-1, F11-2, F11-3, F11-4, etc. in the example of FIG. 13) have a common specific location but were photographed at different times. According to this embodiment, it is easier to grasp the transition of the state of the specific location compared to the case where only specific images photographed at a single time are output.

[0091] In addition, since the site worker W1 takes the images by the imaging device 30 while walking, the moving path during the image taking is not always the same. Therefore, the specific images do not necessarily show the exact same range as the specific location, but it is sufficient that at least a part of the specific location is shown. This is because the situation of the specific location at the work site can be understood by looking at the image of that part.

[0092] As described above, the data output unit 116 functions as an example of a specific output unit that outputs specific image data including a specific image extracted by the image processing unit 114. According to this aspect, a list indicating a specific location can be easily created. In this embodiment, the data output unit 116 outputs, as specific image data, a list in which the specific image extracted by the image processing unit 114 and the predetermined information associated with the specific image by the above-mentioned second operation are arranged.

[0093] When a specific enlargement operation (e.g., a single click) is performed on a specific image F10 included in the checklist H1, the display control unit 411 enlarges and displays the specific image F10. This makes it easier to see the situation at the site. In addition, in this embodiment, the specific image F10 included in the checklist H1 is a link to area image data generated based on the captured image from which the specific image F10 was extracted. More specifically, the specific image F10 is a link to area image data in which a specific location shown in the specific image F10 is displayed as a display image.

[0094] At A41, the supervising terminal 40 receives a predetermined operation (e.g., double-clicking) on ​​the specific image F10 as a link operation via the operation receiving unit 412. The operation receiving unit 412 transmits operation data indicating the specific image F10 for which the link operation has been performed to the supporting server device 10. At A42, the supporting server device 10 outputs area image data of the link destination to the supervising terminal 40 via the data output unit 116. At A43, the supervising terminal 40 displays the output area image data via the display control unit 411.

[0095] For example, when a link operation to the specific image F11-4 shown in Fig. 13 is performed, the area image data generated based on the image captured at mm / d4 is output with the entrance as the display image and displayed on the supervisor terminal 40. The supervisor terminal 40 accepts an operation to move the part shown as the display image of the area image data at A44 as a display operation. For example, the work supervisor W2 cannot fully determine the work situation from the specific image F11-4 alone, so he determines the work situation by manipulating the area image data in this way and viewing the surrounding image.

[0096] As described above, the data output unit 116 outputs data including a link from an image of a specific location (each specific image F10 in the example of FIG. 13) to an image showing that location among images shown by the area image data (for example, an image showing the entrance photographed at mm / d4 in the case of specific image F11-4) as specific image data. According to this aspect, it is easier to confirm an image around the specific location compared to a case where the specific image data does not include the above link.

[0097] When the work supervisor W2 judges the work status by displaying the area image data, he returns to the checklist H1 to check the work status. In this embodiment, the area image data includes a link for switching the display to the specific image data (the checklist H1 in the example of FIG. 13). Fig. 14 is a diagram showing an example of a displayed area image. In the example of Fig. 14, the display control unit 411 displays a display image A41 that represents a part of the area image A4.

[0098] Display image A11 shows link images L14-4 and L15-4 (referred to as "link image L10" when there is no need to distinguish between them). Link image L14-4 is a link to specific image F14-4 shown in FIG. 13. Link image L15-4 is a link to specific image F15-4 shown in FIG. 13. When the supervising terminal 40 accepts a link operation to link image L10 by the operation accepting unit 412 at A51, it transmits operation data indicating the operated link image L10 to the supporting server device 10.

[0099] At A52, the support server device 10 outputs the specific image data, which is the link destination indicated by the transmitted operation data, to the supervisor terminal 40 by the data output unit 116. At A53, the supervisor terminal 40 displays the output specific image data by the display control unit 411. When an operation is performed on the link image L14-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of the specific image F14-4 shown in Fig. 13. Also, when an operation is performed on the link image L15-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of the specific image F15-4.

[0100] As described above, the data output unit 116 outputs data including a link from an image taken at a position corresponding to a specific location to an image of that location in the specific image data as area image data. According to this aspect, even when an area image indicated by area image data is displayed, the image of the specific location can be output more easily than when the area image data does not include a link.

[0101] Next, at A54, the supervisor terminal 40 accepts the check operation through the operation acceptance unit 412, and transmits the operation data to the support server device 10. At A55, the support server device 10 stores the specific image data reflecting the check result indicated by the transmitted operation data in the specific image database DB1 through the DB control unit 111. The captured image processing up to A34 shown in Fig. 8 is executed every time a photograph is taken at the work site. As a result, the area image data and the specific image data are saved and accumulated every time a photograph is taken.

[0102] As described above, the data output unit 116 uses multiple images (photographed images) acquired by the image acquisition unit 113 to output area image data showing the photographed area as seen from each position on the estimated travel path. Similarly, the data output unit 116 outputs specific image data using the photographed images as described above. In this manner, the site management support system 1 utilizes the photographed images for both area image data as shown in FIG. 9 and specific image data as shown in FIG. 13. When photographing an image of the photographed area to generate area image data and specific image data, respectively, the effort of photographing is required twice. In this embodiment, the photographed images are shared as described above, and therefore the effort of the work associated with photographing the site can be reduced compared to the case where the photographed images are not shared.

[0103] In addition, when the site worker W1 takes still images with a camera with a normal angle of view while moving around the shooting area, even if the specific location is taken one by one, the work supervisor W2 does not necessarily take an image that makes it easy to judge the situation, and it is very troublesome to take another image just because it is an undesirable image. Even if the image is taken well, the surrounding situation outside the range of the camera's angle of view cannot be understood. In this embodiment, as described above, a video is taken with a 360-degree camera that captures all directions, and a specific location is extracted, so that a desired image can be selected. In addition, since the omnidirectional image is highly comprehensive, it is rare that a desired image cannot be selected, and even if a specific location that you want to extract occurs after shooting, that location can be extracted.

[0104] In addition, the area image data also shows a specific location, but when one wants to see only the required location, it takes time and effort to find the location from the area image shown by the area image data. In this embodiment, the specific location is extracted and the specific image data is output, so that the image of the specific location can be displayed without such time and effort. On the other hand, by outputting not only the specific image data but also the area image data, even if it is difficult to tell from the specific image data which part of the shooting area the specific image shows, the position of the location shown in the specific image can be ascertained from the area image data.

[0105] Furthermore, as described above, when associating a specific image with predetermined information, by presenting candidate information, it is possible to reduce the effort required to associate appropriate candidate information. Furthermore, in this embodiment, the check points shown in Fig. 10 are always extracted as the specific image. As a result, when the specific image data is used as a checklist, the specific points are always checked at any work site, and the quality of the work can be guaranteed.

[0106] <Accumulation of analysis results> The site acquisition unit 101 shown in Fig. 6 acquires area images and the like at each shooting position indicated by area image data generated by the information processing described in Fig. 7 and subsequent figures as the measurement results of the site by the imaging device 30 (site images in this embodiment). The site acquisition unit 101 acquires images captured by the imaging device 30 capable of capturing images in all directions as the measurement results of the site. According to this aspect, the range shown in the site image is wider than when capturing images by an imaging device that cannot capture images in all directions, so more site information can be digitized.

[0107] The work acquisition unit 102 acquires, as work information, drawing data and the like used in the information processing described in Fig. 7 and subsequent figures. In addition to the above, the work acquisition unit 102 acquires, as work information, a process schedule, material information, component information, cost information, environmental load information, and the like stored in the business server device 60. The work information acquired in this manner includes schedule information (for example, a process schedule or a work plan, etc.) indicating the schedule of the work.

[0108] The site analysis unit 103 executes a process of analyzing the progress of work as a predetermined analysis process based on the site situation shown in the site image acquired by the site acquisition unit 101 and the schedule shown in the work information acquired by the work acquisition unit 102. For example, the site analysis unit 103 analyzes the site image to determine whether each component part of a structure such as a wall, floor, ceiling, and pillar is completed. Then, the site analysis unit 103 generates, as information indicating the progress of the site work, the ratio of the components that have been completed among the components that are shown to be completed according to the schedule in the work information. According to this embodiment, the progress of the work can be managed.

[0109] The work information also includes information about the worker who will perform the work (e.g., a personnel schedule, etc.). The worker here may be an individual worker or an organization (such as a construction company) that undertakes the work. The site analysis unit 103 executes an analysis process to analyze the construction capabilities of the workers indicated by the work information based on the results of the progress analysis process and the acquired work information. The site analysis unit 103 calculates the amount of work that can be completed in a unit period as the analysis result, based on, for example, the amount of work indicated in the schedule of each worker and the progress rate of that worker. According to this embodiment, the construction capabilities of the workers can be grasped.

[0110] The result storage unit 104 stores the results of the analysis process executed by the site analysis unit 103 as described above. As described with reference to Fig. 7 and other figures, site images are photographed and acquired daily, so new analysis processes are performed daily and the analysis results are stored. According to this embodiment, it is possible to promote the creation of data on site work compared to a case where a system such as the site management support system 1 is not introduced.

[0111] <Example of analysis results> The site analysis unit 103 may execute various analysis processes other than those described above. For example, the site analysis unit 103 may not only analyze the progress of work on the site drawings indicated by the work information, but also analyze construction errors. For example, the site analysis unit 103 compares the site drawings with site images (360-degree images) and extracts differences between the drawings and the actual construction. The site analysis unit 103 utilizes the extracted differences to output work progress data and construction error data indicating the progress of work on the drawings.

[0112] The site analysis unit 103 may also output work progress data for a process as an analysis result. For example, the site analysis unit 103 outputs work progress data indicating the progress of work for a planned process by combining the work progress data obtained by comparing the site drawings with the site images (360-degree images) with the process chart.

[0113] The site analysis unit 103 may output the construction ability data of the builder as the analysis result. For example, the site analysis unit 103 analyzes the construction accuracy data of the builder by combining data related to the builder or the builder with construction error data. The site analysis unit 103 also outputs the construction speed data of the builder as the analysis result by combining data related to the builder or the builder, the corresponding work content, and work progress data for the process.

[0114] The site analysis unit 103 may also output data on the cause of construction errors as an analysis result. In this case, for example, a user creates corrective instruction data on the system using a site image (360-degree image) acquired by the site acquisition unit 101, and the result accumulation unit 104 accumulates the created corrective instruction data. Then, the site analysis unit 103 combines the corrective instruction data with a schedule to output data on the cause of construction errors in a specific construction project as an analysis result. The accumulated data on the cause of construction errors is presented, for example, by the dashboard unit 105 to a construction manager who is carrying out a similar construction project, together with the accumulated construction error data.

[0115] Furthermore, the on-site analysis unit 103 may output waiting period data as an analysis result. For example, the on-site analysis unit 103 outputs data on waiting periods during which work is not progressing as an analysis result based on work progress data for a process.

[0116] The site analysis unit 103 may also output the total construction capacity data of a specific area as an analysis result. For example, the site analysis unit 103 outputs the total construction capacity data obtained by aggregating the construction capacity data of the builders in a specific area as an analysis result. Based on the aggregated total construction capacity data, the balance between supply and demand for each area can be optimized.

[0117] In addition, the on-site analysis unit 103 may output an optimal personnel allocation as an analysis result. For example, the on-site analysis unit 103 uses accumulated construction ability data of the constructors to calculate a personnel allocation that can minimize the construction period, and outputs the calculated personnel allocation as an analysis result.

[0118] The on-site analysis unit 103 may also output an optimal process as an analysis result. For example, the on-site analysis unit 103 calculates a process that can appropriately minimize the construction period based on the accumulated progress status data for the process and taking into consideration other projects handled by the construction manager, and outputs the calculated optimal process as an analysis result.

[0119] Furthermore, the site analysis unit 103 may output an optimal material delivery schedule as an analysis result. For example, the site analysis unit 103 calculates an optimized material delivery schedule based on accumulated construction progress data for each process as follows. First, the site analysis unit 103 calculates a material delivery schedule that minimizes waiting time by just-in-time material delivery. The site analysis unit 103 also calculates a material delivery schedule that always maximizes work space by just-in-time material delivery, improves work efficiency, and minimizes the time spent by site workers tidying up materials.

[0120] Furthermore, the site analysis unit 103 calculates a material delivery schedule that minimizes the environmental load. Specifically, the site analysis unit 103 calculates a material delivery schedule that optimizes the amount of materials and minimizes the amount of waste. Furthermore, the site analysis unit 103 calculates a material delivery schedule that minimizes the number of deliveries and minimizes the amount of exhaust. Furthermore, the site analysis unit 103 calculates a material delivery schedule that minimizes the delivery distance and minimizes the amount of exhaust.

[0121] <Other embodiments> The shooting area is not limited to the above. For example, it may be inside an existing building, inside a vehicle, or a predetermined area outdoors. The imaging device 30 is not limited to a 360-degree camera, and may be a wide-angle camera or a camera that shoots at a normal angle of view. The captured images are not limited to moving images, and may be still images captured continuously.

[0122] In the embodiment, the imaging device 30 is an example of a measuring device, but the present invention is not limited to this. The measuring device may be, for example, a device that includes a distance image sensor that measures the distance to an object and outputs point cloud data indicating the measurement result as measurement data. The measuring device may also be a device that includes an infrared sensor that measures the temperature of the object or a millimeter wave sensor that measures the distance to the object or the speed of the object.

[0123] Also, in the embodiment, the measurement results (in the embodiment, site images) generated by the functions shown in Fig. 7 are analyzed by the site analysis unit 103 and stored by the result storage unit 104, but for example, a predetermined analysis process (e.g., a process of estimating a travel route) may be executed by the image processing unit 114 and stored by the DB control unit 111. In this case, the image processing unit 114 is an example of a site analysis unit, and the DB control unit 111 is an example of a storage unit. Also, some of the functions shown in Fig. 7, including the image processing unit 114 and the DB control unit 111, will execute processing in the database creation phase shown in Fig. 6.

[0124] In the embodiment, the work acquisition unit 102 acquires the information stored in the business server device 60 as the work information as it is, but for example, the work information may be acquired after processing the information so that a predetermined analysis process can be easily performed. For example, the work acquisition unit 102 performs a process of classifying the information stored in the business server device 60 (drawing information, process chart, material information, etc.) by site or area as a processing process, and acquires the classified information as the work information.

[0125] Moreover, the area image data is not limited to that shown in FIG. 9. For example, it may be data showing only the area image without any drawing or position image superimposed. In that case, it is sufficient that the viewpoint can be moved along the movement route by an operation different from the operation of selecting the position image. Moreover, the specific image data is not limited to that shown in FIG. 13. For example, it may be data in which a specific location is changed each time shooting is performed, or it may be data showing only a specific image extracted from the latest shooting image.

[0126] Furthermore, the predetermined information is not limited to the name of the check location, but may be, for example, the name of the work site (floor number or room number, etc.), the work process, the name of the photographer or work supervisor, the date and time of shooting, the weather at the time of shooting, or the type of imaging device, etc. In this way, the predetermined information may be information indicating what kind of location the specific location is or what kind of image the specific image is. However, information unrelated to the specific image may also be used as the predetermined information.

[0127] <Candidate image> The image processing unit 114 may extract candidate images in a manner different from that of the embodiment. For example, when a history of a specific range designated by the operator W3 is accumulated, the image processing unit 114 extracts candidate images based on the history. For example, when the same part of the shooting area is designated as a specific range a predetermined number of times or more, the image processing unit 114 extracts an image including the part as a candidate image.

[0128] Furthermore, when multiple rooms with the same layout are photographed as the photographing area, such as in an apartment, the image processing unit 114 extracts images of the range specified in the other rooms with the same layout as candidate images. The server display unit 112 presents the candidate images thus extracted to the user. In this case, the candidate images are images extracted based on the history of the specific range specified by the user. According to this aspect, images similar to the specific image of the range previously specified by the user are presented as candidate images, so that the usefulness of the candidate images can be increased compared to the case where the history of the specific range is not taken into consideration.

[0129] <Candidate information> The association processing unit 118 may determine candidate information in a manner different from that of the embodiment. For example, the association processing unit 118 recognizes a location shown in a specific image extracted by the image processing unit 114, and determines information on the recognized location as candidate information. For example, the association processing unit 118 recognizes doors, stairs, lights, windows, washrooms, etc. by image recognition using a pattern image, and determines their names as candidate information.

[0130] In this case, the candidate information is information that is determined based on a location shown in the specific image extracted by the image processing unit 114. According to this aspect, for example, even if the specific location is changed due to the situation at the work site, information that matches the location is presented as candidate information, so that the usefulness of the candidate information can be increased compared to a case where the candidate information is fixed.

[0131] Furthermore, the association processing unit 118 may determine, for example, predetermined information previously associated with a specific image as candidate information. Furthermore, the association processing unit 118 may determine, as candidate information, predetermined information previously used not only at the same work site but also at work sites with the same layout. In either case, the candidate information is information determined based on past predetermined information. According to such an embodiment, for example, predetermined information added depending on the situation of the work site is also presented as candidate information, so that the usefulness of the candidate information can be increased compared to a case where the candidate information is fixed.

[0132] <Image Filter> In the site management support system 1, both the estimation of the travel path and the extraction of the specific image are performed based on a plurality of images (photographed images) captured by the imaging device 30. At that time, the captured images may be narrowed down to those with better images. The quality of the images may be determined, for example, using a known technique (fast Fourier transform, machine learning, open source, etc.) for determining whether a photograph is out of focus.

[0133] For example, the image processing unit 114 estimates a travel path based on images narrowed down based on a first rule from among the multiple images acquired by the image acquisition unit 113. The image processing unit 114 also extracts a specific image from images narrowed down based on a second rule from among the multiple images acquired by the image acquisition unit 113. The second rule may be a rule different from the first rule.

[0134] The first rule is, for example, a rule for narrowing down the captured images to images that have a good overall appearance. On the other hand, the second rule is a rule for narrowing down the captured images to images that have a good appearance in a specific part. Even in one image, there may be a mixture of good and bad parts. Therefore, it may happen that the overall image is good but a specific part is bad, or the specific part is good but the overall image is bad. Therefore, by using the first and second rules as described above, it is possible to extract a specific image that has a good appearance while improving the estimation accuracy of the travel route, and it is possible to achieve both the estimation accuracy of the travel route and the quality of the extracted image.

[0135] The first and second rules are not limited to the above rules. For example, the first rule is a rule for narrowing down images to those with feature points equal to or greater than a threshold, and the second rule is a rule for narrowing down images regardless of the number of feature points. The more feature points an image has, the higher the accuracy of self-location estimation using VSLAM. On the other hand, when a specific location is a wall or a floor, the number of feature points becomes extremely small. Therefore, by making the first and second rules different as described above, it is possible to achieve both the estimation accuracy of the travel path and the quality of the extracted image.

[0136] The first rule and the second rule may be the same rule. Even in this case, it is possible to improve both the estimation accuracy of the travel route and the quality of the extracted image, compared to the case where the narrowing down is not performed. The narrowing down of the images may be used when extracting candidate images. In this case, the image processing unit 114 narrows down the captured images based on the second rule (which may be the first rule), and extracts candidate images from the narrowed down captured images, as in the embodiment. In this case, it is also possible to improve the quality of the extracted image.

[0137] <Automatic extraction> In the embodiment, the image processing unit 114 extracts a specific image through a manual operation of the user specifying a specific range, but the specific image may be extracted automatically without manual operation. In that case, the image processing unit 114 may extract a specific image, for example, by using the same method for extracting candidate images as it is. For example, the image processing unit 114 extracts a new specific image based on the shooting information of the check point shown in FIG. 10, as in the embodiment.

[0138] Furthermore, the image processing unit 114 extracts a new specific image including the same location as a specific location based on, for example, the history of extracted specific images. Furthermore, the image processing unit 114 extracts a new specific image including a range previously specified as a specific location based on the history of specific ranges specified by the user.

[0139] 15 is a diagram showing an example of a flow diagram of the automatic extraction process. First, in S11, the supporting server device 10 acquires a captured image by the image acquisition unit 113. Next, in S12, the supporting server device 10 estimates a moving path of the imaging device 30 by the image processing unit 114. Next, in S13, the supporting server device 10 generates area image data by the data generation unit 115. Next, in S14, the supporting server device 10 extracts a specific image from the captured image by the image processing unit 114. Next, in S15, the supporting server device 10 determines predetermined information by the association processing unit 118.

[0140] In S14 and S15, unlike the example of Fig. 8, a specific image is extracted and predetermined information is determined without any operation by the user. Next, in S16, the support server device 10 generates specific image data by the data generating unit 115. Then, in S17, the support server device 10 outputs the area image data and the specific image data in response to a request from the supervisor terminal 40 or the operator terminal 50.

[0141] According to this embodiment, the effort required for generating specific image data can be further reduced compared to the case where candidate images are presented. When extracting specific images, the image processing unit 114 may extract images that look good as specific images by narrowing down the captured images using the above-mentioned second rule. Furthermore, the image processing unit 114 may extract specific images by AI (Artificial Intelligence) technology that performs machine learning using past specific images as training data.

[0142] Furthermore, drawing information such as a floor plan of a work site may contain location information indicating the location of a specific location. In this case, when a specific location is specified from the drawing information, the image processing unit 114 may specify a captured image taken from a shooting position where the location is likely to be included in the angle of view based on the location information of the specified location and the estimated moving route. Note that the server display unit 112 may present the drawing information and the moving route to the user, and the user may specify a similar shooting location, thereby specifying a captured image taken from the specified shooting position.

[0143] Then, the image processing unit 114 recognizes a portion of the identified photographed image that shows the specific location, and extracts the recognized portion as a specific image. Note that the server display unit 112 may present the identified photographed image to the user, so that the user may specify a portion that shows the specific location, and the image processing unit 114 may extract the specified portion as a specific image. According to these aspects, if the location is indicated in the drawing information, the specific image can be easily extracted.

[0144] <Automatic mapping> In the embodiment, the association processing unit 118 associates the specific information with the specific image through a manual operation of designating the specific information in association with the specific image, but the specific information may be associated automatically without a manual operation. In that case, the association processing unit 118 determines, for example, the specific information associated with the history of specific images that have captured the same specific location as the extracted specific image, as the specific information to be associated with the extracted specific image.

[0145] Furthermore, when the association processing unit 118 determines the predetermined information to be associated with a specific image as described above, it determines predetermined information existing in the vicinity of the predetermined information as predetermined information to be associated with an image of another location extracted from the same shooting information as the specific image. For example, when the predetermined information "kitchen wall" is associated with a specific image, the association processing unit 118 determines "kitchen floor" or "kitchen ceiling" existing in the vicinity of the "kitchen wall" as predetermined information to be associated with an image of another location (e.g., kitchen floor or ceiling).

[0146] Furthermore, the association processing unit 118 may specify a location shown in the specific image by image recognition technology, and determine the name or the like of the specified location as the predetermined information. Specifically, the association processing unit 118 specifies a location shown in the specific image using a pattern image associated with the predetermined information (if the predetermined information is "entrance", pattern images of the entrance photographed from various angles) and determines the predetermined information associated with the pattern image of the specified location as the predetermined information to be associated with the specific image.

[0147] According to this embodiment, the effort required for generating specific image data can be further reduced compared to the case where candidate information is presented. Note that the association processing unit 118 may associate the specific information by an AI technique that performs machine learning using the specific information associated with the past specific image as training data.

[0148] Furthermore, drawing information such as a floor plan of a work site may contain the name of each location as the predetermined information. In this case, the association processing unit 118 may estimate the predetermined information corresponding to the specific location indicated by the specific image from the drawing information. For example, the association processing unit 118 identifies the position of the specific location on the drawing from the shooting position and angle of view information at which the specific image was shot, and determines the name of the location including that position as the predetermined information.

[0149] The server display unit 112 may present the drawing information to the user, and the association processing unit 118 may determine the name of a place designated by the user (the operator W3 or the work supervisor W2, etc.) from the presented drawing information as the predetermined information. According to these aspects, if the predetermined information is indicated in the drawing information, association with a specific image can be easily performed.

[0150] <Extraction of wide-area images> When extracting a specific image, the image processing unit 114 may extract an image having a range that is a predetermined ratio (hereinafter referred to as "image ratio") wider than the range in which the specific location is shown as the specific image. In this case, the data generating unit 115 generates data showing the specific location and the surroundings of the location when a predetermined operation (such as a flick) is performed as specific image data. This makes it possible to easily display the surroundings of a specific location when the user wishes to see the surroundings of the specific location.

[0151] The image ratio may be a value according to the size of the photographed area, the usage of the outputted area image data, or the usage of the outputted specific image data. For example, the image processing unit 114 increases the image ratio to extract the specific image as the photographed area becomes larger. The larger the photographed area, the larger the space around the specific location becomes, and therefore the larger the image of the surroundings to be confirmed becomes.

[0152] Furthermore, the image processing unit 114 extracts a specific image with a larger image ratio the higher the usage of the area image data or specific image data. The usage here is higher the longer the usage time or the more frequently the area image data or specific image data is used. The higher the usage, the more likely it is that there will be opportunities to look around a specific location, and the more likely it will be that a situation will arise in which a wider range is desired. As described above, by varying the image ratio, it is possible to check the situation in a range as wide as necessary, compared to when the image ratio is constant.

[0153] <Recommendation of specific images> A specific image may be recommended. For example, when a specific range is specified by the user, the image processing unit 114 not only extracts images in the specified range, but also extracts images that may be used as a specific image from areas spatially adjacent to the range as recommended images. For example, when a kitchen wall is specified, the image processing unit 114 extracts images of the kitchen floor and ceiling as recommended images. Also, when a specific range is specified from an image captured at a certain shooting position, the image processing unit 114 extracts images of the same range from images captured at positions before and after the position as recommended images.

[0154] The server display unit 112 presents the recommended image thus extracted to the user together with the specific image. The user performs an operation to instruct the user to set the recommended image as the specific image in addition to or instead of the specific image. The data generating unit 115 generates specific image data including the instructed recommended image as the specific image. According to this embodiment, the usefulness of the specific image data can be increased compared to a case where the recommended image is not presented.

[0155] <Check with area image data> When a check spot is extracted as a specific image, the data generating unit 115 may generate, as area image data, data that is displayed together with a check image for inputting a check result when the specific spot is displayed.

[0156] FIG. 16 is a diagram showing an example of a displayed check image. In the example of FIG. 16, the display control unit 411 displays check images M14-4 and M15-4 on the display image A41. The check image M14-4 is an image for inputting the check result of the specific image F14-4, and the check image M15-4 is an image for inputting the check result of the specific image F15-4. When input is made to the check image, the DB control unit 111 reflects the input contents in the check result of the check list H1 shown in FIG. 13. According to this aspect, it is possible to reduce the effort required to check a specific location, compared to a case where the check result can only be input to the specific image data.

[0157] <Other> When a new range is specified as a specific range, the image processing unit 114 may extract an image of the same range from a captured image taken in the past as a past specific image. The data generating unit 115 generates specific image data including the extracted current specific image and the past specific image. This makes it possible to compare the newly specified range with the past image.

[0158] The data generating unit 115 may generate specific image data that represents specific images showing common points extracted from images taken in multiple shooting areas with a common structure (for example, rooms with the same layout in the same apartment building) in a comparative manner. By arranging comparison targets in this way, it is possible to easily judge the status of the work.

[0159] 13, the server display unit 112 may also display past check results for similar specific images. For example, the server display unit 112 displays an image of an entranceway extracted from other captured images and its check results alongside the specific image F11-4 of the entranceway and its check results. This makes it easier to input the check results.

[0160] Furthermore, for example, the image processing unit 114 may determine the check result for the specific image being checked based on the check results for similar specific images in the past. For example, the image processing unit 114 determines the check result as "○" if the work progress is similar to a specific image marked with "○", and determines the check result as "×" if the work progress is similar to a specific image marked with "×".

[0161] The server display unit 112 may support the user in checking by displaying the judgment result by the image processing unit 114 in a checklist, or by displaying an alert when the check result input by the user differs from the judgment result by the image processing unit 114. Furthermore, the DB control unit 111 may reflect the result of the judgment by the image processing unit 114 as the check result in the checklist H1 even if there is no check by the user. According to these aspects, it is possible to reduce the effort of checking specific points compared to the case where the check result is not judged by the image processing unit 114.

[0162] Also, the imaging device 30 does not have to be installed at the work site. For example, a user such as a field worker W1 takes the field terminal 20 and the imaging device 30 to the work site, takes a picture, transmits the captured image data from the imaging device 30 to the field terminal 20, and connects the field terminal 20 to a WiFi router in an office or at home to transfer the captured image data to the support server device 10 (the captured image data may be transferred by mobile communication). In this case, the field management support system 1 only needs to include a communication line 2, a selfie stick 3, a support server device 10, the field terminal 20, the imaging device 30, a supervisor terminal 40, and an operator terminal 50.

[0163] The imaging device 30 may be, for example, a device with a wide-angle camera attached to a helmet. In this case, the imaging device may have a blind spot vertically below where a person is present, but can capture images in other directions. The captured image data may be transferred from the imaging device 30 to the on-site terminal 20 via a recording medium such as a MicroSD card, rather than wireless communication, or via a communication cable such as Type C. The imaging device 30 may transmit the captured image data directly to the support server device 10, or may transfer the captured image data to the support server device 10 via a device other than the on-site terminal 20.

[0164] In addition, the user who performs the manual operations described in the embodiment (the operation of specifying a specific range and the operation of specifying predetermined information in association with a specific image) is not limited to the operator W3. For example, the operations may be performed by the work supervisor W2 or the field worker W1. In addition, the operations may be performed by other workers belonging to the work business operator or by an outsourced company entrusted to the work business operator.

[0165] <Configuration variations> The configuration shown in FIG. 1 and the like is an example, and other aspects may be adopted as long as there is no inconvenience in implementation. For example, one device may be distributed to two or more devices, or may be replaced by a cloud computing system. Furthermore, the functions of one device may be distributed and realized to two or more devices, or the functions of two or more devices may be concentrated and realized by one device. Furthermore, the operation performed by one function may be distributed to two or more functions, or two or more functions may be integrated into one function. In short, as long as each function required for the entire site management support system 1 is realized, the devices that realize those functions may have any configuration.

[0166] The above-described aspects of the embodiment are an information processing device such as the support server device 10, and an information processing system such as the site management support system 1 including the support server device 10, but may also be an information processing method. The information processing method includes each step executed by the site management support system 1. The above-described aspects of the embodiment may also be a program. The program causes a computer to execute each step executed by the site management support system 1.

[0167] <Other variations> The output destination of information or data (hereinafter referred to as "information, etc.") may be another device, a display, a memory unit (including a built-in memory unit and an external memory unit), etc. Acquisition of information, etc. includes acquiring information, etc. generated by the device itself, in addition to acquiring information, etc. transmitted from another device. The table in which parameters are associated is not limited to the table shown in the figure, and the number of parameters may be reduced or increased. Furthermore, information, etc. corresponding to parameters may be obtained by a formula, a conditional formula, etc., without using a table.

[0168] <Additional Notes> Furthermore, it may be provided in the following aspects:

[0169] (1) An information processing system comprising a processor, wherein in a site acquisition step, the processor acquires measurement results of a site where work related to a structure is being performed by a measuring device, the measuring device having a sensor that measures electromagnetic waves arriving from an object, in an analysis step, the processor executes a predetermined analysis process based on the acquired measurement results, and in a storage step, the processor stores the results of the executed analysis process.

[0170] According to this embodiment, it is possible to promote the digitization of on-site work.

[0171] (2) In the information processing system described in (1) above, the processor acquires work information related to the work in a work acquisition step, and performs the analysis process based on the acquired measurement results and the acquired work information in the analysis step.

[0172] According to this embodiment, a wider variety of items can be converted into data.

[0173] (3) In the information processing system described in (1) or (2) above, in the on-site acquisition step, the processor acquires, as the measurement result, an image captured by an imaging device capable of capturing images in all directions.

[0174] According to this embodiment, more on-site information can be digitized.

[0175] (4) In the information processing system described in (2) above, the work information includes schedule information indicating a schedule for the work, and in the analysis step, the processor executes, as the analysis process, a process for analyzing the progress of the work based on the on-site situation indicated by the acquired measurement results and the schedule indicated by the acquired work information.

[0176] According to this embodiment, the progress of work can be managed.

[0177] (5) In the information processing system described in (4) above, the work information includes information regarding the worker performing the work, and in the analysis step, the processor executes, as the analysis process, a process of analyzing the construction capability of the worker indicated by the work information based on the results of the progress analysis process and the acquired work information.

[0178] According to this embodiment, the construction capabilities of the worker can be grasped. Of course, this is not the case. Furthermore, the above-described embodiments and modifications may be combined in any desired manner.

[0179] Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The new embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments and their modifications are within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0180] 1: On-site management support system 2: Communication lines 3: Bar 4: External battery 5: External power supply 6: Breaker 7: Connector 8: Stand 10: Support server device 11: Control section 20: On-site terminal 21: Control section 30: Imaging device 31: Control section 40: Supervisory terminal 41: Control section 50: Operator terminal 51: Control section 60: Business server equipment 61: Control section 101: Field acquisition department 102: Work acquisition department 103: On-site analysis department 104: Result storage unit 105: Dashboard section 106: Tool Integration Department 111:DB control section 112: Server display section 113: Image acquisition unit 114: Image processing unit 115: Data Generation Unit 116: Data output section 117: User reception 118: Correspondence processing unit 211: Display control unit 212: Operation reception section 213: Operation control unit 214: Transmission control unit 231: First Communications Department 232: Second Communication Department 233: 3rd Communications Department 311: Display control unit 312: Operation reception section 313: Shooting control unit 314: Transmission control section 331: First Communications Department 332: Second Communications Department 411: Display control unit 412: Operation reception section 511: Display control unit 512: Operation reception section

Claims

1. An information processing system, a processor, the processor comprising: In the site acquisition step, an image of the site where the work on the structure is performed is acquired by an imaging device; In the estimation step, a moving path of the imaging device is estimated based on the acquired photographed images; In the generation step, on the basis of the acquired photographed image and the estimated moving path, site image data is generated which shows the photographed image photographed by the imaging device from a position on the moving path as an image of the work site as seen from the position; In the analysis step, a predetermined analysis process is performed based on the generated on-site image data. In the storage step, the generated on-site image data and a result of the analysis process executed based on the on-site image data are stored. Information processing system.

2. 2. The information processing system according to claim 1, The processor, In the work acquisition step, work information relating to the work is acquired, and the work information includes drawing information of the site; In the analysis step, a process of analyzing an error in the work is executed as the analysis process based on a difference between a situation of the work site shown in the acquired photographed image and a drawing of the work site shown in the acquired work information. Information processing system.

3. 3. The information processing system according to claim 2, The work information includes information about a contractor performing the work, The processor, In the analysis step, a process of analyzing the construction accuracy of the constructor indicated by the work information is executed as the analysis process based on a result of the analysis process of the work error and the acquired work information. Information processing system.

4. 2. The information processing system according to claim 1, The processor, In the task acquisition step, task information relating to the task is acquired, and the task information includes schedule information indicating a schedule for the task and information regarding a worker who will perform the task; In the analyzing step, A process of analyzing the progress of the work based on the site situation indicated by the acquired photographic images and the schedule indicated by the acquired work information; A process of analyzing the construction capability of a worker indicated by the work information based on a result of the process of analyzing the progress and the acquired work information; A process of analyzing a total construction capacity obtained by aggregating the construction capacity of the construction workers in a specific area; as the analysis process. Information processing system.

5. 2. The information processing system according to claim 1, The processor, In the task acquisition step, task information relating to the task is acquired, and the task information includes schedule information indicating a schedule for the task; In the analyzing step, A process of analyzing the progress of the work based on the site situation indicated by the acquired photographic images and the schedule indicated by the acquired work information; A process of analyzing a material transportation schedule that minimizes waiting time or material preparation time at the site based on a result of the process of analyzing the progress; as the analysis process. Information processing system.

Images