Information processing system

Abstract

To provide an information processing system which reduces time and effort of work accompanying photographing a site.SOLUTION: The information processing system comprises a processor capable of executing a program so as to perform the following steps. In an acquisition step A11, a server device acquires a plurality of images which an imaging apparatus captures at respective position of a movement route while moving in an image capture area. In an inference step A12, the movement route is inferred on the basis of the plurality of captured images. In an area output step A21, the plurality of captured images are used to output area image data indicating the image capture area viewed from the respective positions of the inferred movement route. In an extraction step A24, a part including a specific position in the image capture area is extracted as a specific image from one or more images included in the plurality of acquired images. In a specifying and output step A35, specific image data including the extracted specific image is outputted.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an information processing system.

Background Art

[0002] In Patent Document 1, in a mobile terminal, a photographed drawing of each construction work to be photographed, the name of each construction work to be photographed, and a construction photo checklist including the construction design value are stored. Photographing data including the photographed drawing, the name of the construction work, the construction design value, and the actually measured value of the corresponding construction work for which an operation is selected from among the construction works of the photographed drawing and the construction photo checklist of each stored construction work is created. Then, in response to a camera photographing operation in a state where the photographing data is displayed, photo data of the construction work is obtained. The photo number and the actually measured value of the photographed construction work are written into the construction photo checklist. Finally, a technique for transmitting the photo data of each construction work and the updated construction photo checklist to an external construction photo editing device is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] At a construction site or the like, operations associated with photographing at the site are performed, such as photographing and recording specific locations at the site. The photographed images have various utility values, such as representing the situation at the site, but in order to obtain the images necessary for the intended use, corresponding labor is required.

[0005] In view of the above circumstances, the present invention aims to reduce the labor involved in operations associated with photographing at the site.

Means for Solving the Problems

[0006] According to one aspect of the present invention, an information processing system is provided. The information processing system includes a processor capable of executing a program so that the following steps are performed. In the acquisition step, an imaging device acquires a plurality of images taken at each position on a movement path while moving a shooting area. In the estimation step, the movement path is estimated based on the plurality of acquired images. In the area output step, area image data indicating the shooting area as seen from each position on the estimated movement path is output using the plurality of acquired images. In the extraction step, a portion in which a specific location in the shooting area is reflected is extracted as a specific image from one or more of the plurality of acquired images. In the specific output step, specific image data including the extracted specific image is output.

[0007] According to such an aspect, the labor of the work associated with on-site shooting can be reduced.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic matters shown in the embodiments shown below can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium readable by a computer, may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its function is realized on a client terminal (so-called cloud computing).

[0011] Also, in this embodiment, the “unit” may include, for example, hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in this embodiment, various information is handled, and these information are represented, for example, by physical values of signal values representing voltage and current, the high and low of signal values as a set of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0012] In addition, a circuit in a broad sense is a circuit realized by appropriately combining at least a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] 1. Hardware Configuration In this section, the hardware configuration of the construction support system according to this embodiment will be described.

[0014] FIG. 1 is a diagram showing the overall configuration of the on-site management support system 1. In FIG. 1, an overview of each device included in the on-site management support system 1 and the users who use those devices is shown. Each overview will be described as needed with reference to other figures.

[0015] The on-site management support system 1 is an information processing system that executes processing for supporting the management of work at a site such as construction, building, or civil engineering. The on-site management support system 1 includes a communication line 2, a selfie stick 3, an external battery 4, an external power supply 5, a breaker 6, a server device 10, an on-site terminal 20, an imaging device 30, a supervision terminal 40, and an operator terminal 50.

[0016] The communication line 2 includes the Internet and the like, and mediates the exchange of data between devices connected to its own line. The server device 10 is connected to the communication line 2 by wire, and the on-site terminal 20 and the supervision terminal 40 are connected to it wirelessly. In this embodiment, the on-site terminal 20 communicates with the communication line 2 via mobile communication. Also, the on-site terminal 20 performs wireless communication with the imaging device 30 using two communication methods. The two communication methods are Wi-Fi communication and BLE (Bluetooth (registered trademark) Low Energy) communication in this embodiment.

[0017] The on-site terminal 20 and the imaging device 30 are terminals used, for example, by the on-site worker W1, installed at the work site, and execute processing related to on-site photography. The supervision terminal 40 is a terminal used, for example, by the work supervisor W2 in charge of the work site, and executes processing for managing the progress of the work and the like. The operator terminal 50 is a terminal used, for example, by the operator W3 of the on-site management support system 1, and accepts various operations related to the images captured by the imaging device 30.

[0018] The on-site terminal 20 is connected to the external power source 5 via the external battery 4 by a cable. The imaging device 30 is detachably connected to the external power source 5 via the external battery 4 by a cable and a connector 7. In other words, the external power source 5 also supplies power to the imaging device 30 that is detachably connected. The external battery 4 has a so-called pass-through function that can supply power while charging, and supplies power to the on-site terminal 20 and the imaging device 30 while being charged by the power supplied from the external power source 5 with the breaker 6 turned on.

[0019] The imaging device 30 is a digital camera equipped with an image sensor, and captures an image indicated by the light measured by the image sensor. In the present embodiment, the imaging device 30 is a 360-degree camera (also referred to as an omnidirectional camera or a full-sphere camera) capable of capturing images in all directions, up, down, left, right, front, and back. The imaging device 30 is attached to the selfie stick 3, and the selfie stick 3 can be inserted into and fixed to the stand 8 installed at the work site.

[0020] By the on-site worker W1 removing the connector 7 and walking around the work site with the selfie stick 3 removed from the stand 8, captured image data showing an image captured by the 360-degree camera with the work site as the imaging area is generated. In the on-site management support system 1, the image captured by the imaging device 30 is a moving image in the present embodiment, but may be a still image captured continuously as long as images of various locations of the work site can be obtained. The imaging device 30 transmits the generated captured image data to the on-site terminal 20.

[0021] The on-site terminal 20 is a terminal that serves as the main user interface for the on-site worker W1, and is, for example, a smartphone. The on-site terminal 20 controls the operation of the imaging device 30 using, for example, one of the above two communication methods (BLE communication in the present embodiment). Further, the on-site terminal 20 transfers the captured image data transmitted from the imaging device 30 by one of the above two communication methods (Wi-Fi communication in the present embodiment) to the server device 10 using yet another wireless communication (mobile communication in the present embodiment).

[0022] The server device 10 performs image processing using the image of the work site indicated by the captured image data transmitted from the on-site terminal 20, and generates, for example, area image data showing the image of the work site as seen from the position where the imaging device 30 performed the imaging as the image of the imaging area. The supervision terminal 40 refers to the generated area image data and displays images of various locations of the work site. The work supervisor W2 grasps the situation at the site from the displayed images of the work site and gives work instructions to the on-site worker W1 at the site as necessary.

[0023] Depending on the work site, the breaker 6 may be turned off for reasons such as power saving after the work is completed. In that case, power is no longer supplied from the external power source 5 after the breaker 6 is turned off. Thus, the external power source 5 can switch between power supply and non - supply. Although the on - site terminal 20 and the imaging device 30 also have built - in batteries and do not stop immediately, there are also processes that take time such as the transmission of image data. Therefore, in this embodiment, an external battery 4 is provided to increase the operating time of the on - site terminal 20 and the imaging device 30 after the breaker 6 is turned off.

[0024] The operator terminal 50 receives an operation by the operator W3 for extracting, as a specific image, a portion in which a specific location is shown from the image of the work site photographed by the imaging device 30. In this embodiment, a location to be checked for confirming the progress of the work process is used as the specific location. The server device 10 extracts a specific image based on the operation of the operator W3 on the operator terminal 50 and outputs specific image data indicating the extracted specific image. The specific image data is, in this embodiment, a checklist listing specific locations. The supervision terminal 40 displays the output specific image data (checklist) and receives, for example, a work progress check operation by the work supervisor W2.

[0025] Figure 2 is a diagram showing the hardware configuration of the server device 10. The server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a bus 14. The bus 14 electrically connects each part included in the server device 10.

[0026] (Control unit 11) The control unit 11 is, for example, a Central Processing Unit (CPU) not shown in the figure. The control unit 11 is a computer that realizes various functions related to the on-site management support system 1 by reading a predetermined program stored in the storage unit 12. That is, the information processing by software stored in the storage unit 12 is specifically realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and may be implemented to have a plurality of control units 11 for each function, or a combination thereof may also be used.

[0027] (Storage unit 12) The storage unit 12 stores various information defined by the foregoing description. This can be implemented, for example, as a storage device such as a Solid State Drive (SSD) that stores various programs and the like related to the on-site management support system 1 executed by the control unit 11, or as a memory such as a Random Access Memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to the calculation of the program. The storage unit 12 stores various programs, variables, etc. related to the on-site management support system 1 executed by the control unit 11.

[0028] (Communication unit 13) The communication unit 13 is configured to be able to transmit various electrical signals from the server device 10 to external components. Also, the communication unit 13 is configured to be able to receive various electrical signals from external components to the server device 10. More preferably, the communication unit 13 has a network communication function, and thereby various information may be communicated between the server device 10 and external devices via the communication line 2.

[0029] Figure 3 is a diagram showing the hardware configuration of the on-site terminal 20. The on-site terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, an internal power supply unit 26, and a bus 27. The bus 27 electrically connects each part included in the on-site terminal 20. The control unit 21 and the storage unit 22 are of the same hardware although there are performance differences from the control unit 11 and the storage unit 12 shown in Figure 2.

[0030] (Communication unit 23) The communication unit 23 includes a first communication unit 231, a second communication unit 232, and a third communication unit 233, and is an example of a wireless communication unit that performs three types of wireless communication. The first communication unit 231 performs wireless communication by Wi-Fi communication as the first wireless communication in this embodiment. The second communication unit 232 performs wireless communication by BLE as the second wireless communication that has a slower communication speed and lower power consumption than the first wireless communication. The third communication unit 233 performs wireless communication by mobile communication as the third wireless communication that has a wider communicable area than the first wireless communication and the second wireless communication.

[0031] (Input unit 24) The input unit 24 has keys, buttons, a touch screen, a mouse, etc., and receives input from the user. (Output unit 25) The output unit 25 has a display (including a touch screen), a speaker, etc., and displays visual information generated in a manner visible to the user, such as a screen, an image, an icon, text, etc. on the display surface, and outputs sounds including voices.

[0032] (Internal power supply unit 26) The internal power supply unit 26 is a battery built into the device itself, that is, a rechargeable battery, and supplies the accumulated power to each part of the device itself. The internal power supply unit 26 is an example of a portable battery that can be carried together with the device itself. The internal power supply unit 26 is charged by the power supplied from the external power supply 5. The internal power supply unit 26 has a pass-through function similar to the external battery 4, and supplies power to each part while being charged by the power supplied from the external power supply 5 when the breaker 6 is turned on.

[0033] FIG. 4 is a diagram showing the hardware configuration of the imaging device 30. The imaging device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, an internal power supply unit 36, an imaging unit 37, and a bus 38. The bus 38 electrically connects each part included in the imaging device 30. Each part from the control unit 31 to the internal power supply unit 36 has the same hardware although there are performance differences from each part from the control unit 21 to the internal power supply unit 26 shown in FIG. 3.

[0034] However, the communication unit 33 includes only a first communication unit 331 and a second communication unit 332. The first communication unit 331 performs wireless communication by Wi-Fi communication as the first wireless communication, similar to the first communication unit 231 of the communication unit 23 in the present embodiment. The second communication unit 332 performs wireless communication by BLE as the second wireless communication having a lower communication speed and lower power consumption than the first wireless communication, similar to the second communication unit 232 of the communication unit 23. Further, the output unit 35 has a light in addition to a display or the like, and irradiates light for ensuring the amount of light necessary for imaging. The input unit 34 has a switch for turning on the light.

[0035] (Imaging unit 37) The imaging unit 37 is a sensor that has an optical system including a lens and an image sensor or the like, and measures light incident from the lens to generate captured image data. In the present embodiment, as described above, the imaging unit 37 uses an ultra-wide-angle lens and a plurality of image sensors to generate captured image data obtained by imaging omnidirectionally in the up-down, left-right, and front-back directions.

[0036] FIG. 5 is a diagram showing the hardware configuration of the supervision terminal 40. The supervision terminal 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, an output unit 45, and a bus 46. The bus 46 electrically connects each unit included in the supervision terminal 40. Each unit from the control unit 41 to the output unit 45 has the same hardware although there are performance differences from each unit from the control unit 31 to the output unit 35 shown in FIG. 4. The operator terminal 50 includes hardware such as a control unit 51 (only the control unit 51 is given a different reference numeral from the supervision terminal 40) in the same manner as the supervision terminal 40.

[0037] 2. Functional Configuration In this section, the functional configuration of the present embodiment will be described. As described above, the information processing by software stored in the storage unit of each device is specifically realized by the control unit which is an example of hardware, so that each functional unit included in the control unit can be executed.

[0038] FIG. 6 is a diagram showing the functional configuration of the control unit of each device. The control unit 11 of the server device 10 includes a DB control unit 111, a server display unit 112, an image acquisition unit 113, an image processing unit 114, a data generation unit 115, a data output unit 116, a user reception unit 117, and an association processing unit 118. The control unit 21 of the on-site terminal 20 includes a display control unit 211, an operation reception unit 212, an operation control unit 213, and a transmission control unit 214. The control unit 31 of the imaging device 30 includes a display control unit 311, an operation reception unit 312, a shooting control unit 313, and a transmission control unit 314. The control unit 41 of the supervision terminal 40 includes a display control unit 411 and an operation reception unit 412. The control unit 51 of the operator terminal 50 includes a display control unit 511 and an operation reception unit 512.

[0039] The DB control unit 111 (DB: Database) of the server device 10 controls the storage of data in the database and the reading of data from the database. The database here may be not only the database stored in the server device 10 but also the database stored in an external device. The database stores, for example, captured image data indicating an image captured by the imaging device 30 and the above-described region image data.

[0040] The server display unit 112 executes processing for displaying a system screen related to the on-site management support system 1 on each terminal. The server display unit 112 performs processing such as generating and transmitting an HTML (Hyper Text Markup Language) file, which is display data, and causes a web page showing the system screen to be displayed on the supervision terminal 40 or the like. Note that the server display unit 112 may perform processing such as generating and transmitting display data of an application for using the on-site management support system 1.

[0041] The image acquisition unit 113 acquires an image of the imaging region captured by the imaging device 30. The image processing unit 114 executes image processing on the image acquired by the image acquisition unit 113. The image processing unit 114 performs processing related to a technique called Visual SLAM (Simultaneous Localization and Mapping), which simultaneously estimates its own position and creates a map of the surrounding environment based on the image of the imaging region, for example.

[0042] The data generation unit 115 generates data based on the result of the image processing by the image processing unit 114. For example, the data generation unit 115 generates the above-described region image data based on the result of Visual SLAM. The data output unit 116 outputs the data generated by the data generation unit 115. For example, the data output unit 116 outputs the generated region image data to the supervision terminal 40. The user reception unit 117 receives a specific operation by the user via a terminal used by the user (for example, the operator terminal 50). The association processing unit 118 executes processing for associating an image with information. The specific operation and the association of the image and information will be described in detail later.

[0043] The display control unit 511 of the operator terminal 50 controls the display process to the display means of its own device. The operation reception unit 512 receives the operation of the user (for example, operator W3). The operation received by the operation reception unit 512 includes the above-described specific operation. The display control unit 411 of the supervision terminal 40 controls the display process to the display means of its own device. The operation reception unit 412 receives the operation of the user (for example, work supervisor W2).

[0044] The display control unit 211 of the on-site terminal 20 controls the display process to the display means of its own device. The operation reception unit 212 receives the operation of the user (for example, on-site worker W1). The operation control unit 213 controls the operation of the imaging device 30. The transmission control unit 214 controls the transmission process of the captured image data by its own device and the imaging device 30.

[0045] The display control unit 311 of the imaging device 30 controls the display process to the display means of its own device. The operation reception unit 312 receives the operation of the user (for example, on-site worker W1). The shooting control unit 313 controls the shooting process by the imaging unit 37. The transmission control unit 314 controls the transmission process of the captured image data by its own device.

[0046] 3. Information Processing In this section, in this embodiment, the information processing that the program causes the computer to execute will be described. The on-site management support system 1 executes imaging image processing that performs various processes based on an image captured with a work site as a shooting area, for example.

[0047] FIG. 7 is an activity diagram showing an example of imaging image processing. The activity shown in FIG. 7 is started when the server device 10 receives the captured image data of the work site captured by the imaging device 30. First, in A11, the server device 10 acquires a plurality of images captured by the imaging device 30 by the image acquisition unit 113. These plurality of images are also simply referred to as "captured images" hereinafter.

[0048] In this embodiment, the captured image is each frame shown by the video data and is an omnidirectional image captured by a 360-degree camera. Further, the captured image is an image captured at each position on the movement path while the imaging device 30 carried by the on-site worker W1 moves through the work site. The work site is an example of a shooting area captured by the imaging device 30. The image acquisition unit 113 is an example of an acquisition unit that acquires a captured image.

[0049] <Generation of Region Image Data> Next, the server device 10 performs processing for generating the above-described region image data. First, in A12, the server device 10 estimates the movement path of the imaging device 30 by the image processing unit 114 based on the captured image acquired by the image acquisition unit 113. The image processing unit 114 is an example of an estimation unit. The image processing unit 114 estimates the movement path using the VSLAM technology. There are, for example, the following two methods for the VSLAM technology.

[0050] The first is a method of indirectly estimating the position (indirect method), which is a method of estimating the position of the feature point and the position of the camera by associating the feature points between a plurality of images. When using the indirect method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the association of the feature points of the objects shown by the acquired plurality of captured image data.

[0051] The second is a method of directly estimating the position (direct method), which is a method of estimating the depth of each pixel and the position of the imaging device 30 by minimizing the error of the luminance values of corresponding pixels among a plurality of images. When using the direct method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the error of the values of corresponding pixels respectively indicated by the plurality of acquired captured image data.

[0052] Subsequently, in A13, the server device 10, based on the captured image acquired in A11 and the movement path estimated in A12 by the data generation unit 115, generates the above-described region image data, that is, data indicating the work site viewed from the position where the imaging device 30 captured images as the captured region. The region image data is output to a terminal such as the supervision terminal 40, for example. The terminal to which the region image data has been output displays the region image indicated by the region image data.

[0053] FIG. 8 is a diagram showing an example of the displayed region image. In FIG. 8, a display image A11 and a plan view C11 are shown. The display image A11 is an image showing a part of a region image A1, which is one of a plurality of images (captured images) capturing the construction site C1. The plan view C11 is a plan view showing the structure of the construction site C1. The plan view C11 is shown superimposed on the display image A11. In the display image A11 and the plan view C11, position images P1, P2, P3, P4, etc. indicating the shooting positions are shown along the estimated movement path.

[0054] The region image A1 shown in FIG. 8 is an omnidirectional image captured by the imaging device 30 from the shooting position indicated by the position image P9. The display image A11 is a rectangular image showing a certain angle of view among the region image A1. Note that although the region image A1 is distorted in order to represent the omnidirectional image as a rectangle, in the display image A11, the distortion is corrected and the objects are imaged in the same way as a normal photograph.

[0055] When an operation is performed to move the display image A11 in the vertical direction, horizontal direction, or diagonal direction, the area image A1 moves in the operated direction, and the portion displayed as the display image A11 changes. Further, when an operation is performed to select the area image A1 and the position image shown in the drawing B1, the area image captured from the shooting position indicated by the selected position image is displayed. By thus displaying various portions of the area image at each shooting position indicated by the area image data as the display image, the overall state of the construction site C1 can be confirmed.

[0056] <Extraction of Specific Image> Next, the server device 10 performs processing for extracting a specific image from the captured image. As described above, the specific image is a portion in which a specific location in the area image is reflected. In the present embodiment, the specific image is extracted through a manual operation in which the user designates a specific range in one image included in the captured image, and an automatic process of presenting a candidate image indicating a candidate for the specific range to the user as a preceding step thereto.

[0057] First, in A21, the server device 10 extracts, by the image processing unit 114, an image candidate of a specific range from one area image included in the captured image as a candidate image. The image processing unit 114 extracts the candidate image, for example, based on information regarding check points defined by an operator performing on-site work for improving work quality. In the present embodiment, the server device 10 stores a specific image database including information regarding check points.

[0058] FIG. 9 is a diagram showing an example of the specific image database. The specific image database DB1 shown in FIG. 9 stores check point information, a history of specific images, the current specific image, a history of predetermined information, the current predetermined information, and specific image data, respectively. The check point information stores a name and shooting information. The name stores a character string indicating the names of check points such as "entrance", "corridor", "stairs", "kitchen wall", and "kitchen floor". Further, the shooting information stores information indicating a shooting position where the check point can be shot.

[0059] The shooting information is, for example, angle information indicating the angle of view of the check point as seen from the height at which the imaging device 30 normally shoots. The range information indicates the range of the angle of view. For example, assuming that the vertical upward direction is 0 degrees and the vertical downward direction is 180 degrees, if it is the ceiling, it is 0 degrees or more and 30 degrees or less; if it is the wall, it is 45 degrees or more and 135 degrees or less; if it is the floor, it is 150 degrees or more and 180 degrees or less. In addition, the shooting information includes the range of distances suitable for shooting the check point. The distance suitable for shooting is, for example, 0 m (directly above or below) for the ceiling or floor, and 1 m to 5 m for the wall, such that the check point is reflected in an appropriate size. Also, in the case of a location facing a specific direction such as a wall, the direction in which the location faces is also included in the shooting information.

[0060] First, the image processing unit 114 identifies a shooting position from which the check point can be shot based on the estimated movement path. For example, the image processing unit 114 identifies, among the movement path, a position where the distance from the check point on the drawing is a distance suitable for shooting, and in the case of a wall or the like, a position where the direction in which the check point faces is the shooting position. Then, the image processing unit 114 extracts, as a candidate image, an image within the angle of view indicated by the shooting information from the area image shot from the identified position.

[0061] In the case of a location with features in the appearance of a door or staircase, etc., a pattern image indicating those locations may be used as the shooting information. In that case, the image processing unit 114 uses a well-known image recognition technique to extract, as a candidate image, an image of a location similar to the pattern image. The server display unit 112 transmits the extracted candidate image to, for example, the operator terminal 50. The operator terminal 50 displays the transmitted candidate image by the display control unit 511 at A22.

[0062] FIG. 10 is a diagram showing an example of a displayed candidate image. In FIG. 10, a display control unit 511 is displaying a candidate image selection screen D1. On the candidate image selection screen D1, there are displayed a string "Please select a specific image from the candidate images.", candidate images E11, E12, E13 (referred to as "candidate image E10" when not distinguishing each), selection buttons B11, B12, B13 (referred to as "selection button B10" when not distinguishing each), and adjustment buttons B21, B22, B23 (referred to as "adjustment button B20" when not distinguishing each).

[0063] In this way, the server display unit 112 and the image processing unit 114 function as an example of a first presenting unit that presents candidates of images in a specific range to the user as candidate images (candidate images E1, E2, E3 in the example of FIG. 10). According to such an aspect, compared with the case where candidate images are not presented, it is possible to make it easier to select an image at a specific location.

[0064] The selection button B10 is a button for receiving an operation of selecting, as a specific image, the candidate image displayed in association with the selection button. The adjustment button B20 is a button for receiving an operation of adjusting the range shown as a candidate image. These operations are received, for example, by the operation reception unit 512 of the operator terminal 50, and operation data indicating the operation content is transmitted to the server device 10. The user reception unit 117 of the server device 10 receives the operation indicated by the transmitted operation data as an operation performed by the user.

[0065] The user reception unit 117 functions as an example of a designation reception unit that receives a first operation of the user for designating a specific range among one of the plurality of images acquired by the image acquisition unit 113. The user reception unit 117 receives, for example, an operation of selecting the candidate image E10 presented by the server display unit 112 (an operation of pressing any of the selection buttons B10) as the first operation. The first operation referred to here is an operation of designating the range indicated by the candidate image E10 as a specific range.

[0066] Specifically, for example, after an operation of pressing the adjustment button B20 is performed and then an operation of dragging the candidate image E10 is performed, the user reception unit 117 receives the user operation, and the server display unit 112 moves the area image displayed as the candidate image E10 in the dragged direction to display a portion of the area image that has not been displayed until now. In this way, the user reception unit 117 receives, as the first operation, an operation of spatially finely adjusting an image indicating a specific location.

[0067] In addition to the above operations, the user reception unit 117 may receive, as the first operation, an operation of temporally finely adjusting an image indicating a specific location. For example, when an operation of dragging the candidate image E1 while a predetermined button (such as the Ctrl key) is pressed is performed, the user reception unit 117 receives the user operation as the first operation, and the server display unit 112 displays the same portion of the area image whose shooting time is before and after the area image displayed as the candidate image E1.

[0068] Further, the user reception unit 117 may receive, as the first operation, an operation of specifying a specific range on the screen where the area image data shown in FIG. 8 is displayed. In that case, first, the data output unit 116 outputs the area image data generated by the data generation unit 115 to, for example, the operator terminal 50. The display control unit 511 of the operator terminal 50 displays the output area image data. The operation reception unit 512 receives an operation of specifying a specific range from the displayed image displayed as shown in FIG. 8, and the user reception unit 117 receives the operation as the first operation.

[0069] When the operator W3 determines that an appropriate range has been specified as the specific image by the above-described first operation, the operator W3 performs an operation of pressing the selection button B10 as the first operation. In A24, the server device 10 receives a series of first operations by the user reception unit 117, and the image processing unit 114 extracts a candidate image selected by the received first operation or an area image of a specified specific range as the specific image. The DB control unit 111 stores the extracted specific image in the specific image database DB1 as the "current specific image" shown in FIG. 9.

[0070] As described above, the image processing unit 114 functions as an extraction unit that extracts, as the specific image, a portion in which a specific location in the shooting area is reflected from one or more images included in the plurality of images acquired by the image acquisition unit 113. In the present embodiment, the image processing unit 114 extracts the image in the range specified by the first operation as described above as the specific image. According to such an aspect, the user's judgment can be reflected in the extraction of the image of the specific location.

[0071] When a new captured image is acquired and new area image data is generated, the DB control unit 111 moves the extracted specific image from the "current specific image" to the "history of specific images". In the history of specific images, the specific images extracted by the image processing unit 114 are accumulated as history images in this way. The image processing unit 114 may extract candidate images based on the history images accumulated in this way. In that case, the image processing unit 114 specifies, for example, the shooting position and the angle of view of each of the accumulated history images.

[0072] Then, the image processing unit 114 extracts, as candidate images, images of the angle of view within the specified range from the images taken at the shooting positions within the specified range among the newly acquired captured images. The server display unit 112 presents the candidate images extracted in this way to the user. As described above, the candidate images may be images extracted based on the history of the extracted specific images. According to such an aspect, since an image close to the actual specific image is presented as the candidate image, the usefulness of the candidate image can be enhanced as compared with the case where the history of the specific image is not considered.

[0073] <Association of Predetermined Information> Next, the server device 10 performs a process for associating the extracted specific image with the predetermined information. The predetermined information is defined as information for association with the specific image in the on-site management support system 1, and is, for example, the name of a specific location. In this embodiment, the predetermined information is associated with the specific image through a manual operation in which the user designates the predetermined information in association with the specific image, and an automatic process of presenting candidate information indicating candidates for the predetermined information to the user as a preliminary step.

[0074] First, in A31, the server device 10 determines, by the association processing unit 118, candidates for the predetermined information to be associated with the specific image as candidate information. The association processing unit 118 determines, for example, the names of the check location information stored in the specific image database DB1 as candidate information. The server display unit 112 transmits display data for displaying the determined candidate information to, for example, the operator terminal 50. In A32, the operator terminal 50 displays the candidate information indicated by the transmitted display data by the display control unit 511.

[0075] FIG. 11 is a diagram showing an example of the displayed candidate information. In FIG. 11, the display control unit 511 is displaying a candidate information selection screen D2. On the candidate information selection screen D2, a string “Please select the predetermined information from the candidate information.”, a specific image F14, a selection button B31, and a candidate information input field J11 are displayed. Below the candidate information input field J11, a candidate information list J12 is displayed as a pull-down list, and the candidate information selected from the candidate information list J12 is displayed in the candidate information input field J11.

[0076] In this way, the server display unit 112 and the association processing unit 118 function as a second presentation unit that presents candidate information indicating candidates for predetermined information to the user. According to such an aspect, compared with the case where candidate information is not presented, the labor of associating a specific image with the predetermined information can be reduced. The operation of selecting the candidate information and pressing the selection button B31 is received as an operation performed by the user by the user reception unit 117 of the server device 10 in the same manner as the above-described first operation.

[0077] The user reception unit 117 functions as an example of a correspondence reception unit that receives a second operation of the user for associating the specific image extracted by the image processing unit 114 with the predetermined information (in the example of FIG. 11, the specific image F14 and the name of the check location "kitchen wall"). The user reception unit 117 receives an operation of selecting the candidate information presented by the server display unit 112 (the candidate information included in the candidate information list J12 in the example of FIG. 11) as the second operation. This second operation is an operation of associating the candidate information with the specific image as the predetermined information.

[0078] When the operator W3 determines that appropriate candidate information to be associated with the specific image is selected by the above-described second operation, the operator W3 performs an operation of pressing the selection button B31 as the second operation. The operator terminal 50 receives, at A33, these second operations as an operation of selecting candidate information by the operation reception unit 512. The server device 10 receives, at A34, a series of second operations by the user reception unit 117, and the DB control unit 111 associates the candidate information associated with the specific image by the received second operation with the specific image as the "predetermined information this time" shown in FIG. 9 and stores it in the specific image database DB1.

[0079] Then, the data generation unit 115 reads out the "current specific image" and the "current predetermined information" from the specific image database DB1, and generates a list in which they are associated and arranged as specific image data. Here, for example, it is assumed that specific image data is requested from the supervision terminal 40. In that case, in A34, the server device 10 outputs the generated specific image data to the supervision terminal 40 by the data output unit 116. In A35, the supervision terminal 40 displays the output specific image data by the display control unit 411.

[0080] FIG. 12 is a diagram showing an example of the displayed specific image data. In the example of FIG. 12, the display control unit 411 is displaying a checklist H1. In the checklist H1, the names of the check locations, namely, "entrance", "corridor", "stairs", "kitchen wall", and "kitchen floor", the specific images F11, F12, F13, F14, and F15 of each check location (referred to as "specific image F10" when not distinguishing them), and the check results of each check location are displayed. The check results show a symbol indicating the quality of work progress and a comment.

[0081] The specific image F10 includes those extracted from the captured images taken on different dates such as "mm / d1", "mm / d2", "mm / d3", and "mm / d4". For example, the specific images F11-1, F11-2, F11-3, and F11-4 are all images extracted with the "entrance" as a specific location, and are extracted from the captured images taken on "mm / d1", "mm / d2", "mm / d3", and "mm / d4".

[0082] In this way, the data output unit 116 outputs a list of a plurality of specific images (checklist H1 in the example of FIG. 12) as specific image data. Those plurality of specific images (specific images F11-1, F11-2, F11-3, and F11-4, etc. in the example of FIG. 12) have a common specific location and different shooting times. According to such a mode, compared with the case where only the specific image of a single shooting time is output, the transition of the state of the specific location can be easily grasped.

[0083] Note that since the imaging device 30 takes pictures while the on-site worker W1 walks, the movement path during imaging is not the same every time. Therefore, in a plurality of specific images, the exactly same range does not necessarily appear as a specific location, but at least a part of the specific location may appear. By looking at a part of these images, the situation of a specific location at the work site can be grasped.

[0084] As described above, the data output unit 116 functions as an example of a specific output unit that outputs specific image data including the specific images extracted by the image processing unit 114. According to such an aspect, a list indicating specific locations can be easily created. In the present embodiment, the data output unit 116 outputs, as specific image data, a list in which the specific images extracted by the image processing unit 114 and the predetermined information associated with the specific images by the above-described second operation are arranged.

[0085] When a predetermined enlargement operation (for example, single click, etc.) is performed on the specific image F10 included in the checklist H1, the display control unit 411 enlarges and displays the specific image F10. Thereby, the situation at the site can be made easier to view. In the present embodiment, the specific image F10 included in the checklist H1 is a link to the region image data generated based on the captured image from which the specific image F10 was extracted. Further, the specific image F10 is a link to the region image data in a state where the specific location shown in the specific image F10 is the display image.

[0086] At A41, the supervision terminal 40 receives, by the operation reception unit 412, a predetermined operation (for example, double click, etc.) on the specific image F10 as a link operation. The operation reception unit 412 transmits operation data indicating the specific image F10 on which the link operation has been performed to the server device 10. At A42, the server device 10 outputs, by the data output unit 116, the linked region image data to the supervision terminal 40. At A43, the supervision terminal 40 displays, by the display control unit 411, the output region image data.

[0087] For example, when a link operation to a specific image F11-4 shown in FIG. 12 is performed, the area image data generated based on the captured image taken at mm / d4 is output with the entrance hall as the display image and displayed on the supervision terminal 40. The supervision terminal 40 receives, at A44, an operation to move the location shown in the display image of the area image data as a display operation. The work supervisor W2 determines the work situation by operating the area image data in this way to view the surrounding images because, for example, the work situation could not be fully judged only from the specific image F11-4.

[0088] As described above, the data output unit 116 outputs, as specific image data, data including a link from an image of a specific location (each specific image F10 in the example of FIG. 12) to the image showing that location among the images shown by the area image data (for example, the image showing the entrance hall taken at mm / d4 in the case of the specific image F11-4). According to such an aspect, the images around a specific location can be easily confirmed as compared with the case where the specific image data does not include the above link.

[0089] When the work supervisor W2 can determine the work situation by displaying the area image data, the work supervisor W2 returns to the checklist H1 to check the work situation. In the present embodiment, the area image data includes a link for switching the display to the specific image data (the checklist H1 in the example of FIG. 12). FIG. 13 is a diagram showing an example of the displayed area image. In the example of FIG. 13, the display control unit 411 is displaying a display image A41 representing a part of the area image A4.

[0090] The display image A11 shows link images L14-4 and L15-4 (referred to as "link image L10" when not distinguishing between them). The link image L14-4 is a link to the specific image F14-4 shown in FIG. 12. The link image L15-4 is a link to the specific image F15-4 shown in FIG. 12. When the supervision terminal 40 receives a link operation to the link image L10 by the operation reception unit 412 at A51, it transmits operation data indicating the operated link image L10 to the server device 10.

[0091] The server device 10 outputs, at A52, specific image data that is the link destination indicated by the transmitted operation data to the supervision terminal 40 by the data output unit 116. The supervision terminal 40 displays, at A53, the output specific image data by the display control unit 411. When an operation is performed on the link image L14-4, the display control unit 411 displays, for example, the checklist H1 shown in FIG. 12 and an enlarged image of the specific image F14-4. Also, when an operation is performed on the link image L15-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of the specific image F15-4.

[0092] As described above, the data output unit 116 outputs, as area image data, data including a link from an image captured at a position corresponding to a specific location to the image of that location in the specific image data. According to such a mode, even when the area image shown by the area image data is being displayed, it is possible to more easily output the image of a specific location compared to the case where the area image data does not include a link.

[0093] Next, in A54, the supervision terminal 40 receives a check operation by the operation reception unit 412 and transmits operation data to the server device 10. In A55, the server device 10 stores, in the specific image database DB1, specific image data reflecting the check result indicated by the transmitted operation data by the DB control unit 111. The shooting image processing up to A34 shown in FIG. 7 is executed each time shooting is performed at the work site. As a result, the area image data and the specific image data are stored and accumulated each time shooting is performed.

[0094] As described above, the data output unit 116 outputs area image data indicating the shooting area seen from each position of the estimated movement path using a plurality of images (shooting images) acquired by the image acquisition unit 113. Also, as described above, the data output unit 116 outputs specific image data using the shooting images as well. In this way, the on-site management support system 1 utilizes the shooting images for both the area image data as shown in FIG. 8 and the specific image data as shown in FIG. 12. When shooting images of the shooting area to generate the area image data and the specific image data respectively, the labor of shooting is required twice. In the present embodiment, since the shooting images are shared as described above, the labor of the work associated with shooting at the site can be reduced as compared with the case where the shooting images are not shared.

[0095] Also, when the on-site worker W1 shoots a still image with a camera having a normal angle of view while moving the shooting area, even if shooting each specific location one by one, it is not always the case that an image that is easy for the work supervisor W2 to judge the situation is shot, and if shooting again because it is an undesirable image, it is very laborious. Also, even if the shooting is successful, the surrounding situation outside the range of the camera's angle of view is not known. In the present embodiment, as described above, a video is shot with a 360-degree camera that reflects the entire direction, and by extracting specific locations, desirable images can be selected. Also, since the omnidirectional images have a high coverage, it is less likely that desirable images cannot be selected, and even if a specific location to be extracted after shooting occurs, that location can be extracted.

[0096] In addition, although specific locations are shown in the area image data, it is time-consuming to search for only the necessary locations from the area image shown in the area image data when desired. In the present embodiment, by extracting specific locations and outputting specific image data, it is possible to display images of specific locations without such effort. On the other hand, by outputting not only specific image data but also area image data, even when it is difficult to understand from the specific image data which part of the shooting area the specific image is showing, the position of the part shown in the specific image can be grasped by the area image data.

[0097] Also, as described above, when associating specific images and predetermined information, by presenting candidate information, the effort for associating appropriate candidate information can be reduced. In addition, in the present embodiment, as the specific image, the check location shown in FIG. 9 is always extracted. Thereby, when using the specific image data as a checklist, specific locations are always checked at any work site, and the quality of the work can be ensured.

[0098] <Other Embodiments> The shooting area is not limited to the above-described one. For example, it may be inside an existing building, inside a vehicle, or a predetermined outdoor area, etc. Also, the imaging device is not limited to a 360-degree camera, and a wide-angle camera or a camera that shoots with a normal angle of view may be used. Further, the image to be shot is not limited to a moving image, and may be a still image that is continuously shot.

[0099] Also, the area image data is not limited to that shown in FIG. 8. For example, it may be data showing only the area image without the drawing or the position image being superimposed. In that case, it is sufficient that the viewpoint can be moved along the movement path by an operation different from the operation of selecting the position image. Also, the specific image data is not limited to that shown in FIG. 12. For example, it may be data with different specific locations for each shooting, or data showing only the specific image extracted from the latest shooting image.

[0100] In addition, the predetermined information is not limited to the name of the check location. For example, it may be the name of the work site (such as the floor number or room number), the work process, the names of the photographer and work supervisor, the shooting date and time, the weather at the time of shooting, or the type of imaging device. Thus, as the predetermined information, it is preferable to use information indicating what kind of location a specific location is or what kind of image a specific image is. However, information unrelated to the specific image may also be used as the predetermined information.

[0101] <Candidate Image> The image processing unit 114 may extract candidate images in a method different from the embodiment. For example, when the history of a specific range designated by the operator W3 is stored, the image processing unit 114 extracts candidate images based on that history. For example, when the same location in the shooting area has been designated as a specific range a predetermined number of times or more, the image processing unit 114 extracts an image including that location as a candidate image.

[0102] In addition, when multiple rooms with the same floor plan like in an apartment are photographed as the shooting area, the image processing unit 114 extracts images of the designated range in other rooms with the same floor plan as candidate images. The server display unit 112 presents the candidate images thus extracted to the user. In this case, the candidate images are images extracted based on the history of a specific range designated by the user. According to such an aspect, since images similar to the specific images of the range designated by the user in the past are presented as candidate images, the usefulness of the candidate images can be enhanced compared to the case where the history of the specific range is not considered.

[0103] <Candidate Information> The association processing unit 118 may determine candidate information in a method different from the embodiment. For example, the association processing unit 118 recognizes the locations shown in the specific image extracted by the image processing unit 114 and determines the information of the recognized locations as candidate information. For example, the association processing unit 118 recognizes doors, stairs, lighting, windows, washrooms, etc. by image recognition using a pattern image and determines their names as candidate information.

[0104] In this case, the candidate information is information determined based on the location shown in the specific image extracted by the image processing unit 114. According to such a mode, for example, even when a specific location is changed due to the situation at the work site, information corresponding to that location is presented as candidate information, so the usefulness of the candidate information can be enhanced compared to the case where the candidate information is fixed.

[0105] Also, the association processing unit 118 may, for example, determine as candidate information the predetermined information associated with the specific image in the past. Further, the association processing unit 118 may determine as candidate information the predetermined information used in the past at the work sites with the same floor plan not only at the same work site. In any case, the candidate information is information determined based on the past predetermined information. According to such a mode, for example, even the predetermined information added due to the situation at the work site is presented as candidate information, so the usefulness of the candidate information can be enhanced compared to the case where the candidate information is fixed.

[0106] <Narrowing down of images> In the on-site management support system 1, the estimation of the movement route and the extraction of the specific image are also performed based on a plurality of images (captured images) captured by the imaging device 30. In that case, narrowing down to the captured images with better visibility may be performed. The visibility can be determined using, for example, well-known techniques (such as fast Fourier transform, machine learning, open source, etc.) for determining the blurriness of a photo.

[0107] For example, the image processing unit 114 estimates the movement route based on the images narrowed down based on the first rule among the plurality of images acquired by the image acquisition unit 113. Also, the image processing unit 114 extracts the specific image from the images narrowed down based on the second rule among the plurality of images acquired by the image acquisition unit 113. And the second rule may be a rule different from the first rule.

[0108] The first rule is, for example, a rule for narrowing down to an image that looks good across the entire captured image. On the other hand, the second rule is a rule for narrowing down to an image that looks good at a specific location within the captured image. Even in a single image, there may be a mixture of good-looking and bad-looking parts. Therefore, it is possible that although it looks good overall, it looks bad at a specific location, or vice versa, that it looks good at a specific location but looks bad overall. Thus, by using the first rule and the second rule as described above, it is possible to increase the estimation accuracy of the movement path while extracting a specific image that looks good, and it is possible to balance the estimation accuracy of the movement path and the quality of the extracted image.

[0109] Note that the first rule and the second rule are not limited to the above rules. For example, the first rule is a rule for narrowing down to an image in which the feature points are equal to or greater than a threshold value, and the second rule is a rule for narrowing down an image regardless of the number of feature points. In self-position estimation by VSLAM, the accuracy improves as the number of feature points in the image increases. On the other hand, when a specific location is a wall or a floor, the number of feature points becomes extremely small. Thus, by making the first rule and the second rule different as described above, it is possible to balance the estimation accuracy of the movement path and the quality of the extracted image.

[0110] Note that the first rule and the second rule may be the same rule. Even in that case, compared to the case where no narrowing down is performed, both the estimation accuracy of the movement path and the quality of the extracted image can be improved. Also, the above narrowing down of the image may be used when extracting candidate images. In that case, the image processing unit 114 narrows down the captured image based on the second rule (the first rule may also be used), and extracts candidate images from the narrowed-down captured image in the same manner as in the embodiment. Also in this case, the quality of the extracted image can be improved.

[0111] <Automatic Extraction> In the embodiment, the image processing unit 114 extracts a specific image through a manual operation in which the user designates a specific range, but it may also extract the specific image automatically without going through the manual operation. In that case, the image processing unit 114 may extract the specific image by using, for example, the same method for extracting candidate images as it is. The image processing unit 114 extracts a new specific image based on the shooting information of the check points shown in FIG. 9, for example, in the same manner as in the embodiment.

[0112] Also, the image processing unit 114 extracts a new specific image including the same location as a specific location based on, for example, the history of the extracted specific images. Further, the image processing unit 114 extracts a new specific image including the range designated in the past as a specific location based on the history of the specific range designated by the user.

[0113] FIG. 14 is a diagram showing an example of a flowchart of the automatic extraction process. First, in S11, the server device 10 acquires a captured image by the image acquisition unit 113. Next, in S12, the server device 10 estimates the movement path of the imaging device 30 by the image processing unit 114. Subsequently, in S13, the server device 10 generates region image data by the data generation unit 115. Next, in S14, the server device 10 extracts a specific image from the captured image by the image processing unit 114. Subsequently, in S15, the server device 10 determines predetermined information by the association processing unit 118.

[0114] In S14 and S15, unlike the example of FIG. 7, the specific image is extracted and the predetermined information is determined without an operation by the user. Subsequently, in S16, the server device 10 generates specific image data by the data generation unit 115. Then, in S17, the server device 10 outputs the region image data and the specific image data in response to a request from the supervision terminal 40 or the operator terminal 50.

[0115] According to such an aspect, compared with the case of presenting candidate images, the labor required for generating specific image data can be further reduced. Note that when extracting a specific image, the image processing unit 114 may extract a good-looking image as the specific image by narrowing down the captured image using the second rule described above. Further, the image processing unit 114 may extract a specific image by means of the technology of AI (Artificial Intelligence) that has performed machine learning using past specific images as teacher data.

[0116] In addition, the drawing information such as the floor plan of the work site may include position information indicating the position of a specific location. In that case, when a specific location is specified from the drawing information, the image processing unit 114 may identify a captured image captured from a shooting position where there is a high possibility that the specified location is included in the shooting angle, based on the position information of the specified location and the estimated movement route. Note that the server display unit 112 may present the drawing information and the movement route to the user, and the user may identify a captured image captured from a similar shooting position, thereby identifying the captured image.

[0117] Then, the image processing unit 114 recognizes the portion in the captured image where the specific location appears, and extracts the recognized portion as the specific image. Note that the server display unit 112 may present the captured image that has been identified to the user, so that the user may specify the portion in which the specific location appears, and the image processing unit 114 may extract the specified portion as the specific image. According to these aspects, a specific image can be easily extracted for any location shown in the drawing information.

[0118] <Automatic association> In the embodiment, the association processing unit 118 associates predetermined information after a manual operation of designating the predetermined information in association with the specific image, but the predetermined information may be automatically associated without a manual operation. In that case, the association processing unit 118 determines, for example, the predetermined information associated with the history of the specific image that has captured the same specific location as the extracted specific image, as the predetermined information to be associated with the extracted specific image.

[0119] Further, when the association processing unit 118 determines the predetermined information to be associated with the specific image as described above, it determines, as the predetermined information to be associated with an image at another location extracted from the same shooting information as the specific image, the predetermined information existing in the vicinity of the predetermined information. For example, when the predetermined information "kitchen wall" is associated with the specific image, the association processing unit 118 determines, as the predetermined information to be associated with an image at another location (for example, the kitchen floor, ceiling), "kitchen floor" or "kitchen ceiling" existing in the vicinity of the "kitchen wall".

[0120] Further, the association processing unit 118 may identify the location shown in the specific image by an image recognition technique and determine the name etc. of the identified location as the predetermined information. Specifically, the association processing unit 118 identifies the location shown in the specific image using a pattern image associated with the predetermined information (if the predetermined information is "entrance", pattern images of the entrance taken from various directions), and determines the predetermined information associated with the pattern image of the identified location as the predetermined information to be associated with the specific image.

[0121] According to such an aspect, compared with the case of presenting candidate information, the labor required for generating the specific image data can be further reduced. Note that the association processing unit 118 may perform the association of the predetermined information by the technique of AI that has performed machine learning using the predetermined information associated with the past specific images as teacher data.

[0122] Further, the drawing information such as the floor plan of the work site may include the name etc. of each location as the predetermined information. In that case, the association processing unit 118 may estimate the predetermined information corresponding to the specific location shown in the specific image from the drawing information. For example, the association processing unit 118 identifies the position on the drawing of the specific location from the shooting position and the angle-of-view information at which the specific image was taken, and determines the name etc. of the location including the position as the predetermined information.

[0123] In addition, the server display unit 112 may present the drawing information to the user, and the association processing unit 118 may determine, as predetermined information, the name of the location specified by the user (such as operator W3 or work supervisor W2, etc.) from the presented drawing information. According to these aspects, for any predetermined information shown in the drawing information, the association with the specific image can be easily performed.

[0124] <Extraction of Wide-Range Image> When extracting the specific image, the image processing unit 114 may extract, as the specific image, an image in a range that is wider than the range in which a specific location is reflected by a predetermined ratio (hereinafter referred to as the "image ratio"). In this case, when a predetermined operation (such as a flick) is performed while indicating a specific location, the data generation unit 115 generates, as the specific image data, data that also indicates the periphery of that location. Thereby, when it is desired to view the periphery of a specific location, the periphery can be easily displayed.

[0125] Further, the above image ratio may be a value corresponding to the size of the shooting area, the utilization rate of the output area image data, or the utilization rate of the output specific image data. For example, the larger the shooting area, the larger the image ratio the image processing unit 114 sets to extract the specific image. The larger the shooting area, the wider the space around a specific location becomes, and thus the larger the surrounding image to be confirmed becomes.

[0126] In addition, the higher the utilization rate of the area image data or the specific image data, the larger the image ratio the image processing unit 114 sets to extract the specific image. Here, the utilization rate increases as the utilization time of the area image data or the specific image data is longer or the utilization frequency is higher. The higher these utilization rates are, the more likely it is that the opportunity to view the periphery of a specific location increases, and the more likely it is that a situation where a wider range needs to be viewed occurs. As described above, by varying the image ratio, it is possible to confirm the situation in a range of the required width as needed, compared to the case where the image ratio is constant.

[0127] <Recommendation of Specific Image> Specific images may be recommended. For example, when a specific range is specified by the user, the image processing unit 114 not only extracts the image of the specified range, but also extracts, as recommended images, images that seem to be usable as specific images from the area that is spatially adjacent to that range. For example, when the wall of a kitchen is specified, the image processing unit 114 extracts the images of the floor and ceiling of the kitchen as recommended images. Also, when a specific range is specified from a captured image captured at a certain shooting position, the image processing unit 114 extracts, as recommended images, images of the same range from the captured images captured at positions before and after that position.

[0128] The server display unit 112 presents the recommended images extracted in this way to the user together with the specific images. The user performs an operation to instruct to use the recommended image as the specific image in addition to or instead of the specific image. The data generation unit 115 generates specific image data including the instructed recommended image as the specific image. According to such an aspect, the usefulness of the specific image data can be enhanced as compared with the case where the recommended image is not presented.

[0129] <Check with area image data> When a check location is extracted as a specific image, the data generation unit 115 may generate, as area image data, data in which a check image for inputting a check result is also displayed when a specific location is displayed.

[0130] FIG. 15 is a diagram showing an example of the displayed check image. In the example of FIG. 15, the display control unit 411 displays the check images M14-4 and M15-4 on the display image A41. The check image M14-4 is an image for inputting the check result of the specific image F14-4, and the check image M15-4 is an image for inputting the check result of the specific image F15-4. When an input is made to the check image, the DB control unit 111 also reflects the input content in the check result of the check list H1 shown in FIG. 12. According to such an aspect, compared with the case where the check result can be input only to the specific image data, the labor for checking a specific location can be reduced.

[0131] <Others> When a new range is specified as a specific range, the image processing unit 114 may extract an image of the same range from the captured images captured in the past as a past specific image. The data generation unit 115 generates specific image data including the extracted current specific image and the past specific image. Thereby, it is possible to compare past images also for the newly specified range.

[0132] Further, the data generation unit 115 may generate, as specific image data, data that represents in a comparable manner specific images indicating common portions extracted from captured images captured in a plurality of imaging regions having a common structure (for example, rooms of the same floor plan in the same condominium). By arranging the comparison targets in this way, it is possible to easily determine the work situation.

[0133] Further, when the server display unit 112 displays a check list as shown in FIG. 12, it may also display the check results for past similar specific images together. For example, the server display unit 112 arranges and displays an image of the entrance and its check result extracted from other captured images beside the specific image F11-4 of the entrance and its check result. Thereby, the input of the check result can be simplified.

[0134] Further, for example, the image processing unit 114 may determine the check result for the specific image to be checked based on the check results for past similar specific images. For example, when the specific image is similar to a specific image for which the work progress has been checked as "○", the image processing unit 114 may determine the check result as "○", and when the specific image is similar to a specific image for which the work progress has been checked as "×", the image processing unit 114 may determine the check result as "×".

[0135] Then, the server display unit 112 may display the determination result by the image processing unit 114 in the checklist, or may display an alert when the check result input by the user is different from the determination result by the image processing unit 114, thereby assisting the user in performing the check. Further, the DB control unit 111 may reflect the result determined by the image processing unit 114 as the check result in the checklist H1 even without a check by the user. According to these aspects, compared with the case where the determination of the check result by the image processing unit 114 is not performed, the labor for checking specific locations can be reduced.

[0136] Further, the imaging device 30 may not be installed at the work site. For example, a user such as the field worker W1 takes the field terminal 20 and the imaging device 30 to the work site for shooting, causes the imaging device 30 to transmit the captured image data to the field terminal 20, and connects the field terminal 20 to a WiFi router in the office or at home to transfer the captured image data to the server device 10 (it may be transferred by mobile communication). In this case, the field management support system 1 only needs to include the communication line 2, the selfie stick 3, the server device 10, the field terminal 20, the imaging device 30, the supervisor terminal 40, and the operator terminal 50.

[0137] Further, the imaging device 30 may be, for example, a device with a wide-angle camera attached to a helmet. In that case, although the vertical lower part where a person is present becomes a blind spot, it can be an imaging device capable of taking pictures in other directions. Also, the transfer of captured image data from the imaging device 30 to the on-site terminal 20 may be performed via a recording medium such as a MicroSD card instead of wireless communication, or may be performed via a communication cable such as a TypeC. Further, the imaging device 30 may directly transmit the captured image data to the server device 10, or may cause the captured image data to be transferred to the server device 10 via a device other than the on-site terminal 20.

[0138] Also, the user who performs the manual operations (the operation of specifying a specific range and the operation of specifying predetermined information in association with a specific image) described in the embodiment is not limited to the operator W3. For example, the work supervisor W2 may perform it, or the on-site worker W1 may perform it. Also, other workers belonging to the work contractor or a subcontractor commissioned by the work contractor may perform it.

[0139] <Variations of the configuration> In the on-site management support system 1, if it is a work site where the breaker 6 is not turned off, the on-site terminal 20 and the imaging device 30 may be connected to the external power supply 5 without passing through the external battery 4. Also, in the embodiment, the on-site terminal 20 transmitted the measurement data to the server device 10 by mobile communication even when the breaker 6 was turned off. However, when the breaker 6 is not turned off, by installing a Wi-Fi router at the work site, the measurement data may be transmitted to the server device 10 by Wi-Fi communication.

[0140] Further, for example, the server device 10 may be distributed among two or more devices, or may be replaced by a cloud computing system. Also, the on-site terminal 20 and the imaging device 30 may be integrated. Also, the functional configuration shown in FIG. 6 is also an example and is not limited thereto. For example, the functions of the server device 10, the on-site terminal 20, and the imaging device 30 may be realized by being distributed among two or more devices respectively.

[0141] Further, the operations performed by one function may be distributed among two or more functions, or two or more functions may be integrated into one function. In short, as long as each function shown in FIG. 6 is realized in the entire on-site management support system 1, the devices for realizing those functions may have any configuration.

[0142] The aspects of the above-described embodiments were information processing apparatuses such as the server apparatus 10 and information processing systems such as the on-site management support system 1 including the server apparatus 10, but may also be information processing methods. The information processing method includes steps of each process executed by the information processing system. Further, the aspects of the above-described embodiments may also be programs. The program causes a computer to execute each process executed by a similar information processing system.

[0143] <Supplementary Note> Furthermore, it may be provided in each of the aspects described below.

[0144] (1) An information processing system, comprising a processor capable of executing a program so that the following steps are performed. In an acquisition step, an imaging device acquires a plurality of images taken at each position of a movement path while moving a shooting area. In an estimation step, the movement path is estimated based on the plurality of acquired images. In a region output step, region image data indicating the shooting area as seen from each position of the estimated movement path is output using the plurality of acquired images. In an extraction step, a portion in which a specific location in the shooting area appears is extracted as a specific image from one or more of the plurality of images included in the acquired images. In a specific output step, specific image data including the extracted specific image is output.

[0145] According to such an aspect, the labor of the work associated with shooting at the site can be reduced.

[0146] (2) In the information processing system according to (1) above, in the specific output step, a list of a plurality of specific images arranged side by side is output as the specific image data, and the plurality of specific images have a common specific location and different shooting times.

[0147] According to such an aspect, the transition of the state of the specific location can be easily grasped.

[0148] (3) In the information processing system according to (1) or (2) above, in the designation reception step, a first operation of a user for designating a specific range in one of the plurality of images included in the acquired images is received, and in the extraction step, an image in the range designated by the first operation is extracted as the specific image.

[0149] According to such an aspect, the user's judgment can be reflected in the extraction of the image of the specific location.

[0150] (4) In the information processing system according to (3) above, in the first presentation step, candidates for the image of the specific range are presented to the user as candidate images, and in the designation reception step, an operation of selecting the presented candidate image is received as the first operation, and the first operation is an operation of designating the range indicated by the candidate image as the specific range.

[0151] According to such an aspect, it is possible to make it easier to select the image of the specific location.

[0152] (5) In the information processing system according to (4) above, the candidate image is determined based on the history of the extracted specific image or the history of the designated specific range.

[0153] According to such an aspect, the usefulness of the candidate image can be enhanced.

[0154] (6) In the information processing system according to any one of (1) to (5) above, in the correspondence reception step, a second operation of the user for associating the extracted specific image with predetermined information is received, and in the specific output step, a list in which the extracted specific image and the predetermined information associated with the specific image by the second operation are arranged is output as the specific image data.

[0155] According to such an aspect, the information judged by a person can be associated with a specific image.

[0156] (7) In the information processing system according to (6) above, in the second presentation step, candidate information indicating candidates for the predetermined information is presented to the user, and in the correspondence reception step, an operation of selecting the presented candidate information is received as the second operation, and the second operation is an operation of associating the candidate information with the specific image as the predetermined information.

[0157] According to such an aspect, the labor of associating a specific image with predetermined information can be reduced.

[0158] (8) In the information processing system according to (7) above, the candidate information is information determined based on a location shown in the extracted specific image or past predetermined information.

[0159] According to such an aspect, the usefulness of the candidate information can be enhanced.

[0160] (9) In the information processing system according to any one of (1) to (8) above, in the estimation step, the estimation is performed based on an image narrowed down based on a first rule among the plurality of acquired images, and in the extraction step, the specific image is extracted from an image narrowed down based on a second rule among the plurality of acquired images, and the second rule is different from the first rule.

[0161] According to such an aspect, it is possible to achieve both the estimation accuracy of the movement route and the quality of the extracted image.

[0162] (10) In the information processing system according to any one of (1) to (9) above, in the specific output step, data including a link from the image of the specific location to the image showing the location among the images shown by the region image data is output as the specific image data.

[0163] According to such an aspect, the image around a specific location can be easily confirmed.

[0164] (11) In the information processing system according to any one of (1) to (10) above, in the region output step, data including a link from the image taken at the position corresponding to the specific location to the image of the specific location of the specific image data is output as the region image data.

[0165] According to such an aspect, the image of the corresponding location can be easily output.

[0166] (12) In the information processing system according to any one of (1) to (11) above, in the extraction step, an image of a range wider than the range where the specific location appears by a predetermined ratio is extracted as the specific image, and the ratio is a value corresponding to the size of the shooting area, the utilization rate of the output region image data, or the utilization rate of the output specific image data.

[0167] According to such an aspect, the situation in a necessary range can be confirmed. Of course, this is not the limit. Also, the above-described embodiments and modification examples may be arbitrarily combined and implemented.

[0168] Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and their modifications are included in the scope and gist of the invention and are also included in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0169] 1: On-site Management Support System 2: Communication Line 10: Server Device 20: On-site Terminal 30: Imaging Device 40: Supervisor Terminal 50: Operator Terminal 111: DB Control Unit 112: Server Display Unit 113: Image Acquisition Unit 114: Image Processing Unit 115: Data Generation Unit 116: Data Output Unit 117: User Reception Unit 118: Association Processing Unit 411: Display Control Unit 412: Operation Reception Unit 511: Display Control Unit 512: Operation Reception Unit

Claims

1. An information processing system, comprising a processor capable of executing a program so that the following steps are performed, In an acquisition step, an imaging device acquires a plurality of images taken at each position on a movement path while moving a shooting area, In an estimation step, the movement path is estimated based on the plurality of acquired images, In a region output step, region image data indicating the shooting area as seen from each position on the estimated movement path is output using the plurality of acquired images, In an extraction step, a portion in which a specific location in the shooting area appears is extracted as a specific image from one or more images included in the plurality of acquired images, In a specific output step, specific image data including the extracted specific image is output, In a first narrowing-down step, the plurality of acquired images are narrowed down to a first image group based on a first rule, In a second narrowing-down step, the plurality of acquired images are narrowed down to a second image group based on a second rule, In the estimation step, the estimation is performed based on the narrowed-down first image group, In the extraction step, the specific image is extracted from the narrowed-down second image group, and the second rule is different from the first rule, The first rule is a rule for narrowing down the plurality of acquired images to images with good visibility throughout the image, The second rule is a rule for narrowing down the plurality of acquired images to images with good visibility of the specific location, An information processing system.

2. An information processing system, comprising a processor capable of executing a program so that the following steps are performed, In an acquisition step, an imaging device acquires a plurality of images taken at each position on a movement path while moving a shooting area, In an estimation step, the movement path is estimated based on the plurality of acquired images, In a region output step, region image data indicating the shooting area as seen from each position on the estimated movement path is output using the plurality of acquired images, In an extraction step, a portion in which a specific location in the shooting area appears is extracted as a specific image from one or more images included in the plurality of acquired images, In a specific output step, specific image data including the extracted specific image is output, In a first narrowing-down step, the plurality of acquired images are narrowed down to a first image group based on a first rule, In the second narrowing-down step, the plurality of acquired images are narrowed down to a second image group based on a second rule. In the estimation step, the estimation is performed based on the narrowed-down first image group. In the extraction step, the specific image is extracted from the narrowed-down second image group, and the second rule is different from the first rule. The first rule is a rule for narrowing down the plurality of acquired images to images in which the feature points are equal to or greater than a threshold value. The second rule is a rule for narrowing down images regardless of the number of feature points in the plurality of acquired images. An information processing system.

3. In the information processing system according to claim 1 or 2, In the designation reception step, a first operation of a user for designating a specific range in one of the plurality of acquired images is received. In the extraction step, an image in the range designated by the first operation is extracted as the specific image. In the first presentation step, candidates for images in the specific range are presented to the user as candidate images. In the designation reception step, an operation of selecting the presented candidate image is received as the first operation, and the first operation is an operation for designating the range indicated by the candidate image as the specific range. The candidate image is determined based on the history of the extracted specific image or the history of the designated specific range. An information processing system.

4. In the information processing system according to claim 1 or 2, In the association reception step, a second operation of a user for associating the extracted specific image with predetermined information is received. In the specific output step, a list in which the extracted specific image and the predetermined information associated with the specific image by the second operation are arranged is output as the specific image data. In the second presentation step, candidate information indicating candidates for the predetermined information is presented to the user. In the association reception step, an operation of selecting the presented candidate information is received as the second operation, and the second operation is an operation for associating the candidate information with the specific image as the predetermined information. An information processing system.

5. In the information processing system according to claim 4, The candidate information is information determined based on a location shown in the extracted specific image or past predetermined information. An information processing system.

6. In the information processing system according to claim 1 or 2, An information processing system that outputs, as the specific image data, data including a link from the image of the specific location to the image indicating the location among the images indicated by the area image data in the specific output step.

7. In the information processing system according to claim 1 or 2, In the area output step, an information processing system that outputs, as the area image data, data including a link from the image taken at the position corresponding to the specific location to the image of the specific location of the specific image data.

8. In the information processing system according to claim 1 or 2, In the extraction step, an information processing system that extracts, as the specific image, an image in a range wider by a predetermined ratio than the range in which the specific location appears, and the ratio is based on the size of the shooting area, the utilization rate of the output area image data or the utilization rate of the output specific image data.

9. A program that causes a computer to execute each step of the information processing system according to claim 1 or 2. Program.

10. An information processing method that causes an information processing system to execute each step according to claim 1 or 2. Information processing method.

Images