Information Processing Systems

Abstract

To provide an information processing system which reduces time and effort of work accompanying photographing a site.SOLUTION: An information processing system comprises a processor capable of executing a program so as to perform the following steps: an acquisition step S11 of acquiring a plurality of images captured by an imaging apparatus at each position of a moving route while moving in an imaging area; an inference step S12 of inferring a moving route on the basis of the captured images; an area output step S13 of using the acquired images and outputting area image data indicating an imaging area viewed from the positions of the inferred moving route; an extraction step S14 of extracting a part including a specific portion in the imaging area, as a specific image, from one or more images included in the acquired images; and a specific output step S17 of outputting the specific image data including the extracted specific image.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an information processing system. [Background technology]

[0002] Patent Document 1 describes a technique in which a mobile terminal stores photographed drawings of each construction project to be photographed, as well as a construction photo checklist containing the project name and design values ​​of each project. Photographed data is then created containing the photographed drawings, project name, design values, and input actual measured values ​​of a selected construction project from the stored photographed drawings of each construction project and the construction photo checklist. Then, photographed data of the construction project is obtained in response to camera photography operations while the photographed data is displayed. The photograph number and actual measured values ​​of the photographed construction project are written into the construction photo checklist. Finally, the technique discloses a technique in which the photographed data of each construction project and the updated construction photo checklist are sent to an external construction photo editing device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-146682 Summary of the Invention [Problem to be solved by the invention]

[0004] At construction sites, etc., work involving on-site photography is carried out, such as photographing and recording specific locations on the site. The photographed images have various uses, such as showing the site conditions, but obtaining the images necessary for the intended purpose requires a considerable amount of effort.

[0005] In view of the above circumstances, the present invention aims to reduce the amount of work involved in photographing the site. [Means for solving the problem]

[0006] According to one aspect of the present invention, an information processing system is provided. The information processing system includes a processor capable of executing a program to perform the following steps: In the acquisition step, an imaging device acquires multiple images captured at each position on a moving path while moving through a shooting area; In the estimation step, the moving path is estimated based on the multiple acquired images; In the area output step, the multiple acquired images are used to output area image data showing the shooting area as seen from each position on the estimated moving path; In the extraction step, a portion showing a specific location in the shooting area is extracted as a specific image from one or more images included in the multiple acquired images; and In the specific output step, specific image data including the extracted specific image is output.

[0007] According to this aspect, the work involved in taking photographs of the site can be reduced. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing the overall configuration of a site management support system 1. FIG. [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of a server device 10. [Figure 3] FIG. 2 is a diagram illustrating a hardware configuration of a field terminal 20. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of an imaging device 30. [Figure 5] FIG. 2 is a diagram showing the hardware configuration of the supervisor terminal 40. [Figure 6] FIG. 2 is a diagram illustrating the functional configuration of a control unit of each device. [Figure 7] FIG. 10 is an activity diagram showing an example of captured image processing. [Figure 8] FIG. 10 is a diagram showing an example of a displayed area image. [Figure 9] FIG. 10 is a diagram illustrating an example of a specific image database. [Figure 10] FIG. 10 is a diagram showing an example of a displayed candidate image. [Figure 11] FIG. 10 is a diagram illustrating an example of displayed candidate information. [Figure 12] FIG. 10 is a diagram showing an example of displayed specific image data. [Figure 13] FIG. 10 is a diagram showing an example of a displayed area image. [Figure 14] FIG. 10 is a diagram illustrating an example of a flow diagram of an automatic extraction process. [Figure 15] FIG. 10 is a diagram showing an example of a displayed check image. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium, or may be provided so that it can be downloaded from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] In this embodiment, the term "unit" may also include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, various types of information are handled in this embodiment, and this information may be represented by, for example, physical values ​​of signal values ​​representing voltages and currents, high and low signal values ​​as a binary bit set consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations may be performed on a circuit in the broad sense.

[0012] In addition, a circuit in the broad sense is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] 1. Hardware Configuration In this section, the hardware configuration of the construction support system according to this embodiment will be described.

[0014] Fig. 1 is a diagram showing the overall configuration of a site management support system 1. Fig. 1 shows an overview of each device provided in the site management support system 1 and the users who use those devices. Each overview will be explained as needed with reference to other figures.

[0015] The site management support system 1 is an information processing system that executes processes to support the management of work at a site such as architecture, construction, or construction work. The site management support system 1 includes a communication line 2, a selfie stick 3, an external battery 4, an external power supply 5, a breaker 6, a server device 10, a site terminal 20, an imaging device 30, a supervisor terminal 40, and an operator terminal 50.

[0016] The communication line 2 includes the Internet or the like, and mediates data exchange between devices connected to the line itself. A server device 10 is connected to the communication line 2 via a wired connection, and a field terminal 20 and a supervisor terminal 40 are connected wirelessly. In this embodiment, the field terminal 20 communicates with the communication line 2 via mobile communication. The field terminal 20 also communicates wirelessly with the imaging device 30 using two communication methods. In this embodiment, the two communication methods are Wi-Fi communication and BLE (Bluetooth (registered trademark) Low Energy) communication.

[0017] The on-site terminal 20 and the imaging device 30 are, for example, terminals used by a site worker W1, are installed at the work site, and perform processing related to photographing the site. The supervisor terminal 40 is, for example, a terminal used by a work supervisor W2 in charge of the work site, and performs processing for managing the progress of work, etc. The operator terminal 50 is, for example, a terminal used by an operator W3 of the site management support system 1, and accepts various operations related to images captured by the imaging device 30.

[0018] The on-site terminal 20 is connected to the external power source 5 via an external battery 4 by a cable. The imaging device 30 is detachably connected to the external power source 5 via the external battery 4 by a cable and a connector 7. In other words, the external power source 5 also supplies power to the detachably connected imaging device 30. The external battery 4 has a so-called pass-through function that allows it to supply power while charging, and when the breaker 6 is on, it supplies power to the on-site terminal 20 and the imaging device 30 while being charged with power supplied from the external power source 5.

[0019] The imaging device 30 is a digital camera equipped with an image sensor, and captures an image represented by light measured by the image sensor. In this embodiment, the imaging device 30 is a 360-degree camera (also called an omnidirectional camera or spherical camera) that can capture images in all directions: up, down, left, right, front, and back. The imaging device 30 is attached to a selfie stick 3, which can be inserted into a stand 8 installed at the work site and fixed there.

[0020] When the site worker W1 removes the connector 7 and walks around the work site holding the selfie stick 3 removed from the stand 8, captured image data is generated showing images captured by the 360-degree camera with the work site as the capture area. In this embodiment, the images captured by the imaging device 30 in the site management support system 1 are moving images, but they may also be still images captured continuously as long as images of various locations on the work site can be obtained. The imaging device 30 transmits the generated captured image data to the site terminal 20.

[0021] The on-site terminal 20 is a terminal that serves as the main user interface for the field worker W1, and is, for example, a smartphone. The on-site terminal 20 controls the operation of the imaging device 30, for example, using one of the two communication methods described above (BLE communication in this embodiment). The on-site terminal 20 also transfers the captured image data transmitted from the imaging device 30 by one of the two communication methods described above (Wi-Fi communication in this embodiment) to the server device 10 by using another wireless communication (mobile communication in this embodiment).

[0022] The server device 10 performs image processing using the images of the work site indicated by the captured image data transmitted from the site terminal 20, and generates area image data that indicates, for example, an image of the work site as seen from the position where the image was taken by the imaging device 30 as an image of the captured area. The supervisor terminal 40 references the generated area image data and displays images of various locations at the work site. The work supervisor W2 understands the situation at the work site from the displayed images and gives instructions to the field worker W1 on site as necessary.

[0023] Depending on the work site, the breaker 6 may be turned off after work is completed for reasons such as power saving. In this case, power is no longer supplied from the external power source 5 after the breaker 6 is turned off. In this way, the external power source 5 can switch between supplying and not supplying power. The on-site terminal 20 and the imaging device 30 also have built-in batteries, so they do not immediately stop, but some processes, such as transmitting image data, take time. Therefore, in this embodiment, an external battery 4 is provided to increase the operating time of the on-site terminal 20 and the imaging device 30 after the breaker 6 is turned off.

[0024] The operator terminal 50 accepts an operation by the operator W3 to extract a portion showing a specific location from an image of the work site captured by the imaging device 30 as a specific image. In this embodiment, the specific location is a location that needs to be checked to confirm the progress of the work process. The server device 10 extracts the specific image based on the operation of the operator W3 on the operator terminal 50, and outputs specific image data showing the extracted specific image. In this embodiment, the specific image data is a checklist that lists the specific locations. The supervisor terminal 40 displays the output specific image data (checklist) and accepts, for example, an operation to check the work progress by the work supervisor W2.

[0025] 2 is a diagram showing the hardware configuration of server device 10. Server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a bus 14. Bus 14 electrically connects the various units included in server device 10.

[0026] (Control unit 11) The control unit 11 is, for example, a central processing unit (CPU) not shown. The control unit 11 is a computer that realizes various functions related to the site management support system 1 by reading out predetermined programs stored in the memory unit 12. In other words, information processing by software stored in the memory unit 12 is specifically realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and it may be implemented by having multiple control units 11 for each function. It may also be a combination of these.

[0027] (Storage unit 12) The memory unit 12 stores various pieces of information defined above. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the site management support system 1 executed by the control unit 11, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program calculations. The memory unit 12 stores various programs, variables, etc. related to the site management support system 1 executed by the control unit 11.

[0028] (Communications Department 13) The communication unit 13 is configured to be able to transmit various electrical signals from the server device 10 to external components. The communication unit 13 is also configured to be able to receive various electrical signals from the external components to the server device 10. More preferably, the communication unit 13 has a network communication function, which allows various information to be communicated between the server device 10 and external devices via the communication line 2.

[0029] 3 is a diagram showing the hardware configuration of the field terminal 20. The field terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, an internal power supply unit 26, and a bus 27. The bus 27 electrically connects the units included in the field terminal 20. The control unit 21 and the storage unit 22 are similar hardware to the control unit 11 and the storage unit 12 shown in FIG. 2, although their performance differs.

[0030] (Communications Department 23) The communication unit 23 includes a first communication unit 231, a second communication unit 232, and a third communication unit 233, and is an example of a wireless communication unit that performs three types of wireless communication. The first communication unit 231 performs wireless communication by Wi-Fi communication in this embodiment as the first wireless communication. The second communication unit 232 performs wireless communication by BLE in this embodiment as the second wireless communication, which has a slower communication speed and consumes less power than the first wireless communication. The third communication unit 233 performs wireless communication by mobile communication in this embodiment as the third wireless communication, which has a wider communication area than the first wireless communication and the second wireless communication.

[0031] (Input unit 24) The input unit 24 includes keys, buttons, a touch screen, a mouse, etc., and receives input from the user. (Output section 25) The output unit 25 has a display (including a touch screen) and a speaker, and displays visual information generated in a manner that is visible to the user, such as a screen, image, icon, text, etc., on the display surface, and outputs sound including voice.

[0032] (Internal power supply section 26) The internal power supply unit 26 is a battery built into the device itself, i.e., a repeatedly rechargeable battery, and supplies stored power to each component of the device. The internal power supply unit 26 is an example of a portable battery that can be carried around with the device itself. The internal power supply unit 26 is charged with power supplied from the external power supply 5. Like the external battery 4, the internal power supply unit 26 has a pass-through function, and when the breaker 6 is on, it is charged with power supplied from the external power supply 5 and supplies power to each component.

[0033] 4 is a diagram showing the hardware configuration of the imaging device 30. The imaging device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, an internal power supply unit 36, an imaging unit 37, and a bus 38. The bus 38 electrically connects the various units included in the imaging device 30. The units from the control unit 31 to the internal power supply unit 36 ​​are similar in hardware to the units from the control unit 21 to the internal power supply unit 26 shown in FIG. 3, although their performance differs.

[0034] However, the communication unit 33 only includes a first communication unit 331 and a second communication unit 332. Like the first communication unit 231 of the communication unit 23, the first communication unit 331 performs wireless communication via Wi-Fi in this embodiment as the first wireless communication. Like the second communication unit 232 of the communication unit 23, the second communication unit 332 performs wireless communication via BLE in this embodiment as the second wireless communication, which has a slower communication speed and lower power consumption than the first wireless communication. Furthermore, the output unit 35 has a light in addition to a display, etc., and emits light to ensure the amount of light required for shooting. The input unit 34 has a switch for turning on the light.

[0035] (Image capture unit 37) The imaging unit 37 is a sensor that has an optical system including a lens, an image sensor, etc., and measures light incident from the lens to generate captured image data. In this embodiment, as described above, the imaging unit 37 uses an ultra-wide-angle lens and multiple image sensors to generate captured image data captured in all directions, including up and down, left and right, and front and back.

[0036] 5 is a diagram showing the hardware configuration of the supervisor terminal 40. The supervisor terminal 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, an output unit 45, and a bus 46. The bus 46 electrically connects the various units included in the supervisor terminal 40. The units from the control unit 41 to the output unit 45 are similar in hardware to the units from the control unit 31 to the output unit 35 shown in FIG. 4, although their performance differs. The operator terminal 50, like the supervisor terminal 40, includes hardware such as a control unit 51 (only the control unit 51 is assigned a different reference number from that of the supervisor terminal 40).

[0037] 2. Functional configuration This section describes the functional configuration of this embodiment. As described above, information processing by software stored in the storage unit of each device is specifically realized by a control unit, which is an example of hardware, so that each functional unit included in the control unit can be executed.

[0038] 6 is a diagram showing the functional configuration of the control unit of each device. The control unit 11 of the server device 10 includes a DB control unit 111, a server display unit 112, an image acquisition unit 113, an image processing unit 114, a data generation unit 115, a data output unit 116, a user reception unit 117, and an association processing unit 118. The control unit 21 of the on-site terminal 20 includes a display control unit 211, an operation reception unit 212, an operation control unit 213, and a transmission control unit 214. The control unit 31 of the imaging device 30 includes a display control unit 311, an operation reception unit 312, an imaging control unit 313, and a transmission control unit 314. The control unit 41 of the supervisor terminal 40 includes a display control unit 411 and an operation reception unit 412. The control unit 51 of the operator terminal 50 includes a display control unit 511 and an operation reception unit 512.

[0039] The DB control unit 111 (DB: Database) of the server device 10 controls the storage of data in the database and the reading of data from the database. The database here may be not only a database stored in the server device 10 but also a database stored in an external device. The database stores, for example, captured image data indicating images captured by the imaging device 30, the above-mentioned area image data, and the like.

[0040] The server display unit 112 executes processing for displaying on each terminal a system screen relating to the site management support system 1. For example, the server display unit 112 performs processing such as generating and transmitting an HTML (Hyper Text Markup Language) file, which is display data, and displays a web page showing the system screen on the supervisor terminal 40 or the like. Note that the server display unit 112 may also perform processing such as generating and transmitting display data for an application for using the site management support system 1.

[0041] The image acquisition unit 113 acquires an image of the shooting area captured by the imaging device 30. The image processing unit 114 executes image processing on the image acquired by the image acquisition unit 113. The image processing unit 114 performs processing related to a technique called Visual SLAM (Simultaneous Localization and Mapping), which simultaneously estimates the self-position and creates an environmental map based on the image of the shooting area, for example.

[0042] The data generation unit 115 generates data based on the results of image processing by the image processing unit 114. The data generation unit 115 generates the above-mentioned area image data based on, for example, the results of Visual SLAM. The data output unit 116 outputs the data generated by the data generation unit 115. The data output unit 116 outputs, for example, the generated area image data to the supervisor terminal 40. The user reception unit 117 receives a specific operation by a user via a terminal used by the user (for example, the operator terminal 50). The association processing unit 118 executes processing for associating images with information. The association of specific operations with images and information will be explained in detail later.

[0043] The display control unit 511 of the operator terminal 50 controls the display processing on the display means of the own device. The operation reception unit 512 receives operations from a user (e.g., operator W3). The operations received by the operation reception unit 512 include the specific operations described above. The display control unit 411 of the supervisor terminal 40 controls the display processing on the display means of the own device. The operation reception unit 412 receives operations from a user (e.g., work supervisor W2).

[0044] The display control unit 211 of the on-site terminal 20 controls the display processing on the display means of the own device. The operation reception unit 212 receives operations from a user (e.g., field worker W1). The operation control unit 213 controls the operation of the imaging device 30. The transmission control unit 214 controls the transmission processing of captured image data by the own device and the imaging device 30.

[0045] The display control unit 311 of the imaging device 30 controls the display processing on the display means of the device itself. The operation reception unit 312 receives operations from a user (e.g., field worker W1). The photography control unit 313 controls the photography processing by the imaging unit 37. The transmission control unit 314 controls the transmission processing of the photographed image data by the device itself.

[0046] 3. Information Processing In this section, information processing that the program causes the computer to execute in this embodiment will be described. The site management support system 1 executes captured image processing that performs various processes based on images captured in a shooting area, for example, a work site.

[0047] Fig. 7 is an activity diagram showing an example of captured image processing. The activity shown in Fig. 7 is started when the server device 10 receives captured image data of a work site captured by the imaging device 30. First, in A11, the server device 10 acquires, via the image acquisition unit 113, a plurality of images captured by the imaging device 30. Hereinafter, these multiple images will also be simply referred to as "captured images."

[0048] In this embodiment, the captured images are each frame indicated by video data, and are omnidirectional images captured by a 360-degree camera. The captured images are also images captured at various positions along the route of movement by the imaging device 30 carried by the site worker W1 while moving around the work site. The work site is an example of an imaging area captured by the imaging device 30. The image acquisition unit 113 is an example of an acquisition unit that acquires the captured images.

[0049] <Generation of area image data> Next, the server device 10 performs processing for generating the above-mentioned area image data. First, in A12, the server device 10 causes the image processing unit 114 to estimate the movement path of the imaging device 30 based on the captured image acquired by the image acquisition unit 113. The image processing unit 114 is an example of an estimation unit. The image processing unit 114 estimates the movement path using VSLAM technology. There are, for example, the following two methods for VSLAM technology.

[0050] The first method is an indirect position estimation method (indirect method), in which the positions of feature points and the camera position are estimated by associating feature points between multiple images. When the indirect method is used, the image processing unit 114 estimates the movement path of the image capture device 30 based on the association of feature points of objects indicated by each of the multiple pieces of captured image data obtained.

[0051] The second method is a direct position estimation method (direct method), which minimizes the error in the brightness values ​​of corresponding pixels between multiple images to estimate the depth of each pixel and the position of the image capture device 30. When using the direct method, the image processing unit 114 estimates the movement path of the image capture device 30 based on the error in the values ​​of corresponding pixels indicated by each of the multiple acquired captured image data.

[0052] Next, in A13, the server device 10 generates the above-mentioned area image data, i.e., data indicating the work site as an image capture area as seen from the position where the image capture device 30 captured the image, using the data generation unit 115 based on the captured image acquired in A11 and the travel path estimated in A12. The area image data is output to a terminal such as the supervisor terminal 40. The terminal to which the area image data has been output displays the area image indicated by the area image data.

[0053] Fig. 8 is a diagram showing an example of a displayed area image. Fig. 8 shows a display image A11 and a floor plan C11. The display image A11 is an image showing a part of the area image A1, which is one of a plurality of images (photographed images) of the construction site C1. The floor plan C11 is a plan view showing the structure of the construction site C1. The floor plan C11 is shown superimposed on the display image A11. The display image A11 and the floor plan C11 show position images P1, P2, P3, P4, etc., which indicate the photographing positions along the estimated travel route.

[0054] 8 is an omnidirectional image captured by the imaging device 30 from the shooting position indicated by the position image P9. The display image A11 is a rectangular image showing a certain angle of view within the area image A1. Note that the area image A1 is distorted because the omnidirectional image is represented as a rectangle, but the distortion is corrected in the display image A11, so that the object is captured in the same way as in a normal photograph.

[0055] When an operation to move the display image A11 up and down, left and right, or diagonally is performed, the area image A1 moves in the direction of the operation, and the portion displayed as the display image A11 changes. Also, when an operation to select the area image A1 and the position image shown in the drawing B1 is performed, the area image photographed from the photographing position indicated by the selected position image is displayed. In this way, by displaying various portions of the area image at each photographing position indicated by the area image data as the display image, the overall state of the construction site C1 can be confirmed.

[0056] <Extracting specific images> Next, the server device 10 performs processing to extract a specific image from the captured image. As described above, the specific image is a portion of the area image that shows a specific location. In this embodiment, the specific image is extracted through a manual process in which the user specifies a specific range in one image included in the captured image, and then through an automatic process in which candidate images that indicate candidates for the specific range are presented to the user.

[0057] First, in A21, the server device 10 uses the image processing unit 114 to extract candidate images of a specific range from one area image included in the captured image. The image processing unit 114 extracts candidate images based on, for example, information about check points determined by a business operator performing on-site work to improve work quality. In this embodiment, the server device 10 stores a specific image database that includes information about check points.

[0058] Fig. 9 is a diagram showing an example of a specific image database. The specific image database DB1 shown in Fig. 9 stores check location information, specific image history, the current specific image, the history of specified information, the current specified information, and specific image data. The check location information stores a name and photographing information. The name stores a character string indicating the name of the check location, such as "entrance," "hallway," "stairs," "kitchen wall," and "kitchen floor." The photographing information also stores information indicating a photographing position where the check location can be photographed.

[0059] The shooting information is, for example, angle information indicating the angle of view of the check point as seen from a height at which the imaging device 30 normally shoots. The range information indicates the range of the angle of view, where 0 degrees is vertically upward and 180 degrees is vertically downward, such as 0 degrees to 30 degrees for a ceiling, 45 degrees to 135 degrees for a wall, and 150 degrees to 180 degrees for a floor. The shooting information also includes a range of distances suitable for shooting the check point. Suitable shooting distances are, for example, 0 m (directly above or below) for a ceiling or floor, and 1 m to 5 m for a wall, which are distances at which the check point appears at an appropriate size. For a location facing a specific direction, such as a wall, the shooting information also includes the direction in which the location faces.

[0060] The image processing unit 114 first identifies a shooting position from the estimated travel route where the check point can be photographed. For example, the image processing unit 114 identifies a position on the travel route that is a distance from the check point on the drawing that is suitable for photography, and that faces the direction of the check point in the case of a wall, etc., as the shooting position. Then, the image processing unit 114 extracts, from among the area images photographed from the identified position, an image with an angle of view indicated by the photography information as a candidate image.

[0061] In addition, in the case of locations with distinctive appearances, such as doors and stairs, pattern images showing those locations may be used as the photographic information. In this case, the image processing unit 114 uses well-known image recognition technology to extract images of locations similar to the pattern image as candidate images. The server display unit 112 transmits the extracted candidate images to, for example, the operator terminal 50. The operator terminal 50 displays the transmitted candidate images via the display control unit 511 at A22.

[0062] Fig. 10 is a diagram showing an example of displayed candidate images. In Fig. 10, the display control unit 511 displays a candidate image selection screen D1. The candidate image selection screen D1 displays a character string saying "Please select a specific image from the candidate images," candidate images E11, E12, and E13 (referred to as "candidate image E10" when not distinguishing between them), selection buttons B11, B12, and B13 (referred to as "selection button B10" when not distinguishing between them), and adjustment buttons B21, B22, and B23 (referred to as "adjustment button B20" when not distinguishing between them).

[0063] In this way, the server display unit 112 and the image processing unit 114 function as an example of a first presentation unit that presents candidate images of a specific range to the user as candidate images (candidate images E1, E2, and E3 in the example of FIG. 10). According to this aspect, it is easier to select an image of a specific location than when candidate images are not presented.

[0064] The selection button B10 is a button for accepting an operation to select a candidate image displayed in association with the selection button as a specific image. The adjustment button B20 is a button for accepting an operation to adjust the range displayed as a candidate image. These operations are accepted, for example, by the operation acceptance unit 512 of the operator terminal 50, and operation data indicating the operation content is transmitted to the server device 10. The user acceptance unit 117 of the server device 10 accepts the operation indicated by the transmitted operation data as an operation performed by the user.

[0065] The user accepting unit 117 functions as an example of a designation accepting unit that accepts a first operation by the user to designate a specific range in one image included in the plurality of images acquired by the image acquiring unit 113. The user accepting unit 117 accepts, for example, an operation to select a candidate image E10 presented by the server display unit 112 (an operation to press one of the selection buttons B10) as the first operation. The first operation here is an operation to designate the range indicated by the candidate image E10 as the specific range.

[0066] Specifically, for example, when an operation of pressing the adjustment button B20 is performed and then an operation of dragging the candidate image E10 is performed, the user accepting unit 117 accepts the user operation, and the server display unit 112 moves the area image displayed as the candidate image E10 in the dragged direction, thereby displaying a previously undisplayed portion of the area image. In this way, the user accepting unit 117 accepts, as a first operation, an operation of spatially fine-tuning the image showing a specific location.

[0067] In addition to the above operations, the user accepting unit 117 may also accept, as the first operation, an operation for finely adjusting an image showing a specific location over time. For example, when an operation for dragging the candidate image E1 is performed while pressing a predetermined button (such as the Ctrl key), the user accepting unit 117 accepts the user operation as the first operation, and the server display unit 112 displays the same part of an area image captured earlier or later than the area image displayed as the candidate image E1.

[0068] Furthermore, the user accepting unit 117 may accept, as the first operation, an operation of designating a specific range on a screen displaying the area image data shown in Fig. 8. In this case, first, the data output unit 116 outputs the area image data generated by the data generating unit 115 to, for example, the operator terminal 50. The display control unit 511 of the operator terminal 50 displays the output area image data. The operation accepting unit 512 accepts an operation of designating a specific range from the display image displayed as shown in Fig. 8, and the user accepting unit 117 accepts the operation as the first operation.

[0069] When operator W3 determines that an appropriate range has been specified as a specific image through the above-mentioned first operation, he or she performs an operation of pressing selection button B10 as a first operation. At A24, server device 10 receives a series of first operations through user reception unit 117, and extracts, as a specific image, a candidate image selected through the received first operation or an area image of the specified specific range through image processing unit 114. DB control unit 111 stores the extracted specific image in specific image database DB1 as "current specific image" shown in FIG. 9.

[0070] In this way, the image processing unit 114 functions as an extraction unit that extracts, as a specific image, a portion of the shooting area that shows a specific location from one or more images included in the multiple images acquired by the image acquisition unit 113. In this embodiment, the image processing unit 114 extracts, as the specific image, an image of the range specified by the first operation as described above. According to this aspect, the user's judgment can be reflected in the extraction of the image of the specific location.

[0071] When a new photographed image is acquired and new area image data is generated, the DB control unit 111 moves the extracted specific image from "current specific image" to "specific image history." In this way, specific images extracted by the image processing unit 114 are stored as history images in the specific image history. The image processing unit 114 may extract candidate images based on the history images thus stored. In this case, the image processing unit 114, for example, identifies the shooting position and angle of view of each of the stored history images.

[0072] Then, the image processing unit 114 extracts, as candidate images, images with a field of view within the specified range from among the newly acquired captured images taken at shooting positions within the specified range. The server display unit 112 presents the candidate images extracted in this manner to the user. In this manner, the candidate images may be images extracted based on the history of extracted specific images. According to this aspect, images that are close to the actual specific image are presented as candidate images, thereby increasing the usefulness of the candidate images compared to when the history of specific images is not taken into consideration.

[0073] <Matching of specified information> Next, the server device 10 performs a process for associating the extracted specific image with predetermined information. The predetermined information is information, such as the name of a specific location, that is defined as information to be associated with the specific image in the site management support system 1. In this embodiment, the specific information is associated with the specific image through a manual process in which the user specifies the predetermined information in association with the specific image, and an automatic process prior to this process in which candidate information indicating candidates for the predetermined information is presented to the user.

[0074] First, in A31, the server device 10 determines, by the association processing unit 118, candidates for predetermined information to be associated with a specific image as candidate information. For example, the association processing unit 118 determines, as candidate information, the name of check point information stored in the specific image database DB1. The server display unit 112 transmits display data for displaying the determined candidate information to, for example, the operator terminal 50. In A32, the operator terminal 50 displays, by the display control unit 511, the candidate information indicated by the transmitted display data.

[0075] Fig. 11 is a diagram showing an example of displayed candidate information. In Fig. 11, the display control unit 511 displays a candidate information selection screen D2. The candidate information selection screen D2 displays a character string saying "Please select specific information from the candidate information," a specific image F14, a selection button B31, and a candidate information input field J11. Below the candidate information input field J11, a candidate information list J12 is displayed as a pull-down list, and candidate information selected from the candidate information list J12 is displayed in the candidate information input field J11.

[0076] In this way, the server display unit 112 and the association processing unit 118 function as a second presentation unit that presents candidate information indicating candidates for predetermined information to the user. According to this aspect, it is possible to reduce the effort required to associate a specific image with predetermined information compared to when candidate information is not presented. The operation of selecting candidate information and pressing the selection button B31 is accepted by the user acceptance unit 117 of the server device 10 as an operation performed by the user, similar to the first operation described above.

[0077] The user accepting unit 117 functions as an example of a correspondence accepting unit that accepts a second operation by the user to associate the specific image extracted by the image processing unit 114 with predetermined information (in the example of FIG. 11, the specific image F14 with the name of the checked portion, "kitchen wall"). The user accepting unit 117 accepts, as the second operation, an operation to select candidate information presented by the server display unit 112 (in the example of FIG. 11, the candidate information included in the candidate information list J12). This second operation is an operation to associate the candidate information with the specific image as predetermined information.

[0078] When the operator W3 determines that appropriate candidate information to be associated with the specific image has been selected through the second operation, the operator W3 performs the second operation of pressing the selection button B31. At A33, the operator terminal 50 receives these second operations through the operation reception unit 512 as selection operations for candidate information. At A34, the server device 10 receives the series of second operations through the user reception unit 117, and stores the candidate information associated with the specific image through the received second operations in the specific image database DB1 as "current specified information" shown in FIG. 9 in association with the specific image through the DB control unit 111.

[0079] Then, the data generation unit 115 reads out the "current specific image" and the "current specified information" from the specific image database DB1 and generates a list of these in association with each other as specific image data. For example, suppose that specific image data is requested from the supervisor terminal 40. In this case, the server device 10 causes the data output unit 116 to output the generated specific image data to the supervisor terminal 40 at A34. The supervisor terminal 40 then causes the display control unit 411 to display the output specific image data at A35.

[0080] Fig. 12 is a diagram showing an example of displayed specific image data. In the example of Fig. 12, the display control unit 411 displays a checklist H1. The checklist H1 displays the names of the check points, "Entrance," "Hallway," "Stairs," "Kitchen Wall," and "Kitchen Floor," as well as specific images F11, F12, F13, F14, and F15 of each check point (referred to as "specific image F10" when not distinguishing between them), and the check results for each check point. The check results include a symbol indicating whether the work progress is good or bad, and a comment.

[0081] The specific image F10 includes images extracted from images taken on different dates, "mm / d1," "mm / d2," "mm / d3," and "mm / d4." For example, specific images F11-1, F11-2, F11-3, and F11-4 are all images extracted with the "entrance" as the specific location, and are extracted from images taken on "mm / d1," "mm / d2," "mm / d3," and "mm / d4."

[0082] In this way, the data output unit 116 outputs a list of multiple specific images (checklist H1 in the example of FIG. 12) as specific image data. These multiple specific images (specific images F11-1, F11-2, F11-3, and F11-4 in the example of FIG. 12) have a common specific location but were photographed at different times. According to this aspect, it is easier to understand the transition of the state of the specific location compared to when only specific images photographed at a single time are output.

[0083] Note that, since the site worker W1 takes the photographs using the imaging device 30 while walking, the route he or she travels during the photographing is not the same each time. Therefore, the specific images do not necessarily show the exact same range as the specific location, but it is sufficient that at least a part of the specific location is shown. This is because by looking at that part of the image, the situation of the specific location at the work site can be grasped.

[0084] As described above, the data output unit 116 functions as an example of a specific output unit that outputs specific image data including a specific image extracted by the image processing unit 114. According to this aspect, a list indicating specific locations can be easily created. In this embodiment, the data output unit 116 outputs, as specific image data, a list that lists the specific image extracted by the image processing unit 114 and the predetermined information associated with the specific image by the second operation described above.

[0085] When a predetermined enlargement operation (e.g., a single click) is performed on a specific image F10 included in the checklist H1, the display control unit 411 enlarges and displays the specific image F10. This makes it easier to see the situation at the site. In addition, in this embodiment, the specific image F10 included in the checklist H1 is a link to area image data generated based on the captured image from which the specific image F10 was extracted. Furthermore, the specific image F10 is a link to area image data in which a specific location shown in the specific image F10 is displayed as a display image.

[0086] In A41, the supervising terminal 40 receives a predetermined operation (e.g., double-clicking) on ​​the specific image F10 as a link operation via the operation receiving unit 412. The operation receiving unit 412 transmits operation data indicating the specific image F10 for which the link operation has been performed to the server device 10. In A42, the server device 10 outputs the linked area image data to the supervising terminal 40 via the data output unit 116. In A43, the supervising terminal 40 displays the output area image data via the display control unit 411.

[0087] For example, when a link operation to specific image F11-4 shown in Fig. 12 is performed, area image data generated based on the image captured at mm / d4 is output with the entrance as the display image and displayed on the supervisor terminal 40. The supervisor terminal 40 accepts, at A44, an operation to move the part shown in the display image of the area image data as a display operation. For example, since the work supervisor W2 is unable to fully determine the status of the work from specific image F11-4 alone, he determines the status of the work by manipulating the area image data in this way and viewing the surrounding image.

[0088] As described above, the data output unit 116 outputs, as specific image data, data including a link from an image of a specific location (each specific image F10 in the example of FIG. 12) to an image showing that location among the images shown by the area image data (for example, in the case of specific image F11-4, an image showing the entrance photographed at mm / d4). According to this aspect, it is easier to check the image around the specific location compared to when the specific image data does not include the link.

[0089] When the work supervisor W2 can judge the work status by displaying the area image data, he / she returns to the checklist H1 and checks the work status. In this embodiment, the area image data includes a link for switching the display to specific image data (checklist H1 in the example of FIG. 12). Fig. 13 is a diagram showing an example of a displayed area image. In the example of Fig. 13, the display control unit 411 displays a display image A41 that represents a part of the area image A4.

[0090] Display image A11 shows link images L14-4 and L15-4 (referred to as "link image L10" when there is no need to distinguish between them). Link image L14-4 is a link to specific image F14-4 shown in FIG. 12. Link image L15-4 is a link to specific image F15-4 shown in FIG. 12. When the operation receiving unit 412 of the supervising terminal 40 receives a link operation to link image L10 at A51, the supervising terminal 40 transmits operation data indicating the operated link image L10 to the server device 10.

[0091] In A52, the server device 10 outputs the specific image data that is the link destination indicated by the transmitted operation data to the supervisor terminal 40 via the data output unit 116. In A53, the supervisor terminal 40 displays the output specific image data via the display control unit 411. When an operation is performed on link image L14-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of specific image F14-4 shown in FIG. 12. Furthermore, when an operation is performed on link image L15-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of specific image F15-4.

[0092] As described above, the data output unit 116 outputs data including a link from an image taken at a position corresponding to a specific location to the image of that location in the specific image data as area image data. According to this aspect, even when the area image indicated by the area image data is displayed, the image of the specific location can be output more easily than when the area image data does not include a link.

[0093] Next, at A54, the supervisor terminal 40 accepts the check operation through the operation accepting unit 412 and transmits the operation data to the server device 10. At A55, the server device 10 stores the specific image data reflecting the check result indicated by the transmitted operation data in the specific image database DB1 through the DB control unit 111. The captured image processing up to A34 shown in FIG. 7 is executed each time a photograph is taken at the work site. As a result, the area image data and specific image data are saved and accumulated each time a photograph is taken.

[0094] As described above, the data output unit 116 uses multiple images (photographed images) acquired by the image acquisition unit 113 to output area image data that indicates the photographed area as seen from each position on the estimated travel path. Furthermore, as described above, the data output unit 116 also uses the photographed images to output specific image data. In this manner, the site management support system 1 utilizes the photographed images for both the area image data shown in FIG. 8 and the specific image data shown in FIG. 12. If images of the photographed area are photographed to generate the area image data and the specific image data, respectively, the effort of photographing the area is required twice. In this embodiment, the photographed images are shared as described above, and therefore the effort of the work involved in photographing the site can be reduced compared to when the photographed images are not shared.

[0095] Furthermore, if a site worker W1 moves around the shooting area and takes still images with a camera with a normal angle of view, taking pictures of specific locations one by one does not necessarily result in an image that makes it easy for the work supervisor W2 to assess the situation, and taking pictures again if an image is undesirable is extremely time-consuming. Even if a good image is taken, the surrounding situation outside the camera's angle of view cannot be seen. In this embodiment, as described above, a video is taken with a 360-degree camera that captures all directions, and a specific location can be extracted to select a desired image. Furthermore, because omnidirectional images are highly comprehensive, it is rare that a desired image cannot be selected, and even if a specific location that needs to be extracted appears after shooting, that location can be extracted.

[0096] Furthermore, although the area image data also shows a specific location, it is time-consuming to search for the location in the area image indicated by the area image data when viewing only the required location. In this embodiment, by extracting the specific location and outputting the specific image data, it is possible to display an image of the specific location without such time-consuming. On the other hand, by outputting not only the specific image data but also the area image data, even if it is difficult to determine from the specific image data which part of the shooting area the specific image depicts, the location of the location shown in the specific image can be determined from the area image data.

[0097] Furthermore, as described above, when associating a specific image with predetermined information, by presenting candidate information, it is possible to reduce the effort required to associate appropriate candidate information. Furthermore, in this embodiment, the check points shown in Fig. 9 are always extracted as the specific image. As a result, when the specific image data is used as a checklist, the specific points are always checked at any work site, thereby ensuring the quality of the work.

[0098] <Other embodiments> The shooting area is not limited to those described above. For example, it may be the inside of an existing building, the inside of a vehicle, or a predetermined outdoor area. Furthermore, the imaging device is not limited to a 360-degree camera, and a wide-angle camera or a camera that shoots at a normal angle of view may also be used. Furthermore, the captured images are not limited to moving images, and may be still images captured continuously.

[0099] Furthermore, the area image data is not limited to that shown in FIG. 8. For example, it may be data showing only an area image without a drawing or a position image superimposed thereon. In that case, it is sufficient if the viewpoint can be moved along the movement route by an operation different from the operation of selecting the position image. Furthermore, the specific image data is not limited to that shown in FIG. 12. For example, it may be data in which a specific location is changed each time photography is performed, or it may be data showing only a specific image extracted from the most recently photographed image.

[0100] Furthermore, the predetermined information is not limited to the name of the check location, but may also be, for example, the name of the work site (such as the floor number or room number), the work process, the name of the photographer or work supervisor, the date and time of the photo, the weather at the time of the photo, or the type of imaging device. In this way, it is preferable to use information indicating what kind of location the specific location is or what kind of image the specific image is. However, information unrelated to the specific image may also be used as the predetermined information.

[0101] <Candidate image> The image processing unit 114 may extract candidate images using a method different from that of the embodiment. For example, when a history of a specific range designated by the operator W3 has been accumulated, the image processing unit 114 extracts candidate images based on the history. For example, when the same part of the shooting area has been designated as a specific range a predetermined number of times or more, the image processing unit 114 extracts an image including that part as a candidate image.

[0102] Furthermore, when multiple rooms with the same layout are photographed as the photographic area, such as in an apartment building, the image processing unit 114 extracts images of the range specified in other rooms with the same layout as candidate images. The server display unit 112 presents the candidate images thus extracted to the user. In this case, the candidate images are images extracted based on the history of the specific range specified by the user. According to this aspect, images similar to the specific image of the range previously specified by the user are presented as candidate images, thereby increasing the usefulness of the candidate images compared to when the history of the specific range is not taken into consideration.

[0103] <Candidate information> The association processing unit 118 may determine candidate information in a manner different from that of the embodiment. For example, the association processing unit 118 recognizes a location shown in a specific image extracted by the image processing unit 114, and determines information about the recognized location as candidate information. For example, the association processing unit 118 recognizes doors, stairs, lights, windows, washrooms, etc. by image recognition using a pattern image, and determines their names as candidate information.

[0104] In this case, the candidate information is information determined based on the location shown in the specific image extracted by the image processing unit 114. According to this aspect, even if the specific location is changed due to the situation at the work site, for example, information appropriate for that location is presented as candidate information, so the usefulness of the candidate information can be increased compared to when the candidate information is fixed.

[0105] Furthermore, the association processing unit 118 may determine, for example, predetermined information that has been previously associated with a specific image as candidate information. Furthermore, the association processing unit 118 may determine, as candidate information, predetermined information that has been used in the past not only at the same work site but also at work sites with the same layout. In either case, the candidate information is information determined based on past predetermined information. According to such an embodiment, for example, predetermined information that has been added depending on the situation at the work site is also presented as candidate information, so that the usefulness of the candidate information can be increased compared to when the candidate information is fixed.

[0106] <Image Filtering> In the site management support system 1, both estimation of the travel path and extraction of the specific image are performed based on a plurality of images (photographed images) captured by the imaging device 30. At this time, the captured images may be narrowed down to those with better image quality. The quality of the image can be determined, for example, using well-known techniques (such as fast Fourier transform, machine learning, and open source) for determining whether a photograph is out of focus.

[0107] For example, the image processing unit 114 estimates the travel path based on images narrowed down based on a first rule from among the multiple images acquired by the image acquisition unit 113. Furthermore, the image processing unit 114 extracts a specific image from the images narrowed down based on a second rule from among the multiple images acquired by the image acquisition unit 113. The second rule may be a rule different from the first rule.

[0108] The first rule, for example, is a rule that narrows down the captured images to images that have a good overall appearance. On the other hand, the second rule is a rule that narrows down the captured images to images that have a good appearance in a specific part. Even a single image may contain a mixture of good and bad parts. Therefore, it is possible that the overall image is good but a specific part is bad, or that a specific part is good but the overall image is bad. Therefore, by using the first and second rules as described above, it is possible to extract specific images that have a good appearance while improving the estimation accuracy of the travel path, thereby achieving both the estimation accuracy of the travel path and the quality of the extracted image.

[0109] The first and second rules are not limited to the above. For example, the first rule is a rule that narrows down images to those with feature points equal to or greater than a threshold, and the second rule is a rule that narrows down images regardless of the number of feature points. The more feature points an image has, the more accurate the self-location estimation using VSLAM becomes. On the other hand, if a specific location is a wall or floor, the number of feature points will be extremely small. Therefore, by differentiating the first and second rules as described above, it is possible to achieve both high accuracy in estimating the travel path and high quality in the extracted image.

[0110] The first rule and the second rule may be the same rule. Even in this case, the estimation accuracy of the travel route and the quality of the extracted image can be improved compared to when no narrowing down is performed. The narrowing down of images described above may also be used when extracting candidate images. In this case, the image processing unit 114 narrows down the captured images based on the second rule (or the first rule), and extracts candidate images from the narrowed down captured images, as in the embodiment. In this case, the quality of the extracted image can also be improved.

[0111] <Automatic extraction> In the embodiment, the image processing unit 114 extracts a specific image through a manual operation in which a user specifies a specific range, but the specific image may be extracted automatically without manual operation. In this case, the image processing unit 114 may extract the specific image, for example, by using the same method for extracting candidate images as is. For example, the image processing unit 114 extracts a new specific image based on the shooting information of the checked area shown in FIG. 9, as in the embodiment.

[0112] Furthermore, the image processing unit 114 extracts a new specific image that includes the same location as a specific location based on the history of extracted specific images, for example. Furthermore, the image processing unit 114 extracts a new specific image that includes a range previously specified as a specific location based on the history of specific ranges specified by the user.

[0113] 14 is a diagram showing an example of a flow diagram of the automatic extraction process. First, in S11, the server device 10 acquires a captured image using the image acquisition unit 113. Next, in S12, the server device 10 estimates the movement path of the imaging device 30 using the image processing unit 114. Next, in S13, the server device 10 generates area image data using the data generation unit 115. Next, in S14, the server device 10 extracts a specific image from the captured image using the image processing unit 114. Next, in S15, the server device 10 determines predetermined information using the association processing unit 118.

[0114] 7, in S14 and S15, the specific image is extracted and the predetermined information is determined without any operation by the user. Subsequently, in S16, the server device 10 generates specific image data using the data generating unit 115. Then, in S17, the server device 10 outputs the area image data and the specific image data in response to a request from the supervisor terminal 40 or the operator terminal 50.

[0115] According to this embodiment, the effort required to generate specific image data can be further reduced compared to when candidate images are presented. Note that when extracting specific images, the image processing unit 114 may narrow down the captured images using the second rule described above to extract images that look good as specific images. Furthermore, the image processing unit 114 may extract specific images using AI (Artificial Intelligence) technology that performs machine learning using past specific images as training data.

[0116] Furthermore, drawing information such as a floor plan of a work site may contain location information indicating the location of a specific location. In this case, when a specific location is specified from the drawing information, the image processing unit 114 may identify a photographed image taken from a photographing position where the specified location is likely to be included in the angle of view, based on the location information of the specified location and the estimated movement route. Note that the server display unit 112 may present the drawing information and the movement route to the user, and the user may specify a similar photographing location, thereby identifying a photographed image taken from the specified photographing position.

[0117] Then, image processing unit 114 recognizes a portion of the identified photographed image that shows the specific location, and extracts the recognized portion as a specific image. Note that server display unit 112 may present the identified photographed image to the user, allowing the user to specify the portion that shows the specific location, and image processing unit 114 may extract the specified portion as a specific image. According to these aspects, a specific image can be easily extracted if it is a location shown in the drawing information.

[0118] <Automatic mapping> In the embodiment, the association processing unit 118 associates the predetermined information with the specific image through a manual operation of specifying the predetermined information in association with the specific image, but the predetermined information may be associated automatically without manual operation. In this case, the association processing unit 118 determines, for example, the predetermined information associated with the history of specific images that have taken the same specific location as the extracted specific image as the predetermined information to be associated with the extracted specific image.

[0119] Furthermore, when the association processing unit 118 determines predetermined information to be associated with a specific image as described above, it determines predetermined information that exists near the predetermined information as predetermined information to be associated with an image of another location extracted from the same photographing information as the specific image. For example, when predetermined information "kitchen wall" is associated with a specific image, the association processing unit 118 determines "kitchen floor" or "kitchen ceiling" that exists near the "kitchen wall" as predetermined information to be associated with an image of another location (for example, kitchen floor or ceiling).

[0120] Furthermore, the association processing unit 118 may identify a location shown in the specific image using an image recognition technology and determine the name of the identified location as the predetermined information. Specifically, the association processing unit 118 identifies a location shown in the specific image using a pattern image associated with the predetermined information (if the predetermined information is "entrance," pattern images of the entrance photographed from various directions) and determines the predetermined information associated with the pattern image of the identified location as the predetermined information to be associated with the specific image.

[0121] According to this aspect, the effort required to generate specific image data can be further reduced compared to when candidate information is presented. Note that the association processing unit 118 may associate the specific information using AI technology that performs machine learning using specific information associated with past specific images as training data.

[0122] Furthermore, drawing information such as a floor plan of a work site may contain the name of each location as predetermined information. In this case, the association processing unit 118 may estimate the predetermined information corresponding to the specific location indicated by the specific image from the drawing information. For example, the association processing unit 118 identifies the position of the specific location on the drawing from the shooting position and angle of view information at which the specific image was taken, and determines the name of the place including that position as predetermined information.

[0123] The server display unit 112 may present the drawing information to the user, and the association processing unit 118 may determine the name of a place designated by the user (operator W3, work supervisor W2, etc.) from the presented drawing information as the predetermined information. According to these aspects, if the predetermined information is indicated in the drawing information, association with a specific image can be easily performed.

[0124] <Extraction of wide-area images> When extracting a specific image, image processing unit 114 may extract an image having a range that is a predetermined ratio (hereinafter referred to as "image ratio") wider than the range in which the specific location is captured as the specific image. In this case, data generation unit 115 generates data as specific image data that indicates the specific location and also indicates the surroundings of the location when a predetermined operation (such as a flick) is performed. This makes it possible to easily display the surroundings of the specific location when the user wishes to see the surroundings of the specific location.

[0125] The image ratio may be set to a value according to the size of the photographed area, the usage of the output area image data, or the usage of the output specific image data. For example, the larger the photographed area, the larger the image ratio the image processing unit 114 uses to extract the specific image. The larger the photographed area, the larger the space around the specific location, and therefore the larger the surrounding image to be confirmed.

[0126] Furthermore, the image processing unit 114 extracts a specific image by increasing the image ratio as the usage of the regional image data or specific image data increases. The usage here increases as the usage time of the regional image data or specific image data increases or the usage frequency increases. The higher the usage, the more likely it is that there will be opportunities to look around a specific location, and the more likely it will be that a situation will arise where you want to view a wider range. As described above, by varying the image ratio, it is possible to check the situation in a range as wide as necessary, compared to when the image ratio is constant.

[0127] <Recommend specific images> A specific image may be recommended. For example, when a specific range is specified by the user, the image processing unit 114 not only extracts images within the specified range, but also extracts images that are likely to be used as the specific image from areas spatially adjacent to the specified range as recommended images. For example, when a kitchen wall is specified, the image processing unit 114 extracts images of the kitchen floor and ceiling as recommended images. Furthermore, when a specific range is specified from an image captured at a certain shooting position, the image processing unit 114 extracts images of the same range from images captured at positions before and after the specified position as recommended images.

[0128] The server display unit 112 presents the extracted recommended image to the user along with the specific image. The user performs an operation to instruct the recommended image to be set as the specific image in addition to or instead of the specific image. The data generation unit 115 generates specific image data that includes the instructed recommended image as the specific image. According to this aspect, the usefulness of the specific image data can be increased compared to when the recommended image is not presented.

[0129] <Check with area image data> When a check point is extracted as a specific image, the data generation unit 115 may generate, as area image data, data that is displayed together with a check image for inputting the check result when the specific point is displayed.

[0130] FIG. 15 is a diagram showing an example of a displayed check image. In the example of FIG. 15, the display control unit 411 displays check images M14-4 and M15-4 on the display image A41. The check image M14-4 is an image for inputting the check result of the specific image F14-4, and the check image M15-4 is an image for inputting the check result of the specific image F15-4. When input is made to the check images, the DB control unit 111 also reflects the input content in the check result of the checklist H1 shown in FIG. 12. This aspect can reduce the effort required to check specific locations compared to when check results can only be input to specific image data.

[0131] <Other> When a new range is designated as a specific range, the image processing unit 114 may extract an image of the same range from a previously captured image as a previous specific image. The data generation unit 115 generates specific image data including the extracted current specific image and the previous specific image. This makes it possible to compare the newly designated range with the previous image.

[0132] The data generating unit 115 may also generate specific image data that represents, in a comparable manner, specific images showing common locations extracted from images taken in multiple shooting areas with a common structure (for example, rooms with the same layout in the same apartment building). By arranging the comparison targets in this way, it becomes easier to determine the status of the work.

[0133] 12, the server display unit 112 may also display check results for similar specific images from the past. For example, the server display unit 112 may display an image of an entranceway extracted from other captured images and its check results alongside the specific entranceway image F11-4 and its check results. This simplifies the input of check results.

[0134] Furthermore, for example, the image processing unit 114 may determine the check result for the specific image being checked based on the check results for similar specific images in the past. For example, the image processing unit 114 determines the check result as "○" if the work progress is similar to a specific image marked with "○", and determines the check result as "×" if the work progress is similar to a specific image marked with "×".

[0135] The server display unit 112 may then support the user in checking by displaying the judgment results by the image processing unit 114 in a checklist, or by displaying an alert when the check results entered by the user differ from the judgment results by the image processing unit 114. Furthermore, the DB control unit 111 may reflect the results of the judgment by the image processing unit 114 as the check results in the checklist H1 even without a check by the user. According to these aspects, it is possible to reduce the effort required to check specific locations compared to when the check results are not judged by the image processing unit 114.

[0136] Furthermore, the imaging device 30 does not have to be installed at the work site. For example, a user such as a field worker W1 can take a field terminal 20 and imaging device 30 to the work site, take a photograph, have the imaging device 30 send the captured image data to the field terminal 20, and connect the field terminal 20 to a Wi-Fi router in the office or at home to transfer the captured image data to the server device 10 (or via mobile communication). In this case, the field management support system 1 only needs to include the communication line 2, the selfie stick 3, the server device 10, the field terminal 20, the imaging device 30, the supervisor terminal 40, and the operator terminal 50.

[0137] Furthermore, the imaging device 30 may be, for example, a device in which a wide-angle camera is attached to a helmet. In this case, the imaging device may have a blind spot vertically below where a person is located, but can capture images in other directions. Furthermore, the captured image data may be transferred from the imaging device 30 to the on-site terminal 20 via a recording medium such as a MicroSD card, or via a communication cable such as a Type C, rather than wireless communication. Furthermore, the imaging device 30 may transmit the captured image data directly to the server device 10, or may transfer the captured image data to the server device 10 via a device other than the on-site terminal 20.

[0138] Furthermore, the user who performs the manual tasks described in the embodiment (the task of specifying a specific range and the task of specifying predetermined information in association with a specific image) is not limited to the operator W3. For example, the tasks may be performed by the work supervisor W2 or the on-site worker W1. Furthermore, the tasks may also be performed by other workers belonging to the work business or by an outsourced company commissioned by the work business.

[0139] <Configuration variations> In the site management support system 1, if the work site is one in which the breaker 6 is not turned off, the site terminal 20 and the imaging device 30 may be connected to the external power source 5 without the external battery 4. In the embodiment, the site terminal 20 transmits measurement data to the server device 10 via mobile communication even when the breaker 6 is turned off, but if the breaker 6 is not turned off, the site terminal 20 may transmit measurement data to the server device 10 via Wi-Fi communication by installing a Wi-Fi router at the work site.

[0140] Also, for example, the server device 10 may be distributed across two or more devices, or may be replaced by a cloud computing system. Also, the on-site terminal 20 and the imaging device 30 may be integrated. Also, the functional configuration shown in FIG. 6 is an example, and is not limited to this. For example, the functions of the server device 10, the on-site terminal 20, and the imaging device 30 may each be distributed across two or more devices.

[0141] In addition, an operation performed by one function may be distributed to two or more functions, or two or more functions may be integrated into one function. In short, as long as the functions shown in Fig. 6 are realized by the entire site management support system 1, the devices that realize those functions may have any configuration.

[0142] The above-described embodiments are described as information processing devices such as the server device 10 and information processing systems such as the site management support system 1 including the server device 10, but may also be information processing methods. The information processing methods include steps of processes executed by the information processing system. The above-described embodiments may also be programs. The programs cause a computer to execute the processes executed by the same information processing system.

[0143] <Additional Notes> Furthermore, it may be provided in the following aspects.

[0144] (1) An information processing system comprising a processor capable of executing a program to perform the following steps: in the acquisition step, a plurality of images taken by an imaging device at each position of a movement path while the imaging device moves through a shooting area are acquired; in the estimation step, the movement path is estimated based on the acquired plurality of images; in the area output step, area image data showing the shooting area as seen from each position of the estimated movement path is output using the acquired plurality of images; in the extraction step, a portion showing a specific location in the shooting area is extracted as a specific image from one or more images included in the acquired plurality of images; and in the specific output step, specific image data including the extracted specific image is output.

[0145] According to this aspect, the work involved in taking photographs of the site can be reduced.

[0146] (2) In the information processing system described in (1) above, in the specific output step, a list of multiple specific images is output as the specific image data, and the multiple specific images have the specific location in common but were taken at different times.

[0147] According to this aspect, it is possible to easily grasp the state transition of a specific location.

[0148] (3) In the information processing system described in (1) or (2) above, the specification receiving step receives a first operation by the user specifying a specific range of one image included in the acquired plurality of images, and the extraction step extracts the image of the range specified by the first operation as the specific image.

[0149] According to this aspect, the user's judgment can be reflected in the extraction of an image of a specific location.

[0150] (4) In the information processing system described in (3) above, in the first presentation step, candidate images of the specific range are presented to the user as candidate images, and in the designation acceptance step, an operation of selecting the presented candidate image is accepted as the first operation, and the first operation is an operation of designating the range indicated by the candidate image as the specific range.

[0151] According to this aspect, it is possible to easily select an image of a specific location.

[0152] (5) In the information processing system described in (4) above, the candidate images are determined based on the history of the extracted specific images or the history of the specified specific range.

[0153] According to this aspect, the usefulness of the candidate images can be improved.

[0154] (6) In the information processing system described in any one of (1) to (5) above, in the response receiving step, a second operation by the user that associates the extracted specific image with specified information is received, and in the specific output step, a list that lists the extracted specific image and the specified information that has been associated with the specific image by the second operation is output as the specific image data.

[0155] According to this aspect, it is possible to associate information determined by a person with a specific image.

[0156] (7) In the information processing system described in (6) above, in the second presentation step, candidate information indicating candidates for the specified information is presented to the user, and in the response reception step, an operation of selecting the presented candidate information is received as the second operation, and the second operation is an operation of associating the candidate information as the specified information with the specific image.

[0157] According to this aspect, it is possible to reduce the effort required to associate a specific image with predetermined information.

[0158] (8) In the information processing system described in (7) above, the candidate information is information determined based on the location shown in the extracted specific image or the past specified information.

[0159] According to this aspect, the usefulness of the candidate information can be improved.

[0160] (9) In the information processing system described in any one of (1) to (8) above, in the estimation step, the estimation is performed based on images narrowed down from the plurality of acquired images based on a first rule, and in the extraction step, the specific image is extracted from images narrowed down from the plurality of acquired images based on a second rule, and the second rule is different from the first rule.

[0161] According to this aspect, it is possible to achieve both high accuracy in estimating the travel route and high quality in the extracted image.

[0162] (10) In the information processing system described in any one of (1) to (9) above, in the specific output step, data including a link from the image of the specific location to an image showing the location among the images indicated by the area image data is output as the specific image data.

[0163] According to this aspect, it is possible to easily check the image of the surroundings of a specific location.

[0164] (11) In the information processing system described in any one of (1) to (10) above, in the area output step, data including a link from an image taken at a position corresponding to the specific location to an image of the location in the specific image data is output as the area image data.

[0165] According to this aspect, an image of the relevant portion can be easily output.

[0166] (12) In the information processing system described in any one of (1) to (11) above, in the extraction step, an image having a range that is a predetermined percentage wider than the range in which the specific location is captured is extracted as the specific image, and the percentage is a value corresponding to the size of the shooting area, the usage rate of the output area image data, or the usage rate of the output specific image data.

[0167] According to this embodiment, it is possible to check the situation in a range as needed. Of course, this is not the case. Furthermore, the above-described embodiments and modifications may be combined in any desired manner.

[0168] Finally, while various embodiments of the present invention have been described, these are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. The embodiments and their modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the appended claims. [Explanation of symbols]

[0169] 1: On-site management support system 2: Communication line 10: Server device 20: On-site terminal 30: Imaging device 40: Supervisory terminal 50: Operator terminal 111:DB control section 112: Server display area 113: Image acquisition unit 114: Image processing unit 115: Data generation unit 116: Data output section 117: User reception 118: Correspondence processing unit 411: Display control unit 412: Operation reception section 511: Display control unit 512: Operation reception unit

Claims

1. An information processing system, A processor capable of executing a program to perform the following steps: In the acquisition step, the imaging device acquires a plurality of images taken at each position along a moving path while moving through an imaging region; In the estimation step, the travel path is estimated based on the plurality of acquired images; In the area output step, area image data indicating the photographed area as viewed from each position on the estimated moving path is output using the acquired plurality of images; In the extraction step, a portion showing a specific location in the photographed area is extracted as a specific image from one or more images included in the plurality of acquired images; In the specific output step, specific image data including the extracted specific image is output; In the first narrowing down step, the acquired images are narrowed down to a first image group based on a first rule; In the second narrowing down step, the acquired images are narrowed down to a second image group based on a second rule; In the estimating step, the estimation is performed based on the narrowed-down first image group; In the extracting step, the specific image is extracted from the narrowed-down second image group; The second rule is different from the first rule, the first rule and the second rule are rules for narrowing down the acquired plurality of images to images having characteristics defined for each rule; Information processing system.

2. 2. The information processing system according to claim 1, In the designation receiving step, a first operation by a user is received to designate a specific range of one image included in the acquired plurality of images; In the extracting step, an image within a range designated by the first operation is extracted as the specific image; In the first presentation step, candidate images of the specific range are presented to the user as candidate images; In the designation receiving step, an operation of selecting the presented candidate image is received as the first operation, and the first operation is an operation of designating a range indicated by the candidate image as the specific range; An information processing system in which the candidate images are determined based on a history of the extracted specific images or a history of the specified specific range.

3. 2. The information processing system according to claim 1, In the correspondence receiving step, a second operation by the user for associating the extracted specific image with predetermined information is received; In the specific output step, a list of the extracted specific images and the predetermined information associated with the specific images by the second operation is output as the specific image data, In the second presentation step, candidate information indicating candidates for the predetermined information is presented to the user; In the response receiving step, an operation of selecting the presented candidate information is received as the second operation, and the second operation is an operation of associating the candidate information with the specific image as the specified information.

4. 4. The information processing system according to claim 3, An information processing system, wherein the candidate information is information determined based on a location shown in the extracted specific image or the past predetermined information.

5. 2. The information processing system according to claim 1, In the specific output step, data including a link from the image of the specific location to an image showing the specific location among images indicated by the regional image data is output as the specific image data.

6. 2. The information processing system according to claim 1, In the area output step, data including a link from an image taken at a position corresponding to the specific location to an image of the specific location in the specific image data is output as the area image data.

7. 2. The information processing system according to claim 1, In the extraction step, an image having a range that is a predetermined percentage wider than the range in which the specific location is captured is extracted as the specific image, and the percentage is a value corresponding to the size of the shooting area, the usage rate of the output area image data, or the usage rate of the output specific image data.

8. A program, A method for making a computer execute each step of the information processing system according to claim 1. program.

9. An information processing method, comprising: causing an information processing system to execute each step of claim 1; Information processing methods.

Images