Information processing system and program

Abstract

To provide an information processing system, or the like, configured to identify a travel path in an imaging area more accurately.SOLUTION: An information processing system includes a processor which is configured to execute a program so that the following steps are to be performed. The program includes: an acquisition step of acquiring, by the processor, images captured by an imaging apparatus in imaging positions on a travel path on which the imaging apparatus moves in an imaging area with a predetermined marker provided there in; an estimation step of estimating the travel path and positional relationship between at least a part of the travel path and the marker, on the basis of the acquired captured images; a display output step of outputting, when an imaging position is designated, display data for displaying an image captured in the imaging position; and a drawing output step of outputting drawing data indicating a drawing of the imaging area and the estimated travel path superimposed on the drawing. In the drawing data, the position of the marker and the travel path in the drawing have the estimated positional relationship.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system and a program.

Background Art

[0002] Patent Document 1 discloses a technique for detecting the shapes of a plurality of target parts existing in a construction site within a building, acquiring shape data of the plurality of target parts based on the detection results, and identifying a reference target part from among the plurality of target parts based on information regarding the dimensions of each of the plurality of target parts, and estimating the self-position within the construction site based on the collation result between the position of the reference target part included in the drawing data and the position of the actual reference target part in the acquired shape data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Simply displaying the captured images taken at each position in the captured area makes it difficult to grasp the movement path in the captured area.

[0005] In view of the above circumstances, the present invention aims to provide an information processing system or the like that can more accurately grasp the movement path in the captured area.

Means for Solving the Problems

[0006] According to one aspect of the present invention, an information processing system is provided. This information processing system includes a processor capable of executing a program such that the following steps are performed. In the acquisition step, the processor acquires a captured image with each position on the movement path along which the imaging device moves the imaging area being a shooting position. A predetermined marker is installed in the imaging area. In the estimation step, based on the acquired captured image, the movement path and the positional relationship between at least a part of the movement path and the marker are estimated. In the display output step, when a shooting position is specified, display data for displaying the captured image taken from the specified shooting position is output. In the drawing output step, drawing data showing a drawing of the imaging area and the movement path estimated to be superimposed on the drawing is output. In the drawing data, the position of the marker in the drawing and the movement path are in the estimated positional relationship.

[0007] According to such an aspect, it is possible to easily grasp the positional relationship of the objects shown in the captured image.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic matters shown in the following embodiments can be combined with each other.

[0010] By the way, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium that can be read by a computer, may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its function is realized on a client terminal (so-called cloud computing).

[0011] In addition, in this embodiment, the "section" may include, for example, hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in this embodiment, various information is handled, and these information are represented, for example, by physical values of signal values representing voltage and current, the high and low of signal values as a set of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0012] In addition, a circuit in a broad sense is a circuit realized by appropriately combining at least a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), programmable logic devices (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] 1. Hardware Configuration In this section, the hardware configuration of the on-site management support system according to this embodiment will be described.

[0014] FIG. 1 is a diagram showing the overall configuration of the on-site imaging system 1. In FIG. 1, an overview of each device included in the on-site imaging system 1 and the users who use those devices is shown. Each overview will be described as needed with reference to other figures.

[0015] The on-site imaging system 1 is an imaging system for photographing images of a site such as a construction site, a building site, or a construction project, and assisting in grasping the situation of the site based on the photographed images. The on-site imaging system 1 includes an information processing system 2, a communication line 3, a selfie stick 4, an external battery 5, an external power source 6, a stand 7, a marker board 8, and an imaging device 30. The information processing system 2 includes a server device 10, an on-site terminal 20, a supervision terminal 40, and an operator terminal 50.

[0016] The communication line 3 includes the Internet and the like, and mediates the exchange of data between devices connected to its own line. The server device 10 is connected to the communication line 3 by wire, and the on-site terminal 20 and the supervision terminal 40 are connected to the communication line 3 wirelessly. In this embodiment, the on-site terminal 20 communicates with the communication line 3 by mobile communication. Further, the on-site terminal 20 performs wireless communication with the imaging device 30 using two communication methods. The two communication methods are Wi-Fi communication and BLE (Bluetooth (registered trademark) Low Energy) communication in this embodiment.

[0017] The on-site terminal 20 and the imaging device 30 are, for example, terminals used by the on-site worker W1, installed at the work site, and execute processing related to on-site photography. The on-site terminal 20 is, for example, a smartphone or a tablet terminal. The supervision terminal 40 is, for example, a terminal used by the work supervisor W2 in charge of the work site, and executes processing for managing the progress of the work. The operator terminal 50 is, for example, a terminal used by the operator W3 of the on-site imaging system 1, and accepts various operations related to the image captured by the imaging device 30.

[0018] The on-site terminal 20 is connected to the external power source 6 via the external battery 5 by a cable. The imaging device 30 is detachably connected to the external power source 6 via the external battery 5 by a cable and a connector. In other words, the external power source 6 also supplies power to the imaging device 30 that is detachably connected. The external battery 5 has a so-called pass-through function that can supply power while charging, and supplies power to the on-site terminal 20 and the imaging device 30 while being charged by the power supplied from the external power source 6 with the breaker turned on.

[0019] The imaging device 30 is a digital camera equipped with an image sensor, and captures an image indicated by the light measured by the image sensor. In the present embodiment, the imaging device 30 is a 360-degree camera (also referred to as an omnidirectional camera or a full-sphere camera) that can capture images in all directions, including up, down, left, right, front, and back. The imaging device 30 is attached to the selfie stick 4, and the selfie stick 4 can be inserted into and fixed to the stand 7 installed at the work site. The stand 7 is an example of a fixing device that fixes the imaging device 30 in a stationary state.

[0020] The marker plate 8 is a plate member on which markers 81 and 82 (collectively referred to as "marker 80" when not distinguishing between them) are represented on the surface, and is fixed to the stand 7. The marker 80 is installed in the imaging area and serves as a reference for the position and size in the three-dimensional space. The marker 80 is, for example, a rectangular pattern whose side lengths are registered in the server device 10. The marker plate 8 is fixed to the stand 7 such that the plane of the rectangle formed by the marker 80 is along the vertical. In this way, the marker plate 8 is an example of a display means for displaying a marker. According to such an aspect, the labor of installing a marker in the imaging area can be reduced.

[0021] By walking around the work site with the selfie stick 4 that the on-site worker W1 has removed the connector from and pulled out of the stand 7, imaging image data indicating an image captured by the 360-degree camera with the work site as the imaging area is generated. In the on-site imaging system 1, the image captured by the imaging device 30 is a moving image in the present embodiment, but may be a continuously captured still image as long as images of various locations of the work site can be obtained. The imaging device 30 transmits the generated imaging image data to the on-site terminal 20.

[0022] The on-site terminal 20 is a terminal that serves as the main user interface for on-site worker W1 and is, for example, a smartphone. The on-site terminal 20 controls the operation of the imaging device 30 using, for example, one of the above two communication methods (BLE communication in this embodiment). Further, the on-site terminal 20 transfers the captured image data transmitted from the imaging device 30 by one of the above two communication methods (Wi-Fi communication in this embodiment) to the server device 10 using yet another wireless communication (mobile communication in this embodiment).

[0023] The server device 10 performs image processing using the image of the work site indicated by the captured image data transmitted from the on-site terminal 20, and for example, generates region image data indicating an image of the work site as an image of the captured area, seen from the position where the imaging device 30 performed the shooting. The supervisor terminal 40 refers to the generated region image data and displays images of various locations of the work site. The work supervisor W2 grasps the situation at the site from the displayed images of the work site and gives work instructions to the on-site worker W1 at the site as necessary.

[0024] Depending on the work site, the breaker may be turned off for reasons such as power saving after the work is completed. In that case, power is no longer supplied from the external power source 6 after the breaker is turned off. Even if the power supply stops, the on-site terminal 20 and the imaging device 30 have built-in batteries, so they do not immediately stop operating, but there are also processes that take time, such as transmitting image data. Therefore, in this embodiment, an external battery 5 is provided to increase the operating time of the on-site terminal 20 and the imaging device 30 after the breaker is turned off.

[0025] The operator terminal 50 accepts operations by operator W3 to assist the server device 10 in the process of generating region image data. Details of the operations by operator W3 will be described later.

[0026] FIG. 2 is a diagram showing the hardware configuration of the server device 10. The server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a bus 14. The bus 14 electrically connects each part included in the server device 10.

[0027] (Control Unit 11) The control unit 11 is, for example, a central processing unit (CPU) not shown in the figure. The control unit 11 is a computer that realizes various functions related to the on-site imaging system 1 by reading a predetermined program stored in the storage unit 12. That is, the information processing by software stored in the storage unit 12 is specifically realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and may be implemented to have a plurality of control units 11 for each function. Or a combination thereof may also be acceptable.

[0028] (Storage Unit 12) The storage unit 12 stores various information defined by the above description. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the on-site imaging system 1 executed by the control unit 11, or as a memory such as a random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to the calculation of the program. The storage unit 12 stores various programs, variables, etc. related to the on-site imaging system 1 executed by the control unit 11.

[0029] (Communication Unit 13) The communication unit 13 is configured to be able to transmit various electrical signals from the server device 10 to external components. Also, the communication unit 13 is configured to be able to receive various electrical signals from external components to the server device 10. More preferably, the communication unit 13 has a network communication function, and thereby, various information may be communicatively implemented between the server device 10 and external devices via the communication line 3.

[0030] FIG. 3 is a diagram showing the hardware configuration of the on-site terminal 20. The on-site terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, an internal power supply unit 26, and a bus 27. The bus 27 electrically connects each part included in the on-site terminal 20. The control unit 21 and the storage unit 22 are of the same hardware although there are performance differences from the control unit 11 and the storage unit 12 shown in FIG. 2.

[0031] (Communication unit 23) The communication unit 23 includes a first communication unit 231, a second communication unit 232, and a third communication unit 233, and is an example of a wireless communication unit that performs three types of wireless communication. The first communication unit 231 performs wireless communication by Wi-Fi communication as the first wireless communication in this embodiment. The second communication unit 232 performs wireless communication by BLE as the second wireless communication that has a slower communication speed and lower power consumption than the first wireless communication. The third communication unit 233 performs wireless communication by mobile communication as the third wireless communication that has a wider communicable area than the first wireless communication and the second wireless communication.

[0032] (Input unit 24) The input unit 24 has keys, buttons, a touch screen, a mouse, etc., and accepts input by the user. (Output unit 25) The output unit 25 has a display (including a touch screen), a speaker, etc., and displays visual information generated in a manner visible to the user, such as a screen, an image, an icon, text, etc. on the display surface, and outputs sounds including voice.

[0033] (Internal power supply unit 26) The internal power supply unit 26 is a battery built into the device itself, that is, a rechargeable battery, and supplies the accumulated power to each part of the device itself. The internal power supply unit 26 is an example of a portable battery that can be carried together with the device itself. The internal power supply unit 26 is charged by the power supplied from the external power supply 6. The internal power supply unit 26 has a pass-through function similar to the external battery 5, and while being charged by the power supplied from the external power supply 6 with the breaker turned on, it supplies power to each part.

[0034] FIG. 4 is a diagram showing the hardware configuration of the imaging device 30. The imaging device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, an internal power supply unit 36, an imaging unit 37, and a bus 38. The bus 38 electrically connects each part included in the imaging device 30. Each part from the control unit 31 to the internal power supply unit 36 is hardware similar to each part from the control unit 21 to the internal power supply unit 26 shown in FIG. 3, although there are performance differences.

[0035] However, the communication unit 33 includes only a first communication unit 331 and a second communication unit 332. The first communication unit 331 performs wireless communication by Wi-Fi communication as the first wireless communication, similar to the first communication unit 231 of the communication unit 23. The second communication unit 332 performs wireless communication by BLE as the second wireless communication that has a slower communication speed and lower power consumption than the first wireless communication, similar to the second communication unit 232 of the communication unit 23. Also, the output unit 35 has a light in addition to a display or the like, and irradiates light for ensuring the light amount necessary for shooting. The input unit 34 has a switch for turning on the light.

[0036] (Imaging unit 37) The imaging unit 37 is a sensor that has an optical system including a lens and an image sensor or the like, and measures the light incident from the lens to generate captured image data. In the present embodiment, as described above, the imaging unit 37 uses an ultra-wide-angle lens and a plurality of image sensors to generate captured image data that captures the entire omnidirectional view in the up-down, left-right, and front-back directions.

[0037] FIG. 5 is a diagram showing the hardware configuration of the supervision terminal 40. The supervision terminal 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, an output unit 45, and a bus 46. The bus 46 electrically connects each part included in the supervision terminal 40. Each part from the control unit 41 to the output unit 45 has the same hardware although there are performance differences from each part from the control unit 31 to the output unit 35 shown in FIG. 4. The operator terminal 50 includes hardware such as a control unit 51 (only the control unit 51 is given a different reference numeral from that of the supervision terminal 40) in the same manner as the supervision terminal 40.

[0038] 2. Functional Configuration In this section, the functional configuration of the present embodiment will be described. As described above, the information processing by software stored in the storage unit of each device is specifically realized by a control unit which is an example of hardware, so that each functional unit included in the control unit can be executed.

[0039] FIG. 6 is a diagram showing an example of the functional configuration of the control unit of each device. The control unit 11 of the server device 10 includes a DB control unit 111, a server display unit 112, an image acquisition unit 113, an image processing unit 114, a data generation unit 115, a data output unit 116, and a user reception unit 117. The control unit 21 of the on-site terminal 20 includes a display control unit 211, an operation reception unit 212, an operation control unit 213, and a transmission control unit 214. The control unit 31 of the imaging device 30 includes a display control unit 311, an operation reception unit 312, a photographing control unit 313, and a transmission control unit 314. The control unit 41 of the supervision terminal 40 includes a display control unit 411 and an operation reception unit 412. The control unit 51 of the operator terminal 50 includes a display control unit 511 and an operation reception unit 512.

[0040] The DB control unit 111 (DB: Database) of the server device 10 controls the storage of data in the database and the reading of data from the database. The database referred to here may be not only the database stored in the server device 10 but also the database stored in an external device. For example, photographed image data indicating an image photographed by the imaging device 30 and the above-described region image data are stored in the database.

[0041] The server display unit 112 executes processing for causing each terminal to display a system screen related to the on-site imaging system 1. The server display unit 112 performs processing such as generating and transmitting an HTML (Hyper Text Markup Language) file which is display data, and causes a web page showing the system screen to be displayed on the supervision terminal 40 or the like. Note that the server display unit 112 may perform processing such as generating and transmitting display data of an application for using the on-site imaging system 1.

[0042] The image acquisition unit 113 acquires an image of the imaging area photographed by the imaging device 30. The image processing unit 114 executes image processing on the image acquired by the image acquisition unit 113. The image processing unit 114 performs processing related to a technique called so-called Visual SLAM (Simultaneous Localization and Mapping) which simultaneously estimates its own position and creates a surrounding map based on the image of the imaging area, for example.

[0043] The data generation unit 115 generates data and the like based on the result of the image processing by the image processing unit 114. The data generation unit 115 generates the above-described area image data based on the result of Visual SLAM, for example. The data output unit 116 outputs the data generated by the data generation unit 115. The data output unit 116 outputs the generated area image data to the supervision terminal 40, for example. The user reception unit 117 receives a specific operation by the user via a terminal used by the user (for example, the operator terminal 50).

[0044] The display control unit 511 of the operator terminal 50 controls the display process to the display means of its own device. The operation reception unit 512 receives the operations of the user (for example, operator W3). The operations received by the operation reception unit 512 include operations for assisting the process of generating the above-described area image data. The display control unit 411 of the supervisor terminal 40 controls the display process to the display means of its own device. The operation reception unit 412 receives the operations of the user (for example, work supervisor W2).

[0045] The display control unit 211 of the site terminal 20 controls the display process to the display means of its own device. The operation reception unit 212 receives the operations of the user (for example, site worker W1). The operation control unit 213 controls the operation of the imaging device 30. The transmission control unit 214 controls the transmission process of the captured image data by its own device and the imaging device 30. The display control unit 311 of the imaging device 30 controls the display process to the display means of its own device. The operation reception unit 312 receives the operations of the user (for example, site worker W1). The imaging control unit 313 controls the imaging process by the imaging unit 37. The transmission control unit 314 controls the transmission process of the captured image data by its own device.

[0046] 3. Information Processing In this section, in this embodiment, the information processing that the program causes the computer to execute will be described. The information processing system includes a processor capable of executing a program so that each of the following steps (each process) is performed.

[0047] FIG. 7 is an activity diagram showing an example of information processing. First, the site terminal 20 receives the operation of the site worker W1 by the operation reception unit 212, and controls the operation of the imaging device 30 by the operation control unit 213 to control the imaging of the imaging area (A11). The imaging device 30 is controlled by the site terminal 20 and performs imaging by the imaging control unit 313 (A12). The site worker W1 moves the imaging area while holding the selfie stick 4 to which the imaging device 30 is attached, and the imaging device 30 images the imaging area.

[0048] After the on-site worker W1 performs an operation to start shooting on the on-site terminal 20 and then lifts the selfie stick 4, since the shooting by the imaging device 30 has already started, the marker 80 shown on the marker plate 8 attached to the stand 7 for fixing the selfie stick 4 will be shot. Also, at the end of shooting, after fixing the selfie stick 4 to the stand 7 and then performing an operation to end shooting on the on-site terminal 20, the marker 80 will also be shot before the end of shooting.

[0049] Next, the imaging device 30 stores the captured image data indicating the captured image by the shooting control unit 313 (A13). Subsequently, the on-site terminal 20 controls the imaging device 30 by the transmission control unit 214 to transmit the captured image data to itself and acquires the captured image (A14). Next, the on-site terminal 20 transfers the acquired captured image to the server device 10 by the transmission control unit 214 (A15). The server device 10 acquires the transferred captured image by the image acquisition unit 113 (A21: acquisition step).

[0050] Next, the server device 10, and the image processing unit 114 estimates the movement path of the imaging device 30 based on the acquired captured image (A22: estimation step). The image processing unit 114 estimates the movement path using the VSLAM technology. There are, for example, the following two methods in the VSLAM technology.

[0051] The first is a method of indirectly estimating the position (indirect method), which is a method of estimating the position of the feature points and the position of the camera by performing the association of feature points between a plurality of images. When using the indirect method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the association of the feature points of the objects shown in the acquired plurality of captured image data.

[0052] The second is a method of directly estimating the position (direct method), which is a method of estimating the depth of each pixel and the position of the imaging device 30 by minimizing the error of the luminance values of corresponding pixels among a plurality of images. When using the direct method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the error of the values of corresponding pixels indicated by the plurality of acquired captured image data.

[0053] Next, the server device 10 estimates the positional relationship between the imaging device 30 and the marker 80 based on the captured image in which the marker 80 appears among the captured images acquired by the image processing unit 114 (A23: estimation step). The server device 10 stores information regarding the size of the marker 80.

[0054] FIG. 8 is a diagram showing an example of the marker 80. In the example of FIG. 8, the horizontal length L11 and the vertical length L12 of the marker 81, and the horizontal length L13 and the vertical length L14 of the marker 82 are shown. Further, the marker 82 is installed at a height L15 from the floor surface 9, and the marker 81 is installed at a position higher than the marker 82 by a height L16. The server device 10 stores this information on the lengths and heights as information regarding the size of the marker 80. When not distinguishing the center point C81 of the marker 81 and the center point C82 of the marker 82, it is referred to as "the center point C80 of the marker 80".

[0055] The image processing unit 114 recognizes the marker 80 appearing in the captured image by, for example, image processing for detecting edges, and calculates the distance from each side length in the captured image to the marker 80 (for example, the distance to the center point C80 of the marker 80). Further, the image processing unit 114 calculates the horizontal component and the vertical component of the angle formed by the plane including the marker 80 and the straight line connecting the center point C80 of the marker 80 and the imaging device 30 from the inclination of each side in the captured image. The image processing unit 114 estimates the positional relationship between the imaging device 30 and the marker 80 by, for example, obtaining the coordinates of the imaging device 30 in a three-dimensional coordinate system with the center point C80 of the marker 80 as the origin based on the distance and angle calculated in this way.

[0056] Subsequently, the server device 10 determines an initial arrangement of the estimated movement path with respect to the drawing of the imaging area by the image processing unit 114 (A24). The drawing referred to here is a drawing showing the imaging area as viewed from directly above. When the imaging area is a construction site, walls, doors, and the spaces (rooms, corridors, storage, etc.) partitioned by them are represented. When the imaging area is a construction site, instead of measuring the imaging area to create a drawing, the construction of the buildings at the site is carried out in accordance with the design drawing called the drawing.

[0057] It is assumed that drawing data showing the drawing of the imaging area is stored in the server device 10 in advance. This drawing data is hereinafter referred to as "basic drawing data". In the basic drawing data, each position on the drawing can be represented by coordinates in a two-dimensional coordinate system with a predetermined position on the drawing as the origin. Further, in the present embodiment, installation location data indicating the installation location of the marker 80 in the imaging area is stored in the server device 10 in advance, and it is assumed that the installation location of the marker 80 is known in advance. The installation location data is, for example, data representing the installation location of the marker 80 by coordinates on the drawing, and is created by the on-site worker W1 or the work supervisor W2 looking at the installation location of the marker 80 at the site and registered in the server device 10. Note that the registration of the installation location of the marker 80 may be performed each time imaging is performed.

[0058] In the present embodiment, the image processing unit 114 determines the initial position of the movement path based on the installation location data stored in advance and the positional relationship between the imaging device 30 and the marker 80 estimated in A23. The image processing unit 114 specifies two or more positions on the drawing for the position of the imaging device 30, that is, the imaging position, which has a positional relationship with the marker 80 installed at the installation location indicated by the installation location data.

[0059] FIG. 9 is a diagram showing an example of a specified photographing position. In the example of FIG. 9, the marker 80 and the movement path R1 as viewed from directly above are shown. The movement path R1 shows photographing positions P1, P2, and P35. These photographing positions are the positions at which the photographed images in which the marker 80 is reflected were taken. The photographing position P1 is a position where the horizontal component of the distance from the center point C80 of the marker 80 is D1, and the horizontal component of the angle formed by the plane S80 including the marker 80 and the straight line E1 connecting the center point C80 of the marker 80 and the photographing position P1 is θ1.

[0060] The photographing position P2 is a position where the horizontal component of the distance from the center point C80 of the marker 80 is D2, and the horizontal component of the angle formed by the plane S80 including the marker 80 and the straight line E2 connecting the center point C80 of the marker 80 and the photographing position P2 is θ2. The photographing position P35 is a position where the horizontal component of the distance from the center point C80 of the marker 80 is D35, and the horizontal component of the angle formed by the plane S80 including the marker 80 and the straight line E35 connecting the center point C80 of the marker 80 and the photographing position P35 is θ35.

[0061] The image processing unit 114 specifies, for example, the photographing positions P1, P2, and P35 as the photographing positions having the positional relationship estimated with the marker 80. Then, the image processing unit 114 overlaps the photographing position P1 at the position having the positional relationship estimated for the photographing position P1 from the position of the marker 80 on the drawing, overlaps the photographing position P2 at the position having the positional relationship estimated for the photographing position P2, and overlaps the photographing position P35 at the position having the positional relationship estimated for the photographing position P35, and arranges the movement path R1. The image processing unit 114 determines this arrangement of the movement path R1 as the initial arrangement.

[0062] Next, the server device 10 generates presentation data indicating the movement path and drawing to be presented to the operator W3 by the data generation unit 115 (A25: Presentation step). The data generation unit 115 generates, as presentation data, data that can edit the arrangement of the movement path on the drawing. Further, the data generation unit 115 generates, as presentation data, data indicating a drawing in which the movement path is arranged in the initial arrangement determined in A24. The data output unit 116 outputs the generated presentation data to the operator terminal 50. The operator terminal 50 displays the drawing and the movement path indicated by the output presentation data by the display control unit 511 (A26).

[0063] FIG. 10 is a diagram showing an example of the presented drawing and movement path. In the example of FIG. 10, the data output unit 116 presents an editing image G1 showing the drawing D1 and the movement path R1. The drawing D1 is a drawing of the imaging area AR1 indicated by the basic drawing data. The movement path R1 is a path along which the imaging device 30 has moved, indicated by a solid line connecting the start point RS1 and the end point RG1. Near the start point RS1 and the end point RG1, a marker position image F80 indicating the installation position of the marker 80 is shown.

[0064] The editing image G1 shows a reduction button B1, an enlargement button B2, up / down / left / right buttons B3, a rotation button B4, and a confirmation button B5. The reduction button B1 is a button for reducing the movement path R1. The enlargement button B2 is a button for enlarging the movement path R1. The up / down / left / right buttons B3 are buttons for moving the movement path R1 up, down, left, and right. The rotation button B4 is a button for rotating the movement path R1. The confirmation button B5 is a button for confirming the arrangement of the movement path R1 in the drawing D1. The operator W3 performs an editing operation on these buttons so that the movement path R1 matches the imaging area AR1 indicated by the drawing D1 (a state where it fits exactly).

[0065] The state where the movement path R1 coincides with the imaging area AR1 means that the movement path R1 does not intersect with the wall and exists only in a movable space (such as inside a room or a corridor). Note that the editing operation may be an operation of pinching in (shrinking), pinching out (enlarging), dragging (up, down, left, or right), or multi-tap rotation on the movement path R1. The operator terminal 50 receives, by the operation reception unit 512, an editing operation on the shrinking button B1 or the like (A31).

[0066] The operation reception unit 512 transmits operation data indicating the received editing operation to the server device 10. The server device 10 generates, by the data generation unit 115, path drawing data indicating the movement path R1 and the drawing D1 edited by the editing operation indicated by the transmitted operation data (A32). By outputting the path drawing data generated by the data output unit 116 to the operator terminal 50, the edited movement path R1 and the drawing D1 are displayed. In this way, when the editing operation is repeatedly performed and the confirmation button B5 is operated, the data generation unit 115 generates path drawing data indicating the movement path R1 and the drawing D1 in the confirmed arrangement.

[0067] Subsequently, the server device 10 generates, by the data generation unit 115, the above-described area image data, that is, data indicating the work site viewed from the position where the imaging device 30 performs imaging as the imaging area, based on the captured image acquired in A21 and the movement path estimated in A22 (A33). Next, the server device 10 generates, by the data generation unit 115, display data to be displayed on the terminal (A34). The display data is output to, for example, the supervisor terminal 40 (A34: display output step and drawing output step). The supervisor terminal 40 displays, by the display control unit 411, the area image data and the path drawing data indicated by the output display data (A35).

[0068] FIG. 11 is a diagram showing an example of the displayed area image data and the route drawing data. In the example of FIG. 11, an area image G2 shown by the area image data and a route drawing image G3 shown by the route drawing data are shown. The route drawing image G3 is an image including a movement route R1 and a drawing D1. On the movement route R1, 35 shooting positions, namely, shooting positions P1, P2, P3, ···, P35 (when not distinguishing each of them, referred to as "shooting position P100"), are superimposed.

[0069] In the area image G2, a shooting area AR1 including a stand 7 and a marker plate 8 is shown. Also, in the area image G2, position images PG1, PG2, PG34, PG35 (when not distinguishing each of them, referred to as "position image PG100") that virtually indicate the shooting positions are shown. The position images PG1, PG2, PG34, PG35 are images showing the shooting positions P1, P2, P34, P35 superimposed on the shooting area AR1.

[0070] When the work supervisor W2 performs an operation of selecting any one of the position images PG100, the display control unit 411 displays the shooting area AR1 photographed from the shooting position indicated by the selected position image PG100. By displaying the shooting area AR1 photographed from various shooting positions, the work supervisor W2 can confirm the state of the shooting area AR1.

[0071] The area image G2 is a rectangular image showing a certain angle of view among the omnidirectional images photographed by the imaging device 30 from the shooting positions. Since the omnidirectional image is an image in which the object is distorted, the data generation unit 115 corrects the distortion so that the object appears in the same way as a normal photograph and generates the area image G2. When an operation of moving the area image G2 in the vertical direction, horizontal direction, or diagonal direction is performed, the shooting area AR1 moves in the operated direction, and the portion displayed as the area image G2 changes.

[0072] Also, when an operation of selecting the position images shown in the area image G2 and the route drawing image G3 is performed, the area image G2 taken from the shooting position indicated by the selected position image is displayed. By displaying various portions of the shooting area AG1 at each shooting position indicated by the area image data as the area image G2 in this way, the overall state of the shooting area AG1 can be confirmed.

[0073] As described above, in the present embodiment, the image acquisition unit 113 is an example of an acquisition unit that acquires the captured images taken at each position (each shooting position P100) of the movement route R1 while the imaging device 30 moves the shooting area AR1. A predetermined marker (marker 80 in the present embodiment) is installed in the shooting area AR1. The image processing unit 114 is an example of an estimation unit that estimates the movement route R1 and the positional relationship between at least a part of the movement route R1 and the marker 80 based on the captured images acquired by the image acquisition unit 113.

[0074] When the shooting position P100 is specified, the data output unit 116 is an example of a display output unit that outputs display data (area display data) for displaying the captured image taken from the shooting position P100. Also, the data output unit 116 is an example of a drawing output unit that outputs drawing data (route drawing data) indicating the drawing D1 of the shooting area AR1 and the estimated movement route R1. In the route drawing data, the movement route R1 is shown superimposed on the drawing D1, and the position of the marker in the drawing D1 and the movement route R1 have the estimated positional relationship.

[0075] Note that in the route drawing data, the position of the marker may or may not be shown (not shown in the example of FIG. 11). Regardless of whether the position of the marker is shown or not, if the position of the marker and the movement route R1 have the estimated positional relationship, a more accurate movement route can be grasped on the drawing compared to the case where such a positional relationship does not exist. Also, since the positional relationships between two or more shooting positions and the position of the marker are estimated, the scales of the drawing and the movement route can be matched.

[0076] Further, the data output unit 116 is an example of a presentation unit that presents the drawing D1 and the estimated movement path R1 to the user in a manner that enables editing of the arrangement. The editing image G1 shown in FIG. 10 is an example of the drawing D1 and the movement path R1 presented in this way. The user reception unit 117 is an example of a reception unit that receives editing of the arrangement of the presented movement path R1 in the drawing D1. The data output unit 116 (an example of a drawing output unit) outputs, as drawing data (path drawing data), data indicating the drawing D1 with the edited movement path R1 superimposed thereon. The path drawing image G3 shown in FIG. 11 is an example of the movement path R1 and the drawing D1 indicated by the output path drawing data. According to such an aspect, the movement path can be arranged on the drawing by human judgment.

[0077] <Other Embodiments> In the above embodiment, the installation location of the marker 80 was known in advance, but the installation location of the marker 80 may not be known. An example of the functional configuration in that case will be described. FIG. 12 is a diagram showing another example of the functional configuration of the control unit of the server device 10. The control unit 11 of the server device 10 includes a position specifying unit 118 in addition to the respective units shown in FIG. 6. The position specifying unit 118 functions as an example of a specifying unit that specifies the position of the marker 80. There are mainly two methods for specifying the position of the marker 80.

[0078] In the first method, first, the operator W3 compares the shape of the imaging region AR1 shown in the drawing D1 presented at A26 in FIG. 7 with the shape of the movement path R1, and edits the size and orientation of the movement path R1 so as to match the imaging region AR1. Then, the position specifying unit 118 specifies, as the position of the marker 80, the position that has an estimated positional relationship with the movement path R1 that matches the imaging region AR1.

[0079] In the second method, first, in A25 shown in FIG. 7, the data generation unit 115 generates, as presentation data, data indicating the captured image acquired in A21 in addition to the movement path and the drawing, and the data output unit 116 outputs the generated presentation data to the operator terminal 50. The operator terminal 50 displays the drawing and the captured image indicated by the presentation data by the display control unit 511 (A26). The operator W3 looks at the captured image, finds the image in which the marker 80 appears, and performs an operation of indicating the position of the marker 80 in the drawing from the background shown in the image. The position specifying unit 118 specifies the position indicated by this operation as the position of the marker 80.

[0080] As described above, the method for the operator W3 to specify the installation position of the marker 80 can be implemented without determining the initial position of the movement path in A24. However, it is easier to specify the installation position of the marker 80 if the initial position of the movement path is determined. Therefore, the data output unit 116 (an example of the presentation unit) may, for example, present the estimated movement path together with the drawing arranged so as to have a positional relationship with the estimated position and the temporary marker position in the drawing when the position of the marker specified in the past by the position specifying unit 118 is used as the temporary marker position in the drawing.

[0081] The specification of the marker position may be performed by either the first method or the second method described above. In either case, since the movement path is arranged at a position that is somewhat correct from the beginning, the editing of the arrangement of the movement path can be facilitated as compared with the case where the arrangement of the movement path based on the temporary marker position is not performed.

[0082] Further, when there are a plurality of temporary marker positions, the data output unit 116 (an example of the presentation unit) may use the position within the region surrounded by the plurality of temporary marker positions as a new temporary marker position. FIG. 13 is a diagram showing an example of a provisional marker position. In the example of FIG. 13, marker position images F80-1, F80-2, and F80-3 indicating the position of marker 80 are shown. When the marker position is specified by the position specifying unit 118, the same position is not always specified, and there are cases where a slightly shifted position is specified as shown in FIG. 13.

[0083] In this case, the position specifying unit 118 specifies, for example, as a new provisional marker position, the marker position indicated by marker position image F80-4 inside the region TR1 of a polygon (in this case, a triangle) connecting the center points C80-1, C80-2, and C80-3 of the markers. The center point C80-4 of the marker indicated by marker position image F80-4 is located inside region TR1. When the specified marker position is used as the provisional marker position on the drawing, the data output unit 116 arranges the estimated movement path so as to have the positional relationship with the provisional marker position and presents it together with the drawing.

[0084] According to such an aspect, the accuracy of the provisional marker position can be improved as compared with the case of using only one specified marker position. Note that as the number of marker positions increases, the shape of region TR1 becomes more complex. Therefore, the position specifying unit 118 may specify a new provisional marker position using only a predetermined number (for example, three or four, etc.) of the most recent marker positions among the marker positions specified in the past.

[0085] Further, the data output unit 116 (an example of a presentation unit) may use, as the new provisional marker position, the position where the latest provisional marker position among the plurality of provisional marker positions is the closest. FIG. 14 is a diagram showing another example of a provisional marker position. In the example of FIG. 14, marker position images F80-1, F80-2, and F80-3 (assuming that F80-3 indicates the latest marker position) shown in FIG. 13 and a marker position image F80-5 indicating a new marker position are shown.

[0086] The distances D11, D12, and D13 between the center point C80-5 of the marker shown in the marker position image F80-5 and the center points C80-1, C80-2, and C80-3 of the markers shown in the marker position images F80-1, F80-2, and F80-3 respectively have a magnitude relationship of D13 < D11 and D13 < D12. That is, the new marker position is the position closest to the latest provisional marker position (the marker position shown by F80-3) among the plurality of provisional marker positions. According to such an aspect, for example, when specifying the marker position by the second method, the operator W3 gradually grasps the marker position accurately. Therefore, compared with the case where the latest marker position is not considered, the accuracy of the provisional marker position can be improved.

[0087] In the above embodiment, the operator W3 always edited the arrangement of the movement path. However, as the accuracy of the provisional marker position improves, the accuracy of the initial arrangement of the movement path determined at A24 also improves. Therefore, the initial arrangement of the movement path may be directly determined as the arrangement of the movement path. In that case, the data output unit 116 may, for example, stop presenting the movement path and the drawing to the user after presenting the movement path and the drawing to the user a predetermined number of times, assuming that the accuracy of the initial arrangement of the movement path has sufficiently improved. Further, when the operator W3 performs an operation indicating that the accuracy of the initial arrangement of the movement path has sufficiently improved and the presentation of the movement path and the drawing is unnecessary, the data output unit 116 may stop presenting the movement path and the drawing thereafter.

[0088] In the imaging area, for example, as the construction progresses, the installation location of the marker 80 may be changed. A new functional configuration for dealing with such a case will be described. FIG. 15 is a diagram showing another example of the functional configuration of the control unit of the server device 10. The control unit 11 of the server device 10 includes a determination processing unit 119 and a movement notification unit 120 in addition to the units shown in FIG. 6.

[0089] For example, when the position of the marker specified by the image processing unit 114 is separated from the position of the previously specified marker by a predetermined distance or more, the determination processing unit 119 functions as an example of a determination unit that determines that the marker has moved. When the marker has moved, some processes premised on the marker being in the same position also need to be changed.

[0090] For example, when it is not determined by the determination processing unit 119 that the marker has moved, the data output unit 116 (an example of the presentation part) does not present the movement path and the drawing, considering that the accuracy of the initial arrangement of the movement path has been sufficiently improved as described above. When it is determined by the determination processing unit 119 that the marker has moved, the movement path and the drawing are presented again. According to such an aspect, since the arrangement of the movement path is edited only when necessary, the output of the drawing data can be made faster.

[0091] Note that the method for determining marker movement is not limited to this. For example, when the movement path initially arranged by the image processing unit 114 is significantly deviated from the imaging area shown in the drawing, the determination processing unit 119 may determine that the marker has moved. The magnitude of the deviation between the movement path and the imaging area is represented, for example, by the ratio of the portion of the movement path that does not overlap the imaging area. When this ratio is equal to or greater than the threshold value, the determination processing unit 119 determines that it is significantly deviated from the imaging area shown in the drawing and determines that the marker has moved.

[0092] In addition, when it is determined by the image processing unit 114 that the marker has moved, the movement notification unit 120 functions as an example of a notification unit that notifies a predetermined destination that the marker has moved. The predetermined destination is a destination for notifying the operator W3, and is, for example, the IP (Internet Protocol) address of the operator terminal 50 or the like. When it is determined that the marker has moved, the data generation unit 115 generates notification data for notifying the fact.

[0093] The movement notification unit 120 stores in advance the IP address of the operator terminal 50, and when notification data is generated, it performs notification by outputting the notification data to that IP address. The data output unit 116 (an example of a presentation unit) presents the movement route and the drawing to the notified destination when the movement notification unit 120 notifies of the marker movement. When the movement notification unit 120 outputs notification data, the data output unit 116 outputs the presentation data generated by the data generation unit 115 to the output destination. According to such an aspect, compared with the case where the above notification is not performed, the work at the time of marker movement can be performed quickly.

[0094] Further, when the determination processing unit 119 determines the movement of the marker, the data output unit 116 may present the movement route and the drawing without performing the initial arrangement of the movement route at A24. In that case, the data output unit 116 presents, for example, the movement route and the drawing side by side horizontally or vertically to the user (operator W3). When the marker moves, if the movement route is initially arranged, the movement route becomes an unnatural arrangement with respect to the drawing. Therefore, by presenting regardless of the positional relationship with the marker, it is possible to prevent presentation in an unnatural arrangement.

[0095] The on-site imaging system 1 is not limited to the configuration shown in FIG. 1. For example, the stand (an example of a fixing device) may fix the imaging device so that the displayed marker image is included in the imaging range. FIG. 16 is a diagram showing the overall configuration of the on-site imaging system 1a. The on-site imaging system 1a includes a on-site terminal 20, an imaging device 30, a selfie stick 4, a stand 7a, and a marker plate 8. The stand 7a includes a support column 71 and an arm 72.

[0096] A marker plate 8 is fixed to the support column 71. A selfie stick 4 is inserted and fixed at the tip of the arm 72. The imaging device 30 and the marker plate 8 are respectively fixed at positions separated from each other by the arm 72. Therefore, the marker 80 represented on the marker plate 8 is included in the imaging range of the imaging device 30. By performing an operation to start shooting on the on-site terminal 20 before removing the selfie stick 4 from the arm 72, the marker can be shot more reliably than in the case of using the stand shown in FIG. 1.

[0097] The marker display means is not limited to the marker plate 8. For example, the on-site terminal 20 (such as a smartphone or a tablet terminal) may be the marker display means. In that case, the on-site terminal 20 is fixed to the stand 7a as shown in FIG. 16. The on-site terminal 20 is fixed to the support column 71 so that the display surface is parallel to the vertical direction, and a marker image indicating the marker is displayed. The on-site terminal 20 has a display surface that emits light, such as a liquid crystal display or an organic EL (Electro Luminescence) display, and has a display means for displaying the marker image with a luminance equal to or higher than a predetermined value. According to such an aspect, the marker image can be shot even when the surroundings are dark.

[0098] Also, since the on-site terminal 20 is operated by a person such as the on-site worker W1, it is installed at a height that is easy for a person to operate (for example, a height above the chest). On the other hand, since the imaging device 30 is carried around by a person, it will move at a height around the person's hand. If the marker is in a low position, it is likely to be shielded by an obstacle, but if it is in a high position like the on-site terminal 20, the displayed marker image is less likely to be shielded, and the marker image can be shot more reliably.

[0099] Note that the display means for displaying the marker image may be a projector. In that case, the projector projects the marker image onto a wall or the like existing in the imaging area. Since the projected image is larger than the image displayed on a normal display, it is possible to estimate the positional relationship or the like using a larger marker image. Further, the display means for displaying the marker image may have a variable display direction of the marker. For example, the column 71 that fixes the on-site terminal 20 rotates, and the orientation of the display surface changes. Thereby, the marker image can be oriented in a direction that is easily captured by the imaging device 30.

[0100] In addition to the above-described devices, the on-site imaging system 1 may include a sensor that detects movement. The sensor that detects movement is, for example, an acceleration sensor, a speed sensor, an angular velocity sensor, or the like. These sensors are, for example, provided in the on-site terminal 20. In this case, the determination processing unit 119 (an example of a determination unit) may determine that the marker has moved based on the output of the sensor. For example, when the sensor is an acceleration sensor, the determination processing unit 119 determines that the marker has moved when an acceleration equal to or greater than a threshold value is measured for a length of a predetermined time or more.

[0101] The stand 7 (an example of a fixing device) fixes the imaging device 30 in a non-moving state. Although the imaging device 30 also includes an acceleration sensor or the like, those sensors usually operate only during imaging. On the other hand, since the sensors provided in the on-site terminal 20 are always operating, even if the stand 7 and the marker plate 8 move when not imaging, it is possible to measure the acceleration or the like and determine the movement. In addition, the on-site terminal 20 may be fixed to the stand 7 or may be fixed to the selfie stick 4.

[0102] When fixed to the selfie stick 4, acceleration or the like will be measured during imaging. However, since the movement of the marker plate 8 is usually performed when not imaging, if the determination processing unit 119 does not determine the movement of the marker during imaging, false determination can be eliminated. Note that when the sensor provided in the imaging device 30 performs measurement other than during imaging, the movement of the marker may be determined using the measurement result of that sensor.

[0103] In addition, the sensor for detecting movement is not limited to a sensor that detects the movement of the sensor itself, and may be a sensor that detects the movement of other objects. The sensor for detecting the movement of other objects is, for example, an image sensor included in the imaging device 30. In that case, for example, the imaging device 30 periodically (every hour or the like) takes a still image. The determination processing unit 119 calculates the feature amount of the taken still image, and when the calculated feature amount changes by a threshold value or more, determines that the installation location of the imaging device 30 has changed, that is, the marker has moved.

[0104] In addition, the sensor for detecting movement may be provided, for example, on the stand 7. In that case, the output of the sensor is transmitted to the on-site terminal 20 by wireless communication such as Bluetooth (registered trademark), and the on-site terminal 20 transfers it to the server device 10. Further, the on-site terminal 20 may display a movement button, and after the on-site worker W1 presses the movement button, the marker 80 may be moved. In that case, the determination processing unit 119 determines the movement of the marker 80 when the movement button is pressed.

[0105] In addition, when the movement of the marker is determined by the determination processing unit 119, the position specifying unit 118 may calculate the movement amount of the marker based on the measurement result by the sensor for detecting movement, and specify the position of the marker after movement. For example, if the position specifying unit 118 is an acceleration sensor, it integrates the acceleration measured in the direction of the acceleration to calculate the movement amount in each direction. The position specifying unit 118 specifies the position obtained by adding the calculated movement amount to the position of the marker before movement as the position of the marker after movement. Thereby, even when the marker moves, the initial arrangement of the movement path can be made.

[0106] Also, the positional relationship with the estimated marker was used for the arrangement of the drawing and the movement path in the above-described embodiment, but it is not limited to this. For example, since the shooting time of the marker is known, when the image processing unit 114 estimates the positional relationship with the marker at two or more locations, it calculates the moving speed of the movement path connecting those locations. When the calculated speed is equal to or higher than the threshold value (when it is an unusually high moving speed), the image processing unit 114 may detect that the estimated movement path includes an error.

[0107] Also, in the example of FIG. 9, for example, the image processing unit 114 can calculate not only the horizontal component but also the vertical component of the angle formed by the plane S80 including the marker 80 and the straight line E1 connecting the center point C80 of the marker 80 and the shooting position P1. Thereby, the image processing unit 114 also estimates the height of the shooting position from the ground (floor surface). When the shooting area is a construction site, the on-site worker W1 may shoot under the floor or in the ceiling space during construction. For movement paths with different heights at those shooting positions, the image processing unit 114 estimates them as movement paths with different height levels.

[0108] The image processing unit 114 calculates, for example, the average value of the heights of the shooting positions on the movement path, and estimates shooting positions with a height difference of x cm or more from the calculated average value as shooting positions on movement paths with different height levels. The image processing unit 114 estimates the movement path, for example, with a height of the average value ±x cm as the "on-floor level", a height lower than the average value by x cm or more as the "under-floor level", and a height higher than the average value by x cm or more as the "ceiling-space level". The data generation unit 115 generates area image data that can select area images of the "on-floor level", "under-floor level", and "ceiling-space level" according to the height of the shooting position even at the same position on the drawing.

[0109] Note that the image processing unit 114 may estimate the height levels of different floors, not limited to those of the same floor. In that case, for example, among the movement paths, the image processing unit 114 determines that a location where the height changes by a predetermined value or more is a movement between floors due to stairs or the like, and estimates the areas before and after that location as movement paths on different floors. As described above, the estimation of the positional relationship with the marker can be used for specifying the scale in a two-dimensional or three-dimensional space.

[0110] Also, for example, assume that the data generation unit 115 extracts an image of a specific location and generates a check sheet. In that case, when the specific location is on the floor or the ceiling, if the height of the shooting position is not known, the specific location cannot be extracted unless the shooting image is taken directly above or below the specific location. However, if the height of the shooting position is known, the depression angle or elevation angle for shooting the specific location can be calculated, so the data generation unit 115 can select an image with a good reflection of the specific location from a number of images in which the specific location is reflected and extract that location.

[0111] Note that the method for specifying the position of the marker 80 is not limited to the two methods described above. For example, when installation location data indicating the installation location of the marker 80 in the shooting area is stored in advance as in the above-described embodiment, the position specifying unit 118 may read out the stored installation location data and specify the installation location indicated by the read installation location data (the position where the marker 80 is installed) as the position of the marker 80. Even in that case, the initial arrangement of the movement path may be performed using the specified marker position as a provisional marker position.

[0112] In the above-described embodiment, the movement of the marker is determined based on the shooting image captured by the imaging device 30. However, by using the above sensors, the movement of the marker can be detected even when no shooting is being performed.

[0113] In the on-site imaging system 1, Wi-Fi communication is used as the first wireless communication and BLE communication is used as the second wireless communication. However, this is not the only option, and other wireless communications (for example, wireless LAN other than Wi-Fi communication) may be used. Further, as the wireless communication between the on-site terminal 20 and the imaging device 30, only the first wireless communication may be used. Further, the imaging device 30 is not limited to a 360-degree camera, and may be a wide-angle camera or the like that can perform higher-resolution imaging.

[0114] In addition, in the on-site imaging system 1, if it is a construction site where the breaker is not turned off, the on-site terminal 20 and the imaging device 30 may be connected to an external power source without passing through the external battery 5. Further, in the embodiment, the on-site terminal 20 transmitted the imaging data to the server device 10 by mobile communication even when the breaker was turned off. However, when the breaker is not turned off, the imaging data may be transmitted to the server device 10 by Wi-Fi communication by installing a Wi-Fi router at the construction site.

[0115] The configuration shown in FIG. 1 and the like is an example, and other modes may be adopted as long as there is no inconvenience in implementation. For example, one device may be distributed among two or more devices, or may be replaced by a cloud computing system. Further, the functions of one device may be realized by being distributed among two or more devices, or the functions of two or more devices may be concentrated and realized by one device. Further, the operations performed by one function may be distributed among two or more functions, or two or more functions may be integrated into one function. In short, as long as each function necessary for the entire on-site imaging system 1 is realized, the devices that realize these functions may have any configuration.

[0116] The output destination of information or data (hereinafter referred to as "information, etc.") may be another device, a display, a storage unit (including a built-in storage unit and an external storage unit), or the like. The acquisition of information, etc. includes, in addition to the mode of acquiring information, etc. transmitted from another device, the mode of acquiring information, etc. generated by the own device. The table associating parameters is not limited to the illustrated table, and the number of parameters may be reduced or increased. Further, information, etc. corresponding to the parameters may be obtained by a mathematical formula, a conditional expression, or the like without using a table.

[0117] The aspects of the above-described embodiments were information processing apparatuses such as the server device 10 and information processing systems such as the on-site imaging system 1 including the server device 10, but may also be information processing methods. The information processing method includes steps of each process executed by the information processing system. Further, the aspects of the above-described embodiments may also be programs. The program causes a computer to execute steps of each process executed by a similar information processing system.

[0118] <Appendix> Furthermore, it may be provided in each of the aspects described below.

[0119] (1) An information processing system, comprising a processor capable of executing a program so that the following steps are performed. In the acquisition step, the processor acquires a captured image with each position on the movement path along which the imaging device moves the imaging area as the imaging position. A predetermined marker is installed in the imaging area. In the estimation step, based on the acquired captured image, the movement path and the positional relationship between at least a part of the movement path and the marker are estimated. In the display output step, when the imaging position is specified, display data for displaying the captured image captured from the imaging position is output. In the drawing output step, drawing data indicating the drawing of the imaging area and the movement path estimated to be superimposed on the drawing is output. In the drawing data, the position of the marker in the drawing and the movement path are in the estimated positional relationship.

[0120] According to such an aspect, the movement path in the imaging area can be grasped more accurately.

[0121] (2) In the information processing system according to (1) above, in the presentation step, the processor presents the drawing and the estimated movement path to the user in a manner that enables editing of the arrangement, in the reception step, receives editing of the arrangement of the movement path in the presented drawing, and in the drawing output step, outputs, as the drawing data, data indicating the drawing with the movement path of the edited arrangement superimposed thereon.

[0122] According to such an aspect, the movement path can be arranged on the drawing based on human judgment.

[0123] (3) In the information processing system according to (2) above, in the specifying step, the processor specifies the position of the marker, and in the presentation step, when the position of the marker specified in the past is used as a temporary marker position on the drawing, arranges the estimated movement path so as to have the positional relationship with the temporary marker position, and presents it together with the drawing.

[0124] According to such an aspect, the editing of the arrangement of the movement path can be facilitated.

[0125] (4) In the information processing system according to (3) above, in the presentation step, when there are a plurality of the temporary marker positions, a position within the area surrounded by the plurality of temporary marker positions is set as a new temporary marker position.

[0126] According to such an aspect, the accuracy of the temporary marker position can be improved.

[0127] (5) In the information processing system according to (4) above, in the presentation step, a position where the latest temporary marker position among the plurality of temporary marker positions is closest is set as the new temporary marker position.

[0128] According to such an aspect, the accuracy of the provisional marker position can be improved.

[0129] (6) In the information processing system according to any one of (1) to (5) above, in the determination step, the processor determines that the marker has moved when the position of the identified marker is separated from the position of the previously identified marker by a predetermined distance or more. In the presentation step, when it is not determined that the marker has moved, the movement path and the drawing are not presented. When it is determined that the marker has moved, the movement path and the drawing are presented. An information processing system.

[0130] According to such an aspect, the output of the drawing data can be speeded up.

[0131] (7) In the information processing system according to (6) above, in the notification step, when it is determined that the marker has moved, the processor notifies a predetermined destination. In the presentation step, when the notification has been made, the movement path and the drawing are presented to the notified destination. An information processing system.

[0132] According to such an aspect, the work when the marker moves can be quickly performed.

[0133] (8) An imaging system including the information processing system according to any one of (1) to (7) above, the imaging device, a fixing device, and a display means, wherein the fixing device fixes the imaging device in a non-moving state, and the display means displays a marker image indicating the marker. An imaging system.

[0134] According to such an aspect, the labor of installing a marker in the imaging area can be reduced.

[0135] (9) In the imaging system according to (8) above, the fixing device fixes the imaging device so that the displayed marker image is included in the imaging range. An imaging system.

[0136] According to such an aspect, the marker can be photographed more reliably.

[0137] (10) In the imaging system according to (8) or (9) above, the display means is an imaging system having means for displaying the marker image with a luminance equal to or higher than a predetermined value.

[0138] According to such an aspect, the marker image can be photographed even when the surroundings are dark.

[0139] (11) An imaging system including the information processing system according to (6) or (7) above, the imaging device, a fixing device, a display means, and a sensor for detecting movement, wherein the fixing device fixes the imaging device in a non-moving state, the display means displays a marker image indicating the marker, and in the determination step, it is determined that the marker has moved based on the output of the sensor.

[0140] According to such an aspect, the movement of the marker can be detected even when no photographing is being performed.

[0141] (12) A program that causes a computer to execute each step of the information processing system according to any one of (1) to (7) above. Of course, this is not the limit. Also, the above-described embodiments and modified examples may be arbitrarily combined and implemented.

[0142] Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and their modifications are included in the scope and gist of the invention and are included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0143] 1: On-site imaging system 1a: On-site imaging system 2: Information processing system 7: Stand 7a: Stand 8: Marker board 10: Server device 11: Control unit 20: On-site terminal 21: Control unit 30: Imaging device 31: Control unit 40: Supervisor terminal 41: Control unit 50: Operator terminal 51: Control unit 71: Column 72: Arm 80: Marker 81: Marker 82: Marker 111: DB control unit 112: Server display unit 113: Image acquisition unit 114: Image processing unit 115: Data generation unit 116: Data output unit 117: User reception unit 118: Position identification unit 119: Judgment processing unit 120: Movement notification unit 211: Display control unit 212: Operation reception unit 213: Operation control unit 214: Transmission control unit 231: First communication unit 232: Second communication unit 233: Third communication unit 311: Display control unit 312: Operation reception unit 313: Shooting control unit 314: Transmission control unit 331: First communication unit 332: Second communication unit 411: Represents the control unit 412: Operation reception unit 511: Represents the control unit 512: Operation reception unit

Claims

1. An information processing system, comprising a processor capable of executing a program so that each of the following steps is performed, wherein the processor in an acquisition step, acquires a captured image with each position on a movement path along which an imaging device moves a shooting area as a shooting position, and a predetermined marker is installed in the shooting area, in an estimation step, estimates the movement path and the positional relationship between at least a part of the movement path and the position of the marker based on the acquired captured image, in a display output step, outputs display data for displaying a captured image taken from the shooting position when the shooting position is specified, in a drawing output step, outputs drawing data showing a drawing of the shooting area and the movement path estimated to be superimposed on the drawing, and in the drawing data, the position of the marker in the drawing and the movement path are in the estimated positional relationship, an information processing system.

2. The information processing system according to claim 1, wherein the processor in a presentation step, presents the drawing and the estimated movement path to the user so that editing of the arrangement is possible, in a reception step, receives editing of the arrangement in the drawing of the presented movement path, and in the drawing output step, outputs, as the drawing data, data showing the drawing with the edited movement path of the arrangement superimposed thereon. an information processing system.

3. The information processing system according to claim 2, wherein the processor in a specification step, specifies the position of the marker, and in the presentation step, when the position of the marker specified in the past is set as a temporary marker position in the drawing, presents the estimated movement path arranged so as to be in the positional relationship with the temporary marker position and the drawing together. an information processing system.

4. The information processing system according to claim 3, wherein in the presentation step, when there are a plurality of the temporary marker positions, a position within a region surrounded by the plurality of temporary marker positions is set as a new temporary marker position. an information processing system.

5. The information processing system according to claim 4, wherein in the presentation step, a position where the latest temporary marker position among the plurality of temporary marker positions is closest is set as the new temporary marker position. an information processing system.

6. The information processing system according to claim 2, wherein the processor In the determination step, when the position of the specified marker is separated from the position of the marker specified last time by a predetermined distance or more, it is determined that the marker has moved. In the presentation step, when it is not determined that the marker has moved, the movement path and the drawing are not presented, and when it is determined that the marker has moved, the movement path and the drawing are presented. Information processing system.

7. In the information processing system according to claim 6, the processor In the notification step, when it is determined that the marker has moved, a notification is sent to a predetermined destination. In the presentation step, when the notification has been made, the movement path and the drawing are presented to the notified destination. Information processing system.

8. An imaging system including the information processing system according to claim 2, the imaging device, a fixing device, and a display means, the fixing device fixes the imaging device in a non-moving state. the display means displays a marker image indicating the marker. Imaging system.

9. In the imaging system according to claim 8, the fixing device fixes the imaging device so that the displayed marker image is included in the imaging range. Imaging system.

10. In the imaging system according to claim 8, the display means is a means for displaying the marker image with a luminance equal to or higher than a predetermined value. Imaging system.

11. An imaging system including the information processing system according to claim 6, the imaging device, a fixing device, a display means, and a sensor for detecting movement, the fixing device fixes the imaging device in a non-moving state. the display means displays a marker image indicating the marker. In the determination step, it is determined that the marker has moved based on the output of the sensor. Imaging system.

12. A program, causing a computer to execute each step of the information processing system according to any one of claims 2 to 7. Program.

Images