Information processing system

Abstract

To provide an information processing system for reducing time and trouble for measurement to show a path of a measuring device in a structure having two or more stories.SOLUTION: An information processing system is a construction support system 1 including a communication line 2, a selfie stick 3, an external battery 4, an external power source 5, a breaker 6, a server device 10, a site terminal 20, an imaging device 30, and a remote terminal 40. A control part of the server device 10 includes a measurement acquisition part for acquiring measurement data measured at each position of a movement course by a measurement device (e.g., the imaging device 30) having a sensor for measuring an electromagnetic wave arriving from an object while moving inside or outside a structure, and a path output part for outputting path data showing a path of the measurement device on the basis of the acquired measurement data. The path has two or more partial paths on which positions measured by the measurement device are respectively placed within two or more different height ranges in the case that the structure has two or more stories.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system.

Background Art

[0002] Patent Document 1 discloses a cleaning machine system including a position sensor that acquires the positional relationship of surrounding objects, a map acquisition unit that acquires a floor map, a self-position estimation unit that estimates the self-position in the floor map based on the position sensor, a boundary information generation unit that acquires boundary information indicating the boundary of a cleaning area, which is an area where the cleaning machine performs cleaning on the floor, based on the self-position, a boundary instruction unit that instructs the boundary to the boundary information generation unit, a cleaning area creation unit that creates a cleaning area based on the boundary information, and a travel route creation unit that creates a travel route for cleaning based on the created cleaning area.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, when estimating the self-position from an image taken at a construction site and transmitting the site situation, if the construction site spans two or more floors, in the technology of Patent Document 1, shooting must be performed for each floor.

[0005] In view of the above circumstances, the present invention aims to reduce the labor of measurement for showing the trajectory of a measuring device in a structure having two or more floors.

Means for Solving the Problems

[0006] According to one aspect of the present invention, an information processing system is provided. In the measurement acquisition step in this information processing system, a measurement device having a sensor for measuring electromagnetic waves arriving from an object acquires measurement data measured at each position on the movement path while moving inside or outside a structure. In the trajectory output step, trajectory data indicating the trajectory of the measurement device is output based on the acquired measurement data. When the structure has two or more floors, the trajectory has two or more partial trajectories in which the positions where the measurement device performs measurements are respectively within ranges of two or more different heights.

[0007] According to such an aspect, it is possible to reduce the labor of measurement for showing the trajectory of the measurement device in a structure having two or more floors.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic matters shown in the embodiments described below can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium that can be read by a computer, may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its function is realized on a client terminal (so-called cloud computing).

[0011] Also, in this embodiment, the "section" may include, for example, hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in this embodiment, various information is handled, and these information are represented, for example, by physical values of signal values representing voltage and current, the high and low of signal values as an aggregate of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0012] In addition, a circuit in a broad sense is a circuit realized by appropriately combining at least a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] 1. Hardware Configuration In this section, the hardware configuration of the construction support system according to this embodiment will be described.

[0014] FIG. 1 is a diagram showing the overall configuration of the construction support system 1. In FIG. 1, an overview of each device included in the construction support system 1 and the users who use those devices is shown. Each overview will be described as needed with reference to other figures.

[0015] The construction support system 1 is an information processing system that executes processing for supporting construction work such as building construction. The construction support system 1 includes a communication line 2, a selfie stick 3, an external battery 4, an external power supply 5, a breaker 6, a server device 10, a field terminal 20, an imaging device 30, and a remote terminal 40.

[0016] The communication line 2 includes the Internet and the like, and mediates the exchange of data between devices connected to its own line. The server device 10 is connected to the communication line 2 by wire, and the on-site terminal 20 and the remote terminal 40 are connected to the communication line 2 wirelessly. In the present embodiment, the on-site terminal 20 communicates with the communication line 2 by mobile communication. Further, the on-site terminal 20 performs wireless communication with the imaging device 30 using two communication methods. The two communication methods are Wi-Fi communication and BLE (Bluetooth (registered trademark) Low Energy) communication in the present embodiment.

[0017] The on-site terminal 20 and the imaging device 30 are installed at the construction site and are used, for example, by the on-site worker W1. The remote terminal 40 is a terminal that is used, for example, by the construction supervisor W2 in charge of the construction site and is assumed to be used even at a location away from the construction site.

[0018] The on-site terminal 20 is connected to the external power supply 5 via the external battery 4 by a cable. The imaging device 30 is detachably connected to the external power supply 5 via the external battery 4 by a cable and a connector 7. In other words, the external power supply 5 also supplies power to the imaging device 30 that is detachably connected. The external battery 4 has a so-called pass-through function that can supply power while charging, and supplies power to the on-site terminal 20 and the imaging device 30 while being charged by the power supplied from the external power supply 5 when the breaker 6 is turned on.

[0019] The imaging device 30 is a digital camera equipped with an image sensor, and takes an image indicated by the light measured by the image sensor. The imaging device 30 is a 360-degree camera that can take images of all directions, up, down, left, right, front, and back, in the present embodiment. The imaging device 30 is an example of a measuring device equipped with a sensor. The imaging device 30 is attached to the selfie stick 3, and the selfie stick 3 can be inserted into and fixed to the stand 8 installed at the construction site.

[0020] The marker plate 9 is a plate member on which markers 91 and 92 (referred to as "marker 90" when not distinguishing between them) are represented on the surface, and is fixed to the stand 8. The marker 90 is installed at the construction site and serves as a reference for the position and size in the three-dimensional space. The marker 90 is, for example, a rectangular pattern with the length of each side registered in the server device 10. The marker plate 9 is fixed to the stand 8 such that the plane of the rectangle formed by the marker 90 is along the vertical direction.

[0021] By having the on-site worker W1 remove the connector 7 and walk around the construction site with the selfie stick 3 removed from the stand 8, image data showing an image of the construction site photographed by the 360-degree camera is generated. In the construction support system 1, the image photographed by the imaging device 30 is a moving image in this embodiment, but may be a still image continuously photographed as long as images of various locations of the construction site can be obtained. The imaging device 30 transmits the generated image data to the on-site terminal 20.

[0022] The on-site terminal 20 is a terminal that serves as the main user interface for the on-site worker W1 and is, for example, a smartphone. The on-site terminal 20 controls the operation of the imaging device 30 using, for example, one of the above two communication methods (BLE communication in this embodiment). Further, the on-site terminal 20 transfers the image data transmitted from the imaging device 30 by one of the above two communication methods (Wi-Fi communication in this embodiment) to the server device 10 using yet another wireless communication (mobile communication in this embodiment).

[0023] The server device 10 performs image processing using the image of the construction site shown by the image data transmitted from the on-site terminal 20 and generates three-dimensional image data showing the construction site three-dimensionally. The remote terminal 40 refers to the generated three-dimensional image data and displays an image of the construction site represented three-dimensionally. The construction supervisor W2 grasps the situation at the site from the displayed image of the construction site and gives work instructions to the on-site worker W1 at the site as necessary.

[0024] Depending on the construction site, the breaker 6 may be turned off after the work is completed for reasons such as power saving. In that case, after the breaker 6 is turned off, power is no longer supplied from the external power source 5. Thus, the external power source 5 can switch between power supply and non - supply. Although the on - site terminal 20 and the imaging device 30 also have built - in batteries and do not stop immediately, there are also processes that take time, such as the transmission of image data. Therefore, in this embodiment, an external battery 4 is provided to increase the operating time of the on - site terminal 20 and the imaging device 30 after the breaker 6 is turned off.

[0025] Figure 2 is a diagram showing the hardware configuration of the server device 10. The server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a bus 14. The bus 14 electrically connects each part included in the server device 10.

[0026] (Control Unit 11) The control unit 11 is, for example, a central processing unit (CPU) not shown in the figure. The control unit 11 is a computer that realizes various functions related to the construction support system 1 by reading a predetermined program stored in the storage unit 12. That is, the information processing by software stored in the storage unit 12 is specifically realized by the control unit 11, which is an example of hardware, and can be executed as each functional part included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and may be implemented to have a plurality of control units 11 for each function, or a combination thereof.

[0027] (Storage Unit 12) The storage unit 12 stores various information defined by the foregoing description. This can be implemented, for example, as a storage device such as a Solid State Drive (SSD) that stores various programs related to the construction support system 1 executed by the control unit 11, or as a memory such as a Random Access Memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to the calculation of programs. The storage unit 12 stores various programs, variables, etc. related to the construction support system 1 executed by the control unit 11.

[0028] (Communication unit 13) The communication unit 13 is configured to be able to transmit various electrical signals from the server device 10 to external components. Also, the communication unit 13 is configured to be able to receive various electrical signals from external components to the server device 10. More preferably, the communication unit 13 has a network communication function, and thereby various information may be communicated between the server device 10 and external devices via the communication line 2.

[0029] FIG. 3 is a diagram showing the hardware configuration of the on-site terminal 20. The on-site terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, an internal power supply unit 26, and a bus 27. The bus 27 electrically connects each part included in the on-site terminal 20. The control unit 21 and the storage unit 22 are of the same hardware although there are performance differences from the control unit 11 and the storage unit 12 shown in FIG. 2.

[0030] (Communication unit 23) The communication unit 23 includes a first communication unit 231, a second communication unit 232, and a third communication unit 233, and is an example of a wireless communication unit that performs three types of wireless communication. The first communication unit 231 performs wireless communication by Wi-Fi communication as the first wireless communication in this embodiment. The second communication unit 232 performs wireless communication by BLE as the second wireless communication that has a lower communication speed and lower power consumption than the first wireless communication. The third communication unit 233 performs wireless communication by mobile communication as the third wireless communication that has a wider communicable area than the first wireless communication and the second wireless communication.

[0031] (Input unit 24) The input unit 24 has keys, buttons, a touch screen, a mouse, etc., and receives input from the user. (Output unit 25) The output unit 25 has a display (including a touch screen), a speaker, etc., and displays visual information generated in a manner visible to the user, such as a screen, an image, an icon, text, etc. on the display surface, and outputs sounds including voice.

[0032] (Internal power supply unit 26) The internal power supply unit 26 is a battery built into the device itself, that is, a rechargeable battery, and supplies the accumulated power to each part of the device itself. The internal power supply unit 26 is an example of a portable battery that can be carried together with the device itself. The internal power supply unit 26 is charged by the power supplied from the external power supply 5. The internal power supply unit 26 has a pass-through function similar to the external battery 4, and supplies power to each part while being charged by the power supplied from the external power supply 5 when the breaker 6 is turned on.

[0033] Figure 4 is a diagram showing the hardware configuration of the imaging device 30. The imaging device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, an internal power supply unit 36, an imaging unit 37, and a bus 38. The bus 38 electrically connects each part included in the imaging device 30. Each part from the control unit 31 to the internal power supply unit 36 is hardware similar to each part from the control unit 21 to the internal power supply unit 26 shown in Figure 3, although there are performance differences.

[0034] However, the communication unit 33 includes only the first communication unit 331 and the second communication unit 332. Similar to the first communication unit 231 of the communication unit 23, the first communication unit 331 performs wireless communication by Wi-Fi communication as the first wireless communication in this embodiment. Similar to the second communication unit 232 of the communication unit 23, the second communication unit 332 performs wireless communication by BLE as the second wireless communication with a lower communication speed and lower power consumption than the first wireless communication. Further, the output unit 35 has a light in addition to a display or the like, and irradiates light for ensuring the amount of light necessary for imaging. The input unit 34 has a switch for turning on the light.

[0035] (Imaging unit 37) The imaging unit 37 includes an optical system including a lens, an image sensor, etc., and is a sensor that measures light incident from the lens and generates image data. In this embodiment, as described above, the imaging unit 37 uses an ultra-wide-angle lens and a plurality of image sensors to generate image data obtained by imaging the entire omnidirectional range of up and down, left and right, and front and back.

[0036] FIG. 5 is a diagram showing the hardware configuration of the remote terminal 40. The remote terminal 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, an output unit 45, and a bus 46. The bus 46 electrically connects each unit included in the remote terminal 40. Each unit from the control unit 41 to the output unit 45 is of the same hardware although there are performance differences from each unit from the control unit 31 to the output unit 35 shown in FIG. 4.

[0037] 2. Functional configuration In this section, the functional configuration of this embodiment will be described. As described above, the information processing by software stored in the storage unit of each device is specifically realized by the control unit which is an example of hardware, so that each functional unit included in the control unit can be executed.

[0038] FIG. 6 is a diagram showing the functional configuration of the control unit of each device. The control unit 11 of the server device 10 includes an information storage unit 111, an image processing unit 112, a data generation unit 113, a data output unit 114, a measurement acquisition unit 115, a trajectory output unit 116, a hierarchy acquisition unit 117, a correspondence output unit 118, a feature identification unit 119, an association unit 120, and an image output unit 121. The control unit 21 of the on-site terminal 20 includes a display control unit 211, an operation reception unit 212, an operation control unit 213, and a transmission control unit 214. The control unit 31 of the imaging device 30 includes a display control unit 311, an operation reception unit 312, a shooting control unit 313, and a transmission control unit 314. The control unit 41 of the remote terminal 40 includes a display control unit 411 and an operation reception unit 412.

[0039] The information storage unit 111 of the server device 10 stores image data showing an image of the construction site taken by the imaging device 30 and the above-described three-dimensional image data. The image processing unit 112 performs processing related to a technique called so-called SLAM (Simultaneous Localization and Mapping) that simultaneously estimates its own position and creates an environmental map based on the image of the construction site. The data generation unit 113 generates the above-described three-dimensional image data based on the processing performed by the image processing unit 112. The data output unit 114 outputs the generated three-dimensional image data.

[0040] The measurement acquisition unit 115 acquires measurement data measured at each position of the movement path while the imaging device 30 moves inside or outside the structure. Inside or outside the structure is, for example, the construction site. The trajectory output unit 116 outputs trajectory data indicating the trajectory of the imaging device 30. The hierarchy acquisition unit 117 acquires hierarchy data indicating the hierarchy of the structure. The hierarchy data is, for example, data indicating the floors (such as the 1st floor, 2nd floor, and 3rd floor) of a building. Note that the hierarchy data may be data indicating the ceiling space, floor surface, and floor below of the same floor as a hierarchy. The correspondence output unit 118 outputs correspondence data indicating the correspondence relationship between the hierarchy indicated by the hierarchy data acquired by the hierarchy acquisition unit 117 and the partial trajectory.

[0041] When the hierarchical data indicating the drawings of each layer of the structure is acquired, the feature specifying unit 119 specifies the locations having a predetermined feature in the drawing. The association unit 120 associates the position of the end of the partial locus corresponding to the layer indicated by the hierarchical data with the location specified in the drawing of that layer. The image output unit 121 outputs an image represented in a manner corresponding to the height of the position for each position measured by the imaging device 30 indicated by the partial locus.

[0042] The display control unit 411 of the remote terminal 40 controls the display process to the display means of its own device. The display control unit 411 causes, for example, an image of the construction site indicated by the stereoscopic image data output from the server device 10 to be displayed. The operation reception unit 412 receives an operation by the user (for example, the construction supervisor W2). The construction supervisor W2 performs an operation to move the displayed location in the construction site to check the current situation of the site, and instructs the construction procedure and the like to the on-site workers W1 and the like at the site.

[0043] The display control unit 211 of the on-site terminal 20 controls the display process to the display means of its own device. The operation reception unit 212 receives an operation by the user (for example, the on-site worker W1). The operation control unit 213 controls the operation of the imaging device 30. The transmission control unit 214 controls the transmission process of the image data by its own device and the imaging device 30.

[0044] The display control unit 311 of the imaging device 30 controls the display process to the display means of its own device. The operation reception unit 312 receives an operation by the user (for example, the on-site worker W1). The shooting control unit 313 controls the shooting process by the imaging unit 37. The transmission control unit 314 controls the transmission process of the image data by its own device.

[0045] 3. Information Processing In this section, in this embodiment, the information processing that the program causes the computer to execute will be described.

[0046] In the construction support system 1, the measurement acquisition unit 115 acquires measurement data measured at each position on the movement path while an imaging device 30 having a sensor for measuring electromagnetic waves reaching from an object moves inside or outside the structure. The trajectory output unit 116 outputs trajectory data indicating the trajectory of the imaging device 30 based on the measurement data acquired by the measurement acquisition unit 115. When the structure has two or more floors, the trajectory of the imaging device 30 has two or more partial trajectories in which the positions where the imaging device 30 performs measurements are respectively within ranges of two or more different heights.

[0047] The measurement data is, for example, data indicating a set of measurement results at each position on the movement path. The trajectory data is, for example, data indicating three-dimensional coordinates or two-dimensional coordinates (with or without scale). The partial trajectory is, for example, generated by dividing the measurement data and then generating a trajectory. In that case, the division method is a method of recognizing a staircase from the measurement data and dividing before and after the staircase.

[0048] Also, the partial trajectory is obtained by generating a trajectory based on the measurement data and then dividing the trajectory. The division method in that case is a method of dividing by the average value of the height if there are two floors or a method using a well-known clustering method such as k-means. Note that the captured images other than the trajectory data, the number of floors, and the floor drawings (with scale and initial value) will be described later.

[0049] FIG. 7 is a diagram showing an example of a partial trajectory. In FIG. 7, a graph G10 of a three-dimensional coordinate system having an X-axis, a Y-axis, and a Z-axis is shown. The graph G10 shows an overall trajectory C10 having a partial trajectory C11 and a partial trajectory C12. The partial trajectory C11 is a trajectory in which the position where the imaging device 30 performs measurements is within a range where the coordinate in the Z-axis direction (i.e., height) is less than z1 (m: meter). Also, the partial trajectory C12 is a trajectory in which the position where the imaging device 30 performs measurements is within a range where the coordinate in the Z-axis direction is z1 (m: meter) or more.

[0050] The hierarchical acquisition unit 117 acquires hierarchical data indicating the hierarchy of the structure. The corresponding output unit 118 outputs correspondence data indicating the correspondence relationship between the hierarchy indicated by the hierarchical data acquired by the hierarchical acquisition unit 117 and the partial trajectory.

[0051] The hierarchical data is, for example, data indicating the number and order of hierarchies, the height of the hierarchy, or the drawings of each hierarchy. The correspondence data is, for example, data indicating a table showing the correspondence relationship, partial trajectory data with the number of hierarchies added to the header, or the original video thereof. The partial trajectory indicates a higher hierarchy as the statistical value (average value, etc.) of the height coordinate is larger. If the number of partial trajectories matches the number of hierarchies, the correspondence relationship can be simply determined.

[0052] The hierarchical data indicates the drawings of each hierarchy of the structure. The feature identification unit 119 identifies the locations having a predetermined feature in the drawing. The association unit 120 associates the end positions of the partial trajectories corresponding to the hierarchy with the locations identified in the drawing of that hierarchy.

[0053] The association unit 120 identifies the connection path connecting between the hierarchies shown in the drawing as a location having a predetermined feature in the drawing. The connection path connecting between the hierarchies shown in the drawing is, for example, a staircase or a slope.

[0054] The feature identification unit 119 identifies the start position where the measurement by the imaging device 30 starts in the drawing as a location. The association unit 120 associates the end positions of the partial trajectories corresponding to the hierarchy where the start position is identified by the feature identification unit 119 with the start position.

[0055] The start position where the measurement starts is, for example, the installation position of a marker, the installation position of a camera, or a manually set position.

[0056] The feature identification unit 119 identifies, as the start position, the position where the feature amount defined as the start position among the feature amounts indicated by the measurement data acquired by the measurement acquisition unit 115 is measured.

[0057] The characteristic quantities indicated by the measurement data are the length of a straight line, the angle formed by the straight line, etc. When a marker is installed at the starting position, if the length and angle of the straight line indicated by the characteristic quantity indicate the marker shape, the position where the measurement data indicating the characteristic quantity was measured is specified as the starting position.

[0058] The hierarchical data shows the drawings of each layer of the structure. When the number of partial trajectories is less than the number of layers indicated by the hierarchical data acquired by the hierarchy acquisition unit 117, the correspondence output unit 118 outputs, as correspondence data, data indicating the correspondence relationship between the layer with the highest degree of matching when the partial trajectories are arranged on the drawing of the layer and the partial trajectories.

[0059] The correspondence output unit 118 arranges the partial trajectories without overlapping them on the wall in the drawing. In that case, the larger the scale, the less likely it is that the partial trajectories can be arranged because they do not match the shape of the drawing. Therefore, the correspondence output unit 118 determines that the higher the maximum scale, the higher the degree of matching when the scale of the trajectory arranged on the drawing is increased.

[0060] FIG. 8 is a diagram showing an example of drawing data. In FIG. 8, a drawing D10 of a floor that is a certain construction site is shown. In the drawing D10, a staircase D11, which is a connection path connecting between each layer, is shown. In the example of FIG. 8, the partial trajectory C21 is arranged without overlapping the wall of the construction site shown in the drawing D10, and among the positions C211 and C212 at the ends of the partial trajectory C21, the position C211 is associated with the staircase D11. Since the partial trajectory C21 passes through almost all regions of the floor shown in the drawing D10, it is determined that the scale can be increased and the degree of matching is the highest compared to the case where it is arranged on the drawings of other floors.

[0061] The image output unit 121 outputs an image represented in a manner corresponding to the height of each position where the imaging device 30 indicated by the partial trajectory performed measurement.

[0062] The image output unit 121 outputs, for example, a mark image with a color and shape corresponding to the height (ceiling space / underfloor). Also, the image output unit 121 may represent warp points (measurement points represented in the 360-degree image or adjacent points) in a manner corresponding to the height of the measurement position. Further, the image output unit 121 may represent image points (measurement points represented in the drawing) in a manner corresponding to the height of the measurement position.

[0063] The image output unit 121 may output an image represented in a manner corresponding to the height of the measurement position based on the trajectory data of only one floor. Also, the image output unit 121 outputs image data showing an image of these images as seen from above, and when, for example, a user performs an operation to change the viewing point from looking down from above to looking from the side, outputs image data showing an image of these images as seen from the side.

[0064] The construction support system 1 first executes an acquisition process of acquiring measurement data indicating the measurement results obtained by a measurement device (for example, the imaging device 30) for the interior of a structure (for example, the interior of a building under construction).

[0065] FIG. 9 is an activity diagram showing an example of the acquisition process. The activity shown in FIG. 9 starts when the on-site worker W1 performs a measurement start operation to cause the on-site terminal 20 to start measurement by the imaging device 30. Note that the imaging by the imaging device 30 is the measurement of the light intensity using an image sensor. First, the on-site terminal 20 receives (A11) the measurement start operation by the on-site worker W1 by the operation reception unit 212.

[0066] When the operation reception unit 212 receives the measurement start operation, it transmits instruction data instructing the start of measurement (imaging in the example of FIG. 9) to the imaging device 30. When the imaging device 30 receives the transmitted instruction data, it starts measurement, that is, imaging (A12) by the imaging control unit 313. The imaging device 30 moves around the construction site while being moved by the on-site worker W1 and stores (A13) the captured image as measurement data indicating the measurement results in the storage unit 32 inside the device itself.

[0067] Next, when the on-site worker W1 performs a measurement end operation to end the measurement on the on-site terminal 20 after the measurement at the construction site is completed, the on-site terminal 20 receives the measurement end operation by the operation reception unit 212 (A14). When the operation reception unit 212 receives the measurement end operation, it transmits instruction data indicating an instruction to end the measurement to the imaging device 30. When the imaging device 30 receives the transmitted instruction data, the imaging control unit 313 ends the measurement (A15). By performing the processes from A11 to A15, the measurement data at the work site is stored in the imaging device 30.

[0068] Next, the on-site terminal 20 transmits, by the transmission control unit 214, instruction data indicating an instruction to transmit the measurement data to its own device to the imaging device 30 at a predetermined timing (A21). The predetermined timing is, for example, the timing when a predetermined time has elapsed after the measurement has ended, or the timing when an operation of transmission is performed by the on-site worker W1. When the imaging device 30 receives the instruction data, the imaging control unit 313 reads out the measurement data stored in the storage unit 32 and transmits it to the on-site terminal 20 (A22).

[0069] The on-site terminal 20 receives, by the transmission control unit 214, the transmitted measurement data and stores it in the storage unit 32 of its own device (A23). Next, the on-site terminal 20 reads out the measurement data from the storage unit 32 at a predetermined timing by the transmission control unit 214 and transmits it to the server device 10 (A31). The predetermined timing is, for example, the timing when a predetermined time has elapsed. The server device 10 acquires the transmitted measurement data by the measurement acquisition unit 115 (A32). The server device 10 executes an analysis process for analyzing the trajectory of the imaging device 30 based on the acquired measurement data.

[0070] FIG. 10 is a flowchart showing an example of the analysis process. The analysis process starts with the measurement acquisition unit 115 acquiring measurement data (S11 = A32). In the example of FIG. 10, the server device 10 first generates trajectory data indicating the trajectory of the imaging device 30 based on the acquired measurement data by the trajectory output unit 116 (S12). For example, when the measurement data is an image as described above, the trajectory output unit 116 generates trajectory data using the VisualSLAM technique.

[0071] As described above, the trajectory data is data indicating the trajectory by three-dimensional coordinates or two-dimensional coordinates. In the example of FIG. 7, data indicating the trajectory by three-dimensional coordinates is generated as the trajectory data. FIG. 11 is a diagram showing an example of the trajectory data. The trajectory data D1 shown in FIG. 11 is data showing the measurement time and the three-dimensional coordinates indicating the measurement position of the measurement data at that measurement time in association with each other. In the trajectory data D1, the three-dimensional coordinates with earlier measurement times are arranged in order.

[0072] Next, the server device 10 executes sub-trajectory processing by the trajectory output unit 116 (S13). For example, as described above, if there are two floors, the trajectory output unit 116 divides the trajectory into parts indicating each of the two floors by a method of dividing by the average value of the height. Also, if there are three or more floors, the trajectory output unit 116 divides the trajectory into parts indicating each of the three or more floors by a well-known clustering method such as k-means. The trajectory output unit 116 generates trajectory data indicating the divided sub-trajectories.

[0073] FIG. 12 is a diagram showing an example of trajectory data indicating partial trajectories. The trajectory data D2 shown in FIG. 12 is data in which the measurement times and three-dimensional coordinates shown in the trajectory data D1 are associated with the hierarchy indicated by the partial trajectories to which each three-dimensional coordinate belongs. In the example of FIG. 12, the hierarchies indicating three partial trajectories, namely, "first hierarchy", "second hierarchy", and "third hierarchy", are associated in order from the three-dimensional coordinates with earlier measurement times. Hereinafter, the three-dimensional coordinates associated with each hierarchy shall be referred to as E01 to E99 (first hierarchy), F01 to F99 (second hierarchy), and G01 to G99 (third hierarchy).

[0074] Subsequently, the server device 10 outputs, by the trajectory output unit 116, trajectory data indicating the divided partial trajectories (for example, data showing the measurement time, three-dimensional coordinates, and hierarchy in association with each other as in FIG. 12) (S14). The trajectory output unit 116 outputs the trajectory data to a predetermined area of the storage unit 12, for example. This area is an area secured for work for performing analysis processing.

[0075] Next, the server device 10 acquires, by the hierarchy acquisition unit 117, hierarchy data indicating the hierarchy of the structure (S15). The hierarchy data is data indicating information regarding each hierarchy of the structure (the number and order of the hierarchies, the height of the hierarchies, or the drawings of each hierarchy), as described above. The hierarchy data is input in advance by, for example, a user or an operator of the construction support system 1 and stored in the server device 10.

[0076] FIG. 13 is a diagram showing an example of the hierarchy data. The hierarchy data D3 shown in FIG. 13 is data in which the number of hierarchies, the floor number (order of the hierarchies), the height, and the drawings are associated with each other. In the hierarchy data D3, the number of hierarchies "3", the floor numbers "5th floor", "4th floor", and "3rd floor", the height "xxm", and the respective drawing data are associated with each other. This hierarchy data D3 indicates that, for example, the third to fifth floors of a high-rise structure such as an apartment are the construction site. Subsequently, the server device 10 outputs, by the correspondence output unit 118, correspondence data indicating the correspondence between the hierarchy indicated by the acquired hierarchy data and the partial trajectories (S16).

[0077] FIG. 14 is a diagram showing an example of corresponding data. The corresponding data D4 shown in FIG. 14 is data in which the measurement time, three-dimensional coordinates, and the hierarchy of partial trajectories shown in the trajectory data D2 are associated with the floor numbers. In the example of FIG. 14, the corresponding output unit 118 associates "5th floor" with "first hierarchy", "4th floor" with "second hierarchy", and "3rd floor" with "third hierarchy". The corresponding output unit 118 determines the vertical relationship between the hierarchies, for example, from the z coordinate indicating the height among the three-dimensional coordinates.

[0078] Then, the corresponding output unit 118 determines which floor number each hierarchy corresponds to based on the vertical relationship between the hierarchies. In the example of FIG. 14, the corresponding output unit 118 determines that the "first hierarchy" is the highest and the "third hierarchy" is the lowest, and determines that the highest "first hierarchy" is the "5th floor" and the lowest "third hierarchy" is the "3rd floor". According to such an aspect, it is possible to grasp which hierarchy the partial trajectory belongs to.

[0079] Note that the number of hierarchies indicated by the hierarchy data does not always match the number of hierarchies of the partial trajectories. For example, the number of hierarchies of the partial trajectories may be "3" (the number of hierarchies only at the construction site), while the number of hierarchies indicated by the hierarchy data may be "10" (for example, the number of hierarchies of the entire structure). In that case, the corresponding output unit 118 arranges the partial trajectories on the drawings of each hierarchy indicated by the hierarchy data, and calculates the degree of coincidence between those drawings and the partial trajectories.

[0080] For the degree of coincidence, for example, as described above, the maximum scale of the partial trajectories arranged without overlapping the wall on the drawing is used (the larger the maximum scale, the larger the degree of coincidence). Note that the method of obtaining the degree of coincidence is not limited to this. The corresponding output unit 118, for example, specifies two coordinates with the longest distance between each other among the coordinates indicating the partial trajectories, and sets the longest distance between the specified coordinates to 70% to 90% of the same longest distance in the internal space of the structure shown in the drawing to match the real space and the scale.

[0081] The reason why the longest distance of the partial trajectory is shorter than the longest distance of the internal space of the drawing is that the imaging device 30 rarely reaches the end of the internal space and mostly moves inside the end of the internal space. Then, the correspondence output unit 118 calculates, as the degree of coincidence, a value obtained by subtracting from 1 the ratio of the partial trajectory that most overlaps with the wall when overlaid on the drawing in a scaled state. In this case, the degree of coincidence of the partial trajectory that does not overlap with the wall on the drawing at all in the scaled state is the maximum value of "1", and the larger the ratio of overlap with the wall, the smaller the value and the smaller the degree of coincidence.

[0082] By thus determining the correspondence between the hierarchy indicated by the hierarchical data and the partial trajectory based on the degree of coincidence, even when only some of the hierarchies formed by the structure are measured, the hierarchy of the partial trajectory can be grasped.

[0083] Next, the server device 10 specifies, by the feature specifying unit 119, locations having a predetermined feature (hereinafter referred to as "feature locations") in the drawings of each hierarchy (S17). The feature specifying unit 119 specifies, for example, the start position where measurement by the imaging device 30 starts and the end position where it ends in the drawing as the feature locations.

[0084] Specifically, the feature specifying unit 119 specifies, as the start position, the position where the feature amount defined as the start position (hereinafter referred to as "start feature amount") among the feature amounts indicated by the measurement data acquired by the measurement acquisition unit 115 is measured. The start feature amount is, for example, the feature amount indicated by the marker 90 of the marker plate 9 shown in FIG. 1.

[0085] FIG. 15 is a diagram showing an example of the marker 90. In the example of FIG. 15, the horizontal length L11 and the vertical length L12 of the marker 91, and the horizontal length L13 and the vertical length L14 of the marker 92 are shown. Further, the marker 92 is installed at a height L15 from the floor surface 100, and the marker 91 is installed at a position higher by a height L16 than the marker 92. The server device 10 stores the information on these lengths and heights as information regarding the size of the marker 90.

[0086] The feature identification unit 119 identifies, for example, the contour shown in the captured image through image processing that detects edges. When two rectangles arranged vertically are shown by the recognized contour, the feature identification unit 119 identifies the capture position of the captured image as the start position where the start feature amount indicating the marker 90 is measured. In this way, the feature identification unit 119 identifies the start position where the start feature amount is measured by recognizing the marker 90 whose features are known in advance. Note that the feature identification unit 119 may also recognize the marker 90 based on the aspect ratio of the recognized rectangles and the distance between the rectangles. According to such an aspect, the alignment of the movement trajectory can be easily performed as compared with the case where the start feature amount is not used.

[0087] Further, the feature identification unit 119 identifies the last measurement position in the entire estimated trajectory as the end position where the measurement is completed. In the example of FIG. 14, the on-site worker W1 starts the measurement from the 5th floor and ends the measurement on the 3rd floor. However, there may be a case where the worker turns back on the 3rd floor and returns to the installation position of the stand 8 on the 5th floor to end the shooting. In that case, the feature identification unit 119 may use the start feature amount as the end feature amount and identify the position where the end feature amount is measured as the end position.

[0088] Further, as described above, the feature identification unit 119 identifies the connection path connecting between the respective layers shown in the drawing as a location having a predetermined feature in the drawing. The feature identification unit 119 identifies, for example, the location where a staircase (represented by continuous rectangles and arrows) is shown in the drawing as a feature location.

[0089] Next, the server device 10 causes the association unit 120 to associate the position of the end of the partial trajectory corresponding to the layer with the feature location identified in the drawing of that layer (S18). The association unit 120 indicates the association of the feature location, for example, in the association data as shown in FIG. 14.

[0090] FIG. 16 is a diagram showing an example of corresponding data. The corresponding data D5 shown in FIG. 16 is data obtained by adding a column of characteristic portions to the corresponding data D4 shown in FIG. 14. For example, the first three-dimensional coordinates E01 with the earliest measurement time and the last three-dimensional coordinates E99 with the latest measurement time in the first layer (5th floor) where the start position is specified indicate the positions of the ends of the partial trajectories corresponding to the first layer, respectively. The association unit 120 associates the three-dimensional coordinates E01 indicating the end with the start position specified for the first layer, and associates the three-dimensional coordinates E99 with the connection path specified for the first layer.

[0091] Also, the three-dimensional coordinates F01 indicating the first measurement position and the three-dimensional coordinates F99 indicating the last measurement position in the second layer (4th floor) indicate the positions of the ends of the partial trajectories corresponding to the second layer, respectively. The association unit 120 associates both the three-dimensional coordinates F01 and F99 with the connection path specified for the second layer. Specifically, the association unit 120 associates the three-dimensional coordinates F01 with an earlier measurement time with the connection path to the first layer measured earlier, and associates the three-dimensional coordinates F99 with a later measurement time with the connection path to the third layer measured later.

[0092] Also, the three-dimensional coordinates G01 indicating the first measurement position and the three-dimensional coordinates G99 indicating the last measurement position in the third layer (3rd floor) indicate the positions of the ends of the partial trajectories corresponding to the third layer, respectively. The association unit 120 associates the three-dimensional coordinates G01 and G99 with the connection path specified for the third layer. Specifically, the association unit 120 associates the three-dimensional coordinates G01 with the connection path specified for the third layer. Also, the association unit 120 associates the three-dimensional coordinates G99 with the end position specified for the third layer.

[0093] In this way, by associating the positions of the ends of the partial trajectories with the characteristic portions, the positions of the movement trajectories in the drawing can be automatically aligned. Also, by associating the connection paths with the positions of the ends of the partial trajectories as characteristic portions, the higher the number of connection paths, the higher the alignment accuracy can be improved.

[0094] Then, the server device 10 outputs, via the image output unit 121, a site image showing the site photographed at each position measured by the imaging device 30 indicated by the partial trajectory. As described above, the image output unit 121 outputs an image represented in a manner corresponding to the height of each position measured by the imaging device 30. For example, when it is indicated that the photographing position is lower than the floor surface, the image output unit 121 creates and outputs an image (for example, an image including the string "under the floor") indicating that the photographing position is under the floor.

[0095] Also, when it is indicated that the photographing position is higher than the ceiling, the image output unit 121 creates and outputs an image (for example, an image including the string "above the ceiling") indicating that the photographing position is above the ceiling. Further, when the photographing position is higher than the floor surface and lower than the ceiling, the image output unit 121 creates and outputs an image (for example, an image including the string "inside the room") indicating that the photographing position is inside the room. Since the measurement positions indicated by the partial trajectory are measurement positions on the same floor, by outputting the images as described above, the difference in height in the trajectory can be shown even on the same floor.

[0096] In the analysis process shown in FIG. 10, the trajectory output unit 116 generates trajectory data and then executes partial trajectory processing, but the order of these processes may be reversed. In that case, the trajectory output unit 116, for example, recognizes a staircase from the acquired measurement data and divides it before and after the staircase, and generates trajectory data indicating the partial trajectories of each floor based on the divided measurement data. In any method, since the on-site worker W1 only needs to photograph the construction site once with the imaging device 30, the movement trajectory for each floor of the structure can be obtained by a single measurement.

[0097] <Configuration Variations> Further, the construction support system 1 may include a measuring device different from the imaging device 30. The measuring device may include, for example, a distance image sensor that measures the distance to an object and outputs point cloud data indicating the measurement result as measurement data. Further, the measuring device may include an infrared sensor that measures the temperature of the object or a millimeter-wave sensor that measures the distance to the object and the speed of the object. Further, the measuring device may be a wide-angle camera or the like that enables higher-resolution imaging.

[0098] Further, if the construction support system 1 is a construction site where the breaker 6 is not turned off, the on-site terminal 20 and the imaging device 30 may be connected to the external power source 5 without passing through the external battery 4. Further, in the embodiment, the on-site terminal 20 transmitted the measurement data to the server device 10 by mobile communication even when the breaker 6 was turned off. However, when the breaker 6 is not turned off, the measurement data may be transmitted to the server device 10 by Wi-Fi communication by installing a Wi-Fi router at the construction site.

[0099] Further, for example, the server device 10 may be distributed among two or more devices or may be replaced by a cloud computing system. Further, the on-site terminal 20 and the imaging device 30 may be integrated. Also, the functional configuration shown in FIG. 4 is merely an example and is not limited thereto. For example, the functions of the server device 10, the on-site terminal 20, and the imaging device 30 may be realized by being distributed among two or more devices respectively.

[0100] Further, the operations performed by one function may be distributed among two or more functions, or two or more functions may be integrated into one function. In short, as long as the functions shown in FIG. 6 are realized in the entire construction support system 1, the devices that realize these functions may have any configuration.

[0101] The aspects of the above-described embodiments were information processing apparatuses such as the server apparatus 10 and information processing systems such as the construction support system 1 including the server apparatus 10, but may also be information processing methods. The information processing method includes steps of each process executed by the information processing system. Further, the aspects of the above-described embodiments may also be programs. The program causes a computer to execute each process executed by a similar information processing system.

[0102] <Appendix> Furthermore, it may be provided in each of the aspects described below.

[0103] (1) An information processing system, in a measurement acquisition step, acquires measurement data measured at each position of a movement path while a measurement device having a sensor that measures electromagnetic waves reaching from an object moves inside or outside a structure, and in a trajectory output step, outputs trajectory data indicating the trajectory of the measurement device based on the acquired measurement data, and the trajectory has two or more partial trajectories in which positions where the measurement device has performed measurement are respectively within ranges of two or more different heights when the structure has two or more floors.

[0104] According to such an aspect, the movement trajectory for each floor of the structure can be obtained by a single measurement.

[0105] (2) In the information processing system according to (1) above, in a floor acquisition step, acquires floor data indicating the floor of the structure, and in a correspondence output step, outputs correspondence data indicating a correspondence relationship between the floor indicated by the acquired floor data and the partial trajectory.

[0106] According to such an aspect, it is possible to grasp which floor the partial trajectory belongs to.

[0107] (3) In the information processing system according to (2) above, the hierarchical data shows the drawings of each layer of the structure. In the specific step, a location having a predetermined feature in the drawing is specified, and in the association step, the position of the end of the partial locus corresponding to the layer is associated with the location specified in the drawing of the layer.

[0108] According to such an aspect, the position in the drawing of the movement locus can be aligned.

[0109] (4) In the information processing system according to (3) above, in the specific step, the connection path connecting between each layer shown in the drawing is specified as the location.

[0110] According to such an aspect, the higher the number of connection paths, the higher the alignment accuracy can be.

[0111] (5) In the information processing system according to (3) or (4) above, in the specific step, the start position where the measurement is started in the drawing is specified as the location, and in the association step, the position of the end of the partial locus corresponding to the layer where the start position is specified is associated with the start position.

[0112] According to such an aspect, the position in the drawing of the movement locus can be aligned.

[0113] (6) In the information processing system according to (5) above, in the specific step, the position where the feature amount determined as the start position among the feature amounts shown in the acquired measurement data is measured is specified as the start position.

[0114] According to such an aspect, the alignment of the movement locus can be easily performed.

[0115] (7) In the information processing system according to any one of (2) to (6) above, the hierarchical data shows the drawings of each layer of the structure, and in the corresponding output step, when the number of the partial trajectories is smaller than the layer shown by the acquired hierarchical data, data indicating the correspondence relationship between the layer with the highest matching degree when the partial trajectories are arranged in the drawing of the layer and the partial trajectories is output as the corresponding data.

[0116] According to such an aspect, even when only some layers are measured, the layer of the partial trajectories can be grasped.

[0117] (8) In the information processing system according to any one of (1) to (7) above, in the image output step, an image represented in a manner corresponding to the height of the position is output for each position where the measuring device indicated by the partial trajectories has performed measurement.

[0118] According to such an aspect, even on the same layer, the difference in height in the trajectory can be shown. Of course, this is not the limit. Also, the above-described embodiments and modified examples may be arbitrarily combined and implemented.

[0119] Finally, although various embodiments according to the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0120] 1: Construction Support System 10: Server Device 11: Control Unit 20: On-site Terminal 21: Control Unit 30: Imaging Device 31: Control Unit 40: Remote Terminal 41: Control Unit 111: Information Storage Unit 112: Image Processing Unit 113: Data Generation Unit 114: Data Output Unit 115: Measurement Acquisition Unit 116: Trajectory Output Unit 117: Hierarchy Acquisition Unit 118: Corresponding Output Unit 119: Feature Identification Unit 120: Association Unit 121: Image Output Unit 211: Display Control Unit 212: Operation Reception Unit 213: Operation Control Unit 214: Transmission Control Unit 311: Display Control Unit 312: Operation Reception Unit 313: Shooting Control Unit 314: Transmission Control Unit 411: Display Control Unit 412: Operation Reception Unit

Claims

1. An information processing system, In the measurement acquisition step, measurement data measured at each position on the movement path is acquired while a measurement device having a sensor for measuring electromagnetic waves reaching from an object moves inside or outside a structure, In the trajectory output step, trajectory data indicating the trajectory of the measurement device is output based on the acquired measurement data. When the structure has two or more floors, the trajectory has two or more partial trajectories in which the positions where the measurement device performs measurements are respectively within ranges of two or more different heights, In the floor acquisition step, floor data including drawings showing layouts partitioned by walls for each floor of the structure is acquired, In the identification step, a location having a predetermined feature in the drawing is identified, In the association step, the position of the end of the partial trajectory corresponding to the floor is associated with the location identified in the drawing of the floor, In the association output step, association data showing the correspondence between the floors indicated by the acquired floor data and the partial trajectories is output.

2. In the information processing system according to Claim 1, In the identification step, a connection path connecting between each floor shown in the drawing is identified as the location.

3. In the information processing system according to Claim 1, In the identification step, a start position at which the measurement is started in the drawing is identified as the location, In the association step, the position of the end of the partial trajectory corresponding to the floor where the start position is identified is associated with the start position.

4. In the information processing system according to Claim 3, In the identification step, a position where the feature amount defined as the start position among the feature amounts indicated by the acquired measurement data is measured is identified as the start position.

5. In the information processing system according to claim 1, the hierarchical data shows the drawings of each layer of the structure, in the corresponding output step, when the number of the partial trajectories is smaller than the layer indicated by the acquired hierarchical data, data indicating the correspondence relationship between the layer that gives the highest matching degree when the partial trajectories are arranged in the drawing of the layer and the partial trajectories is output as the corresponding data.

6. In the information processing system according to claim 1, in the image output step, an image represented in a manner corresponding to the height of the position where the measuring device indicated by the partial trajectory performs measurement is output for each position where the measurement is performed.

7. A program, which causes a computer to execute each step of the information processing system according to any one of claims 1 to 6. Program.

8. An information processing method, which causes an information processing system to execute each step according to any one of claims 1 to 6. Information processing method.

Images