Information processing method, information processing device, and information processing program

Abstract

This information processing method includes: calculating, on the basis of sensor information, an estimated current position of a photographer during a photographing operation for photographing a predetermined space using a photographic device; determining, on the basis of the estimated current position, whether or not the photographer has arrived at a checkpoint; presenting to the photographer an input request of the current position each time the photographer is determined to have arrived at the checkpoint; acquiring an input current position, which is the current position input by the photographer in response to the input request; and, for images photographed after the checkpoint was reached, acquiring a photographing point through a self-position estimating process using the input current position.

Description

Information processing method, information processing device, and information processing program

[0001] The present disclosure relates to a technology for determining a shooting point of an image taken in a predetermined space.

[0002] Patent Document 1 discloses a method for estimating the relative position and orientation of a camera from a map image contained in a captured image using various camera calibration techniques, acquiring data of a drawing that is identical to the drawing contained in the captured image, and generating guidance information in which an icon indicating the position and orientation of the camera is superimposed on the acquired drawing.

[0003] However, in the conventional technology disclosed in Patent Document 1, the relative position and relative orientation of the image capturing device are estimated using camera calibration technology, and further improvements are required to more accurately calculate the image capturing point using self-position estimation processing.

[0004] Japanese Patent Application Laid-Open No. 2021-38959

[0005] The present disclosure has been made to solve such problems, and aims to provide a technology for more accurately calculating a shooting point using a self-position estimation process.

[0006] An information processing method according to one aspect of the present disclosure includes acquiring an estimated current position of a photographer who is photographing a specified space using an imaging device based on sensor information, determining whether the photographer has reached a checkpoint based on the estimated current position, presenting the photographer with a request to input a current position each time it is determined that the photographer has reached the checkpoint, acquiring an input current position which is the current position input by the photographer in response to the input request, and acquiring a photographing location for an image photographed after the checkpoint has been reached based on a self-position estimation process using the input current position.

[0007] According to the present disclosure, the imaging point can be calculated more accurately using the self-position estimation process.

[0008] FIG. 1 is an overall configuration diagram of an information processing system in embodiment 1. FIG. 2 is a block diagram showing a detailed configuration of an information terminal. FIG. 3 is a flowchart showing processing of the information processing system in embodiment 1. FIG. 4 is a diagram showing an example of a display screen of a design drawing onto which a shooting location icon is mapped. FIG. 5 is a block diagram showing a detailed configuration of an information terminal in embodiment 2. FIG. 6 is a flowchart showing processing of the information processing system in embodiment 2.

[0009] (Findings underlying the present disclosure) A user interface is being developed that displays multiple photography location icons indicating locations where actual photographs of a construction site were taken on a two-dimensional blueprint of the construction site, etc., and when one photography location icon is selected, displays an image taken at the photography location indicated by that icon. This allows a construction site manager to understand the situation at the construction site without visiting the construction site.

[0010] In such user interfaces, the shooting locations of multiple images taken by the camera are generally calculated by a self-position estimation process using the shooting start point input by the photographer at the start of the shooting operation.

[0011] However, if the point input by the photographer is only the shooting start point, the difference between the shooting point calculated by the self-position estimation process and the actual shooting point may increase as the shooting operation progresses.

[0012] Therefore, the inventors discovered that if a checkpoint is set during the shooting operation, the photographer is asked to input their current location at that checkpoint, and the input current location is used to calculate the shooting location of images taken thereafter using a self-position estimation process, the shooting location can be calculated with high accuracy, which led to the present disclosure.

[0013] (1) An information processing method according to one aspect of the present disclosure includes: acquiring, based on sensor information, an estimated current position of a photographer who is photographing a specified space using an imaging device; determining, based on the estimated current position, whether the photographer has reached a checkpoint; presenting the photographer with a request to input a current position each time it is determined that the photographer has reached the checkpoint; acquiring an input current position, which is the current position input by the photographer in response to the input request; and acquiring, for images photographed after reaching the checkpoint, a photographing location based on a self-position estimation process using the input current position.

[0014] With this configuration, the photographer inputs the current position every time the photographer reaches a checkpoint, and the self-position estimation process uses this input current position to calculate the shooting point of the image captured thereafter. This reduces the discrepancy between the shooting point acquired by the self-position estimation process and the actual shooting point, allowing for more accurate calculation of the shooting point.

[0015] (2) In the information processing method described in (1) above, the checkpoints may include a first checkpoint, which is a point where the photographer has moved a predetermined distance from the most recent checkpoint, and a second checkpoint, which is a point where the photographer has moved a predetermined time from the most recent checkpoint.

[0016] According to this configuration, the photographer inputs the current position every time the photographer moves a predetermined distance or a predetermined time, so that the deviation between the calculated photographing location and the actual photographing location can be further reduced.

[0017] (3) In the information processing method described in (2) above, determining whether or not the destination has been reached may include increasing the specified distance or the specified time if the difference between the estimated current position at the first checkpoint and the input current position is smaller than a reference value.

[0018] According to this configuration, if the difference between the estimated current position at the first checkpoint and the input current position is smaller than a reference value, the predetermined distance or the predetermined time is increased, thereby reducing the frequency of inputting the input current position and improving convenience for the photographer.

[0019] (4) In the information processing method described in (3) above, the sensor device that detects the sensor information may be an IMU sensor carried by the photographer, and the method may further include correcting the estimated current position using a correction formula for suppressing drift included in the estimated current position calculated from the sensor information of the IMU sensor, and if the difference is greater than the reference value, modifying the correction formula based on the estimated current position and the input current position.

[0020] According to this configuration, if the difference between the estimated current position and the input current position is greater than a reference value, the correction formula for suppressing the drift included in the estimated current position is modified to reduce the drift, thereby enabling the estimated current position to be calculated more accurately.

[0021] (5) In the information processing method described in any one of (1) to (4) above, the checkpoint may include a third checkpoint that is a predetermined point in the predetermined space.

[0022] According to this configuration, it is possible to reliably acquire images taken at predetermined points such as important points.

[0023] (6) In the information processing method described in any one of (1) to (5) above, the method may further include detecting, based on the shooting location and a bird's-eye view of the specified space, whether the space in which the photographer is located has changed from a first subspace to a second subspace different from the first subspace, and the checkpoint may include a fourth checkpoint, which is the point at which the change from the first subspace to the second subspace is detected.

[0024] According to this configuration, the shooting point of an image captured in the second subspace can be accurately calculated.

[0025] (7) In the information processing method described in (5) above, the third checkpoint may be a location where radio wave conditions are poor.

[0026] According to this configuration, it is possible to accurately calculate the shooting location of an image taken at a location with poor radio wave conditions.

[0027] (8) In the information processing method described in any one of (1) to (7) above, the checkpoint includes a fifth checkpoint, and the determining step further includes calculating a movement trajectory of the photographer by plotting the shooting location on a bird's-eye view of the specified space in real time, and detecting a characteristic point of the movement trajectory as the fifth checkpoint, and the characteristic point may include a point where the photographer has turned.

[0028] According to this configuration, the point where the accuracy of calculation of the shooting point by the self-position estimation process becomes low is set as the fifth checkpoint, and the photographer can be prompted to input the input current position at the fifth checkpoint.

[0029] (9) In the information processing method described in any one of (1) to (8) above, the sensor device that detects the sensor information is a communication device that is communicatively connected to a plurality of wireless communication devices installed in the specified space and carried by the photographer, and the estimated current location may be calculated by triangulation using the received signal strengths of each of a plurality of wireless signals transmitted by the plurality of wireless communication devices and the known installation locations of each of the plurality of wireless communication devices.

[0030] According to this configuration, the estimated current location can be accurately calculated based on the radio wave intensity of the radio signal received from the wireless communication device installed in the predetermined space and the known installation position of the wireless communication device.

[0031] (10) In the information processing method described in any one of (1) to (8) above, the sensor device that detects the sensor information may be a barcode reader carried by the photographer, and the barcode reader may read a barcode installed at the checkpoint that indicates the position of the checkpoint, and the estimated current position may be calculated based on the barcode read by the barcode reader.

[0032] According to this configuration, the estimated current location can be calculated by reading the barcode installed at the checkpoint with a barcode reader.

[0033] (11) In another aspect of the present disclosure, an information processing device includes a processor that executes the following operations: acquiring an estimated current position of a photographer who is photographing a specified space using an imaging device based on sensor information; determining whether the photographer has reached a checkpoint based on the estimated current position; presenting the photographer with a request to input a current position each time it is determined that the photographer has reached the checkpoint; acquiring an input current position, which is the current position input by the photographer in response to the input request; and acquiring a photographing location based on a self-position estimation process using the input current position, for images photographed after reaching the checkpoint.

[0034] According to this configuration, it is possible to provide an information processing device that can more accurately calculate the shooting point.

[0035] (12) In another aspect of the present disclosure, an information processing program causes a computer to perform the following: calculate, based on sensor information, an estimated current position of a photographer who is photographing a specified space using an imaging device; determine, based on the estimated current position, whether the photographer has reached a checkpoint; present the photographer with a request to input their current position each time it is determined that the photographer has reached the checkpoint; obtain an input current position, which is the current position input by the photographer in response to the input request; and, for images photographed after reaching the checkpoint, obtain the photographing location based on a self-position estimation process using the input current position.

[0036] According to this configuration, it is possible to provide an information processing program that can more accurately calculate the photographing point.

[0037] The present disclosure can also be realized as an information processing system operated by such an information processing program. Needless to say, such a computer program can be distributed on a non-transitory computer-readable recording medium such as a CD-ROM or via a communication network such as the Internet.

[0038] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all of the embodiments, the respective contents can be combined.

[0039] (Embodiment 1) FIG. 1 is an overall configuration diagram of an information processing system 1 according to Embodiment 1. The information processing system 1 is a system that improves the accuracy of calculating the shooting location by having a photographer input their current location each time the photographer reaches a checkpoint and calculating the shooting location of each image using the input current location. The information processing system 1 includes a server 10, an information terminal 20, a photographing device 30, and an administrator terminal 40. The server 10, the information terminal 20, and the administrator terminal 40 are connected to each other so as to be able to communicate with each other via a network NT. An example of the network NT is the Internet. The server 10 is, for example, a cloud server configured with one or more computers. However, this is just one example, and the server 10 may be configured as an edge server or may be implemented in the information terminal 20. The aspect in which the server 10 is implemented in the information terminal 20 is an example of an aspect in which the information terminal 20 is configured as an information processing device.

[0040] The information terminal 20 is configured as a portable information terminal such as a smartphone or a tablet computer, and connects the photographing device 30 to the network NT. The information terminal 20 and the photographing device 30 are connected via a wireless communication path such as Bluetooth (registered trademark) or a wireless LAN. The information terminal 20 is carried by the photographer, and acquires the current location input by the photographer at checkpoints.

[0041] The information terminal 20 includes a first communication unit 21 , a processor 22 , a display 23 , an operation unit 24 , an IMU (Inertial Measurement Unit) sensor 25 , and a second communication unit 26 .

[0042] The first communication unit 21 is a communication interface that connects the information terminal 20 to the network NT. The first communication unit 21 acquires photographing information (described later) that is transmitted from the photographing device 30 and received by the second communication unit 26, and transmits the acquired photographing information to the server 10.

[0043] The processor 22 is configured by, for example, a central processing unit (CPU), and displays a display screen of the design drawing of the predetermined space transmitted from the server 10 on the display 23. Details of the processor 22 will be described later with reference to FIG.

[0044] The display 23 is configured with various display devices such as a liquid crystal display and an organic EL display, and displays various display screens under the control of the processor 22. In this embodiment, the display 23 displays a display screen of a design drawing of a predetermined space.

[0045] The operation unit 24 is configured by, for example, a touch panel, and accepts various instructions input by the user. In this embodiment, the operation unit 24 accepts input of a shooting start point and a shooting end point from the photographer.

[0046] The IMU sensor 25 includes an acceleration sensor and a gyro sensor, and detects the speed and angular velocity of the information terminal 20. The IMU sensor 25 is used to calculate an estimated current position of the photographer.

[0047] The second communication unit 26 is a communication device for communicating with a wireless communication device installed in a predetermined space and is used to calculate an estimated current location of the photographer. The second communication unit 26 communicates with the wireless communication device using a wireless communication standard such as Wireless LAN (registered trademark) or Bluetooth (registered trademark). The wireless communication device is, for example, an IoT device, a wireless LAN access point, or a beacon radio. The second communication unit 26 receives shooting information transmitted from the shooting device 30.

[0048] The image capturing device 30 is, for example, an omnidirectional camera that captures images at a predetermined frame rate. An omnidirectional camera is also called a 360-degree camera, and is a camera that can capture images in all directions of 360 degrees. The image capturing device 30 is, for example, a portable image capturing device carried by a photographer. The photographer is, for example, a worker or site supervisor at a construction site. The image capturing device 30 may also be a regular camera.

[0049] The photographer moves within a predetermined space while photographing the construction site with the photographing device 30. When the photographing is completed, the photographing device 30 transmits photographing information including the series of photographed images to the server 10 via the information terminal 20.

[0050] The photographing information includes an image photographed by the photographing device 30, a photographing start point, a photographing end point, and a blueprint ID corresponding to the specified space. The blueprint ID is an identifier that identifies the blueprint of the specified space. The photographing information also includes an input current position, which is the current position input by the photographer at the checkpoint. The photographing start point, photographing end point, and input current position are added by the information terminal 20.

[0051] At the start point of shooting, the photographer points the camera 30 in a predetermined direction, presses the camera button on the camera 30, and starts the shooting operation. When the photographer reaches the end point of shooting, he or she presses the camera button again to end the shooting operation. The predetermined direction is a direction that aligns with the blueprint displayed on the display 23. The predetermined direction is, for example, north, but is not limited to this.

[0052] The start and end points of photography are identified by the photographer inputting instructions specifying the positions on the display screen of the blueprint of the specified space displayed on the display 23 of the information terminal 20. Two-dimensional coordinate axes are defined on the blueprint. Therefore, the start and end points of photography are each defined by two-dimensional coordinate values. The start and end points of photography are used by the server 10 to identify the position of each photography point and the photography direction of each image.

[0053] The administrator terminal 40 is a terminal device carried by an administrator who remotely manages a predetermined space. The predetermined space is, for example, a construction site. However, this is just one example, and the predetermined space may also be a construction site, factory, store, office, etc. The administrator terminal 40 may be configured as a portable computer such as a smartphone or tablet computer, or as a stationary computer.

[0054] In the example of FIG. 1, one information terminal 20 and one administrator terminal 40 are shown, but a plurality of information terminals 20 and a plurality of administrator terminals 40 may be connected to the server 10 via the network NT.

[0055] The server 10 includes a processor 11, a memory 12, and a communication unit 13. The processor 11 is configured, for example, by a central processing unit (CPU). The processor 11 includes an acquisition unit 111, a photography location calculation unit 112, and a display control unit 113. The acquisition unit 111 to the display control unit 113 may be realized by the processor 11 executing an information processing program, or may be configured by dedicated hardware circuits.

[0056] The acquisition unit 111 acquires the photography information transmitted from the information terminal 20 using the communication unit 13. The acquisition unit 111 acquires a display instruction for the blueprint from the information terminal 20 or the manager terminal 40 using the communication unit 13. The acquisition unit 111 acquires a selection instruction for the photography location icon displayed on the blueprint from the manager terminal 40 using the communication unit 13.

[0057] The photographing point calculation unit 112 calculates (acquires) the photographing point of each of the multiple images photographed by the photographing device 30 using the photographing information acquired by the acquisition unit 111. The photographing point calculation unit 112 calculates the photographing point of each image using a self-position estimation process. Visual simultaneous localization and mapping (VSLAM) can be used as the self-position estimation process.

[0058] The display control unit 113 displays a display screen of the design drawing on the display 23 of the information terminal 20 or the display of the manager terminal 40 in response to a display instruction from the information terminal 20 or the manager terminal 40. The display control unit 113 displays an image taken at the photography location indicated by the photography location icon on the display of the manager terminal 40 in response to an instruction to select a photography location icon.

[0059] The blueprint is an example of a bird's-eye view. The blueprint display instruction includes a blueprint ID. The display control unit 113 reads the blueprint indicated by the blueprint ID from the blueprint information storage unit 121 and displays the display screen of the read blueprint on the display 23.

[0060] A blueprint is a bird's-eye view of a building, including columns, walls, or materials. A blueprint is a drawing showing the design of a construction site, and may be a floor plan, blueprint, map, or perspective drawing of the construction site. A bird's-eye view can also be called a bird's-eye view, and may be a view from above or a view from a high place.

[0061] The memory 12 is configured as a non-volatile rewritable storage device such as a hard disk drive or a solid state drive. The memory 12 includes a blueprint information storage unit 121 and an image information storage unit 122. The blueprint information storage unit 121 stores blueprint information. The blueprint information is an image showing a blueprint of a specified space. The blueprint information is associated with a blueprint ID that identifies the blueprint.

[0062] The image information storage unit 122 stores image information including the photography information transmitted from the photography device 30. The image information includes multiple images included in the photography information transmitted from the photography device 30, the photography start point, the photography end point, the input current position, and the blueprint ID, as well as meta information for each image. The image information is generated each time the photography information described above is transmitted. The meta information includes the photography ID, photography period ID, photography time, photography direction, and photography location.

[0063] The shooting ID is an identifier of the shooting location. The shooting period ID is an identifier of the shooting period. The shooting period refers to the period from the start to the end of the shooting operation. The shooting time is the date and time when the image was shot. The shooting direction is the shooting direction of each image in the shooting device 30. The shooting location is composed of a position (two-dimensional coordinate value) that indicates the shooting location of the image.

[0064] The photography ID is an identifier of the photography location. The photography period ID is an identifier of the photography period. The blueprint ID is an identifier of the blueprint of the specified space corresponding to the image.

[0065] The communication unit 13 is a communication interface that connects the server 10 to the network NT.

[0066] 2 is a block diagram showing a detailed configuration of the information terminal 20. The processor 22 of the information terminal 20 includes an acquisition unit 221, an estimated current position calculation unit 222, an arrival determination unit 223, an input request presentation unit 224, an input current position acquisition unit 225, and an output unit 226. The acquisition unit 221 to the output unit 226 may be realized by the processor 22 executing an information processing program, or may be configured by dedicated hardware circuits.

[0067] The acquisition unit 221 acquires the shooting start point and shooting end point input by the photographer using the operation unit 24 .

[0068] The estimated current position calculation unit 222 calculates the estimated current position of a photographer who is moving around and photographing a predetermined space using the imaging device 30, based on sensor information detected by a sensor device carried by the photographer. The sensor device includes, for example, the IMU sensor 25, the second communication unit 26, and the imaging device 30.

[0069] The estimated current position calculation unit 222 may calculate the estimated current position using at least one of the following first to third estimation methods.

[0070] (First Estimation Method) The first estimation method is a method of calculating an estimated current position from acceleration detected by the IMU sensor 25. In the first estimation method, the estimated current position calculation unit 222 estimates the estimated current position, which is the current position of the information terminal 20, at a predetermined sampling interval by performing a second-order integration of acceleration data detected by the IMU sensor 25 at a predetermined sampling interval. The estimated current position calculation unit 222 may also calculate the orientation of the information terminal 20 at a predetermined sampling interval by integrating angular velocity data detected by the IMU sensor 25 at a predetermined sampling interval. For example, a photographer positions the image capture device 30 and the information terminal 20 facing north and starts a photographing operation. This allows the estimated current position calculation unit 222 to calculate the orientation of the information terminal 20 based on the north orientation. In the following description, the position of the information terminal 20 refers to the position of the image capture device 30 and the position of the photographer.

[0071] (Second Estimation Method) The second estimation method is a method for calculating an estimated current location by communicating with multiple wireless communication devices installed in a predetermined space. In the second estimation method, the estimated current location calculation unit 222 detects the received signal strength of wireless signals transmitted from multiple wireless communication devices installed in a predetermined space at the second communication unit 26. The estimated current location calculation unit 222 may calculate the estimated current location by triangulation using the received signal strength of each of the multiple wireless communication devices and the known installation positions of each of the multiple wireless communication devices. Note that the estimated current location calculation unit 222 may calculate the estimated current location when wireless signals from at least two wireless communication devices are received by the second communication unit 26. The wireless signals from the wireless communication devices include identifiers of the wireless communication devices. Meanwhile, the information terminal 20 stores, in a memory (not shown), coordinates of the installation positions of each wireless communication device in association with the identifier of the communication device. Therefore, the estimated current location calculation unit 222 can identify the installation position of the wireless communication device that transmitted the wireless signal from the identifier included in the wireless signal. The estimated current position calculation unit 222 calculates the distance between the information terminal 20 and each wireless communication device from the installation position of each wireless communication device and the received signal strength of the wireless signal from each wireless communication device. The estimated current position calculation unit 222 sets, for each wireless communication device, multiple circles whose radii are the distance between the information terminal 20 and each wireless communication device, and calculates the intersection of the multiple circles as the estimated current position.

[0072] In addition, if BIM (Building Information Modeling) information for a specified space is stored in the memory (not shown) of the information terminal 20, the estimated current position calculation unit 222 may identify the coordinates of the installation position of the wireless communication device from the BIM information. BIM information indicates a three-dimensional model that recreates a specified space in a computer space. The BIM information stores the coordinates of the installation position of the wireless communication device installed in the specified space in association with an identifier. Therefore, the estimated current position calculation unit 222 can identify the installation position of the wireless communication device from the BIM information.

[0073] (Third Estimation Method) The third estimation method is a method of calculating an estimated current location by reading a barcode attached to a predetermined checkpoint (hereinafter referred to as the third checkpoint) within a specified space using a barcode reader. The image capture device 30 is an example of a barcode reader that reads barcodes. The barcode is attached to, for example, a building material (e.g., a wall) located at the third checkpoint. The barcode includes coordinates of the barcode's installation location. Therefore, if a barcode is included in an image captured by the image capture device 30, the estimated current location calculation unit 222 identifies the coordinates of the third checkpoint by decoding the barcode. The estimated current location calculation unit 222 may calculate the estimated current location based on the coordinates of the identified third checkpoint and the position where the barcode appears in the image. For example, a QR Code (registered trademark) can be used as the barcode.

[0074] The arrival determination unit 223 determines whether or not the photographer has reached a checkpoint during the shooting operation based on the estimated current position calculated by the estimated current position calculation unit 222 .

[0075] The checkpoints include a first checkpoint, which is a location a predetermined distance away from the nearest checkpoint. The checkpoints also include a second checkpoint, which is a location to which the photographer has traveled a predetermined time from the nearest checkpoint. In this case, the arrival determination unit 223 may calculate the travel distance from the nearest checkpoint using the estimated current position calculated by the estimated current position calculation unit 222, and determine that the photographer has reached the first checkpoint when the calculated travel distance reaches the predetermined distance. The arrival determination unit 223 may also measure the travel time from the nearest checkpoint, and determine that the photographer has reached the second checkpoint when the measured travel time reaches the predetermined time.

[0076] The third checkpoint has the barcode set thereon. Therefore, the arrival determination unit 223 determines that the photographer has reached the third checkpoint when the estimated current position calculated by the estimated current position calculation unit 222 using the barcode is within a predetermined distance from the barcode setting position. The third checkpoint may be, for example, a location with poor radio wave reception.

[0077] The input request presentation unit 224 presents the photographer with a request to input the current position every time the arrival determination unit 223 determines that the photographer has reached a checkpoint. The input request presentation unit 224 may present the input request by displaying on the display 23 a message prompting the photographer to input the current position.

[0078] The input current position acquisition unit 225 acquires an input current position, which is the current position input by the photographer in response to an input request, via the operation unit 24. The photographer inputs an operation to specify the current position on the display screen of the blueprint displayed on the display 23. The operation to specify the current position includes, for example, an operation of tapping the current position using the operation unit 24 on the display screen of the blueprint displayed on the display 23, an operation of inputting the coordinate values ​​of the current position using the operation unit 24, etc.

[0079] The output unit 226 acquires the shooting information transmitted from the shooting device 30 using the second communication unit 26. The output unit 226 identifies an image that was shot at a checkpoint from among multiple images included in the acquired shooting information, and links the identified image to the current input position. The output unit 226 outputs the shooting information including the image linked to the current input position to the server 10 using the first communication unit 21. The output unit 226 may identify the image whose shooting time is closest to the input time of the current input position as the image that was shot at the checkpoint.

[0080] FIG. 3 is a flowchart showing the processing of the information processing system 1 according to the first embodiment.

[0081] In step S1, the operation unit of the photographing device 30 receives an instruction to start photographing. For example, the photographing device 30 may determine that an instruction to start photographing has been input when the photographer presses the photographing button. The instruction to start photographing is transmitted from the photographing device 30 to the information terminal 20 and acquired by the acquisition unit 221 of the information terminal 20. This allows the information terminal 20 to determine that the photographing operation has started.

[0082] Next, in step S2, the acquisition unit 221 of the information terminal 20 accepts an operation to input a shooting start point. For example, the photographer inputs an operation to specify a start point on the display screen of the blueprint displayed on the display 23. The photographer may specify the shooting start point by tapping on the display screen of the blueprint, or may specify the coordinate values ​​of the shooting start point.

[0083] Next, in step S3, the estimated current position calculation unit 222 calculates the estimated current position of the photographer using one or more of the first to third estimation methods described above. For example, the estimated current position calculation unit 222 may basically calculate the estimated current position using either the first estimation method or the second estimation method, and if the estimated current position cannot be calculated using one of the methods, may calculate the estimated current position using the other method. Furthermore, since the barcode is placed at the third checkpoint, if the estimated current position is calculated using the third estimation method, the estimated current position calculation unit 222 may prioritize that estimated current position. This makes it possible to avoid a situation where the photographer's arrival at the third checkpoint goes undetected.

[0084] Next, in step S4, the arrival determination unit 223 uses the estimated current position calculated in step S3 to determine whether the photographer has reached any of the first to third checkpoints.

[0085] The reach determination unit 223 calculates the distance traveled by the photographer from the nearest checkpoint by accumulating the estimated current position calculated in step S3, and determines that the photographer has reached the first checkpoint when the traveled distance reaches a predetermined distance.

[0086] The arrival determination unit 223 determines that the photographer has reached the second checkpoint when the travel time of the photographer from the nearest checkpoint reaches a predetermined time.

[0087] The arrival determination unit 223 determines that the photographer has reached the third checkpoint when the estimated current position calculated using the third estimation method is within a predetermined distance from the barcode installation position.

[0088] The reach determination unit 223 may use either the first checkpoint or the second checkpoint, and the third checkpoint as checkpoints.

[0089] If it is determined that the photographer has reached the checkpoint (YES in step S4), the process proceeds to step S5. If it is determined that the photographer has not reached the checkpoint (NO in step S4), the process proceeds to step S7.

[0090] Next, in step S5, the input request presentation unit 224 presents the photographer with a request to input the current position by displaying on the display 23 a message prompting the photographer to input the current position.

[0091] Next, in step S6, the input current position acquisition unit 225 acquires the input current position input by the photographer via the operation unit 24 in response to the input request.

[0092] Next, in step S7, the output unit 226 determines whether the photographing operation has ended. When the photographing operation has ended, the photographer inputs a photographing end instruction to the photographing device 30. The photographing end instruction is output from the photographing device 30 to the information terminal 20. This allows the output unit 226 to recognize that the photographing operation has ended. If the photographing operation has not ended (NO in step S7), the process returns to step S3. On the other hand, if the photographing operation has ended (YES in step S7), the process proceeds to step S8.

[0093] Next, in step S8, the acquisition unit 221 acquires an operation to specify the shooting end point via the operation unit 24. The operation to specify the shooting end point includes, for example, an operation to tap the current position using the operation unit 24 on the display screen of the design drawing displayed on the display 23, an operation to input the coordinate values ​​of the current position using the operation unit 24, and the like.

[0094] Next, in step S9, the output unit 226 acquires the shooting information transmitted from the shooting device 30 via the second communication unit 26 and uploads the acquired shooting information to the server 10 via the first communication unit 21. At this time, the output unit 226 links the input position to the image captured at the checkpoint among the multiple images included in the shooting information. Furthermore, the output unit 226 adds the shooting start point and shooting end point to the shooting information. The shooting information is received by the communication unit 13 of the server 10 and input to the shooting position calculation unit 112 of the server 10.

[0095] Next, in step S10, the photographing point calculation unit 112 calculates the photographing points of the multiple images included in the photographing information using a self-position estimation process.

[0096] For images from the shooting start point to the point immediately before the first checkpoint, the shooting point calculation unit 112 calculates the shooting point of each image by self-position estimation processing using the shooting start point.For images from the first checkpoint to the point immediately before the second checkpoint, the shooting point calculation unit 112 calculates the shooting point of each image by self-position estimation processing using the input current position associated with the image of the first checkpoint.For images from the second checkpoint to the point immediately before the third checkpoint, the shooting point calculation unit 112 calculates the shooting point of each image by self-position estimation processing using the input current position associated with the image of the second checkpoint.In this way, for images taken after reaching a checkpoint, the shooting point calculation unit 112 calculates the shooting point by self-position estimation processing using the most recent input current position.Note that the most recent input current position means the input current position with the closest input time in the past from each image.

[0097] Details of the self-position estimation process by the photographing point calculation unit 112 are as follows. The photographing point calculation unit 112 detects feature points from each of the multiple images included in the photographing information through image processing. Feature points are characteristic points that indicate the outline of an object included in the image. The photographing point calculation unit 112 performs self-position estimation processing on each of the multiple images using at least two of the multiple images in which feature points are set, and calculates relative coordinate values ​​of the photographing point, which is the self-position corresponding to each image. The photographing point calculation unit 112 aligns the direction of the photographing point corresponding to each image by utilizing the fact that the photographing device 30 is set facing north at the start of the photographing operation. The photographing point calculation unit 112 calculates absolute coordinate values ​​of the photographing point corresponding to each image from the relative coordinate values ​​of the photographing point corresponding to each image, based on the photographing start point included in the photographing information or the most recently input current position, and calculates the calculated absolute coordinate values ​​as the coordinate values ​​of the photographing point corresponding to each image.

[0098] 4 is a diagram showing an example of a display screen 400 of a blueprint onto which a photography location icon 410 is mapped. The display screen 400 includes a blueprint of a predetermined space. The photography location icon 410 is superimposed on the display screen 400.

[0099] The photography location icon 410 is an icon indicating a photography location, and is associated with an image taken by the photography device 30. In this example, the photography location icon 410 is configured as a circular image. The multiple photography location icons 410 displayed here correspond to multiple photography locations belonging to one photography period. The route tracing from the photography location icon 410 at the beginning to the photography location icon 410 at the end indicates the trajectory followed by the photographer during this photography period.

[0100] When the acquisition unit 111 receives an instruction to select one of the multiple shooting location icons 410 (for example, shooting location icon 410n), the display control unit 113 displays an image corresponding to the shooting location icon 410n on the display 23.

[0101] In this way, according to this embodiment, the photographer inputs the current position every time the photographer reaches a checkpoint, and the self-position estimation process uses this input current position to calculate the shooting point of the image captured thereafter. This reduces the discrepancy between the shooting point calculated by the self-position estimation process and the actual shooting point, allowing the shooting point to be calculated more accurately.

[0102] (Embodiment 2) In embodiment 2, the frequency of inputting the input current location is changed depending on the accuracy of the estimated current location. Figure 5 is a block diagram showing a detailed configuration of an information terminal 20 in embodiment 2. Note that the same components as in embodiment 2 will be described with the same reference numerals.

[0103] The processor 22 further includes an estimated current position calculation unit 222A, an arrival determination unit 223A, and a correction unit 227.

[0104] The estimated current position calculation unit 222A calculates the estimated current position using the first estimation method (method of calculating the estimated current position using the IMU sensor 25) and the third estimation method (method of calculating the estimated current position using a barcode) described in embodiment 1, rather than using the second estimation method (method of calculating the estimated current position using a wireless communication device) described in embodiment 1. Therefore, the second communication unit 26 is omitted from the sensor device.

[0105] The arrival determination unit 223A increases the predetermined distance or the predetermined time when the difference between the estimated current position at the first checkpoint and the input current position at the first checkpoint is smaller than a reference value. Here, the arrival determination unit 223A may increase the predetermined distance or the predetermined time by a predetermined value. The reference value may be a predetermined value that indicates that the difference is within an allowable range.

[0106] If the difference between the estimated current position at the first checkpoint and the input current position at the first checkpoint is greater than a reference value, the correction unit 227 corrects, based on the estimated current position and the input current position, a correction formula for suppressing drift included in the estimated current position calculated from the sensor information of the IMU sensor 25. Thereafter, the correction unit 227 corrects the estimated current position calculated by the estimated current position calculation unit 222A using the corrected correction formula.

[0107] Drift refers to a phenomenon in which the sensor information detected by the IMU sensor 25 becomes unstable over time. The estimated current position is calculated by integrating acceleration data, but if the drift contained in the acceleration data is large, the drift is integrated and the estimated current position deviates significantly from the actual current position. Therefore, the correction unit 227 corrects the estimated current position using a correction formula.

[0108] If the estimated current position at the first checkpoint is Sn(Xn', Yn') and the input current position is Un(Xn, Yn), the correction formula H is expressed by the following formula.

[0109] ΔX={(Xn'-Xn-1)-(Xn-Xn-1)}×ΔT / (Tn-Tn-1) ΔY={(Yn'-Yn-1)-(Yn-Yn-1)}×ΔT / (Tn-Tn-1) n is the sampling number that defines the first check point. ΔT is the elapsed time from the first check point. Tn is the time of the first check point. ΔX is the correction amount for the X component of the estimated current position, and ΔY is the correction amount for the Y component of the estimated current position.

[0110] FIG. 6 is a flowchart showing the processing of the information processing system 1 according to the second embodiment.

[0111] Steps S11, S12, and S13 are the same as steps S1, S2, and S3 in FIG.

[0112] In step S14, the correction unit 227 corrects the estimated current position calculated in step S13 using the correction formula H. That is, if the estimated current position Sn is X and Y, the correction unit 227 corrects the estimated current position by X-ΔX and Y-ΔY.

[0113] Steps S15, S16, and S17 are the same as steps S4, S5, and S6 in FIG.

[0114] Next, in step S18, the arrival determination unit 223A determines whether the difference between the estimated current position Sn and the input current position Un is greater than a reference value. If the distance between the estimated current position Sn and the input current position Un is greater than the reference value, the arrival determination unit 223A may determine that the difference is greater than the reference value.

[0115] If the difference is equal to or greater than the reference value (YES in step S18), the process proceeds to step S20, and if the difference is less than the reference value (NO in step S18), the process proceeds to step S19.

[0116] Next, in step S19, the arrival determination unit 223A increases the predetermined distance and the predetermined time by a first predetermined value and a second predetermined value, respectively, thereby increasing the interval to the next first checkpoint and reducing the photographer's effort in inputting the current input position.

[0117] In step S20, the correction unit 227 corrects the correction formula H by using the input current position Un accepted in step S17 and the estimated current position Sn corresponding to the input current position Un. That is, the correction unit 227 corrects the correction formula H by substituting the input current position Un (Xn, Yn) and the estimated current position Sn (Xn', Yn') into the right-hand side of the correction formula H.

[0118] The processes in steps S21, S22, S23, and S24 are the same as those in steps S7, S8, S9, and S10 in FIG.

[0119] Thus, according to the second embodiment, if the difference between the estimated current position and the input current position is smaller than the reference value, the predetermined distance or the predetermined time is increased, thereby reducing the frequency of inputting the input current position and improving convenience for the photographer.

[0120] Furthermore, according to the second embodiment, if the difference between the estimated current position Sn and the input current position Un is greater than a reference value, the correction formula H is modified to reduce the drift, thereby enabling the estimated current position to be calculated more accurately.

[0121] The present disclosure can employ the following modifications.

[0122] (Variation 1) The arrival determination unit 223 may detect whether the space in which the photographer is located has changed from the first subspace to a second subspace different from the first subspace, based on the shooting location and a blueprint of the specified space. In this case, the arrival determination unit 223 plots the estimated current position on the blueprint in real time each time the estimated current position is calculated. The arrival determination unit 223 may detect the point at which the plotted estimated current position switches from the first subspace to the second subspace as the fourth checkpoint. For example, if the estimated current position Sn belongs to the first space and the estimated current position Sn+1 belongs to the second space, the arrival determination unit 223 may determine that the estimated current position has switched from the first subspace to the second subspace. When the arrival determination unit 223 detects the fourth checkpoint, it causes the input request presentation unit 224 to present an input request.

[0123] The first sub-space refers to a space within a predetermined space such as a room, a hallway, a passageway, an atrium, a staircase, etc. The second sub-space refers to a space within a predetermined space such as a room, a hallway, a passageway, an atrium, a staircase, etc., and is a sub-space adjacent to the first sub-space.

[0124] 1, the photography location calculation unit 112 is provided in the server 10, but the photography location calculation unit 112 may be provided in the processor 22 of the information terminal 20 instead of the server 10. In this case, the photography location calculation unit 112 may calculate the photography location of each image in real time using self-location estimation processing every time the photography device 30 captures an image. Furthermore, in this case, the photography device 30 may transmit the image to the information terminal 20 in real time every time the photography device 30 captures an image.

[0125] (Variation 3) The arrival determination unit 223 may implement the aspect of Variation 1 by using the photography point calculated in real time by the photography point calculation unit 112. That is, the arrival determination unit 223 may detect the switch from the first subspace to the second subspace by plotting the photography point calculated by the photography point calculation unit 112 on the design drawing in real time.

[0126] (Variation 4) The third checkpoint may be a location input by the user in advance. In this case, the user may set a location in a predetermined space where the estimated current position previously calculated has a large error as the third checkpoint. Furthermore, the user may set the number of checkpoints in advance based on the distance between the shooting start point and the shooting end point.

[0127] (Variation 5) When Variation 2 is employed, in which the imaging point calculation unit 112 calculates the imaging point in real time, the arrival determination unit 223 may detect a feature point of the photographer's movement trajectory as the fifth checkpoint. Then, when the arrival determination unit 223 detects the fifth checkpoint, it may cause the input request presentation unit 224 to present an input request to the photographer. The feature point of the movement trajectory may be, for example, a point where the photographer has turned, or a representative point of a curved section where the photographer is moving in a curved line. In a section where the photographer is moving in a straight line, the accuracy of the calculation of the imaging point by the self-location estimation process is high. However, when the photographer turns or moves in a curved line, the accuracy of the calculation of the imaging point by the self-location estimation process decreases. Therefore, by detecting the feature point as the fifth checkpoint, the accuracy of the calculation of the imaging point by the self-location estimation process can be improved.

[0128] For example, the arrival determination unit 223 determines the movement trajectory of the photographer by plotting the shooting points calculated in real time by the shooting point calculation unit 112 on a blueprint. Then, the arrival determination unit 223 may detect points where the direction of the movement trajectory changes as points where the photographer has turned. When the arrival determination unit 223 detects a curved section from the movement trajectory, it may detect points that divide the curved section at regular intervals as representative points of the curved section.

[0129] (Variation 6) A beacon radio may be installed at the third checkpoint. In this case, when the second communication unit 26 receives a wireless signal from the beacon radio, the estimated current position calculation unit 222 may calculate an estimated current position based on this wireless signal. The wireless signal from the beacon radio includes an identifier of the beacon radio. The coordinates of the installation position of the beacon radio are stored in a memory (not shown) of the information terminal 20 in association with the identifier of the beacon radio. Therefore, the estimated current position calculation unit 222 can identify the coordinates of the beacon radio that transmitted this wireless signal from the identifier included in this wireless signal. Alternatively, the estimated current position calculation unit 222 may identify the coordinates of the beacon radio from BIM information that stores the identifier of the beacon radio in association with the coordinates. Then, the estimated current position calculation unit 222 may calculate the coordinates of the beacon radio as the estimated current position. The arrival determination unit 223 may determine that the photographer has reached the third checkpoint if the coordinates of the estimated current position calculated by the estimated current position calculation unit 222 match the coordinates of the beacon radio.

[0130] (Variation 7) If the answer to step S18 in FIG. 6 is YES, i.e., if the difference between the estimated current position Sn and the input current position Un is equal to or greater than the reference value, the correction unit 227 corrects the correction formula H in step S20. In this case, the correction unit 227 may re-correct the estimated current position Sn corrected using the pre-correction correction formula H using the corrected correction formula H. Alternatively, the correction unit 227 may re-correct the estimated current position Sn corrected using the pre-correction correction formula H using the time gradient of the deviation between the pre-correction correction formula H and the corrected correction formula H. This makes it possible to bring the estimated current position Sn estimated when the deviation of the IMU sensor 25 has increased closer to a more appropriate value.

[0131] The present disclosure is useful in the technical field of remote monitoring of a site.

Claims

1. An information processing method in a computer, comprising: acquiring an estimated current position of a photographer who is photographing a specified space using an imaging device based on sensor information; determining whether or not the photographer has reached a checkpoint based on the estimated current position; each time it is determined that the photographer has reached the checkpoint, presenting the photographer with a request to input a current position; acquiring an input current position, which is the current position input by the photographer in response to the input request; and acquiring a shooting location for an image photographed after the checkpoint is reached based on a self-position estimation process using the input current position.

2. An information processing method as described in claim 1, wherein the checkpoints include a first checkpoint which is a point where the photographer has moved a predetermined distance from the nearest checkpoint, and a second checkpoint which is a point where the photographer has moved a predetermined time from the nearest checkpoint.

3. The information processing method according to claim 2, wherein determining whether or not the target has been reached includes increasing the specified distance or the specified time if the difference between the estimated current position at the first checkpoint and the input current position is smaller than a reference value.

4. The information processing method of claim 3, further comprising: correcting the estimated current position using a correction equation for suppressing drift contained in the estimated current position calculated from the sensor information of the IMU sensor; and if the difference is greater than the reference value, revising the correction equation based on the estimated current position and the input current position.

5. The information processing method according to claim 1 or 2, wherein the checkpoints include a third checkpoint which is a predetermined point in the specified space.

6. An information processing method according to claim 1 or 2, further comprising detecting whether or not the space in which the photographer is located has changed from a first subspace to a second subspace different from the first subspace based on the shooting location and the bird's-eye view of the specified space, and the checkpoints include a fourth checkpoint which is a point at which the change from the first subspace to the second subspace is detected.

7. The information processing method according to claim 5, wherein the third checkpoint is a location with poor radio wave conditions.

8. An information processing method as described in claim 1 or 2, wherein the checkpoint includes a fifth checkpoint, and the determining step further includes: calculating a movement trajectory of the photographer by plotting the shooting location on a bird's-eye view of the specified space in real time; and detecting a characteristic point of the movement trajectory as the fifth checkpoint, and the characteristic point includes a point where the photographer has turned.

9. An information processing method according to claim 1 or 2, wherein the sensor device that detects the sensor information is a communication device that is communicatively connected to a plurality of wireless communication devices installed in the specified space and carried by the photographer, and the estimated current location is calculated by triangulation using the received signal strengths of each of a plurality of wireless signals transmitted by the plurality of wireless communication devices and the known installation positions of each of the plurality of wireless communication devices.

10. An information processing method as claimed in claim 1 or 2, wherein the sensor device that detects the sensor information is a barcode reader carried by the photographer, the barcode reader reads a barcode installed at the checkpoint and indicating the position of the checkpoint, and the estimated current position is calculated based on the barcode read by the barcode reader.

11. An information processing device including a processor that executes the following operations: acquiring an estimated current position of a photographer during a photographing operation of photographing a specified space using an imaging device based on sensor information; determining whether the photographer has reached a checkpoint based on the estimated current position; presenting the photographer with a request to input a current position each time it is determined that the photographer has reached the checkpoint; acquiring an input current position, which is the current position input by the photographer in response to the input request; and acquiring the photographing location for images photographed after reaching the checkpoint based on a self-position estimation process using the input current position.

12. An information processing program that causes a computer to perform the following operations: calculate an estimated current position of a photographer during a photographing operation in which a specified space is photographed using an imaging device, based on sensor information; determine whether the photographer has reached a checkpoint based on the estimated current position; present the photographer with a request to input his / her current position each time it is determined that the photographer has reached the checkpoint; obtain an input current position, which is the current position input by the photographer in response to the input request; and obtain the photographing location for images photographed after the checkpoint is reached, based on self-position estimation processing using the input current position.