Information processing method, information processing device, and information processing program

Abstract

This information processing device generates an imaging trajectory including a plurality of floors from an imaging start point to an imaging end point from a plurality of bird's-eye views including a plurality of floors and a plurality of images captured at a plurality of imaging points including the imaging start point and the imaging end point, associates an imaging point belonging to a first imaging section, of which the height is constant, with a first bird's-eye view in the plurality of bird's-eye views on the basis of height information indicating the height of each of the plurality of imaging points, and associates an imaging point belonging to a second imaging section, of which the height is constant after having changed, with a second bird's-eye view different from the first bird's-eye view in the plurality of bird's-eye views.

Description

Information processing method, information processing device, and information processing program

[0001] The present disclosure relates to a technique for mapping a shooting trajectory.

[0002] Patent Literature 1 discloses a management system that associates movement information of a site worker obtained by a pedestrian autonomous navigation means with the detection results of site equipment and identification tags by a detection means and stores them in a database by time. This management system discloses that a site worker carries a pedestrian autonomous navigation means and a photographing and recording device and photographs multiple site equipment located within the construction site while walking and moving within the construction site, and that the site worker moves within each floor starting from a location near where an identification tag is installed on each floor. Height information indicating the horizontal position and vertical position (floor) within the construction site where the identification tag is installed is linked to the identification tag.

[0003] However, in the technology of Patent Document 1, in order to obtain movement information for each of a plurality of floors, it is necessary to place an identification tag at the position that is the movement starting point for each of a plurality of floors.

[0004] The present disclosure is intended to solve the above problem, and aims to provide a technology that can obtain a shooting trajectory for each of multiple floors without having to input a shooting start point for each of the multiple floors.

[0005] Japanese Patent Application Laid-Open No. 2022-92364

[0006] An information processing method in one aspect of the present disclosure is an information processing method on a computer, which includes generating a shooting trajectory including the multiple floors from a shooting start point to a shooting end point from a multiple bird's-eye view including the multiple floors and a multiple images taken at a multiple shooting point including the shooting start point and the shooting end point, and linking a shooting point belonging to a first shooting section in which the height is constant to a first bird's-eye view among the multiple bird's-eye views based on height information indicating the height of each of the multiple shooting points, and linking a shooting point belonging to a second shooting section in which the height is constant after the height changes to a second bird's-eye view among the multiple bird's-eye views that is different from the first bird's-eye view.

[0007] According to the present disclosure, a shooting trajectory for each of a plurality of floors can be obtained without inputting a shooting start point for each of the plurality of floors.

[0008] Fig. 4 is an overall configuration diagram of an information processing system according to embodiment 1. Fig. 5 is a flowchart showing the processing of the information processing system according to embodiment 1. Fig. 6 is a diagram showing an example of a design drawing screen. Fig. 7 is a flowchart showing the processing of the information processing system according to embodiment 2. Fig. 8 is a flowchart continuing from Fig. 4. Fig. 9 is a flowchart showing the processing of embodiment 3.

[0009] (Findings underlying the present disclosure) A user interface is being developed that displays multiple photography location icons indicating locations where actual photographs of a construction site were taken on a two-dimensional blueprint of the construction site, etc., and when one photography location icon is selected, displays an image taken at the photography location indicated by that icon. This allows a construction site manager to understand the situation at the construction site without visiting the construction site.

[0010] In such a user interface, the shooting locations are registered as follows: First, the photographer inputs the shooting start location on the blueprint screen, and takes images of the construction site with the camera while moving around the site. When the shooting is finished, the photographer inputs the shooting end location on the blueprint screen, and uploads the captured images to the server. The server performs a self-location estimation process using the uploaded images and the shooting start and end locations to calculate the shooting locations for each image, and plots the calculated shooting locations on the blueprint screen. This results in a shooting trajectory that connects multiple shooting locations.

[0011] The coordinate values ​​of the shooting location calculated using self-localization technology are two-dimensional and do not include height information. Therefore, to generate shooting trajectories for multiple floors, the photographer is required to input the shooting start point for each floor, which is a burden on the photographer.

[0012] Therefore, the inventor discovered that if height information indicating the height of each shooting point is obtained during shooting, it is possible to identify which floor each captured image was taken on, and thereby it is possible to generate shooting trajectories for multiple floors without having to input the shooting start point for each floor, which led to the present disclosure.

[0013] (1) An information processing method in one aspect of the present disclosure is an information processing method in a computer, which includes generating a shooting trajectory including the multiple floors from a shooting start point to a shooting end point from a multiple bird's-eye view including the multiple floors and a multiple images taken at a multiple shooting point including the shooting start point and the shooting end point, and linking a shooting point belonging to a first shooting section in which the height is constant to a first bird's-eye view among the multiple bird's-eye views based on height information indicating the heights of each of the multiple shooting points, and linking a shooting point belonging to a second shooting section in which the height is constant after the height changes to a second bird's-eye view among the multiple bird's-eye views that is different from the first bird's-eye view.

[0014] According to this configuration, a shooting location belonging to a first shooting section where the height is constant is linked to a first bird's-eye view among the multiple bird's-eye views, and a shooting location belonging to a second shooting section where the height indicated by the height information changes and the height remains constant is linked to a second bird's-eye view among the multiple bird's-eye views, which is different from the first bird's-eye view. This allows a shooting trajectory for each of multiple floors to be obtained without having to input a shooting start point for each of multiple floors. Note that in the present disclosure, a constant height also includes a state in which the height indicated by the height information falls within a predetermined range within which the height can be considered constant.

[0015] (2) In the information processing method described in (1) above, the altitude information may be information detected by an air pressure sensor or a pressure sensor.

[0016] In this case, accurate height information can be obtained by an air pressure sensor or a pressure sensor.

[0017] (3) In the information processing method described in (1) or (2) above, the shooting trajectory may be generated after all of the plurality of images have been acquired, and the linking may be performed after the process of generating the shooting trajectories for all of the plurality of images has been completed.

[0018] In this case, after the photographing operation is completed, photographing trajectories for each of a plurality of floors can be obtained all at once.

[0019] (4) In the information processing method described in (1) or (2) above, the shooting trajectory may be generated each time one of the multiple images is acquired, and the linking may be performed each time the shooting trajectory is generated.

[0020] In this case, the shooting trajectory can be obtained in real time during the shooting operation.

[0021] (5) In the information processing method described in any one of (1) to (4) above, the generating may include calculating the multiple shooting points by performing a self-position estimation process using the shooting start point and the multiple images, and generating the shooting trajectory from the calculated multiple shooting points.

[0022] In this case, since a plurality of shooting points are calculated using the self-position estimation technology, the shooting trajectory can be determined with high accuracy.

[0023] (6) The information processing method described in (4) above may further include acquiring coordinate values ​​of a marking point, which is a shooting point designated by the photographer during the shooting period, and the shooting trajectory after the marking point is acquired may be generated by performing a self-position estimation process using the marking point each time an image is acquired.

[0024] In this case, since the marking points are input during shooting, the subsequent shooting trajectory can be generated with high accuracy.

[0025] (7) In another aspect of the present disclosure, an information processing device includes a processor that executes the following operations: generating a shooting trajectory including the multiple floors from a shooting start point to a shooting end point from a multiple bird's-eye view including the multiple floors and a multiple images taken at a multiple shooting point including the shooting start point and the shooting end point; and linking a shooting point belonging to a first shooting section in which the height is constant to a first bird's-eye view among the multiple bird's-eye views based on height information indicating the height of each of the multiple shooting points; and linking a shooting point belonging to a second shooting section in which the height is constant after the height changes to a second bird's-eye view among the multiple bird's-eye views that is different from the first bird's-eye view.

[0026] In this case, an information processing device can be provided that can obtain a shooting trajectory for each of a plurality of floors without having to input a shooting start point for each of the plurality of floors.

[0027] (8) In yet another aspect of the present disclosure, an information processing program causes a computer to generate a shooting trajectory including the multiple floors from a shooting start point to a shooting end point from multiple bird's-eye views including multiple floors and multiple images taken at multiple shooting points including a shooting start point and a shooting end point, and link a shooting point belonging to a first shooting section in which the height is constant to a first bird's-eye view among the multiple bird's-eye views based on height information indicating the height of each of the multiple shooting points, and link a shooting point belonging to a second shooting section in which the height is constant after the height changes to a second bird's-eye view among the multiple bird's-eye views that is different from the first bird's-eye view.

[0028] In this case, an information processing program is provided that can obtain a photography trajectory for each of a plurality of floors without inputting a photography start point for each of the plurality of floors.

[0029] The present disclosure can also be realized as an information processing system operated by such an information processing program. Needless to say, such a computer program can be distributed on a non-transitory computer-readable recording medium such as a CD-ROM or via a communication network such as the Internet.

[0030] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all of the embodiments, the respective contents can be combined.

[0031] (First Embodiment) FIG. 1 is an overall configuration diagram of an information processing system 1 according to a first embodiment. The information processing system 1 is a system that calculates a photographing trajectory of a photographer's movement from multiple images of a given space captured by the photographer while moving across multiple floors, and associates the photographing trajectory with each of the floor plans. The information processing system 1 includes a server 10, an information terminal 20, a photographing device 30, a communication device 40, and an altitude sensor 50. The server 10 (an example of an information processing device and a computer), the information terminal 20, and the communication device 40 are connected to each other via a network NT so as to be able to communicate with each other. An example of a network is the Internet. The server 10 is, for example, a cloud server composed of one or more computers. However, this is merely an example, and the server 10 may be configured as an edge server or may be implemented in the information terminal 20. The implementation of the server 10 in the information terminal 20 is an example of the implementation of the information terminal 20 in an information processing device.

[0032] The information terminal 20 is owned by a user. The user is, for example, a manager of a predetermined space in which the image capturing device 30 captures images. The predetermined space is, for example, a construction site. However, this is just one example, and the predetermined space may be a construction site, a factory, a store, an office, etc. The information terminal 20 may be configured as a portable computer such as a smartphone or a tablet computer, or as a stationary computer. The information terminal 20 displays images on a display under the control of the server 10. Although one information terminal 20 is illustrated in the example of FIG. 1, multiple information terminals may be connected to the server 10 via a network.

[0033] The communication device 40 is configured, for example, by a portable information terminal such as a smartphone or a tablet computer, and connects the photographing device 30 and the altitude sensor 50 to the network NT. The communication device 40 and the photographing device 30 are connected via a short-range wireless communication path such as Bluetooth (registered trademark). The communication device 40 and the altitude sensor 50 are connected via a short-range wireless communication path such as Bluetooth (registered trademark). The communication device 40 is carried by the photographer.

[0034] The altitude sensor 50 is a sensor carried by the photographer for acquiring altitude information of the shooting location. The altitude sensor 50 is configured, for example, by an air pressure sensor or a variable pressure sensor. The altitude sensor 50 periodically detects altitude information indicating the altitude of the shooting location during the shooting period. The shooting period refers to the period from when the photographer starts shooting to when he or she finishes shooting.

[0035] In detail, the altitude sensor 50 includes a sensor unit, a processor, and a communication unit. The sensor unit is composed of an air pressure detection unit or a pressure transformer detection unit. The processor converts the air pressure value or pressure transformer value detected by the sensor unit into height relative to the ground to generate altitude information. The communication unit transmits the altitude information generated by the processor to the server 10 via the communication device 40.

[0036] 1, the altitude sensor 50 is connected to the network NT via the communication device 40, but this is just an example. For example, the altitude sensor 50 may be provided in the communication device 40.

[0037] The image capturing device 30 is, for example, an omnidirectional camera that captures images at a predetermined frame rate. An omnidirectional camera is also called a 360-degree camera, and is a camera that can capture images in all directions of 360 degrees. The image capturing device 30 is, for example, a portable image capturing device carried by a photographer. The photographer is, for example, a worker or site supervisor at a construction site. The image capturing device 30 may also be a regular camera.

[0038] The photographer moves within a predetermined space while photographing the construction site with the photographing device 30. When photographing is completed, the photographing device 30 transmits photographing information including the series of photographed images to the server 10 via the communication device 40. When photographing is completed, the altitude sensor 50 transmits height information detected during the photographing period to the server 10 via the communication device 40.

[0039] The photography information includes the photographed image, the photography start point, the photography end point, and a blueprint ID corresponding to the predetermined space. The blueprint ID is an identifier for identifying the blueprint of the predetermined space.

[0040] At the shooting start position, the photographer points the shooting direction of the camera 30 northward, presses the shooting start button of the camera 30, starts shooting, and when the shooting end point is reached, presses the shooting button again to end shooting. In particular, in this embodiment, the photographer performs shooting operations across multiple floors.

[0041] The start and end points of photography are specified by the photographer inputting instructions specifying the positions on a blueprint screen of a predetermined space displayed on the display of the communication device 40. Two-dimensional coordinate axes are defined on the blueprint screen. Therefore, the start and end points of photography are each defined by two-dimensional coordinate values. The start and end points of photography are used by the server 10 to specify the location of each photography point and the photography direction of each image.

[0042] The server 10 includes a processor 11, a memory 12, and a communication unit 13. The processor 11 is configured, for example, by a central processing unit (CPU). The processor 11 includes an acquisition unit 111, a trajectory generation unit 112, and an allocation unit 113. The acquisition unit 111 to the allocation unit 113 may be realized by the processor 11 executing an information processing program, or may be configured by a dedicated hardware circuit such as an ASIC.

[0043] The acquisition unit 111 acquires, using the communication unit 13, the imaging information transmitted from the imaging device 30 via the communication device 40 and the height information transmitted from the altitude sensor 50 via the communication device 40. The acquired imaging information is stored in the imaging information storage unit 122. The acquired height information is stored in the height information storage unit 123.

[0044] The acquisition unit 111 acquires design drawing information corresponding to the photographing information from the design drawing information storage unit 121. The acquisition unit 111 may acquire the design drawing information corresponding to the photographing information using the design drawing ID included in the photographing information as a key. In this way, design drawing information of the specified space photographed by the photographer can be obtained.

[0045] The trajectory generation unit 112 generates a shooting trajectory including multiple floors from the shooting start point to the shooting end point from multiple design drawings (an example of a bird's-eye view) including multiple floors and multiple images taken at multiple shooting points including the shooting start point and the shooting end point.

[0046] The trajectory generation unit 112 performs a self-location estimation process using the shooting start point, shooting end point, and multiple images included in the shooting information to calculate each shooting point. Visual simultaneous localization and mapping (VSLAM) can be used as the self-location estimation process.

[0047] The shooting trajectory is a trajectory connecting multiple shooting locations. The shooting trajectory indicates the movement route of the photographer. The position of each shooting location is represented by two-dimensional coordinate values ​​on the design drawing. The shooting direction of each image is also identified using a self-position estimation process. The shooting direction of each image may be represented by, for example, a three-dimensional polar coordinate vector. Alternatively, the shooting direction of each image may be represented by a two-dimensional polar coordinate vector represented in a two-dimensional polar coordinate space parallel to the horizontal direction.

[0048] The trajectory generating unit 112 generates a shooting trajectory after all of the multiple images shot during the shooting period have been acquired.

[0049] The trajectory generation unit 112 detects feature points from each of the multiple images included in the shooting information through image processing. Feature points are characteristic points that indicate the outline of an object included in the image. The trajectory generation unit 112 performs a self-position estimation process on each of the multiple images using at least two of the multiple images in which feature points are set, and calculates the relative coordinate values ​​of the shooting point, which is the self-position corresponding to each image. The trajectory generation unit 112 aligns the direction of the shooting point corresponding to each image by utilizing the fact that the shooting device 30 is set facing north at the start of the shooting operation. The trajectory generation unit 112 calculates the absolute coordinate values ​​of the shooting point corresponding to each image from the relative coordinate values ​​of the shooting point corresponding to each image, based on the coordinate values ​​of the shooting start point and shooting end point included in the shooting information, and calculates the calculated absolute coordinate values ​​as the coordinate values ​​of the shooting point corresponding to each image. The trajectory generation unit 112 generates a trajectory connecting this series of shooting points as a shooting trajectory.

[0050] The allocating unit 113 links a shooting location belonging to a first shooting section, where the height is constant, to a first bird's-eye view among the plurality of bird's-eye views, based on height information indicating the heights of the respective shooting locations. Furthermore, the allocating unit 113 links a shooting location belonging to a second shooting section, where the height indicated by the height information is constant after the height has changed, to a second bird's-eye view among the plurality of bird's-eye views, which is different from the first bird's-eye view.

[0051] The first shooting section refers to the section photographed by the photographer on the first floor, and the second shooting section refers to the section photographed by the photographer on the second floor. The first bird's-eye view is a blueprint of the first floor, and the second bird's-eye view is a blueprint of the second floor. Note that the bird's-eye view may be an overhead image of a specified space other than the blueprint, or may be an overhead image of the specified space using computer graphics. The first and second floors are not intended to be limited to two floors, but rather the specified space may have multiple floors. In other words, the present disclosure is applicable even when the specified space has three or more floors. The first floor may be the floor below the second floor or the floor above the second floor. This allows the shooting trajectory to be linked to each floor. The blueprint is a diagram showing the design of the construction site, and may be a floor plan, blueprint, map, perspective view, etc. of the construction site. The bird's-eye view can also be referred to as a bird's-eye view, and may be a diagram viewed from above or from a high vantage point.

[0052] After the trajectory generating unit 112 has completed the process of generating the photographing trajectories for all of the multiple images, the allocating unit 113 executes a process of allocating the photographing points to the respective floors.

[0053] The memory 12 is configured as a non-volatile rewritable storage device such as a hard disk drive or a solid state drive. The memory 12 includes a blueprint information storage unit 121, a photography information storage unit 122, and a height information storage unit 123. The blueprint information storage unit 121 stores blueprint information. The blueprint information is image information showing a blueprint of a specified space. The blueprint information is associated with a blueprint ID that identifies the blueprint.

[0054] The photography information storage unit 122 stores photography information transmitted from the photography device 30. The photography information includes meta information for each image, in addition to the multiple images, photography start point, photography end point, and blueprint ID included in the photography information transmitted from the photography device 30. The photography information is generated each time the photography information described above is transmitted. The meta information includes the photography ID, photography period ID, photography date and time, photography direction, and photography location.

[0055] The shooting ID is an identifier of the shooting location. The shooting period ID is an identifier of the shooting period. The shooting date and time is the date and time when the image was shot. The shooting direction is the shooting direction of each image by the shooting device 30. The shooting location is composed of a position (two-dimensional coordinate value) that indicates the shooting location of the image.

[0056] The height information storage unit 123 stores the height information transmitted from the height sensor 50 via the communication device 40. The height information is linked to a blueprint ID and a detection date and time. This allows the trajectory generation unit 112 to identify the height information at the shooting location by comparing the blueprint ID and detection date and time linked to the height information with the blueprint ID and shooting date and time linked to the image.

[0057] The communication unit 13 is a communication interface that connects the server 10 to a network.

[0058] 2 is a flowchart showing the processing of the information processing system 1 according to the first embodiment. In this flowchart, the predetermined space includes a first floor and a second floor. Therefore, the design drawing of this predetermined space is composed of a design drawing of the first floor and a design drawing of the second floor.

[0059] In step S1, the operation unit of the communication device 40 accepts an operation to input a shooting start point. For example, the photographer inputs an operation to specify the shooting start point on a blueprint screen displayed on the display of the communication device 40. The photographer may specify the shooting start point by tapping on the blueprint screen, or may specify the coordinate values ​​of the shooting start point.

[0060] Next, in step S2, the camera device 30 starts taking multiple images from multiple shooting locations, and the altitude sensor 50 starts processing to periodically detect altitude information. On the first floor, the photographer points the camera device 30 in the shooting direction north and inputs a shooting start command to the camera device 30 to start shooting in the specified space.

[0061] The photographer first takes a photograph of the first floor. After finishing photographing the first floor, the photographer moves to the second floor using an ascending or descending means such as stairs or an elevator without ending the photographing, and reaches the photographing end point. At this time, the photographer inputs a photographing end instruction into the photographing device 30, and ends the photographing operation.

[0062] Next, in step S3, the operation unit of the communication device 40 accepts an operation to input the shooting end point. For example, the photographer inputs an operation to specify the shooting end point on the blueprint screen displayed on the display of the communication device 40. The photographer may specify the shooting end point by tapping on the blueprint screen, or may specify the shooting end point by inputting the coordinate values ​​of the shooting end point.

[0063] Next, in step S4, the communication device 40 acquires from the photographing device 30 photographing information including multiple images photographed by the photographing device 30 during the photographing period, and acquires from the altitude sensor 50 height information detected periodically by the altitude sensor 50 during the photographing period, and transmits the acquired photographing information and acquired height information to the server 10.

[0064] Next, in step S5, the acquisition unit 111 acquires the shooting information and height information transmitted from the communication device 40 using the communication unit 13. The acquired shooting information is provided with meta information and stored in the shooting information storage unit 122, and the acquired height information is stored in the height information storage unit 123.

[0065] Next, in step S6, the trajectory generation unit 112 calculates the shooting points of each of the multiple images included in the shooting information acquired in step S5 using a self-position estimation process, and generates a shooting trajectory from the calculated multiple shooting points.

[0066] Next, in step S7, the trajectory generating unit 112 associates the multiple image capturing points generated in step S6 with the altitude information corresponding to each image capturing point.

[0067] Next, in step S8, the allocating unit 113 extracts one photography location (interesting photography location) from the multiple photography locations generated in step S6 in order of photography date and time, and determines whether the height indicated by the height information linked to the interest photography location falls within a first height range H1. The first height range H1 is defined as a range equal to or greater than a lower limit value H11 and equal to or less than an upper limit value H12. The lower limit value H11 can be the height of the floor surface of the first floor from the ground. The upper limit value H12 can be the height from the ground obtained by adding a predetermined margin to the height of the altitude sensor 50 carried by the photographer standing on the first floor. Predetermined values ​​are used for the lower limit value H11 and the upper limit value H12.

[0068] In addition, the allocation unit 113 may determine that the target shooting location is on the first floor if the difference in height between the previous target shooting location and the current target shooting location is less than a predetermined value, and may determine that the target shooting location is not on the first floor if the difference is greater than or equal to the predetermined value.

[0069] If the height of the target photographing point belongs to the first height range H1 (YES in step S8), processing proceeds to step S9; if the height of the target photographing point does not belong to the first height range H1 (NO in step S8), processing proceeds to step S10.

[0070] Next, in step S9, the allocating unit 113 allocates the interesting photographing point to the first floor.

[0071] Next, in step S10, the allocating unit 113 determines whether the height of the target photography point belongs to a second height range H2. The second height range H2 is defined as a range equal to or greater than a lower limit H21 and equal to or less than an upper limit H22. The lower limit H21 can be the height of the second floor from the ground. The upper limit H22 can be the height from the ground obtained by adding a predetermined margin to the height of the altitude sensor 50 carried by the photographer standing on the second floor. Predetermined values ​​are used for the lower limit H21 and the upper limit H22.

[0072] If the height of the target shooting point belongs to the second height range H2 (YES in step S10), processing proceeds to step S11; if the height of the target shooting point does not belong to the second height range H2 (NO in step S10), processing proceeds to step S12.

[0073] Next, in step S11, the allocating unit 113 allocates the interesting photographing point to the second floor.

[0074] Next, in step S12, the allocating unit 113 allocates the image capturing points of interest that do not belong to either the first floor or the second floor to the stairs or the elevator. The allocating unit 113 may treat the image capturing the stairs or the elevator as an image of a moving part. Furthermore, the allocating unit 113 may treat the image capturing the stairs or the elevator as an image of the construction process of the stairs or the elevator.

[0075] Next, in step S13, the allocating unit 113 determines whether the allocating is complete. For example, if the allocating process using the shooting end point as the shooting point of interest has been completed, the allocating unit 113 determines YES in step S13 and ends the process. On the other hand, if the allocating process using the final end point as the shooting point of interest has not been completed, the allocating unit 113 determines NO in step S13 and returns the process to step S8.

[0076] Next, in step S14, the allocating unit 113 assigns a floor identifier indicating that the photographing locations assigned to the first floor are the first floor and stores them in the photographing information storage unit 122, and assigns a floor identifier indicating that the photographing locations assigned to the second floor are the second floor and stores them in the photographing information storage unit 122. The allocating unit 113 may assign an identifier indicating that the photographing locations are the stairs or elevator to the photographing locations assigned to the stairs or elevator and store them in the photographing information storage unit 122.

[0077] 2, the predetermined space includes two floors, but may include three or more floors. In this case, the flowchart only needs to include n processes for determining whether the target photographing point belongs to the nth height range (n is an integer equal to or greater than 1).

[0078] 3 is a diagram showing an example of a design drawing screen, in which design drawing screens 301, 302, and 303 for the first, second, and third floors are shown.

[0079] The blueprint screen 301 includes a shooting trajectory 311. The shooting trajectory 311 is made up of lines connecting the shooting locations assigned to the first floor. The blueprint screen 302 includes a shooting trajectory 312. The shooting trajectory 312 is made up of lines connecting the shooting locations assigned to the second floor. The blueprint screen 303 includes a shooting trajectory 313. The shooting trajectory 313 is made up of lines connecting the shooting locations assigned to the third floor. Note that the shooting trajectory 311 may include a shooting location icon indicating the shooting location.

[0080] When a user of the information terminal 20 inputs an instruction to display a design drawing screen using the operation unit of the information terminal 20, the processor 11 displays design drawing screens 301 to 303 on the display of the information terminal 20. As a result, the design drawing screens 301 to 303 shown in FIG. 3 are displayed on the display of the information terminal 20. Alternatively, when the user inputs an instruction to specify a floor in accordance with the instruction to display the design drawing using the operation unit of the information terminal 20, the processor 11 may display the design drawing screen of the specified floor on the display of the information terminal 20. As a result, the design drawing screen of the specified floor is displayed on the display of the information terminal 20.

[0081] When the user inputs an instruction to select a photography location icon using the operation unit of the information terminal 20, the processor 11 displays an image corresponding to the photography location icon on the display of the information terminal 20. This allows the user to check the situation around the selected photography location.

[0082] In this way, according to this embodiment, the shooting location belonging to the first shooting section, which has a constant height, is linked to the blueprint of the first floor among the multiple blueprints, and the shooting location belonging to the second shooting section, which has a constant height after the height indicated by the height information changes, is linked to the blueprint of the second floor. This makes it possible to obtain shooting trajectories for multiple floors without having to input the shooting start point for each floor.

[0083] (Embodiment 2) In embodiment 2, the allocation of image capture locations to floors is performed in real time. In this embodiment, the same components as in embodiment 1 are given the same reference numerals, and their explanations will be omitted. In this embodiment, the block diagram in FIG. 1 is used.

[0084] 1, each time an image is captured, the image capturing device 30 transmits image capturing information including the image to the server 10 via the communication device 40.

[0085] The trajectory generating unit 112 generates a shooting trajectory every time one of the multiple images is acquired by the acquiring unit 111 .

[0086] The trajectory generation unit 112 detects feature points of the images in the same manner as in the first embodiment. The trajectory generation unit 112 performs a self-position estimation process using at least two images, including the currently acquired image and one or more previously acquired images, each time multiple images are acquired, and calculates the relative coordinate values ​​of the shooting location for each image. The trajectory generation unit 112 aligns the direction of the shooting location by utilizing the fact that the camera device 30 was set to face north at the start of shooting. The trajectory generation unit 112 calculates absolute coordinate values ​​of the self-position for each image using multiple marking points corresponding to multiple shooting locations specified by the photographer on the blueprint screen at the start of the shooting operation, and calculates the calculated absolute coordinate values ​​as the coordinate values ​​of the shooting location for each image. Each time the trajectory generation unit 112 calculates the shooting location for each image, it displays a shooting icon indicating the shooting location on the blueprint displayed on the display of the information terminal 20, and displays a line connecting the shooting location icons calculated so far as a shooting trajectory.

[0087] The allocating unit 113 executes a process of allocating each shooting point to a floor after the process of generating a shooting trajectory is executed by the trajectory generating unit 112. In this way, the shooting trajectory is linked to each floor.

[0088] 4 is a flowchart showing the processing of the information processing system 1 in embodiment 2. In step S21, the operation unit of the communication device 40 accepts an operation to input marking points corresponding to multiple shooting points. For example, the photographer inputs an operation to specify multiple shooting points on a blueprint screen displayed on the display of the communication device 40. The photographer may specify multiple shooting points by tapping on the blueprint screen, or may specify coordinate values ​​of multiple shooting points. The marking points may include the shooting start point.

[0089] Step S22 is the same process as step S2.

[0090] Next, in step S23, the communication device 40 acquires, from the imaging device 30, imaging information including the multiple images captured while the multiple marking points were being designated by the imaging device 30, and also acquires from the altitude sensor 50 altitude information that has not been transmitted during the period from the start of imaging to the present, and transmits the acquired imaging information and altitude information to the server 10. This imaging information includes the coordinate values ​​of the multiple marking points, the imaging date and time for each of the multiple images, the blueprint ID, etc.

[0091] Step S24 is the same process as step S5.

[0092] Next, in step S25, the trajectory generation unit 112 calculates the shooting locations of each of the multiple images included in the shooting information acquired in step S24 using a self-position estimation process, thereby generating a shooting trajectory from the start of shooting to the shooting location corresponding to the latest image.

[0093] In detail, the trajectory generation unit 112 calculates the relative coordinate values ​​of the shooting points from the multiple images acquired in step S24. The trajectory generation unit 112 aligns the direction of the shooting points corresponding to each image by utilizing the fact that the shooting device is set facing north at the shooting start point. The trajectory generation unit 112 calculates the absolute coordinate values ​​of the shooting points corresponding to each image from the relative coordinate values ​​of the shooting points corresponding to each image by utilizing the coordinate values ​​of the marking points. The trajectory generation unit 112 generates a line connecting the calculated multiple shooting points as a shooting trajectory.

[0094] The processes of steps S26, S27, S28, S29, S30, and S31 are the same as those of steps S7, S8, S9, S10, S11, and S12.

[0095] Step S32 is the same process as step S14, except that the processing targets are not all the photographing points during the photographing period, but a plurality of photographing points including the marking point generated in step S25.

[0096] Fig. 5 is a continuation of the flowchart in Fig. 4. In step S33, the communication device 40 acquires, from the image capturing device 30, image capturing information including the most recent image captured by the image capturing device 30, and acquires, from the altitude sensor 50, altitude information detected by the altitude sensor 50 in the frame period corresponding to this image, and transmits the acquired image capturing information and altitude information to the server 10. This image capturing information includes the date and time of the most recent image, the blueprint ID, etc.

[0097] Step S34 is the same process as step S24.

[0098] Step S35 is the same as step S25 except that the processing target is the most recent single image.

[0099] Steps S36, S37, S38, S39, S40, and S41 are the same processes as steps S26, S27, S28, S29, S30, and S31. That is, the process of allocating the latest photographing points to each floor is executed.

[0100] Step S42 is the same process as step S14, except that the latest photographing location generated in step S35 is the processing target, rather than all photographing locations during the photographing period.

[0101] In step S43, the allocating unit 113 determines whether the allocating is complete. For example, if the allocating process using the shooting end point as the target shooting point has been completed, the allocating unit 113 determines YES in step S43 and ends the process. On the other hand, if the allocating process using the shooting end point as the target shooting point has not been completed, the allocating unit 113 determines NO in step S43 and returns the process to step S33.

[0102] In this way, according to this embodiment, a shooting trajectory can be obtained in real time during shooting operation.

[0103] (Embodiment 3) In embodiment 3, a shooting trajectory is generated using marking points acquired during shooting in embodiment 2. In this embodiment, the same components as in embodiments 1 and 2 are given the same reference numerals, and description thereof will be omitted. In this embodiment, the block diagram in FIG. 1 is used.

[0104] 6 is a flowchart showing the process of embodiment 3. First, in step S51, the information processing system 1 executes process A. Process A is the process shown in steps S21 to S32 in FIG.

[0105] Next, in step S52, the information processing system 1 executes process A', which is the process shown in steps S33 to S42 in FIG.

[0106] Next, in step S53, the trajectory generation unit 112 determines whether or not the calibration timing has arrived. The calibration timing may be, for example, a fixed cycle, or may be the timing when a predetermined number of shooting points are calculated, or the timing when the deviation of the shooting trajectory (coordinates) exceeds a predetermined value. If the calibration timing has arrived (YES in step S53), the process proceeds to step S54. If the calibration timing has not arrived (NO in step S53), the process proceeds to step S55.

[0107] Next, in step S54, the information processing system 1 executes process A. In this case, in the process of step S21, one or more shooting locations where the photographer photographed a predetermined space during the shooting period are designated as marking locations, rather than multiple shooting locations including the shooting start location. Furthermore, in step S25, absolute coordinate values ​​of the shooting locations are generated from the relative coordinate values ​​of the shooting locations using the one or more marking locations designated in step S21. After acquiring a marking location, the trajectory generation unit 112 calculates the shooting location by performing a self-location estimation process using the marking location each time an image is acquired. Furthermore, when the next marking location is acquired, the trajectory generation unit 112 calculates the shooting location by performing a self-location estimation process using the next marking location each time an image is acquired.

[0108] As described above, according to this embodiment, the marking point of the photography location is acquired during the photography period, and therefore, when the photography location is calculated in real time, the calculation accuracy of the photography location is further improved.

[0109] The present disclosure is useful in the technical field of managing the progress of a site from a remote location.

Claims

1. An information processing method on a computer, comprising: generating a shooting trajectory including the multiple floors from a shooting start point to a shooting end point from a shooting start point and a shooting end point from a multiple bird's-eye view including the multiple floors and a multiple images taken at a multiple shooting point including a shooting start point and a shooting end point; and linking a shooting point belonging to a first shooting section in which the height is constant to a first bird's-eye view among the multiple bird's-eye views based on height information indicating the height of each of the multiple shooting points, and linking a shooting point belonging to a second shooting section in which the height is constant after the height changes to a second bird's-eye view different from the first bird's-eye view among the multiple bird's-eye views.

2. The information processing method according to claim 1, wherein the altitude information is information detected by an air pressure sensor or a pressure sensor.

3. The information processing method according to claim 1, wherein the shooting trajectory is generated after all of the plurality of images have been acquired, and the linking is performed after the shooting trajectory generation process for all of the plurality of images has been completed.

4. The information processing method according to claim 1, wherein the shooting trajectory is generated each time one of the plurality of images is captured, and the linking is performed each time the shooting trajectory is generated.

5. An information processing method according to claim 1 or 2, wherein the generating step includes calculating the multiple shooting points by performing a self-position estimation process using the shooting start point and the multiple images, and generating the shooting trajectory from the calculated multiple shooting points.

6. An information processing method as described in claim 4, further comprising acquiring coordinate values ​​of a marking point, which is a shooting point designated by the photographer during a shooting period, and a shooting trajectory after the marking point is acquired is generated by performing a self-position estimation process using the marking point each time an image is acquired.

7. An information processing device including a processor that executes the following operations: generating a shooting trajectory including the multiple floors from a shooting start point to a shooting end point from a shooting start point and a shooting end point from a multiple bird's-eye view including a multiple floors and a multiple images taken at a multiple shooting point including a shooting start point and a shooting end point; and linking a shooting point belonging to a first shooting section in which the height is constant to a first bird's-eye view among the multiple bird's-eye views based on height information indicating the heights of each of the multiple shooting points, and linking a shooting point belonging to a second shooting section in which the height is constant after the height changes to a second bird's-eye view different from the first bird's-eye view among the multiple bird's-eye views.

8. An information processing program that causes a computer to execute the following steps: generate a shooting trajectory including the multiple floors from a shooting start point to a shooting end point from a shooting start point and a shooting end point from a shooting start point and a shooting end point, based on height information indicating the heights of each of the multiple shooting points; link a shooting point belonging to a first shooting section in which the height is constant to a first bird's-eye view among the multiple bird's-eye views, and link a shooting point belonging to a second shooting section in which the height is constant after the height changes to a second bird's-eye view among the multiple bird's-eye views that is different from the first bird's-eye view.

Images