Information processing method, information processing program and information processing device

Abstract

To provide an information processing method, an information processing program and an information processing device capable of utilizing both three-dimensional data and an image of a measurement object of the three-dimensional data as records of construction site.SOLUTION: An information processing method includes a generation step for executing three-dimensional scanning of a measurement object to generate three-dimensional data for specifying a three-dimensional shape of the measurement object, an imaging step for generating a captured image of the measurement object at a resolution higher than a resolution of the three-dimensional data while executing the three-dimensional scanning, and a storage step for associating at least a portion of the captured image with the three-dimensional data to store them.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing program, and an information processing device. [Background technology]

[0002] As described in Patent Document 1, a system is known that can detect instantaneous deformation of the tunnel excavation surface in real time with high accuracy and that is equipped with an alarm function. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-91582 Summary of the Invention [Problem to be solved by the invention]

[0004] The system of Patent Document 1 does not support the use of recording construction sites. It is required to utilize both 3D data and images of the object to be measured for the 3D data for recording construction sites.

[0005] An object of the present disclosure is to provide an information processing method, an information processing program, and an information processing device that can utilize both three-dimensional data and images of objects to be measured using the three-dimensional data as records of a construction site. [Means for solving the problem]

[0006] (1) An information processing method according to one embodiment of the present disclosure includes a generation step of performing a three-dimensional scan of a measurement object to generate three-dimensional data that identifies the three-dimensional shape of the measurement object, a photographing step of generating a photographed image of the measurement object at a resolution higher than the resolution of the three-dimensional data while the three-dimensional scan is being performed, and a storage step of storing at least a portion of the photographed image in association with the three-dimensional data.

[0007] (2) In the storing step of the information processing method described in (1) above, an input specifying an image to be linked to the three-dimensional data from among the captured images may be received.

[0008] (3) In the storing step of the information processing method described in (2) above, an input may be received to set an identifier that identifies at least one position of the three-dimensional data to the three-dimensional data, and the captured image may be linked to the three-dimensional data by linking it to the identifier.

[0009] (4) In the storing step of the information processing method described in (3) above, an identifier setting interface for receiving input for setting the identifier may be displayed.

[0010] (5) In the photographing step of the information processing method described in any one of (1) to (4) above, a photographing interface that accepts photographing operation input may be displayed on a screen that displays the object to be measured while the 3D scan is being performed, and the photographed image may be generated in accordance with the operation input to the photographing interface.

[0011] (6) In the information processing method described in (5) above, in the photographing step, an identifier for identifying the photographing position of the photographed image may be set at a position where the three-dimensional data was generated when the photographed image was generated. In the storing step, the photographed image may be linked to the identifier.

[0012] (7) In the storage step of the information processing method described in any one of (1) to (6) above, the captured image may be linked to the three-dimensional data by being stored in a folder corresponding to the three-dimensional data.

[0013] (8) In the storing step of the information processing method described in (3), (4), or (6) above, information about the captured image may be linked to the identifier.

[0014] (9) In the storing step of the information processing method according to any one of (1) to (8) above, images of the captured images that are not to be linked to the three-dimensional data may be discarded.

[0015] (10) In the storing step of the information processing method described in (9) above, after the three-dimensional scanning is completed, images to be discarded may be determined from among the captured images.

[0016] (11) In the storage step of the information processing method described in (9) or (10) above, among the captured images, images that have not been linked within a predetermined time after capture or images that have not been designated as images to be linked to the three-dimensional data may be determined to be images to be discarded.

[0017] (12) In the information processing method according to any one of (1) to (11) above, the captured image may be corrected in the capturing step or the saving step.

[0018] (13) An information processing program according to an embodiment of the present disclosure causes a processor to execute the information processing method according to any one of (1) to (12) above.

[0019] (14) An information processing device according to an embodiment of the present disclosure executes the information processing method according to any one of (1) to (12) above. [Effects of the Invention]

[0020] According to an information processing method, an information processing program, and an information processing device according to an embodiment of the present disclosure, both three-dimensional data and an image of an object to be measured in three-dimensional data are utilized as a record of a construction site. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a block diagram illustrating a configuration example of an information processing system according to the present disclosure. [Figure 2]1 is a flowchart illustrating an example of a procedure of an information processing method according to the present disclosure. [Figure 3] FIG. 10 is a diagram showing an example of a screen displayed during execution of a three-dimensional scan. [Figure 4] FIG. 10 is a diagram showing an example of a screen displaying three-dimensional data. [Figure 5] FIG. 10 is a diagram showing an example of a screen displaying a captured image. DETAILED DESCRIPTION OF THE INVENTION

[0022] As a construction record, three-dimensional data specifying the three-dimensional shape of a measured object at a construction site may be recorded. Also, as a construction record, photographed images of the measured object may be recorded. The measured object may include an object present at the construction site or the topography of the construction site. The measured object may include, for example, materials such as pipes present at the construction site. The measured object may include, for example, topography such as holes or trenches provided at the construction site.

[0023] 3D data includes point cloud data and depth data. The resolution of the 3D data of the object being measured is often lower than the resolution of the image of the object being measured. Although 3D data is useful for analyzing dimensions or shape, it is not suitable for use by humans to intuitively grasp the situation at a construction site.

[0024] In order to make the 3D data more similar to the human eye, texture mapping is sometimes performed on the 3D data to generate data in which the points of the point cloud data or depth data are interpolated with images. Although texture-mapped images are closer to the actual appearance of a construction site than point cloud data or depth data, they often do not have sufficient resolution for humans to visually judge the condition of the construction site.

[0025] On the other hand, although images of measurement objects at construction sites are useful for humans to intuitively understand the situation at the construction site, they can sometimes make it difficult to accurately grasp, for example, the dimensions or shape of the measurement object. In other words, even if images of measurement objects are useful for humans to visually judge the situation at the construction site, they may not provide sufficient information as a record of the construction work.

[0026] It is possible to record both the 3D data of the object being measured and the photographed image of the object being measured, but simply recording the data makes it unclear how the 3D data relates to the photographed image, making it difficult to use as a record of the construction work.

[0027] Hereinafter, an information processing system 1 and an information processing device 10 (see FIG. 1), as well as an information processing method and an information processing program, that can utilize both three-dimensional data and images of measurement objects at a construction site as records of the construction site will be described.

[0028] (Configuration example of information processing system 1) As shown in FIG. 1, an information processing system 1 according to an embodiment of the present disclosure includes an information processing device 10, a measurement device 20, an imaging device 30, and a database 40. The information processing device 10, the measurement device 20, the imaging device 30, and the database 40 may be communicatively connected to each other, or may be communicatively connected via a network. The network may be wired or wireless. The network may be configured in various ways, such as the Internet or an intranet. A specific configuration example of the information processing system 1 will be described below.

[0029] <Information processing device 10> The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, and a display unit 15.

[0030] The control unit 11 controls each component of the information processing device 10 to realize the functions of the information processing device 10. The control unit 11 may realize the functions of the information processing device 10 by executing a program. The control unit 11 may be configured to include one or more processors. The control unit 11 may include, but is not limited to, a general-purpose processor or a dedicated processor specialized for specific processing. The control unit 11 may be configured to include one or more dedicated circuits. The dedicated circuits may include, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 11 may be replaced by a dedicated circuit, or may be replaced by a configuration that uses a processor and a dedicated circuit in combination.

[0031] The memory unit 12 stores various information or programs for controlling each component of the information processing device 10 to realize the functions of the information processing device 10. The memory unit 12 may store system programs, application programs, etc. The memory unit 12 may be configured to include, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. The memory unit 12 may be configured to include an electromagnetic storage medium such as a magnetic disk. The memory unit 12 may be configured to include a non-transitory computer-readable medium. The memory unit 12 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The memory unit 12 may function as a work memory for the control unit 11. The memory unit 12 may be configured integrally with the control unit 11 or may be configured separately from the control unit 11.

[0032] The communication unit 13 may include a communication interface. The communication interface may include a communication module configured to be able to communicate with other devices such as the measurement device 20, the imaging device 30, or the database 40. The communication module may be configured to be connectable to a network, or may be configured to be able to connect directly to other devices. The communication module may be compatible with mobile communication standards such as 4G (4th Generation) or 5G (5th Generation). The communication module may be compatible with communication standards such as LAN (Local Area Network). The communication module may be compatible with wired or wireless communication standards. The communication module is not limited to these and may be compatible with various communication standards.

[0033] The input unit 14 may include an input device. The input device may include, for example, a touch sensor or a pointing device such as a mouse. The input device may be configured as a touch display integrated with a display device described below. The input device may include physical keys. The input device may include an audio input device such as a microphone. The input unit 14 may be configured to be connectable to an external input device.

[0034] The display unit 15 may include a display device. The display device may include various displays, such as a liquid crystal display, an organic EL (Electro-Luminescence) or inorganic EL display, or an LED (Light Emitting Diode) display. The display unit 15 may be configured to be connectable to an external display device.

[0035] The information processing device 10 may include an output device. The output device may include, for example, a speaker, headphones, earphones, etc. The information processing device 10 may include an interface that can be connected to an external output device.

[0036] The information processing device 10 may be realized as, for example, a computer such as a notebook PC (Personal Computer) or a tablet PC, or a mobile terminal such as a smartphone or a tablet. The information processing device 10 is not limited to these examples and may include various devices.

[0037] The information processing device 10 may be realized using a cloud service or in an on-premise environment. The information processing device 10 may include a part that executes information processing and a part that provides a user interface, which are separate pieces of hardware.

[0038] <Measuring Device 20 and Imaging Device 30> The measuring device 20 acquires three-dimensional data of the measurement object by measuring the measurement object. The measuring device 20 may be configured to include, for example, a LiDAR (Light Detection And Ranging) or a depth camera. The measuring device 20 may also be configured to include a camera that captures an image of the measurement object.

[0039] The image capturing device 30 captures an image of the measurement object to generate a captured image. The captured image is an image of the measurement object. The image capturing device 30 may include, for example, a camera that captures RGB images. The resolution of the captured image is higher than the resolution of the three-dimensional data measured by the measuring device 20.

[0040] The measurement operation by the measuring device 20 and the photographing operation by the photographing device 30 may be executed independently of each other. For example, while the measuring device 20 is measuring three-dimensional data of the measurement object, the photographing device 30 may photograph the measurement object in parallel.

[0041] The measuring device 20 or the photographing device 30 may be provided with a user interface that accepts operation input from the user. The measuring device 20 or the photographing device 30 may be configured so that the user can operate it through the information processing device 10. When the user can operate the measuring device 20 or the photographing device 30 through the information processing device 10, the information processing device 10 may accept input from the user to operate the measuring device 20 or the photographing device 30 via the input unit 14, and transmit control information corresponding to the operation input to the measuring device 20 or the photographing device 30 from the communication unit 13.

[0042] At least one of the measuring device 20 and the imaging device 30 may be configured as a part of the information processing device 10. In other words, the information processing device 10 may include at least one of the measuring device 20 and the imaging device 30.

[0043] <Database 40> The database 40 may be configured to include at least one server. The database 40 may be configured to include, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like. The database 40 may be configured to include an electromagnetic storage medium such as a magnetic disk. The database 40 may be configured to include a communication module configured to be able to communicate with other devices such as the information processing device 10 or the second information processing device 20.

[0044] The database 40 may be realized using a cloud service or in an on-premise environment. The database 40 may be configured as the storage unit 12 of the information processing device 10. That is, the information processing device 10 may include the database 40.

[0045] (Example of operation of information processing system 1) In the information processing system 1, the measuring device 20 generates 3D data of the object to be measured at the construction site, the photographing device 30 generates photographed images of the object to be measured, and the information processing device 10 associates the 3D data of the object to the photographed images and saves them. The user can use the information that associates the 3D data of the object to be measured with the photographed images as a construction record. An example of the operation of the information processing system 1 in which the 3D data of the object to be measured and the photographed images are associated and saved will be described below.

[0046] The information processing system 1 may execute an information processing method including the steps of the flowchart illustrated in Fig. 2 so that three-dimensional data of a measurement object and a captured image of the measurement object can be linked and saved. The information processing method may be realized as an information processing program executed by each component of the information processing system 1. The information processing program may be stored in a non-transitory computer-readable medium.

[0047] The measuring device 20 starts to perform a 3D scan of the object to be measured (step S1). The measuring device 20 generates 3D data of the object to be measured. At any point during the 3D scan, the measuring device 20 may output the 3D data that has been generated up to that point, i.e., the 3D data in the process of being generated, to the information processing device 10. The information processing device 10 may acquire the 3D data in the process of being generated from the measuring device 20 and display it on the display unit 15.

[0048] When the measuring device 20 is controlled by the information processing device 10, the input unit 14 of the information processing device 10 may accept an operation input from a user to start a 3D scan. In response to the operation input to start a 3D scan, the control unit 11 of the information processing device 10 may transmit control information to the measuring device 20 to start performing a 3D scan of the measurement object.

[0049] When the measuring device 20 is not controlled by the information processing device 10, that is, when the user directly operates the measuring device 20, the measuring device 20 may notify the control unit 11 that a 3D scan of the measurement object has started in response to the user's operation. When the control unit 11 receives a notification from the measuring device 20 that a 3D scan of the measurement object has started, the control unit 11 may request the measuring device 20 to transmit 3D data, or may acquire the 3D data transmitted from the measuring device 20 as needed.

[0050] The imaging device 30 generates a captured image of the measurement object while the measuring device 20 is performing a 3D scan of the measurement object (step S2). The procedure of generating a captured image in step S2 is also referred to as the imaging step. The imaging of the measurement object by the imaging device 30 is performed in parallel with the 3D scan of the measurement object by the measuring device 20. Furthermore, the resolution of the captured image obtained by imaging the measurement object by the imaging device 30 is higher than the resolution of the 3D data obtained by performing the 3D scan. In other words, the imaging device 30 generates a captured image by capturing an image of the measurement object at a resolution higher than the resolution of the 3D data while performing a 3D scan of the measurement object.

[0051] When the imaging device 30 is controlled by the information processing device 10, the input unit 14 of the information processing device 10 may accept an imaging operation input from a user. The control unit 11 of the information processing device 10 may transmit control information for causing the imaging device 30 to capture an image of the measurement object in response to the imaging operation input. The control unit 11 may acquire information regarding the execution status of the 3D scan in the measuring device 20 from the measuring device 20, and identify the portion of the measurement object that was the target of the 3D scan when the imaging device 30 generated the captured image. The control unit 11 may link the captured image with the portion that was the target of the 3D scan when the captured image was captured.

[0052] When the photographing device 30 is not controlled by the information processing device 10, that is, when the user directly operates the photographing device 30, the photographing device 30 may notify the control unit 11 that an image of the measurement object has been captured in response to the user's operation. When the control unit 11 receives a notification from the photographing device 30 that an image of the measurement object has been captured, the control unit 11 may request the photographing device 30 to transmit the captured image, or may acquire the captured image transmitted from the photographing device 30 at any time.

[0053] When the information processing device 10 controls the measuring device 20 and the imaging device 30, the control unit 11 may acquire an image of the measurement object from the measuring device 20 while the measuring device 20 is performing a 3D scan. The control unit 11 may display a 3D scan execution screen 80, as illustrated in FIG. 3, on the display unit 15. The 3D scan execution screen 80 is a screen displayed during 3D scanning and may include a screen displaying the measurement object. In the example of FIG. 3, the measurement object is a staircase in a building. The measurement object may be replaced with another object, terrain, or the like. The 3D scan execution screen 80 may display at least a portion of the scan range of the measurement object. The scan range is the range in which the 3D scan of the measurement object is performed. The display range of the 3D scan execution screen 80 may match the scan range of the measurement object or may be a portion of the scan range of the measurement object. The 3D scan execution screen 80 may include a display of an uncompleted 3D scan range 84 and a completed 3D scan range 85. The uncompleted 3D scan area 84 is an area where a 3D scan of the measurement object has not yet been performed. The completed 3D scan area 85 is an area where a 3D scan of the measurement object has been completed. The 3D scan execution screen 80 may display 3D data generated by the 3D scan in the completed 3D scan area 85. The 3D scan execution screen 80 may display an image such as spots in the uncompleted 3D scan area 84, indicating that a 3D scan has not yet been performed.

[0054] The 3D scan execution screen 80 includes an operation button 81 for receiving an operation input from the user to start or stop the 3D scan. The 3D scan starts or stops when an operation such as touching or pressing the operation button 81 is input. The operation button 81 may be displayed in a manner that indicates the progress rate of the 3D scan in progress as the length of a circumference.

[0055] The 3D scan execution screen 80 includes a capture button 82 for receiving an operation input from the user to capture an image of the measurement object during 3D scanning. When an operation such as touching or pressing the capture button 82 is input, a captured image of the measurement object is generated during 3D scanning. The capture button 82 is also referred to as a capture interface. That is, the control unit 11 may display the capture interface on the screen (execution screen 80) displaying the measurement object during 3D scanning. The input unit 14 may accept a user's input to the capture interface as an operation input to capture an image of the measurement object. The control unit 11 may transmit control information to the capture device 30 in response to the operation input to the capture interface. The capture device 30 may capture an image of the measurement object and generate a captured image in response to the control information from the control unit 11. Displaying the capture interface on the screen during 3D scanning and accepting a user's capture operation input makes it easier for the user to capture high-resolution images during 3D scanning.

[0056] The photographing device 30 may automatically generate a photographed image while the measuring device 20 is performing a 3D scan. The photographing device 30 may appropriately perform camera shake correction, illumination correction, or the like when generating a photographed image. When the information processing device 10 acquires a photographed image from the photographing device 30 while the measuring device 20 is performing a 3D scan, the control unit 11 may correct the photographed image.

[0057] The measuring device 20 finishes the 3D scan of the measurement object (step S3). When the measuring device 20 finishes the 3D scan, it may generate 3D data based on the results of measurement performed by the 3D scan. The procedure of generating 3D data in steps S1 and S3 is also referred to as a generation step. The measuring device 20 may output the generated 3D data to the information processing device 10. The information processing device 10 may acquire the generated 3D data from the measuring device 20 and display it on the display unit 15. As illustrated in FIG. 4, the information processing device 10 may display a 3D data display screen 60 on the display unit 15. The 3D data display screen 60 is a screen that displays the generated 3D data and includes a 3D data display frame 62. The 3D data display frame 62 is a frame that displays the 3D data.

[0058] When the measuring device 20 is controlled by the information processing device 10, the input unit 14 of the information processing device 10 may accept an operation input from a user to end the 3D scan. In response to the operation input to end the 3D scan, the control unit 11 of the information processing device 10 may transmit control information to the measuring device 20 to end the 3D scan of the measurement object.

[0059] When the measuring device 20 is not controlled by the information processing device 10, that is, when the user directly operates the measuring device 20, the measuring device 20 may notify the control unit 11 that the 3D scan of the measurement object has been completed in response to the user's operation. When the control unit 11 receives a notification from the measuring device 20 that the 3D scan of the measurement object has been completed, the control unit 11 may request the measuring device 20 to transmit the generated 3D data, or may wait for the measurement device 20 to transmit the generated 3D data.

[0060] The control unit 11 of the information processing device 10 associates and saves the 3D data of the measurement object generated by the measuring device 20 and the captured image of the measurement object generated by the imaging device 30 (step S4). The saving procedure of step S4 is also referred to as a saving step. The control unit 11 may save the data, in which the 3D data of the measurement object and the captured image of the measurement object are associated, in the database 40 or the storage unit 12 of the information processing device 10. When a pin is placed, the 3D position information required to associate the 3D data with the captured image is the position information of the pin in the 3D data. In this case, the position of the pin in the 3D data is considered to be the position in the 3D data of the captured image. Furthermore, when no pin is specified, the 3D data is displayed as a mesh superimposed on the background of the imaging screen, and therefore the position corresponding to the mesh at the center of the screen at the time of imaging is considered to be the position in the 3D data of the captured image.

[0061] When the imaging device 30 generates multiple captured images of the measurement object during the execution of the 3D scan, the control unit 11 may associate at least one of the multiple captured images with the 3D data and store it. In other words, the control unit 11 may associate at least a part of the captured images of the measurement object with the 3D data and store it.

[0062] The input unit 14 of the information processing device 10 may receive an input specifying an image to be linked to the three-dimensional data from among the captured images. The control unit 11 may store the specified image in the database 40 or the storage unit 12, linking it to the three-dimensional data. User convenience is improved by allowing the user to specify the captured image to be linked to the three-dimensional data.

[0063] The input unit 14 may accept an input to set an identifier for identifying at least one position in the three-dimensional data to the three-dimensional data. The control unit 11 may link at least a portion of the captured image to the three-dimensional data by linking the identifier. The identifier identifies at least one point in the three-dimensional data. The identifier may be displayed, for example, as a pin 64 or an icon 66 (see FIG. 4 ) when the three-dimensional data is displayed. Linking the captured image to the three-dimensional data using the identifier makes it easy for the user to understand which part of the three-dimensional data the captured image is linked to.

[0064] The control unit 11 may display an identifier setting interface that accepts an input for setting an identifier on the display unit 15. The input unit 14 may accept a user input to the identifier setting interface as an input for setting an identifier. The control unit 11 may set the identifier at a position where the 3D scan is being performed when the input for setting the identifier is accepted. The input unit 14 may accept an input that further specifies a position in the 3D data where the identifier is to be set. The control unit 11 may set the identifier at the specified position in the 3D data.

[0065] The control unit 11 may display an identifier setting button 83 as an identifier setting interface on a 3D scan execution screen 80 shown in Fig. 3, for example. When an operation such as touching or pressing the identifier setting button 83 is input, an identifier is set in the 3D data. The control unit 11 may display the identifier set in response to the operation input on the 3D scan execution screen 80. After the 3D scan is completed, the control unit 11 may display the identifier as a pin 64 superimposed on the 3D data displayed in the 3D data display frame 62 on a 3D data display screen 60 shown in Fig. 4.

[0066] By using the identifier setting interface, the user can easily set the identifier.

[0067] When the control unit 11 causes the imaging device 30 to capture an image of the measurement object in response to an operation input to the imaging interface, the control unit 11 may set an identifier to identify the imaging position of the captured image at the position where the 3D data was generated when the captured image of the measurement object was generated in the imaging step. That is, the control unit 11 may set an identifier to the 3D data being generated during the 3D scan. The control unit 11 may link the captured image to the identifier in the storage step. This facilitates linking the 3D data and the captured image. The control unit 11 may link the captured image to the identifier during the 3D data capture step. The captured image linked to the identifier may be displayed as an icon 66 connected to a pin 64 on the 3D data display screen 60 illustrated in FIG. 4. The icon 66 may be a thumbnail of the captured image linked to the pin 64.

[0068] When linking a captured image to an identifier, the control unit 11 may further link information about the captured image to the identifier. The information about the captured image may include, for example, a comment about the captured image. The control unit 11 may accept an input to select an identifier set in the three-dimensional data and an input of information about the captured image via the input unit 14, and link the selected identifier to the input information. While accepting the input to select an identifier, the control unit 11 may also acquire information about the captured image by a separate means. The control unit 11 may link the acquired information to the selected identifier. Linking information about the captured image to the identifier makes it easier for the user to recognize the captured image linked to the three-dimensional data.

[0069] The control unit 11 may store the captured images acquired during the 3D scan of the measurement object in a folder that stores the 3D data generated by the 3D scan. The control unit 11 may store the captured images acquired during the 3D scan in a folder associated with the 3D data generated by the 3D scan. In other words, the control unit 11 may store the captured images acquired during the 3D scan in a folder corresponding to the 3D data generated by the 3D scan, thereby linking the images to the 3D data. The folders may be associated by sharing a common folder name. Linking the 3D data and the captured images by the folders in which they are stored makes it easier for the user to handle the data.

[0070] The control unit 11 may display the captured image linked to the three-dimensional data on the display unit 15 as a captured image display screen 70 exemplified in FIG. 5. The captured image display screen 70 is a screen that displays the captured image and may include a photo display frame 72. The photo display frame 72 is a frame that displays the captured image. The control unit 11 may display the captured image display screen 70 of FIG. 5 when an operation input is received for the pin 64 or the icon 66 on the three-dimensional data display screen 60 of FIG. 4. The control unit 11 may display the captured image linked to the pin 64 for which the operation was input in the photo display frame 72. The captured image display screen 70 may include a thumbnail display frame 74. When multiple captured images are linked to the pin 64, the control unit 11 may display thumbnails of the multiple captured images linked to the pin 64 in the thumbnail display frame 74. When the control unit 11 receives an operation input to the thumbnail display frame 74, it may display in the photo display frame 72 a captured image corresponding to the thumbnail for which the operation was input.

[0071] When linking a captured image to three-dimensional data, the control unit 11 may appropriately perform camera shake correction or illumination correction on the image to be linked. Also, as described above, the control unit 11 may acquire the corrected captured image while performing the three-dimensional scan. By saving the corrected captured image, the usefulness of the captured image as a construction record is increased. In other words, it becomes easier for users to use the captured image as a construction record.

[0072] As described above, the control unit 11 may store some of the captured images generated during the 3D scan by linking them to the 3D data. Conversely, the control unit 11 may not link at least some of the captured images generated during the 3D scan to the 3D data. The control unit 11 may discard captured images that are not to be linked to the 3D data, rather than storing them in the database 40 or the storage unit 12. By discarding images that are not to be linked to the 3D data, the data stored in the database 40 or the storage unit 12 is reduced.

[0073] After the 3D scan is completed, the control unit 11 may determine which images to discard without linking them to the 3D data. In other words, the control unit 11 may retain the captured images without discarding them at least while the 3D scan is being performed. In this manner, the user can decide whether to link the captured images to the 3D data after the 3D scan is completed. In other words, the user does not need to be aware of linking the captured images while the 3D scan is being performed.

[0074] The control unit 11 may determine as an image to be discarded an image that was not designated as an image to be linked to 3D data. Here, the user may input an operation to link a captured image to 3D data after a considerable amount of time has passed since the 3D scan was performed. The control unit 11 may wait a predetermined time after capture before determining as an image to be discarded an image that was not designated as an image to be linked to 3D data after the 3D scan was performed, rather than immediately determining as an image to be discarded. In other words, the control unit 11 may determine as an image to be discarded an image that was not designated as an image to be linked to 3D data within a predetermined time after capture. The control unit 11 may accept an input from the user to set the predetermined time, and set the predetermined time. The control unit 11 may set the predetermined time based on the frequency of performing 3D scans or the frequency of generating captured images. In this way, by determining as an image to be discarded an image that was not designated as an image to be linked to 3D data after a predetermined time has passed since capture, or regardless of the time elapsed since capture, the user is not required to specify the image to be discarded.

[0075] Returning to FIG. 2, after executing the procedure of step S4, the control unit 11 ends the execution of the procedure of the flowchart in FIG.

[0076] (summary) As described above, according to the information processing system 1, information processing method, and information processing program of this embodiment, the 3D data of the measurement object and the captured image are linked and saved. Data linking the 3D data and the captured image is useful as a construction record and also useful for visually checking the construction site. As a result, both the 3D data and the captured image of the measurement object at the construction site are utilized.

[0077] Furthermore, the information processing system 1, the information processing method, and the information processing program according to the present embodiment provide various interfaces for linking and saving the 3D data of the measurement object and the captured image, thereby improving user convenience.

[0078] Although embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present disclosure. For example, functions included in each component or step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined or divided into one. Although the embodiments of the present disclosure have been described mainly in terms of an apparatus, the embodiments of the present disclosure can also be realized as a method including steps executed by each component of the apparatus. The embodiments of the present disclosure can also be realized as a method, a program executed by a processor included in the apparatus, or a storage medium on which a program is recorded. It should be understood that these are also encompassed within the scope of the present disclosure. [Explanation of symbols]

[0079] 1. Information Processing Systems 10 information processing device (11: control unit, 12: storage unit, 13: communication unit, 14: input unit, 15: display unit) 20 Measuring Equipment 30 Imaging equipment 40 databases 60 3D data display screen (62: 3D data display frame, 64: pin, 66: icon) 70 Shooting image display screen (72: Photo display frame, 74: Thumbnail display frame) 80 3D scan execution screen (81: operation button, 82: shooting button, 83: identifier setting button, 84: unexecuted range, 85: completed range)

Claims

1. a generating step of performing a three-dimensional scan of a measurement object to generate three-dimensional data that identifies the three-dimensional shape of the measurement object; an imaging step of generating an image of the measurement object at a resolution higher than the resolution of the three-dimensional data during the three-dimensional scan; a storing step of storing at least a part of the captured image in association with the three-dimensional data; An information processing method, including:

2. The information processing method according to claim 1 , wherein the storing step receives an input specifying an image to be linked to the three-dimensional data from among the captured images.

3. 3. The information processing method according to claim 2, wherein in the saving step, an input is accepted to set an identifier that identifies at least one position of the three-dimensional data in the three-dimensional data, and the captured image is linked to the identifier, thereby linking the captured image to the three-dimensional data.

4. 4. The information processing method according to claim 3, wherein said saving step displays an identifier setting interface for accepting input of said identifier setting.

5. 2. The information processing method according to claim 1, wherein, in the photographing step, a photographing interface that accepts photographing operation input is displayed on a screen that displays the measurement object while the 3D scan is being performed, and the photographed image is generated in accordance with the operation input to the photographing interface.

6. In the photographing step, an identifier for identifying a photographing position of the photographed image is set at a position where the three-dimensional data was generated when the photographed image was generated; In the storing step, the captured image is associated with the identifier. The information processing method according to claim 5 .

7. The information processing method according to claim 1 , wherein in the saving step, the captured image is linked to the three-dimensional data by saving the captured image in a folder corresponding to the three-dimensional data.

8. The information processing method according to claim 3 , wherein in the storing step, information about the captured image is linked to the identifier.

9. The information processing method according to claim 1 , wherein, in the saving step, images of the captured images that are not to be linked to the three-dimensional data are discarded.

10. 10. The information processing method according to claim 9, wherein, in the storing step, images to be discarded are determined from among the captured images after the three-dimensional scanning is completed.

11. The information processing method according to claim 9, wherein in the saving step, among the captured images, images that have not been linked within a predetermined time after capture or images that have not been designated as images to be linked to the three-dimensional data are determined to be images to be discarded.

12. The information processing method according to claim 1 , wherein the captured image is corrected in the capturing step or the saving step.

13. An information processing program that causes a processor to execute the information processing method according to any one of claims 1 to 12.

14. An information processing device that executes the information processing method according to any one of claims 1 to 12.

Images