Information processing device, information processing method, and program

Abstract

Even if a shooting location does not appear in the 3D data of the object, it will be possible to align it with the corresponding location in the 3D point cloud data. [Solution] The information processing device 10 includes a data acquisition unit 11 that acquires first video data from a first specific location to a second specific location of an object, and second video data from the second specific location to the inspection location; a camera pose calculation unit 12 that calculates the camera pose at the first specific location on the 3D data (first 3D camera pose), calculates the camera pose at the second specific location on the 3D data (second 3D camera pose) from the camera pose in the frame of the second specific location and the first 3D camera pose, and calculates the camera pose at the inspection location on the 3D data (third 3D camera pose) from the camera pose in the frame of the inspection location and the second 3D camera pose; and a position identification unit 13 that identifies the position of the inspection location on the 3D data using the third 3D camera pose.

Description

Technical Field

[0004] , , , , , , , , , ,

[0005] , , , , , ,

[0001] The present disclosure relates to an information processing apparatus and an information processing method for collating three-dimensional point cloud data of an object with a photographed image of the object, and further relates to a program for realizing these.

Background Art

[0002] In recent years, it has been required to efficiently manage infrastructure such as bridges. For this reason, along with the improvement of image processing technology in recent years, a technology for managing deteriorated parts of infrastructure using three-dimensional point cloud data has been proposed (see, for example, Patent Document 1).

[0003] Specifically, Patent Document 1 discloses an apparatus that can display three-dimensional point cloud data of infrastructure on a screen and attach an image taken at the time of inspection to the corresponding part of the three-dimensional point cloud data on the screen. According to the apparatus disclosed in Patent Document 1, an administrator can easily grasp deteriorated parts of infrastructure and efficiently manage the infrastructure.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in order to efficiently manage the apparatus disclosed in Patent Document 1, it is necessary to accurately align the photographed part of the image taken at the time of inspection with the corresponding part of the three-dimensional point cloud data. However, in an infrastructure having a cavity inside, such as a box girder bridge, there may be a case where a defective part exists at a position where it does not appear on the three-dimensional point cloud data. In such a case, it is difficult to display the defective part on the three-dimensional point cloud data with the apparatus disclosed in Patent Document 1.

[0006] One example of the purpose of this disclosure is to enable alignment between the 3D point cloud data and the corresponding location in the 3D data of the object, even if that location does not appear in the 3D data of the object. [Means for solving the problem]

[0007] To achieve the above objective, the information processing device in one aspect of this disclosure is: A data acquisition unit acquires first video data generated by video recording a portion of an object from a first specific location to a second specific location, and maintaining the camera's orientation during video recording for each frame, and second video data generated by video recording a portion of the object from a second specific location to an inspection location, and maintaining the camera's orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the 3D data of the object, and the camera pose at the location corresponding to the first specific location on the 3D data is calculated as the first 3D camera pose. Using the camera pose held in the frame containing the second specific location and the first 3D camera pose, the camera pose at the location corresponding to the second specific location on the 3D data is calculated as the second 3D camera pose, and further, A camera pose calculation unit calculates a third three-dimensional camera pose using the camera pose held in the frame containing the inspection location in the second video data and the second three-dimensional camera pose, and the camera pose at the location corresponding to the inspection location on the three-dimensional data. A position identification unit that uses the third three-dimensional camera orientation to identify the location of the area corresponding to the inspection point on the three-dimensional data, It is characterized by having the following features.

[0008] Furthermore, in order to achieve the above objectives, the information processing method in one aspect of this disclosure is: A data acquisition step involves acquiring first video data, which is generated by video recording a portion of the object from a first specific location to a second specific location, and which maintains the camera's orientation during video recording for each frame; and second video data, which is generated by video recording a portion of the object from a second specific location to an inspection location, and which maintains the camera's orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the 3D data of the object, and the camera pose at the location corresponding to the first specific location on the 3D data is calculated as the first 3D camera pose. Using the camera pose held in the frame containing the second specific location and the first 3D camera pose, the camera pose at the location corresponding to the second specific location on the 3D data is calculated as the second 3D camera pose, and further, A camera pose calculation step, which involves using the camera pose held in the frame containing the inspection location in the second video data and the second 3D camera pose to calculate the camera pose at the location corresponding to the inspection location on the 3D data as the third 3D camera pose, A position identification step in which the location of the location corresponding to the inspection location on the 3D data is identified using the third 3D camera orientation, It is characterized by having the following:

[0009] Furthermore, in order to achieve the above objectives, the program in one aspect of this disclosure is On the computer, A data acquisition step involves acquiring first video data, which is generated by video recording a portion of the object from a first specific location to a second specific location, and which maintains the camera's orientation during video recording for each frame; and second video data, which is generated by video recording a portion of the object from a second specific location to an inspection location, and which maintains the camera's orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the 3D data of the object, and the camera pose at the location corresponding to the first specific location on the 3D data is calculated as the first 3D camera pose. Using the camera pose held in the frame containing the second specific location and the first 3D camera pose, the camera pose at the location corresponding to the second specific location on the 3D data is calculated as the second 3D camera pose, and further, A camera pose calculation step, which involves using the camera pose held in the frame containing the inspection location in the second video data and the second 3D camera pose to calculate the camera pose at the location corresponding to the inspection location on the 3D data as the third 3D camera pose, A position identification step in which the location of the location corresponding to the inspection location on the 3D data is identified using the third 3D camera orientation, It is characterized by causing the execution of [the specified action]. [Effects of the Invention]

[0010] As described above, according to this disclosure, even if a shooting location does not appear in the 3D data of the object, it is possible to align it with the corresponding location in the 3D point cloud data. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a schematic diagram showing an example of an information processing device. [Figure 2] Figure 2 is a diagram illustrating the configuration of an example of an information processing device. [Figure 3] Figure 3 shows an example of the object. [Figure 4] Figure 4 shows an example of how video data used in an information processing device is captured. [Figure 5] Figure 5 illustrates an example of processing in the camera pose calculation unit 12. [Figure 6] Figure 6 is a flowchart showing an example of the operation of the information processing device 10. [Figure 7] FIG. 7 is a block diagram showing an example of a computer that realizes an information processing apparatus.

Embodiment of the Invention

[0012] (Embodiment) Hereinafter, an information processing apparatus, an information processing method, and a program in the embodiment will be described with reference to FIGS. 1 to 7.

[0013] [Device Configuration] First, a schematic configuration of an example of the information processing apparatus will be described using FIG. 1. FIG. 1 is a configuration diagram showing a schematic configuration of an example of the information processing apparatus.

[0014] The information processing apparatus 10 shown in FIG. 1 is an image matching apparatus that performs matching between three-dimensional point cloud data of an object and a captured image of the object. As shown in FIG. 1, the information processing apparatus 10 includes a data acquisition unit 11, a camera pose calculation unit 12, and a position identification unit 13.

[0015] The data acquisition unit 11 acquires first video data and second video data. The first video data is generated by capturing a video from a first specific location to a second specific location of the object. Also, the first video data holds the camera pose at the time of video capture for each frame. The second video data is generated by capturing a video from a second specific location to an inspection location of the object. Similar to the first video data, the second video data also holds the camera pose at the time of video capture for each frame.

[0016] The camera pose calculation unit 12 first collates a frame including the first specific location in the first video data with the three-dimensional data of the object, and calculates the camera pose (hereinafter referred to as "first three-dimensional camera pose") at the location corresponding to the first specific location on the three-dimensional data.

[0017] Next, the camera pose calculation unit 12 uses the camera pose held in the frame containing the second specific location of the first or second video data and the first 3D camera pose to calculate the camera pose at the location corresponding to the second specific location on the 3D data (hereinafter referred to as the "second 3D camera pose").

[0018] Furthermore, the camera pose calculation unit 12 uses the camera pose held in the frame containing the inspection point in the second video data and the second 3D camera pose to calculate the camera pose at the location corresponding to the inspection point on the 3D data (hereinafter referred to as the "third 3D camera pose").

[0019] The position identification unit 13 uses the orientation of the third 3D camera to identify the location of the area corresponding to the inspection point on the 3D data.

[0020] Thus, even if data corresponding to an inspection point does not exist in the 3D data, the information processing device 10 uses the first and second video data to calculate the camera orientation (third 3D camera orientation) of the location corresponding to the inspection point in the 3D data. Therefore, according to the information processing device 10, even if a shooting location does not appear in the 3D data of the object, it is possible to align it with the corresponding location in the 3D point cloud data.

[0021] In the embodiments, when simply referred to as "camera pose," it means the camera pose for each frame of the video data. When referred to as "3D camera pose," it means the camera pose on the 3D data.

[0022] Next, we will specifically explain the configuration and functions of an example of the information processing device 10 using Figures 2 to 4. Figure 2 is a configuration diagram specifically showing the configuration of an example of the information processing device. Figure 3 is a diagram showing an example of an object. Figure 4 is a diagram showing an example of how video data used in the information processing device is captured.

[0023] As shown in Figure 2, the information processing device 10 includes a display unit 14 in addition to the data acquisition unit 11, camera pose calculation unit 12, and position identification unit 13 described above. Furthermore, the information processing device 10 is connected to the database 20 and the user's terminal device 30 for data communication.

[0024] Database 20 stores the 3D data 21 of the object, the first video data 22, and the second video data 23. In this embodiment, the object is a box girder bridge. However, the object is not limited to a box girder bridge; any structure with a cavity inside is acceptable. Other examples of objects include buildings, factories, large tanks, etc.

[0025] As the 3D data 21, for example, 3D point cloud data composed of a set of feature points of the object can be used. 3D point cloud data can be generated, for example, by the Structure from Motion (SfM) method, which utilizes numerous 2D images of the object. Furthermore, 3D point cloud data may also be generated by a depth sensor (LiDAR, point cloud scanner, etc.).

[0026] Here, an example of an object will be explained using Figure 3. As shown in Figure 3, in this embodiment, the object is a box girder bridge 40. The box girder bridge has an internal space 42. The inspection points (see the ★ marks in Figure 4) are located in the internal space 42. Also, the internal space 42 does not appear in the 3D data 21.

[0027] Next, we will explain how to capture video data using Figure 4. As also shown in Figure 4, inspection point 50 (marked with a ★ in Figure 4) is located in the internal space 42 of the box girder bridge 40. The first specific point 51 (marked with a ▲ in Figure 4) is set on the bridge pier 41. The first specific point 51 can be any location in the 3D point cloud data that is not missing. The location of the first specific point on the box girder bridge 40 may have a specific marking added beforehand.

[0028] The second designated location 52 (marked with a ● in Figure 4) is also located within the interior space 42. The second designated location 52 is positioned near the entrance 43 of the interior space 42 of the box girder bridge 40 so that the photographer 60 can take photographs without entering the interior space 42. A specific marking may also be added in advance to the location corresponding to the second designated location 52 of the box girder bridge 40.

[0029] Therefore, as shown in Figure 4, the photographer 60 uses the imaging device 61 to record video from the first specific location 51 of the bridge pier 41 through the entrance 43 to the second specific location 52 in the interior space 42. This creates the first video data. Next, the photographer 60 enters the interior space 42 from the entrance 43 and uses the imaging device 61 to record video from the second specific location 52 to the inspection location 50 while moving through the interior space 42. This creates the second video data. In the example in Figure 4, a smartphone with a camera is used as the imaging device 61.

[0030] The imaging device 61, a smartphone, is equipped with various sensors, including an IMU (Inertial Measurement Unit). During video recording, the smartphone calculates the camera orientation for each frame using sensor data output from the IMU. The camera orientation may also be calculated as VIO (Visual-Inertial Odometry), which is obtained by combining the sensor data output from the IMU with the image analysis results for each frame. The smartphone then adds information indicating the identified camera orientation to the frame. The camera orientation is relatively associated as an external parameter for each frame of the camera that recorded the video.

[0031] In the example shown in Figure 4, the first video data and the second video data are transmitted from the smartphone, which is the imaging device 61, to the database 20. In the example shown in Figure 4, the first video data and the second video data are captured separately, but they may also be a single video data sequence. Alternatively, the first video data and the second video data may be created by splitting a single video data sequence.

[0032] In this embodiment, the data acquisition unit 11 acquires 3D data 21 of the object, first video data 22, and second video data 23 from the database 20. The data acquisition unit 11 also outputs the acquired 3D data 21, first video data 22, and second video data 23 to the camera posture calculation unit 12.

[0033] The camera pose calculation unit 12 first compares the feature points of the frame containing the first specific location in the first video data with the feature points of the 3D data, thereby identifying multiple corresponding points for both the 3D data and the frame image containing the first specific location. As a result, the region corresponding to the specific location on the 3D data is identified.

[0034] Specifically, the camera pose calculation unit 12 first calculates features such as Haar-Like features, HOG features, and SIFT features in the frame containing the first specific location. Next, the camera pose calculation unit 12 extracts points where the feature values ​​are greater than or equal to a predetermined value as feature points.

[0035] The frame of the first specific location may be specified in advance by the user. The camera pose calculation unit 12 can also identify the frame of the first specific location by extracting feature points from all frames of the first video data 22 and then searching for feature points of the specified feature quantity.

[0036] Furthermore, the camera pose calculation unit 12 performs matching between the feature points of the frame containing the first specific location and each point constituting the 3D point cloud data, which is the 3D data 21. Existing methods are used for the matching process between feature points. Based on the matching results, the camera pose calculation unit 12 identifies multiple feature points corresponding to the 3D data 21 and the frame image containing the first specific location, respectively. Note that the feature points are identified in both the 3D data 21 and the frame image and correspond to each other. Feature points that correspond to each other will be referred to as "corresponding points" below.

[0037] Next, the camera pose calculation unit 12 uses the identified multiple corresponding points to calculate the 3D camera pose (first 3D camera pose) at the location on the 3D data 21 that includes these multiple corresponding points (corresponding to the location corresponding to the first identified location). The first 3D camera pose is the camera pose in the 3D data 21 as a world coordinate system. Specifically, the camera pose calculation unit 12 uses the identified multiple corresponding points and the intrinsic camera parameters at the time of shooting the frame containing the first identified location to calculate the extrinsic parameters at the time of shooting the frame containing the identified location as the first 3D camera pose.

[0038] Next, the camera pose calculation unit 12 identifies the camera pose held in the frame containing the first specific location from the first video data 22, and identifies the camera pose held in the frame containing the second specific location from either the first video data 22 or the second video data 23.

[0039] The second specific location frame may also be specified in advance by the user. Furthermore, the camera pose calculation unit 12 can identify the second specific location frame by extracting feature points from all frames of the first video data 22 or the second video data 23, and then searching for feature points of the specified feature quantity.

[0040] Next, the camera pose calculation unit 12 calculates the difference between the camera pose held in the frame containing the first specific location and the camera pose held in the frame containing the second specific location. Then, the camera pose calculation unit 12 adds the calculated difference to the first 3D camera pose. The resulting camera pose corresponds to the camera pose at the location corresponding to the second specific location in the 3D data, i.e., the second 3D camera pose.

[0041] Next, the camera posture calculation unit 12 identifies, from the second video data, the camera posture held in the frame containing the second specific location and the camera posture held in the frame containing the inspection location.

[0042] Furthermore, the frame of the inspection point may also be specified in advance by the user. In addition, the camera posture calculation unit 12 can identify the frame of the inspection point by extracting feature points from all frames of the second video data 23 and then searching for feature points of the specified feature quantity.

[0043] Next, the camera posture calculation unit 12 calculates the difference between the camera posture held in the frame containing the second specific location and the camera posture held in the frame containing the inspection location. Then, the camera posture calculation unit 12 adds the calculated difference to the second 3D camera posture. The resulting 3D camera posture corresponds to the 3D camera posture at the location corresponding to the inspection location in the 3D data, i.e., the third 3D camera posture. Figure 5 is a diagram illustrating an example of the processing in the camera posture calculation unit 12.

[0044] In this embodiment, the positioning unit 13 uses the third 3D camera orientation (external parameter) at the location corresponding to the inspection point on the 3D data and the 3D data 21 to calculate the coordinates of the location corresponding to the inspection point on the 3D data. Specifically, the positioning unit 13 uses the third 3D camera orientation at the location corresponding to the inspection point on the 3D data to identify the area included in the camera's field of view, and sets the position (coordinates) of the identified area as the position of the location corresponding to the inspection point on the 3D data.

[0045] The display unit 14 displays the 3D data of the object on, for example, the screen of the terminal device 30. The display unit also uses the coordinates of the locations corresponding to the inspection points, calculated by the location identification unit 13, to display the areas indicating the inspection points on the screen, superimposed on the 3D data. [Device operation] Next, an example of the operation of the information processing device 10 will be explained using Figure 6. Figure 6 is a flowchart showing an example of the operation of the information processing device 10. In the following explanation, Figures 1 to 5 will be referred to as appropriate. In this embodiment, the information processing method is implemented by operating the information processing device 10. Therefore, the explanation of the information processing method in this embodiment will be replaced by the following explanation of the operation of the information processing device 10.

[0046] First, as a prerequisite, 3D data 21 of the object is constructed and stored in the database 20. In addition, first video data 22 obtained by video recording from a first specific point to a second specific point of the object, and second video data 23 obtained by video recording from the second specific point to the inspection point are also stored in the database 20.

[0047] As shown in Figure 6, first, the data acquisition unit 11 acquires 3D data 21 of the object, first video data 22, and second video data 23 from the database 20 (step A1). The data acquisition unit 11 also outputs the acquired 3D data 21, first video data 22, and second video data 23 to the camera pose calculation unit 12.

[0048] Next, the camera pose calculation unit 12 compares the feature points of the frame containing the first specific location in the video data 21 with the feature points of the 3D data to identify multiple corresponding points for the 3D data and the frame image containing the first specific location (step A2).

[0049] Specifically, in step A2, the camera pose calculation unit 12 first extracts feature points from the frame containing the specific location, and then performs matching between the extracted feature points and each point that makes up the 3D data 21. Based on the matching results, the camera pose calculation unit 12 then identifies multiple corresponding points on the 3D data 21 and in each of the frame images containing the specific location.

[0050] Next, the camera pose calculation unit 12 uses the multiple corresponding points identified in step A2 to calculate the 3D camera pose (first 3D camera pose) at the location on the 3D data 21 that includes these multiple corresponding points (corresponding to the location corresponding to the first identified location) (step A3).

[0051] Specifically, in step A3, the camera pose calculation unit 12 uses the multiple corresponding points identified in step A2 and the camera's internal parameters at the time of shooting the frame containing the specific location to calculate the camera's external parameters at the time of shooting the frame containing the specific location as the first 3D camera pose.

[0052] Next, the camera pose calculation unit 12 uses the camera pose held in the frame containing the second specific location and the first 3D camera pose calculated in step A3 to calculate the second 3D camera pose at the location corresponding to the second specific location on the 3D data (step A4).

[0053] Specifically, in step A4, as shown in Figure 5, the camera pose calculation unit 12 identifies the camera pose held in the frame containing the first specific location and the camera pose held in the frame containing the second specific location from the first video data 22. Furthermore, the camera pose calculation unit 12 calculates the difference between the two identified camera poses. Then, the camera pose calculation unit 12 adds the calculated difference to the first 3D camera pose to calculate the 3D camera pose (second 3D camera pose) at the location corresponding to the second specific location on the 3D data.

[0054] Next, the camera pose calculation unit 12 uses the camera pose held in the frame containing the inspection point in the second video data and the second 3D camera pose calculated in step A4 to calculate the 3D camera pose (third 3D camera pose) at the location corresponding to the inspection point on the 3D data (step A5).

[0055] Specifically, in step A5, the camera pose calculation unit 12 first identifies the camera pose held in the frame containing the second specific location and the camera pose held in the frame containing the inspection location from the second video data. Then, the camera pose calculation unit 12 calculates the difference between the camera pose held in the frame containing the second specific location and the camera pose held in the frame containing the inspection location. After that, the camera pose calculation unit 12 adds the calculated difference to the second 3D camera pose. The resulting 3D camera pose corresponds to the third 3D camera pose.

[0056] Next, the positioning unit 13 uses the third 3D camera orientation at the location corresponding to the inspection location on the 3D data, calculated in step A5, to identify the location of the location corresponding to the inspection location on the 3D data (step A6).

[0057] Specifically, in step A6, the positioning unit 13 uses the third 3D camera orientation calculated in step A5 and the 3D data 21 to calculate the coordinates of the location corresponding to the inspection point on the 3D data.

[0058] Next, the display unit 14 displays the 3D data of the object on the screen of the terminal device 30, and further displays the area indicating the inspection point by superimposing it on the 3D data using the coordinates of the location identified in step A6 (step A7).

[0059] [Effects in the embodiment] Thus, in this embodiment, even if the inspection point is located inside the object, does not appear in the 3D data, and no data corresponding to the inspection point exists, the 3D camera orientation (third 3D camera orientation) of the location corresponding to the inspection point in the 3D data can be calculated by using the first video data and the second video data. Therefore, according to the information processing device 10, even if the shooting location does not appear in the 3D data of the object, it is possible to align it with the corresponding location in the 3D point cloud data. Furthermore, in this embodiment, the area indicating the inspection point that does not appear in the 3D data is displayed on the screen superimposed on the 3D data, so the person in charge of the object can easily identify the inspection point.

[0060] [Differentiation] Here, we will describe three modified examples of the embodiment below.

[0061] Variation 1: In Modification 1, 3D interpolation is performed on the 3D data 21, and the interpolated 3D data 21 is stored in the database 20. 3D interpolation is performed to add faces to the parts of the data where there is missing information.

[0062] One method of 3D interpolation is to use a machine learning model. In this case, the machine learning model is constructed by using a partially missing 3D model and complete 3D data (training data) through machine learning. Another method of 3D interpolation is to assume that the missing portion is a continuous surface from its vicinity and then interpolate the data.

[0063] In the first modified example, the position identification unit 13 first identifies an area that can be included in the camera's field of view of the 3D data 21, using the 3D camera orientation at the location corresponding to the inspection point on the 3D data 21. Then, the position identification unit 13 determines the location of the area included in the 3D data from the identified area to be the location of the inspection point.

[0064] In the case of Modification 1, the location of the area corresponding to the inspection point on the 3D data can be identified with even greater accuracy.

[0065] Variation 2: In Modification 2, the video data stores depth information for each frame, including the camera orientation and the depth from the imaging device 61 to the object. In Modification 2, the imaging device 61 is equipped with a depth sensor such as LiDAR in addition to a regular camera, and measures the depth to the subject each time a picture is taken. In Modification 2, the depth information to the subject only needs to be added to frames that include the inspection area.

[0066] Accordingly, in the modified example 2, the position identification unit 13 uses the 3D camera orientation at the location corresponding to the inspection location on the 3D data and the depth information held in the frame containing the inspection location to identify the location of the location corresponding to the inspection location on the 3D data.

[0067] In the case of Modification 2, the location of the area corresponding to the inspection point on the 3D data can be identified with even greater accuracy.

[0068] Variation 3: In Modification 3, the depth information used in Modification 2 is calculated from multiple frame images including the inspection point. In this case, the relative shooting position of each frame image is known from the sensor data from the IMU. Therefore, depth information for the frame including the inspection point can be obtained using the principle of triangulation, and the coordinates of the location corresponding to the inspection point on the 3D data can be calculated.

[0069] In the case of Modification 3, the location of the area corresponding to the inspection point on the 3D data can be identified with even greater accuracy.

[0070] [program] The program in this embodiment can be any program that causes a computer to execute steps A1 to A7 shown in Figure 6. By installing and running this program on a computer, the information processing device 10 and the information processing method can be realized. In this case, the computer's processor functions as the data acquisition unit 11, camera pose calculation unit 12, position identification unit 13, and display unit 14, and performs the processing. In addition to general-purpose PCs and server computers, smartphones and tablet terminal devices can also be used as computers.

[0071] Furthermore, the program in the embodiment may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: a data acquisition unit 11, a camera pose calculation unit 12, a position identification unit 13, and a display unit 14.

[0072] [Physical configuration] Here, a computer that implements the information processing device 10 by executing the program in the embodiment will be described using Figure 7. Figure 7 is a block diagram showing an example of a computer that implements the information processing device.

[0073] As shown in Figure 7, the computer 110 comprises a CPU (Central Processing Unit) 111, main memory 112, storage device 113, input interface 114, display controller 115, data reader / writer 116, and communication interface 117. Each of these components is connected to the others via a bus 121, enabling data communication.

[0074] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to, or instead of, the CPU 111. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

[0075] The CPU 111 loads the program in the embodiment, which consists of a set of codes stored in the storage device 113, into the main memory 112, and performs various calculations by executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

[0076] Furthermore, the program in this embodiment is provided stored on a computer-readable recording medium 120. The program in this embodiment may also be distributed over the internet via a communication interface 117.

[0077] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0078] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0079] Furthermore, specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0080] Furthermore, the information processing device 10 can be implemented not only by a computer on which a program is installed, but also by using hardware corresponding to each part, such as electronic circuits. Moreover, the information processing device 10 may be partially implemented by a program and the remaining part by hardware. In the embodiment, the computer is not limited to the computer shown in Figure 6.

[0081] Some or all of the embodiments described above can be expressed by (Appendix 1) to (Appendix 18) described below, but are not limited to the following descriptions.

[0082] (Note 1) A data acquisition unit acquires first video data generated by video recording a portion of an object from a first specific location to a second specific location, and maintaining the camera's orientation during video recording for each frame, and second video data generated by video recording a portion of the object from a second specific location to an inspection location, and maintaining the camera's orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the 3D data of the object, and the camera pose at the location corresponding to the first specific location on the 3D data is calculated as the first 3D camera pose. Using the camera pose held in the frame containing the second specific location and the first 3D camera pose, the camera pose at the location corresponding to the second specific location on the 3D data is calculated as the second 3D camera pose, and further, A camera pose calculation unit calculates a third three-dimensional camera pose using the camera pose held in the frame containing the inspection location in the second video data and the second three-dimensional camera pose, and the camera pose at the location corresponding to the inspection location on the three-dimensional data. A position identification unit that uses the third three-dimensional camera orientation to identify the location of the area corresponding to the inspection point on the three-dimensional data, An information processing device characterized by having the following features.

[0083] (Note 2) The camera attitude calculation unit, The difference between the camera pose held in the frame containing the first specific location in the first video data and the camera pose held in the frame containing the second specific location in the first video data is calculated, and the calculated difference is added to the first 3D camera pose to calculate the second 3D camera pose. The difference between the camera pose held in the frame containing the second specific location in the second video data and the camera pose held in the frame containing the inspection location in the second video data is calculated, and the calculated difference is added to the second 3D camera pose to calculate the third 3D camera pose. The information processing device described in Appendix 1.

[0084] (Note 3) The camera attitude calculation unit, By comparing the feature points of the frame containing the first specific location in the first video data with the feature points of the 3D data, a comparison is performed to identify a plurality of feature points corresponding to the 3D data and the frame image containing the first specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the first 3D camera pose. The information processing device described in Appendix 1.

[0085] (Note 4) The information processing apparatus according to Appendix 1, wherein the position identification unit uses the third three-dimensional camera orientation to identify an area included in the camera's field of view of the three-dimensional data, and sets the position of the identified area to the position of the location corresponding to the inspection point on the three-dimensional data.

[0086] (Note 5) The second video data further contains depth information that identifies the depth from the camera to the object, at least in the frame including the inspection location. The positioning unit uses the third three-dimensional camera orientation and the depth information held in the frame including the inspection location to determine the position of the location corresponding to the inspection location on the three-dimensional data. The information processing device described in Appendix 1.

[0087] (Note 6) The system further includes a display unit that displays the aforementioned 3D data on the screen. The display unit also displays the locations corresponding to the inspection locations superimposed on the three-dimensional data on the screen. The information processing device described in Appendix 1.

[0088] (Note 7) A data acquisition step involves acquiring first video data, which is generated by video recording a portion of the object from a first specific location to a second specific location, and which maintains the camera's orientation during video recording for each frame; and second video data, which is generated by video recording a portion of the object from a second specific location to an inspection location, and which maintains the camera's orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the 3D data of the object, and the camera pose at the location corresponding to the first specific location on the 3D data is calculated as the first 3D camera pose. Using the camera pose held in the frame containing the second specific location and the first 3D camera pose, the camera pose at the location corresponding to the second specific location on the 3D data is calculated as the second 3D camera pose, and further, A camera pose calculation step, which involves using the camera pose held in the frame containing the inspection location in the second video data and the second 3D camera pose to calculate the camera pose at the location corresponding to the inspection location on the 3D data as the third 3D camera pose, A position identification step in which the location of the location corresponding to the inspection location on the 3D data is identified using the third 3D camera orientation, An information processing method characterized by having the following:

[0089] (Note 8) In the camera posture calculation step, The difference between the camera pose held in the frame containing the first specific location in the first video data and the camera pose held in the frame containing the second specific location in the first video data is calculated, and the calculated difference is added to the first 3D camera pose to calculate the second 3D camera pose. The difference between the camera pose held in the frame containing the second specific location in the second video data and the camera pose held in the frame containing the inspection location in the second video data is calculated, and the calculated difference is added to the second 3D camera pose to calculate the third 3D camera pose. The information processing method described in Appendix 7.

[0090] (Note 9) In the camera posture calculation step, By comparing the feature points of the frame containing the first specific location in the first video data with the feature points of the 3D data, a comparison is performed to identify a plurality of feature points corresponding to the 3D data and the frame image containing the first specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the first 3D camera pose. The information processing method described in Appendix 7.

[0091] (Note 10) The information processing method according to Appendix 7, wherein in the position identification step, the third three-dimensional camera orientation is used to identify an area included in the camera's field of view of the three-dimensional data, and the position of the identified area is set to the position of the location corresponding to the inspection point on the three-dimensional data.

[0092] (Note 11) The second video data further contains depth information that identifies the depth from the camera to the object, at least in the frame including the inspection location. In the position identification step, the position of the location corresponding to the inspection location on the 3D data is identified using the third 3D camera orientation and the depth information held in the frame including the inspection location. The information processing method described in Appendix 7.

[0093] (Note 12) The system further includes a display step of displaying the three-dimensional data on the screen. In the display step, the screen also displays the area corresponding to the inspection point, superimposed on the 3D data. The information processing method described in Appendix 7.

[0094] (Note 13) On the computer, A data acquisition step involves acquiring first video data, which is generated by video recording a portion of the object from a first specific location to a second specific location, and which maintains the camera's orientation during video recording for each frame; and second video data, which is generated by video recording a portion of the object from a second specific location to an inspection location, and which maintains the camera's orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the 3D data of the object, and the camera pose at the location corresponding to the first specific location on the 3D data is calculated as the first 3D camera pose. Using the camera pose held in the frame containing the second specific location and the first 3D camera pose, the camera pose at the location corresponding to the second specific location on the 3D data is calculated as the second 3D camera pose, and further, A camera pose calculation step, which involves using the camera pose held in the frame containing the inspection location in the second video data and the second 3D camera pose to calculate the camera pose at the location corresponding to the inspection location on the 3D data as the third 3D camera pose, A position identification step in which the location of the location corresponding to the inspection location on the 3D data is identified using the third 3D camera orientation, A program that executes something.

[0095] (Note 14) In the camera posture calculation step, The difference between the camera pose held in the frame containing the first specific location in the first video data and the camera pose held in the frame containing the second specific location in the first video data is calculated, and the calculated difference is added to the first 3D camera pose to calculate the second 3D camera pose. The difference between the camera pose held in the frame containing the second specific location in the second video data and the camera pose held in the frame containing the inspection location in the second video data is calculated, and the calculated difference is added to the second 3D camera pose to calculate the third 3D camera pose. The program described in Appendix 13.

[0096] (Note 15) In the camera posture calculation step, By comparing the feature points of the frame containing the first specific location in the first video data with the feature points of the 3D data, a comparison is performed to identify a plurality of feature points corresponding to the 3D data and the frame image containing the first specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the first 3D camera pose. The program described in Appendix 13.

[0097] (Note 16) The program as described in Appendix 13, wherein in the position identification step, the third three-dimensional camera orientation is used to identify an area included in the camera's field of view of the three-dimensional data, and the position of the identified area is set to the position of the location corresponding to the inspection point on the three-dimensional data.

[0098] (Note 17) The second video data further contains depth information that identifies the depth from the camera to the object, at least in the frame including the inspection location. In the position identification step, the position of the location corresponding to the inspection location on the 3D data is identified using the third 3D camera orientation and the depth information held in the frame including the inspection location. The program described in Appendix 13.

[0099] (Note 18) The system further includes a display step of displaying the three-dimensional data on the screen. In the display step, the screen also displays the area corresponding to the inspection point, superimposed on the 3D data. The program described in Appendix 13. [Industrial applicability]

[0100] As described above, this disclosure makes it possible to align photographic locations that do not appear in the 3D data of the object with corresponding locations in the 3D point cloud data. This disclosure is useful in fields where matching 3D data with images is required, such as the management of infrastructure structures. [Explanation of Symbols]

[0101] 10 Information Processing Devices 11 Data Acquisition Unit 12 Camera posture calculation unit 13 Location identification part 14 Display section 20 Databases 21 3D data 22 Video Data 30 Terminal devices 40 Box girder bridge 41 Bridge piers 42 Interior space 43 Entrance 50 inspection points 51. First specific location 52 Second specific location 60 Photographer 61 Imaging device 110 Computer 111 CPU 112 Main Memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Readers / Writers 117 Communication Interface 118 Input devices 119 Display device 120 recording media 121 Bus

Claims

1. A data acquisition unit acquires first video data generated by video recording a portion of an object from a first specific location to a second specific location, and maintaining the camera's orientation during video recording for each frame, and second video data generated by video recording a portion of the object from a second specific location to an inspection location, and maintaining the camera's orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the three-dimensional data of the object, and the camera orientation at the location corresponding to the first specific location on the three-dimensional data is calculated as the first three-dimensional camera orientation. Using the camera pose held in the frame including the second specific location and the first three-dimensional camera pose, the camera pose at the location corresponding to the second specific location on the three-dimensional data is calculated as the second three-dimensional camera pose, and further, A camera pose calculation unit calculates a third three-dimensional camera pose using the camera pose held in the frame containing the inspection location in the second video data and the second three-dimensional camera pose, and the camera pose at the location corresponding to the inspection location on the three-dimensional data. A position identification unit that uses the third three-dimensional camera orientation to identify the location of the area corresponding to the inspection point on the three-dimensional data, An information processing device characterized by having the following features.

2. The camera attitude calculation unit, The difference between the camera pose held in the frame containing the first specific location in the first video data and the camera pose held in the frame containing the second specific location in the first video data is calculated, and the calculated difference is added to the first three-dimensional camera pose to calculate the second three-dimensional camera pose. The difference between the camera pose held in the frame containing the second specific location in the second video data and the camera pose held in the frame containing the inspection location in the second video data is calculated, and the calculated difference is added to the second three-dimensional camera pose to calculate the third three-dimensional camera pose. The information processing apparatus according to claim 1.

3. The camera attitude calculation unit, By comparing the feature points of the frame containing the first specific location in the first video data with the feature points of the three-dimensional data, a comparison is performed, thereby identifying a plurality of feature points corresponding to the three-dimensional data and the frame image containing the first specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the first three-dimensional camera pose. The information processing apparatus according to claim 1.

4. The information processing apparatus according to claim 1, wherein the positioning unit uses the third three-dimensional camera orientation to identify a region included in the camera's field of view of the three-dimensional data, and sets the position of the identified region to the position of the location corresponding to the inspection point on the three-dimensional data.

5. The second video data further contains depth information that identifies the depth from the camera to the object, at least in the frame including the inspection location. The positioning unit uses the third three-dimensional camera orientation and the depth information held in the frame including the inspection location to determine the position of the location corresponding to the inspection location on the three-dimensional data. The information processing apparatus according to claim 1.

6. The system further includes a display unit that displays the aforementioned three-dimensional data on the screen. The display unit also displays the locations corresponding to the inspection locations superimposed on the three-dimensional data on the screen. The information processing apparatus according to claim 1.

7. A data acquisition step involves acquiring first video data, which is generated by video recording a portion of the object from a first specific location to a second specific location, and which maintains the camera orientation during video recording for each frame; and second video data, which is generated by video recording a portion of the object from a second specific location to an inspection location, and which maintains the camera orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the three-dimensional data of the object, and the camera orientation at the location corresponding to the first specific location on the three-dimensional data is calculated as the first three-dimensional camera orientation. Using the camera pose held in the frame including the second specific location and the first three-dimensional camera pose, the camera pose at the location corresponding to the second specific location on the three-dimensional data is calculated as the second three-dimensional camera pose, and further, A camera pose calculation step, which involves using the camera pose held in the frame containing the inspection location in the second video data and the second three-dimensional camera pose to calculate the camera pose at the location corresponding to the inspection location on the three-dimensional data as the third three-dimensional camera pose, A position identification step in which the location of the location corresponding to the inspection location on the 3D data is identified using the third 3D camera orientation, An information processing method characterized by having the following:

8. In the camera posture calculation step, The difference between the camera pose held in the frame containing the first specific location in the first video data and the camera pose held in the frame containing the second specific location in the first video data is calculated, and the calculated difference is added to the first three-dimensional camera pose to calculate the second three-dimensional camera pose. The difference between the camera pose held in the frame containing the second specific location in the second video data and the camera pose held in the frame containing the inspection location in the second video data is calculated, and the calculated difference is added to the second three-dimensional camera pose to calculate the third three-dimensional camera pose. The information processing method according to claim 7.

9. In the camera posture calculation step, By comparing the feature points of the frame containing the first specific location in the first video data with the feature points of the three-dimensional data, a comparison is performed, thereby identifying a plurality of feature points corresponding to the three-dimensional data and the frame image containing the first specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the first three-dimensional camera pose. The information processing method according to claim 7.

10. The information processing method according to claim 7, wherein in the position identification step, the third three-dimensional camera orientation is used to identify an area included in the camera's field of view of the three-dimensional data, and the position of the identified area is set to the position of the location corresponding to the inspection point on the three-dimensional data.

11. The second video data further contains depth information that identifies the depth from the camera to the object, at least in the frame including the inspection location. In the position identification step, the position of the location corresponding to the inspection location on the three-dimensional data is identified using the third three-dimensional camera orientation and the depth information held in the frame including the inspection location. The information processing method according to claim 7.

12. The system further includes a display step of displaying the three-dimensional data on the screen. In the display step, the screen also displays the area corresponding to the inspection point, superimposed on the three-dimensional data. The information processing method according to claim 7.

13. On the computer, A data acquisition step involves acquiring first video data, which is generated by video recording a portion of the object from a first specific location to a second specific location, and which maintains the camera orientation during video recording for each frame; and second video data, which is generated by video recording a portion of the object from a second specific location to an inspection location, and which maintains the camera orientation during video recording for each frame. The frames containing the first specific location in the first video data are compared with the three-dimensional data of the object, and the camera orientation at the location corresponding to the first specific location on the three-dimensional data is calculated as the first three-dimensional camera orientation. Using the camera pose held in the frame including the second specific location and the first three-dimensional camera pose, the camera pose at the location corresponding to the second specific location on the three-dimensional data is calculated as the second three-dimensional camera pose, and further, A camera pose calculation step, which involves using the camera pose held in the frame containing the inspection location in the second video data and the second three-dimensional camera pose to calculate the camera pose at the location corresponding to the inspection location on the three-dimensional data as the third three-dimensional camera pose, A position identification step in which the location of the location corresponding to the inspection location on the 3D data is identified using the third 3D camera orientation, A program that executes something.

14. In the camera posture calculation step, The difference between the camera pose held in the frame containing the first specific location in the first video data and the camera pose held in the frame containing the second specific location in the first video data is calculated, and the calculated difference is added to the first three-dimensional camera pose to calculate the second three-dimensional camera pose. The difference between the camera pose held in the frame containing the second specific location in the second video data and the camera pose held in the frame containing the inspection location in the second video data is calculated, and the calculated difference is added to the second three-dimensional camera pose to calculate the third three-dimensional camera pose. The program according to claim 13.

15. In the camera posture calculation step, By comparing the feature points of the frame containing the first specific location in the first video data with the feature points of the three-dimensional data, a comparison is performed, thereby identifying a plurality of feature points corresponding to the three-dimensional data and the frame image containing the first specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the first three-dimensional camera pose. The program according to claim 13.

16. The program according to claim 13, wherein in the position identification step, the third three-dimensional camera orientation is used to identify a region included in the camera's field of view of the three-dimensional data, and the position of the identified region is set to the position of the location corresponding to the inspection point on the three-dimensional data.

17. The second video data further contains depth information that identifies the depth from the camera to the object, at least in the frame including the inspection location. In the position identification step, the position of the location corresponding to the inspection location on the three-dimensional data is identified using the third three-dimensional camera orientation and the depth information held in the frame including the inspection location. The program according to claim 13.

18. The system further includes a display step of displaying the three-dimensional data on the screen. In the display step, the screen also displays the area corresponding to the inspection point, superimposed on the three-dimensional data. The program according to claim 13.

Images