Information processing device, information processing method, and program

Abstract

This enables alignment between the location of an image taken of an object and the corresponding location in the 3D data, without being affected by the state of the object's 3D data. [Solution] The information processing device 10 includes a data acquisition unit 11 that acquires video data generated by video recording a specific part of an object from a specific part to an inspection point, and which holds the camera posture at the time of video recording as a first camera posture for each frame; a camera posture calculation unit 12 that compares the frame of the specific part with the 3D data of the object, calculates the camera posture at the specific part on the 3D data as a second camera posture, and uses the first camera posture of the frame of the inspection point and the second camera posture of the specific part on the 3D data to calculate the second camera posture of the inspection point on the 3D data; and a position identification unit 13 that uses the second camera posture of the inspection point on the 3D data to identify the position of the inspection point on the 3D data.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and an information processing method for collating three-dimensional point cloud data of an object and a captured image of the object, and further relates to a program for realizing these.

Background Art

[0002] In recent years, it has been required to efficiently manage infrastructure such as bridges. For this reason, along with the improvement of image processing technology in recent years, a technique for managing deteriorated parts of infrastructure using three-dimensional point cloud data has been proposed (see, for example, Patent Document 1).

[0003] Specifically, Patent Document 1 discloses an apparatus that can display three-dimensional point cloud data of infrastructure on a screen and attach an image captured at the time of inspection to the corresponding part of the three-dimensional point cloud data on the screen. According to the apparatus disclosed in Patent Document 1, a manager can easily grasp deteriorated parts of infrastructure and efficiently manage the infrastructure.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in order to efficiently manage the apparatus disclosed in Patent Document 1, it is necessary to accurately align the photographed part of the image captured at the time of inspection with the corresponding part of the three-dimensional point cloud data. And in order to accurately perform this alignment, it is necessary to construct three-dimensional point cloud data of infrastructure so that there are no missing parts.

[0006] However, because infrastructure structures are enormous, constructing 3D point cloud data without any missing sections requires measurements at a very large number of locations, which takes an enormous amount of time. Therefore, constructing 3D point cloud data of infrastructure structures without any missing sections is extremely difficult.

[0007] One example of the purpose of this disclosure is to enable alignment between the location of an image taken of an object and the corresponding location in the 3D data, without being affected by the state of the object's 3D data. [Means for solving the problem]

[0008] To achieve the above objective, the information processing device in one aspect of this disclosure is: A data acquisition unit acquires video data generated by video recording from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the 3D data, the second camera pose at the location corresponding to the inspection location in the 3D data is calculated. Camera pose calculation unit, A position identification unit that identifies the location of the location corresponding to the inspection location on the 3D data using a second camera orientation at the location corresponding to the inspection location on the 3D data, It is characterized by having the following features.

[0009] Furthermore, in order to achieve the above objectives, the information processing method in one aspect of this disclosure is: The data acquisition step involves acquiring video data, which is generated by video recording the area from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the 3D data, the second camera pose at the location corresponding to the inspection location in the 3D data is calculated. Camera pose calculation step, A position identification step involves using a second camera orientation at a location corresponding to the inspection location on the 3D data to identify the position of the location corresponding to the inspection location on the 3D data, It is characterized by having the following:

[0010] Furthermore, in order to achieve the above objectives, the program in one aspect of this disclosure is On the computer, The data acquisition step involves acquiring video data, which is generated by video recording the area from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the 3D data, the second camera pose at the location corresponding to the inspection location in the 3D data is calculated. Camera pose calculation step, A position specifying step of specifying a position of a location corresponding to the inspection location on the three-dimensional data by using a second camera pose at a location corresponding to the inspection location on the three-dimensional data; which is characterized by causing the above to be executed.

Advantages of the Invention

[0011] As described above, according to the present disclosure, it is possible to align the imaging location of an image of an object with the corresponding location of the three-dimensional data without being affected by the state of the three-dimensional data of the object.

Brief Description of the Drawings

[0012] [Figure 1] FIG. 1 is a configuration diagram showing a schematic configuration of an example of an information processing apparatus. [Figure 2] FIG. 2 is a configuration diagram specifically showing a configuration of an example of an information processing apparatus. [Figure 3] FIG. 3 is a diagram showing an example of a state of shooting video data used in an information processing apparatus. [Figure 4] FIG. 4 is a diagram for explaining an example of processing in a camera pose calculation unit 12. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the information processing apparatus 10. [Figure 6] FIG. 6 is a block diagram showing an example of a computer for realizing an information processing apparatus.

Modes for Carrying Out the Invention

[0013] (Embodiment) Hereinafter, an information processing apparatus, an information processing method, and a program in an embodiment will be described with reference to FIGS. 1 to 6.

[0014] [Device Configuration] First, a schematic configuration of an example of an information processing apparatus will be described using FIG. 1. FIG. 1 is a configuration diagram showing a schematic configuration of an example of an information processing apparatus.

[0015] The information processing device 10 shown in Figure 1 is a device for comparing three-dimensional point cloud data of an object with a captured image of the object, and is an image matching device. As shown in Figure 1, the information processing device 10 includes a data acquisition unit 11, a camera pose calculation unit 12, and a position identification unit 13.

[0016] The data acquisition unit 11 acquires video data of the target object. The video data is generated by filming a video from a specific point on the target object to the inspection point. In addition, the video data retains the camera orientation at the time of video recording as the first camera orientation for each frame.

[0017] The camera pose calculation unit 12 compares frames containing a specific location in the video data with the 3D data of the object. Then, using the results of the comparison, the camera pose calculation unit 12 calculates the camera pose at the location corresponding to the specific location on the 3D data as a second camera pose.

[0018] Furthermore, the camera posture calculation unit 12 uses the first camera posture held in the frame including the inspection point and the second camera posture at the location corresponding to the specific location in the 3D data to calculate the second camera posture at the location corresponding to the inspection point in the 3D data.

[0019] The position identification unit 13 uses the second camera orientation at the location corresponding to the inspection location in the 3D data to identify the location of the location corresponding to the inspection location in the 3D data.

[0020] Thus, in the information processing device 10, the second camera orientation of a location corresponding to an inspection point on the object in the 3D data is calculated from the first camera orientation of the inspection point obtained from the video data and the second camera orientation of the specific location. In other words, even if the data for a location corresponding to an inspection point on the 3D data is missing in the information processing device 10, the second camera orientation of this location can be calculated. Therefore, according to the information processing device 10, it is possible to align the location of the image taken of the object with the corresponding location in the 3D data, without being affected by the state of the object's 3D data.

[0021] Next, we will specifically explain the configuration and functions of an example of the information processing device 10 using Figures 2 to 4. Figure 2 is a configuration diagram specifically showing the configuration of an example of the information processing device. Figure 3 is a diagram showing an example of how video data used in the information processing device is captured.

[0022] As shown in Figure 2, the information processing device 10 includes a display unit 14 in addition to the data acquisition unit 11, camera pose calculation unit 12, and position identification unit 13 described above. Furthermore, the information processing device 10 is connected to the database 20 and the user's terminal device 30 for data communication.

[0023] Database 20 stores 3D data 21 and video data 22 of the object. In this embodiment, the object is a bridge. However, the object is not limited to a bridge; it may be other infrastructure structures, as well as buildings, factories, large tanks, etc.

[0024] Furthermore, the 3D data 21 may be, for example, 3D point cloud data composed of a set of feature points of the object. This 3D point cloud data can be generated, for example, by the Structure from Motion (SfM) method, which utilizes numerous 2D images of the object. Additionally, the 3D point cloud data may be generated by a depth camera (LiDAR, point cloud scanner, etc.).

[0025] Furthermore, as shown in Figure 3, in this embodiment, the video data is obtained by the photographer 52 using the imaging device 51 to film a portion of the bridge deck 41 from the bridge pier 42 of the bridge 40. In the example shown in Figure 3, a smartphone with a camera is used as the imaging device 51.

[0026] The imaging device 51, a smartphone, is equipped with various sensors, including an IMU (Inertial Measurement Unit). During video recording, the smartphone calculates the camera orientation for each frame using sensor data output from the IMU. The camera orientation may also be calculated as VIO (Visual-Inertial Odometry), which is obtained by combining the sensor data output from the IMU with the image analysis results for each frame. The smartphone then adds information to each frame that identifies the specified camera orientation (first camera orientation). The first camera orientation is relatively associated as an external parameter for each frame of the camera that recorded the video.

[0027] In the example in Figure 3, the inspection point is a part of the bottom surface of the deck slab 41 (marked with a ★ in Figure 3). The specific location is set on a part of the bridge pier of the bridge 40 (marked with a ▲ in Figure 3). The specific location is any location in the 3D point cloud data that is not missing. In addition, specific markings may be added to the specific location in advance. Furthermore, since the specific location can be anywhere in the 3D point cloud data that is not missing, accuracy may be improved by setting multiple specific locations, calculating the position corresponding to the inspection point from each specific location, and averaging these multiple calculated positions.

[0028] As shown in Figure 3, the photographer 52 uses a smartphone, which is the imaging device 51, to film a video from a specific point on the bridge pier 42 to the inspection point on the deck slab 41. This generates video data. The generated video data is transmitted from the smartphone, which is the imaging device 51, to the database 20.

[0029] In this embodiment, the data acquisition unit 11 acquires 3D data 21 and video data 22 of the object from the database 20. The data acquisition unit 11 also outputs the acquired 3D data 21 and video data 22 to the camera pose calculation unit 12.

[0030] In this embodiment, the camera pose calculation unit 12 first compares the feature points of a frame containing a specific location in the video data with the feature points of the 3D data to identify the region corresponding to the specific location in the 3D data.

[0031] Specifically, the camera pose calculation unit 12 first calculates features such as Haar-Like features, HOG features, and SIFT features in frames containing a specific location. Next, the camera pose calculation unit 12 extracts points where the value of the feature is greater than or equal to a predetermined value as feature points. The frame containing the specific location is specified in advance by the user. Alternatively, the frame containing the specific location may be identified by extracting feature points from all frames and then searching for feature points of the specified feature.

[0032] Furthermore, the camera pose calculation unit 12 performs matching between the feature points of the frame containing the specific location and each point that constitutes the 3D point cloud data, which is the 3D data 21. Existing methods are used for the matching process between feature points. Based on the matching results, the camera pose calculation unit 12 identifies multiple feature points corresponding to the 3D data 21 and the frame image containing the specific location. Note that the feature points are identified in both the 3D data 21 and the frame image and correspond to each other. Feature points that correspond to each other will be referred to as "corresponding points" below.

[0033] Next, the camera pose calculation unit 12 uses the identified multiple corresponding points to calculate the camera pose (second camera pose) at the location in the 3D data 21 that includes these multiple corresponding points (corresponding to the location corresponding to the identified location). The second camera pose is the camera pose in the 3D data 21 as a world coordinate system. Specifically, the camera pose calculation unit 12 uses the identified multiple corresponding points and the intrinsic camera parameters at the time of shooting the frame containing the identified location to calculate the extrinsic parameters at the time of shooting the frame containing the identified location.

[0034] Next, the camera pose calculation unit 12 identifies from the video data 22 a first camera pose held in a frame containing a specific location and a first camera pose held in a frame containing an inspection location. The frame containing the inspection location is specified in advance by the user. Furthermore, the camera pose calculation unit 12 calculates the difference between the first camera pose held in the frame containing the specific location and the first camera pose held in the frame containing the inspection location.

[0035] The camera pose calculation unit 12 then adds the calculated difference to the second camera pose at a location corresponding to a specific point on the 3D data. The resulting second camera pose corresponds to the second camera pose at a location corresponding to an inspection point on the 3D data. Figure 4 illustrates an example of the processing in the camera pose calculation unit 12.

[0036] In this embodiment, the positioning unit 13 uses a second camera orientation (external parameter) at the location corresponding to the inspection point on the 3D data and the 3D data 21 to calculate the coordinates of the location corresponding to the inspection point on the 3D data. Specifically, the positioning unit 13 uses the second camera orientation at the location corresponding to the inspection point on the 3D data to set an area that can be included in the camera's field of view, and defines the area where this area intersects with the 3D data 21 as the area corresponding to the inspection point on the 3D data.

[0037] Furthermore, the position identification unit 13 uses the coordinates of the locations corresponding to the calculated inspection locations to identify the image or point cloud corresponding to the inspection locations on the 3D data, and outputs information indicating the identified image or point cloud to the display unit 14. In this embodiment, even if a portion of the location corresponding to the inspection location is missing in the 3D data, the position obtained by advancing from the camera position determined by the second camera orientation by a distance tangent to the 3D data in the direction of the camera orientation is considered to be the position on the 3D data.

[0038] The display unit 14 displays the 3D data of the object on, for example, the screen of the terminal device 30. The display unit also uses the information output by the position identification unit 13 to display the locations corresponding to the inspection points on the screen, superimposed on the 3D data.

[0039] [Device operation] Next, an example of the operation of the information processing device 10 will be explained using Figure 5. Figure 5 is a flowchart showing an example of the operation of the information processing device 10. In the following explanation, Figures 1 to 4 will be referred to as appropriate. In this embodiment, the information processing method is implemented by operating the information processing device 10. Therefore, the explanation of the information processing method in this embodiment will be replaced by the following explanation of the operation of the information processing device 10.

[0040] First, as a prerequisite, 3D data 21 of the object is constructed, and this constructed 3D data 21 is stored in the database 20. In addition, video data of the inspection area filmed from a specific point on the object is also stored in the database 20.

[0041] As shown in Figure 5, first, the data acquisition unit 11 acquires 3D data 21 and video data 22 of the object from the database 20 (step A1). The data acquisition unit 11 also outputs the acquired 3D data 21 and video data 22 to the camera pose calculation unit 12.

[0042] Next, the camera pose calculation unit 12 compares the feature points of the frame containing the specific location in the video data 21 with the feature points of the 3D data to identify multiple corresponding points for both the 3D data and the frame image containing the specific location (Step A2).

[0043] Specifically, in step A2, the camera pose calculation unit 12 first extracts feature points from the frame containing the specific location, and then performs matching between the extracted feature points and each point that makes up the 3D data 21. Based on the matching results, the camera pose calculation unit 12 then identifies multiple corresponding points on the 3D data 21 and in each of the frame images containing the specific location.

[0044] Next, the camera pose calculation unit 12 uses the multiple corresponding points identified in step A2 to calculate the camera pose (second camera pose) at the location on the 3D data 21 that includes these multiple corresponding points (corresponding to the location corresponding to the identified location) (step A3).

[0045] Specifically, in step A3, the camera pose calculation unit 12 uses the multiple corresponding points identified in step A2 and the camera's internal parameters at the time of shooting the frame containing the specific location to calculate the camera's external parameters (second camera pose) at the time of shooting the frame containing the specific location.

[0046] Next, the camera pose calculation unit 12 uses the first camera pose held in the frame including the inspection point and the second camera pose calculated in step A3 to calculate the second camera pose at the location corresponding to the inspection point on the 3D data (step A4).

[0047] Specifically, in step A4, as shown in Figure 4, the camera pose calculation unit 12 identifies a first camera pose held in a frame containing a specific location and a first camera pose held in a frame containing an inspection location from the video data 22. Furthermore, the camera pose calculation unit 12 calculates the difference between the two identified first camera poses. Then, the camera pose calculation unit 12 adds the calculated difference to the second camera pose at the location corresponding to the specific location on the 3D data to calculate the second camera pose at the location corresponding to the inspection location on the 3D data.

[0048] Next, the position identification unit 13 uses the second camera orientation at the location corresponding to the inspection location on the 3D data, calculated in step A4, to identify the location of the location corresponding to the inspection location on the 3D data (step A5).

[0049] Specifically, in step A5, the positioning unit 13 uses the second camera orientation calculated in step A4 and the 3D data 21 to calculate the coordinates of the location corresponding to the inspection point on the 3D data. The positioning unit 13 also uses the calculated coordinates to identify the image or point cloud corresponding to the inspection point on the 3D data and outputs information indicating the identified image or point cloud to the display unit 14.

[0050] Next, the display unit 14 displays the 3D data of the object on the screen of the terminal device 30, and also displays the location identified in step A5 superimposed on the 3D data (step A6).

[0051] [Effects in the embodiment] Thus, in this embodiment, the second camera orientation of a location corresponding to an inspection point on the object in the 3D data is calculated from the two first camera orientations obtained from the video data and the second camera orientation of the specific location. Therefore, even if the data for a location corresponding to an inspection point on the 3D data is missing, the second camera orientation for that location can be calculated. Thus, according to this embodiment, it is possible to align the location of the image taken of the object with the corresponding location in the 3D data, without being affected by the state of the object's 3D data.

[0052] [Differentiation] Here, we will describe three modified examples of the embodiment below.

[0053] Variation 1: In Modification 1, 3D interpolation is performed on the 3D data 21, and the interpolated 3D data 21 is stored in the database 20. 3D interpolation is performed to add faces to the parts of the data that are missing.

[0054] One method of 3D interpolation is to use a machine learning model. In this case, the machine learning model is constructed by using a partially missing 3D model and complete 3D data (training data) through machine learning. Another method of 3D interpolation is to assume that the missing portion is a continuous surface from its vicinity and then interpolate the data.

[0055] In the first modified example, the positioning unit 13 first uses a second camera orientation at a location corresponding to the inspection point on the 3D data 21 to identify an area of ​​the 3D data 21 that may be included in the camera's field of view. Then, the positioning unit 13 identifies the area that intersects with the 3D data from the identified area as the location of the inspection point.

[0056] In the case of Modification 1, the location of the area corresponding to the inspection point on the 3D data can be identified with even greater accuracy.

[0057] Variation 2: In Modification 2, the video data stores, for each frame, depth information that identifies the depth from the camera to the object, in addition to the camera's first camera orientation. In Modification 2, the imaging device 51 is equipped with a depth camera such as LiDAR in addition to a normal camera, and measures the depth to the subject each time a shot is taken. In Modification 2, depth information to the subject is only required for the frame shot that includes the inspection point.

[0058] Accordingly, in the modified example 2, the position identification unit 13 uses the second camera orientation at the location corresponding to the inspection location on the 3D data and the depth information held in the frame containing the inspection location to identify the location of the location corresponding to the inspection location on the 3D data.

[0059] In the case of Modification 2, the location of the area corresponding to the inspection point on the 3D data can be identified with even greater accuracy.

[0060] Variation 3: In Modification 3, the depth information used in Modification 2 is calculated from multiple frame images including the inspection point. In this case, the relative shooting position of each frame image is known from the sensor data from the IMU. Therefore, depth information for the frame including the inspection point can be obtained using the principle of triangulation, and the coordinates of the location corresponding to the inspection point on the 3D data can be calculated.

[0061] In the case of Modification 3, the location of the area corresponding to the inspection point on the 3D data can be identified with even greater accuracy.

[0062] [program] The program in this embodiment can be any program that causes a computer to execute steps A1 to A6 shown in Figure 5. By installing and running this program on a computer, the information processing device 10 and the information processing method can be realized. In this case, the computer's processor functions as the data acquisition unit 11, camera pose calculation unit 12, position identification unit 13, and display unit 14, and performs the processing. In addition to general-purpose PCs and server computers, smartphones and tablet terminal devices can also be used as computers.

[0063] Furthermore, the program in the embodiment may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: a data acquisition unit 11, a camera pose calculation unit 12, a position identification unit 13, and a display unit 14.

[0064] [Physical configuration] Here, a computer that implements the information processing device 10 by executing the program in the embodiment will be described with reference to Figure 6. Figure 6 is a block diagram showing an example of a computer that implements the information processing device.

[0065] As shown in Figure 6, the computer 110 comprises a CPU (Central Processing Unit) 111, main memory 112, storage device 113, input interface 114, display controller 115, data reader / writer 116, and communication interface 117. Each of these components is connected to the others via a bus 121, enabling data communication.

[0066] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to, or instead of, the CPU 111. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

[0067] The CPU 111 loads the program in the embodiment, which consists of a set of codes stored in the storage device 113, into the main memory 112, and performs various calculations by executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

[0068] Furthermore, the program in this embodiment is provided stored on a computer-readable recording medium 120. The program in this embodiment may also be distributed over the internet via a communication interface 117.

[0069] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0070] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0071] Furthermore, specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0072] Furthermore, the information processing device 10 can be implemented not only by a computer on which a program is installed, but also by using hardware corresponding to each part, such as electronic circuits. Moreover, the information processing device 10 may be partially implemented by a program and the remaining part by hardware. In the embodiment, the computer is not limited to the computer shown in Figure 6.

[0073] Some or all of the embodiments described above can be expressed by (Appendix 1) to (Appendix 18) described below, but are not limited to the following descriptions.

[0074] (Note 1) A data acquisition unit acquires video data generated by video recording from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the 3D data, the second camera pose at the location corresponding to the inspection location in the 3D data is calculated. Camera pose calculation unit, A position identification unit that identifies the location of the location corresponding to the inspection location on the 3D data using a second camera orientation at the location corresponding to the inspection location on the 3D data, An information processing device characterized by having the following features.

[0075] (Note 2) The camera posture calculation unit calculates the difference between the first camera posture held in the frame including the specific location and the first camera posture held in the frame including the inspection location. Then, the calculated difference is added to the second camera posture at the location corresponding to the specific location on the 3D data to calculate the second camera posture at the location corresponding to the inspection location on the 3D data. The information processing device described in Appendix 1.

[0076] (Note 3) The camera pose calculation unit compares feature points of the frame containing the specific location in the video data with feature points of the 3D data to perform a match, thereby identifying a plurality of feature points corresponding to the 3D data and the frame image containing the specific location, and calculates the camera pose at the location containing the identified plurality of feature points as the second camera pose. The information processing device described in Appendix 1.

[0077] (Note 4) The information processing apparatus according to Appendix 1, wherein the position identification unit uses a second camera orientation at a location corresponding to the inspection location on the three-dimensional data to identify an area included in the camera's field of view of the three-dimensional data, and sets the position of the identified area to the position of the location corresponding to the inspection location on the three-dimensional data.

[0078] (Note 5) The video data further contains depth information that identifies the depth from the camera to the object, at least in the frame containing the inspection location. The position identification unit identifies the position of the location corresponding to the inspection location on the 3D data using the second camera orientation at the location corresponding to the inspection location on the 3D data and the depth information held in the frame including the inspection location. The information processing device described in Appendix 1.

[0079] (Note 6) The system further includes a display unit that displays the aforementioned 3D data on the screen. The display unit also displays the locations corresponding to the inspection locations superimposed on the three-dimensional data on the screen. The information processing device described in Appendix 1.

[0080] (Note 7) The data acquisition step involves acquiring video data, which is generated by video recording the area from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the 3D data, the second camera pose at the location corresponding to the inspection location in the 3D data is calculated. Camera pose calculation step, A position identification step involves using a second camera orientation at a location corresponding to the inspection location on the 3D data to identify the position of the location corresponding to the inspection location on the 3D data, An information processing method characterized by having the following:

[0081] (Note 8) In the camera posture calculation step, the difference between the first camera posture held in the frame including the specific location and the first camera posture held in the frame including the inspection location is calculated. Then, the calculated difference is added to the second camera posture at the location corresponding to the specific location on the 3D data to calculate the second camera posture at the location corresponding to the inspection location on the 3D data. The information processing method described in Appendix 7.

[0082] (Note 9) In the camera pose calculation step, a comparison is made by comparing the feature points of the frame containing the specific location in the video data with the feature points of the 3D data, thereby identifying a plurality of feature points corresponding to the 3D data and the frame image containing the specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the second camera pose. The information processing method described in Appendix 7.

[0083] (Note 10) The information processing method according to Appendix 7, wherein in the position identification step, a second camera orientation at a location corresponding to the inspection location on the three-dimensional data is used to identify a region included in the camera's field of view of the three-dimensional data, and the position of the identified region is set to the position of the location corresponding to the inspection location on the three-dimensional data.

[0084] (Note 11) The video data further contains depth information that identifies the depth from the camera to the object, at least in the frame containing the inspection location. In the position identification step, the position of the location corresponding to the inspection location on the 3D data is identified using the second camera orientation at the location corresponding to the inspection location on the 3D data and the depth information held in the frame including the inspection location. The information processing method described in Appendix 7.

[0085] (Note 12) The system further includes a display step of displaying the aforementioned 3D data on the screen, In the display step, the screen also displays the area corresponding to the inspection point, superimposed on the 3D data. The information processing method described in Appendix 7.

[0086] (Note 13) On the computer, The data acquisition step involves acquiring video data, which is generated by video recording the area from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the 3D data, the second camera pose at the location corresponding to the inspection location in the 3D data is calculated. Camera pose calculation step, A position identification step involves using a second camera orientation at a location corresponding to the inspection location on the 3D data to identify the position of the location corresponding to the inspection location on the 3D data, A program that executes something.

[0087] (Note 14) In the camera posture calculation step, the difference between the first camera posture held in the frame including the specific location and the first camera posture held in the frame including the inspection location is calculated. Then, the calculated difference is added to the second camera posture at the location corresponding to the specific location on the 3D data to calculate the second camera posture at the location corresponding to the inspection location on the 3D data. The program described in Appendix 13.

[0088] (Note 15) In the camera pose calculation step, a comparison is made by comparing the feature points of the frame containing the specific location in the video data with the feature points of the 3D data, thereby identifying a plurality of feature points corresponding to the 3D data and the frame image containing the specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the second camera pose. The program described in Appendix 13.

[0089] (Note 16) The program as described in Appendix 13, wherein in the position identification step, a second camera orientation at a location corresponding to the inspection location on the 3D data is used to identify a region included in the camera's field of view of the 3D data, and the position of the identified region is set to the position of the location corresponding to the inspection location on the 3D data.

[0090] (Note 17) The video data further contains depth information that identifies the depth from the camera to the object, at least in the frame containing the inspection location. In the position identification step, the position of the location corresponding to the inspection location on the 3D data is identified using the second camera orientation at the location corresponding to the inspection location on the 3D data and the depth information held in the frame including the inspection location. The program described in Appendix 13.

[0091] (Note 18) The system further includes a display step of displaying the aforementioned 3D data on the screen, In the display step, the screen also displays the area corresponding to the inspection point, superimposed on the 3D data. The program described in Appendix 13. [Industrial applicability]

[0092] As described above, this disclosure makes it possible to align the location of an image taken of an object with the corresponding location in the 3D data, without being affected by the state of the object's 3D data. This disclosure is useful in fields where matching 3D data with images is required, such as the management of infrastructure structures. [Explanation of Symbols]

[0093] 10 Information Processing Devices 11 Data Acquisition Unit 12 Camera posture calculation unit 13 Location identification part 14 Display section 20 Databases 21 3D data 22 Video Data 30 Terminal devices 40 Bridges 41 Floor slab 42 Bridge piers 51 Imaging device 52 Photographer 110 Computer 111 CPU 112 Main Memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Readers / Writers 117 Communication Interface 118 Input devices 119 Display device 120 recording media 121 Bus

Claims

1. A data acquisition unit acquires video data generated by video recording the area from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the three-dimensional data, the second camera pose at the location corresponding to the inspection location in the three-dimensional data is calculated. Camera pose calculation unit, A position identification unit that identifies the location of the location corresponding to the inspection location on the 3D data using a second camera orientation at the location corresponding to the inspection location on the 3D data, An information processing device characterized by having the following features.

2. The camera posture calculation unit calculates the difference between the first camera posture held in the frame including the specific location and the first camera posture held in the frame including the inspection location. Then, the calculated difference is added to the second camera pose at the location corresponding to the specific location on the three-dimensional data to calculate the second camera pose at the location corresponding to the inspection location on the three-dimensional data. The information processing apparatus according to claim 1.

3. The camera pose calculation unit compares feature points of the frame containing the specific location in the video data with feature points of the 3D data to perform a match, thereby identifying a plurality of feature points corresponding to the 3D data and the frame image containing the specific location, and calculates the camera pose at the location containing the identified plurality of feature points as the second camera pose. The information processing apparatus according to claim 1.

4. The information processing apparatus according to claim 1, wherein the position identification unit uses a second camera orientation at a location corresponding to the inspection location on the three-dimensional data to identify a region included in the camera's field of view of the three-dimensional data, and sets the position of the identified region to the position of the location corresponding to the inspection location on the three-dimensional data.

5. The video data further contains depth information that identifies the depth from the camera to the object, at least in the frame containing the inspection location. The position identification unit identifies the position of the location corresponding to the inspection location on the three-dimensional data using the second camera orientation at the location corresponding to the inspection location on the three-dimensional data and the depth information held in the frame including the inspection location. The information processing apparatus according to claim 1.

6. The system further includes a display unit that displays the aforementioned three-dimensional data on the screen. The display unit also displays the locations corresponding to the inspection locations superimposed on the three-dimensional data on the screen. The information processing apparatus according to claim 1.

7. The data acquisition step involves acquiring video data, which is generated by video recording the area from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the three-dimensional data, the second camera pose at the location corresponding to the inspection location in the three-dimensional data is calculated. Camera pose calculation step, A position identification step involves using a second camera orientation at a location corresponding to the inspection location on the three-dimensional data to identify the position of the location corresponding to the inspection location on the three-dimensional data, An information processing method characterized by having the following:

8. In the camera posture calculation step, the difference between the first camera posture held in the frame including the specific location and the first camera posture held in the frame including the inspection location is calculated. Then, the calculated difference is added to the second camera pose at the location corresponding to the specific location on the three-dimensional data to calculate the second camera pose at the location corresponding to the inspection location on the three-dimensional data. The information processing method according to claim 7.

9. In the camera pose calculation step, a comparison is performed by comparing the feature points of the frame containing the specific location in the video data with the feature points of the 3D data, thereby identifying a plurality of feature points corresponding to the 3D data and the frame image containing the specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the second camera pose. The information processing method according to claim 7.

10. The information processing method according to claim 7, wherein in the position identification step, a second camera orientation at a location corresponding to the inspection location on the three-dimensional data is used to identify a region included in the camera's field of view of the three-dimensional data, and the position of the identified region is set to the position of the location corresponding to the inspection location on the three-dimensional data.

11. The video data further contains depth information that identifies the depth from the camera to the object, at least in the frame containing the inspection location. In the position identification step, the position of the location corresponding to the inspection location on the three-dimensional data is identified using the second camera orientation at the location corresponding to the inspection location on the three-dimensional data and the depth information held in the frame including the inspection location. The information processing method according to claim 7.

12. The system further includes a display step of displaying the three-dimensional data on the screen, In the display step, the screen also displays the area corresponding to the inspection point, superimposed on the three-dimensional data. The information processing method according to claim 7.

13. On the computer, The data acquisition step involves acquiring video data, which is generated by video recording the area from a specific point on the object to the inspection point, and which maintains the camera orientation at the time of video recording as the first camera orientation for each frame. The frames containing the specific location in the video data are compared with the 3D data of the object, and the camera orientation at the location corresponding to the specific location on the 3D data is calculated as a second camera orientation. Using the first camera pose held in the frame including the inspection location and the second camera pose at the location corresponding to the specific location in the three-dimensional data, the second camera pose at the location corresponding to the inspection location in the three-dimensional data is calculated. Camera pose calculation step, A position identification step involves using a second camera orientation at a location corresponding to the inspection location on the three-dimensional data to identify the position of the location corresponding to the inspection location on the three-dimensional data, A program that executes something.

14. In the camera posture calculation step, the difference between the first camera posture held in the frame including the specific location and the first camera posture held in the frame including the inspection location is calculated. Then, the calculated difference is added to the second camera pose at the location corresponding to the specific location on the three-dimensional data to calculate the second camera pose at the location corresponding to the inspection location on the three-dimensional data. The program according to claim 13.

15. In the camera pose calculation step, a comparison is performed by comparing the feature points of the frame containing the specific location in the video data with the feature points of the 3D data, thereby identifying a plurality of feature points corresponding to the 3D data and the frame image containing the specific location, and the camera pose at the location containing the identified plurality of feature points is calculated as the second camera pose. The program according to claim 13.

16. The program according to claim 13, wherein in the position identification step, a second camera orientation at a location corresponding to the inspection location on the three-dimensional data is used to identify a region included in the camera's field of view of the three-dimensional data, and the position of the identified region is set to the position of the location corresponding to the inspection location on the three-dimensional data.

17. The video data further contains depth information that identifies the depth from the camera to the object, at least in the frame containing the inspection location. In the position identification step, the position of the location corresponding to the inspection location on the three-dimensional data is identified using the second camera orientation at the location corresponding to the inspection location on the three-dimensional data and the depth information held in the frame including the inspection location. The program according to claim 13.

18. The system further includes a display step of displaying the three-dimensional data on the screen, In the display step, the screen also displays the area corresponding to the inspection point, superimposed on the three-dimensional data. The program according to claim 13.

Images