Information processing device, information processing method and program

Abstract

To precisely calculate a camera position.SOLUTION: An information processing device has: an image generation part which generates a projection image corresponding to photography from a camera position and a depth image based upon the camera position of a query image obtained by photographing a scene by a camera from the camera position and three-dimensional information on the scene; and a camera position correction part which corrects the camera position based upon correspondence relation between the query image and the projection image, and the depth image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program that estimate a camera position. [Background technology]

[0002] There is a technology for estimating the three-dimensional information of a camera from an image in order to estimate the self-position of a robot and perform three-dimensional analysis of a subject. With this technology, if the internal parameters of the camera (e.g., focal length, lens distortion, etc.) and the three-dimensional coordinates that constitute the captured scene are both known, the rotation matrix and translation vector of the camera can be estimated. In the following, the combination of the rotation matrix and translation vector of the camera will be simply referred to as the camera position.

[0003] One method for estimating the camera position from an image is to solve the PnP (Perspective-n-point) problem. To solve the PnP problem, first, a 3D point cloud of the scene is acquired in advance using a 3D (3 Dimension) scanner. Next, feature points are calculated from an image of the scene captured by a camera, and corresponding 3D points (2D-3D corresponding points) are determined. Finally, the PnP problem is solved using the given multiple 2D-3D corresponding points, and the camera position is estimated. Patent Document 1 describes a method for obtaining a global optimal solution to the PnP problem. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2012 / 157342 Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional camera position estimation method, when the lighting conditions change or the 3D points contain large errors, it is difficult to stably and accurately estimate the camera position. Note that the method of Patent Document 1 may not be able to accurately estimate the camera position.

[0006] An example of an objective of the present disclosure is to accurately calculate a camera position. [Means for solving the problem]

[0007] In order to achieve the above object, an information processing device according to one aspect of the present disclosure includes: an image generating unit that generates a projection image corresponding to the shooting from the camera position and a depth image based on a camera position for a query image obtained by shooting a scene from the camera position and three-dimensional information about a front scene; a camera position correction unit that corrects the camera position based on a correspondence relationship between the query image and the projection image and the depth image; The present invention is characterized by having the following.

[0008] In order to achieve the above object, an information processing method according to one aspect of the present disclosure includes: An information processing device, Based on a camera position for a query image obtained by photographing a scene from a camera position and three-dimensional information about a previous scene, a projection image corresponding to the photographing from the camera position and a depth image are generated; modifying the camera position based on the correspondence between the query image and the projection image and the depth image; It is characterized by:

[0009] Furthermore, in order to achieve the above object, a program according to one aspect of the present disclosure includes: On the computer, Based on a camera position for a query image obtained by photographing a scene from a camera position and three-dimensional information about a previous scene, a projection image corresponding to the photographing from the camera position and a depth image are generated; correcting the camera position based on the correspondence between the query image and the projection image and the depth image; It is characterized by: Effect of the Invention

[0010] According to the present disclosure, the camera position can be calculated with high accuracy. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a system having an information processing device. [Diagram 3] FIG. 3 is a diagram for explaining an example of learning a depth image correction model. [Figure 4] FIG. 4 is a diagram illustrating an example of the operation of the information processing device. [Diagram 5] FIG. 5 is a diagram for explaining the first modification. [Figure 6] FIG. 6 is a diagram illustrating an example of a computer that realizes the information processing device in the embodiment and the first modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment will be described with reference to the drawings. In the drawings described below, elements having the same or corresponding functions are denoted by the same reference numerals, and repeated description thereof may be omitted. (Embodiment) The configuration of an information processing device will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an example of an information processing device.

[0013] [Device configuration] 1 is a device that calculates a camera position (a rotation matrix and a translation vector) with high accuracy. As shown in FIG. 1, the information processing device 10 includes an image generating unit 11 and a camera position correcting unit 12.

[0014] The image generation unit 11 generates a projection image corresponding to the shooting from the camera position and a depth image based on the camera position for a query image obtained by shooting a scene from the camera position and three-dimensional information about the previous scene.

[0015] The camera position is information that represents the rotation matrix and translation vector of the camera, estimated using the internal parameters of the camera (for example, focal length, lens distortion, etc.) and the three-dimensional coordinates that constitute the scene.

[0016] A query image is an image obtained by capturing a scene with a camera, and is an input image used to estimate the camera position. The input image may be a color image represented by RGB values, or a grayscale image represented only by brightness values.

[0017] The three-dimensional information is, for example, implicit three-dimensional information. The implicit three-dimensional information is information that represents three dimensions using a nonlinear implicit function such as NeRF (Neural Radiance Fields) or SRN (Scene Representation Network). The three-dimensional information also has the same color representation (RGB or grayscale) as the query image in addition to the three-dimensional shape information of the scene.

[0018] The camera position correction unit 12 corrects the camera position based on the correspondence between the query image and the projected image, and on the depth image.

[0019] In this manner, in the embodiment, the camera position is corrected using the corrected depth image and the first corresponding point, so that the camera position can be calculated with high accuracy.

[0020] [System Configuration] The configuration of the information processing device 10 will be described more specifically with reference to Fig. 2. Fig. 2 is a diagram for explaining an example of a system having an information processing device.

[0021] 2, the system 100 in the embodiment includes an information processing device 10, a camera 20, and a storage device 30. The information processing device 10 includes a camera (imaging device) 20, a storage device 30, an input device 40, and an output device 50, which are electrically connected via a network.

[0022] The network is a general network constructed using communication lines such as the Internet, a LAN (Local Area Network), a dedicated line, a telephone line, an in-house network, a mobile communication network, Bluetooth (registered trademark), or Wi-Fi (Wireless Fidelity) (registered trademark).

[0023] The information processing device 10 is, for example, an information processing device such as a CPU (Central Processing Unit), a programmable device such as an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a circuit equipped with any one or more of them, a server computer, a personal computer, a mobile terminal, or the like.

[0024] The camera 20 outputs images captured in time series to the information processing device 10. The camera 20 may be, for example, a monocular camera (for example, a wide-angle camera, a fisheye camera, a celestial sphere camera, etc.), a compound eye camera (for example, a stereo camera, a multi-camera, etc.), an RGB-D camera (for example, a depth camera, a ToF camera, etc.), etc. The camera 20 may be provided in the information processing device 10.

[0025] The storage device 30 is a database, a server computer, a circuit having a memory, etc. In the example of FIG. 2, the storage device 30 is provided outside the information processing device 10, but may be provided inside the information processing device 10.

[0026] The input device 40 is, for example, a device (user interface) such as a touch panel, a mouse, a keyboard, etc. The input device 40 may be provided in the information processing device 10.

[0027] The output device 50 acquires output information, which will be described later, that has been converted into an outputtable format, and outputs generated images and sounds based on the output information. The output device 50 is, for example, an image display device using a liquid crystal, an organic EL (Electro Luminescence), or a CRT (Cathode Ray Tube). Furthermore, the image display device may include an audio output device such as a speaker. The output device 50 may be a printing device such as a printer. The output device 50 may be provided in the information processing device 10.

[0028] The information processing device will now be described in detail. The information processing device 10 includes an image generating unit 11 , a camera position correcting unit 12 , a depth image correcting unit 13 , a detecting unit 14 , and an output information generating unit 15 .

[0029] The image generation unit 11 uses a camera position initial value representing the initial state of the camera position of the query image and three-dimensional information of the scene captured by the camera to project the three-dimensional information onto an image plane based on the camera position, and generates a projected image and a depth image.

[0030] The initial camera position may be, for example, a random rotation matrix and translation vector, or the origin of the scene's 3D coordinate system.

[0031] In order to automatically obtain the camera position as accurately as possible, existing methods such as VLAD (Locally Aggregated Descriptor) and BoW (Bag of word) may be used.

[0032] The projected image is an image in which the 3D information is projected onto the image plane of the camera based on the input camera position, and has the same color representation as the query image.

[0033] A depth image is an image in which each pixel of the image has a depth value in a coordinate system with the camera as the origin. Note that the depth value may be directly stored, or the inverse of the depth value may be stored.

[0034] Specifically, the image generating unit 11 first acquires an initial camera position value of a query image for estimating a camera position and three-dimensional information about a scene captured by the camera from the storage device 30. Next, the image generating unit 11 uses the initial camera position value and the three-dimensional information to project the three-dimensional information onto an image plane based on the camera position to generate a projection image and a depth image.

[0035] The depth image correction unit 13 corrects errors in the depth image by performing a depth image correction process using the projection image and the depth image.

[0036] Specifically, the depth image correction unit 13 first acquires a projection image and a depth image. Next, the depth image correction unit 13 corrects an error included in the depth image by a depth image correction process using the projection image and the depth image.

[0037] Errors are inaccurate depth values ​​that occur in the generated depth image, also known as artifacts.

[0038] The depth image correction process is a process of correcting a depth image based on a projection image. Specifically, the depth image correction process is a process of correcting a depth image using a depth image correction model that corrects an error contained in the depth image using, for example, a neural network.

[0039] The depth image correction process is not limited to a process using a neural network, and may be a machine learning method such as a support vector machine or a random forest. The process of correcting the depth image is not limited to a machine learning method, and may be a process of correcting the depth image according to a correction rule.

[0040] The depth image correction model is a machine learning model that receives the generated projection image and depth image as input and outputs both the noise-removed projection image and depth image, or the depth image.

[0041] Learning of the depth image correction model will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining an example of learning of the depth image correction model.

[0042] 3, in the learning process, first, various scenes captured in the past by the camera 20 are obtained from the storage device 30. Next, noise is added to each scene using computer graphics.

[0043] If the sensor noise distribution of the camera is known, the noise is a random value based on that distribution. If the sensor noise distribution is unknown, the noise is substituted with a random value based on a Gaussian distribution.

[0044] Next, we generate multiple true-value pairs by adding noise to each of the projection image and the depth image, and use them as training data. Next, we input the multiple training data to a depth image correction model that removes noise from the projection image and the depth image, and perform supervised learning.

[0045] It is not necessary to use computer graphics. For example, multiple projection images and depth images learned using NeRF for each of various scenes may be generated, and correct pairs of projection images and depth images corresponding to each scene may be generated.

[0046] The detection unit 14 detects a first image feature point included in the query image and a second image feature point included in the projection image, and detects first corresponding points (2D-2D corresponding points) where the first image feature point and the second image feature point correspond to each other.

[0047] The 2D-2D corresponding points are information that represents a set of matched first image feature points and second image feature points. As a method for acquiring the 2D-2D corresponding points, for example, a method that is robust to illumination variations, such as SIFT (Scale Invariant Feature Transform) or SuperPoint, is used.

[0048] The 2D-2D corresponding points may be manually specified by the user, or may be obtained by a method that accepts removal of erroneous corresponding points, such as RANSAC (Random Sample Consensus).

[0049] Specifically, the detection unit 14 first acquires a query image and a projection image. Next, the detection unit 14 detects a first image feature point included in the query image and a second image feature point included in the projection image. Next, the detection unit 14 detects 2D-2D corresponding points (first corresponding points) where the detected first image feature point and second image feature point correspond to each other.

[0050] The camera position correction unit 12 first obtains a corrected depth image and a 2D-2D corresponding point (first corresponding point). Next, the camera position correction unit 12 calculates a correction value for correcting the camera position by a camera position correction process using the corrected depth image and the 2D-2D corresponding point.

[0051] In the camera position correction process, first, the corrected depth image and the 2D-2D corresponding points (first corresponding points) are used to obtain the 2D-3D corresponding points (second corresponding points) between the query image and the projection image. Specifically, each pixel of the projection image is associated with the depth value of each pixel of the corrected depth image. That is, the 2D-3D corresponding points between the query image and the projection image are obtained.

[0052] Next, the 2D-3D corresponding points (second corresponding points) are used to calculate a correction value by solving a PnP (Perspective-n-point) problem. Specifically, an existing solution to the PnP problem is used to calculate the camera position of the query image in a camera coordinate system with the projection image as the origin. Here, the calculated camera position is equivalent to the difference (correction value) of the camera position from the projection image to the query image. Therefore, when the correction value (difference) is small, the difference of the rotation matrix approaches a unit matrix, and the difference of the translation vector approaches a zero vector.

[0053] Next, if the correction value is greater than a preset threshold, the camera position correction unit 12 corrects the camera position using the correction value. Specifically, the camera position correction unit 12 adds the correction value to the initial camera position or the current camera position.

[0054] Next, the camera position correction unit 12 outputs the corrected camera position to the image generation unit 11. On the other hand, if the correction value is equal to or less than the threshold, the corrected camera position is adopted. In other words, the process is repeated until the correction value becomes equal to or less than the threshold.

[0055] The output information generation unit 15 generates output information for displaying on the output device 50 by combining, for example, one or more of a query image, a projection image, a depth image, and a three-dimensional space display of a camera position. Then, the output information generation unit 15 outputs the output information to the output device 50. However, the output information generation unit 15 does not need to be provided in the information processing device 10.

[0056] [Device operation] Next, the operation of the information processing device in the embodiment will be described with reference to FIG. 4. FIG. 4 is a diagram for explaining an example of the operation of the information processing device. In the following description, the diagram will be referred to as appropriate. Also, in the embodiment, the information processing method is implemented by operating the information processing device. Therefore, the description of the information processing method in the embodiment will be replaced with the description of the operation of the information processing device below.

[0057] As shown in FIG. 4, first, using a camera position initial value representing the initial state of the camera position of a query image to be estimated and three-dimensional information of a scene captured by the camera, the three-dimensional information is projected onto an image plane based on the camera position to generate a projected image and a depth image (step A1).

[0058] Next, the depth image correction unit 13 corrects errors in the depth image by performing a depth image correction process using the projection image and the depth image (step A2).

[0059] Next, the detection unit 14 detects a first image feature point included in the query image and a second image feature point included in the projection image, and detects 2D-2D corresponding points (first corresponding points) where the first image feature point and the second image feature point correspond to each other (step A3). Note that the processing order of steps A2 and A3 described above may be reversed.

[0060] Next, the camera position correction unit 12 acquires 2D-3D corresponding points (second corresponding points) between the query image and the projection image using the corrected depth image and the first corresponding points (step A4).

[0061] Next, the camera position correction unit 12 calculates a correction value by a process of solving the PnP problem using the 2D-3D corresponding points (step A5).

[0062] Next, the camera position correction unit 12 determines whether the correction value is greater than the threshold value (step A6). If the correction value is greater than the threshold value (step A6: Yes), the camera position is corrected using the correction value (step A7).

[0063] Specifically, in step A7, the camera position correction unit 12 adds the correction value to the initial camera position or the current camera position, and then executes the process from step A1 again using the corrected camera position.

[0064] On the other hand, if the correction value is equal to or smaller than the threshold value (step A6: No), the corrected camera position is adopted and the process ends. After that, the output information generating unit 15 generates output information and outputs it to the output device 50 (step A8).

[0065] [Effects of the embodiment] As described above, according to the embodiment, the camera position is corrected using the corrected depth image and the 2D-2D corresponding points, so that the camera position can be calculated with high accuracy. Also, the present invention is suitable for use in estimating the camera position from an image.

[0066] (Variation 1) Modification 1 will be described with reference to Fig. 5. Fig. 5 is a diagram for explaining Modification 1. The depth image correction model of the depth image correction unit 13 described above is constructed by supervised learning using a neural network, but in Modification 1, the depth image correction model is constructed by learning using a neural network that does not require the true value of the depth image.

[0067] The depth image correction model of variant example 1 is a machine learning model that, when a generated projection image and depth image are input, outputs both a noise-removed projection image and a depth image, or a depth image.

[0068] In the example of Fig. 5, data with noise added to the depth image and the projection image may be input as training data, and the corrected value of the camera position of the camera position correction unit 12 may be machine-learned as training data for output. That is, the difference between the true value of the camera position and the estimated value is learned by the backpropagation method. As a result, the corrected depth image is learned implicitly.

[0069] According to the first variant, it is no longer necessary to generate true values ​​of a depth image in advance, so that not only computer graphics simulations but also actual data captured by an RGB-D sensor can be used for learning.

[0070] [program] The program in the embodiment may be a program that causes a computer to execute steps A1 to A8 shown in Fig. 4. By installing and executing this program in a computer, the information processing device and the information processing method in the embodiment can be realized. In this case, the processor of the computer functions as an image generating unit 11, a camera position correcting unit 12, a depth image correcting unit 13, a detecting unit 14, and an output information generating unit 15, and performs processing.

[0071] The program in the embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the image generating unit 11, the camera position correcting unit 12, the depth image correcting unit 13, the detecting unit 14, and the output information generating unit 15.

[0072] [Physical configuration] Here, a computer that realizes an information processing device by executing a program in the embodiment and modification 1 will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining an example of a computer that realizes an information processing device in the embodiment and modification 1.

[0073] 6, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU or an FPGA in addition to the CPU 111 or instead of the CPU 111.

[0074] The CPU 111 loads the programs in the embodiment and the first modified example, which are configured from a group of codes stored in the storage device 113, into the main memory 112, and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0075] Moreover, the program in the embodiment and the first modified example is provided in a state stored in a computer-readable recording medium 120. Note that the program in the embodiment may be distributed on the Internet connected via the communication interface 117.

[0076] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0077] Data reader / writer 116 mediates data transmission between CPU 111 and recording medium 120, reads programs from recording medium 120, and writes processing results in computer 110 to recording medium 120. Communication interface 117 mediates data transmission between CPU 111 and other computers.

[0078] Specific examples of recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or optical recording media such as a CD-ROM (Compact Disk Read Only Memory).

[0079] The information processing device 10 in the embodiment and the first modification can be realized not by a computer with a program installed, but by hardware corresponding to each part, for example, an electronic circuit. Furthermore, the information processing device 10 may be partially realized by a program and the remaining part by hardware. In the embodiment and the first modification, the computer is not limited to the computer shown in FIG. 6.

[0080] [Note] The following supplementary notes are further disclosed with respect to the above-described embodiment and Modification 1. A part or all of the above-described embodiment can be expressed by (Supplementary Note 1) to (Supplementary Note 21) described below, but is not limited to the following description.

[0081] (Appendix 1) an image generating unit that generates a projection image corresponding to the shooting from the camera position and a depth image based on a camera position for a query image obtained by shooting a scene from the camera position and three-dimensional information about a front scene; a camera position correction unit that corrects the camera position based on a correspondence relationship between the query image and the projection image and the depth image; An information processing device having the above configuration.

[0082] (Appendix 2) Further comprising a depth image correction unit that corrects an error in the depth image by a depth image correction process using the projection image and the depth image. 2. An information processing device according to claim 1.

[0083] (Appendix 3) The depth image correction process includes: A machine learning model that inputs the projection image and the depth image and outputs the modified depth image with noise removed, or both the modified depth image and the projection image. 3. An information processing device according to claim 2.

[0084] (Appendix 4) The machine learning model is Input training data with noise added to the depth image and projection image to obtain a corrected depth image; The corrected depth image is input to the camera position correction unit, and machine learning is performed using the noise-free camera position correction value as output training data. 4. The information processing device according to claim 3.

[0085] (Appendix 5) a detection unit that detects a first image feature point included in the query image and a second image feature point included in the projection image, and detects a first corresponding point where the first image feature point and the second image feature point correspond to each other; 2. An information processing device according to claim 1.

[0086] (Appendix 6) The camera position correction means obtaining second corresponding points between the query image and the projection image using the corrected depth image and first corresponding points between the first image feature points and the second image feature points; Calculating the correction value by solving a perspective-n-point (PnP) problem using the second corresponding point; If the correction value is greater than a preset threshold, the correction value is added to the initial camera position or the current camera position; If the correction value is equal to or less than the threshold value, adopt the corrected camera position. 2. An information processing device according to claim 1.

[0087] (Appendix 7) The three-dimensional information is information that represents a three-dimensional space using a nonlinear implicit function of the scene. 5. An information processing device according to any one of claims 1 to 4.

[0088] (Appendix 8) An information processing device, Based on a camera position for a query image obtained by photographing a scene from a camera position and three-dimensional information about a previous scene, a projection image corresponding to the photographing from the camera position and a depth image are generated; modifying the camera position based on the correspondence between the query image and the projection image and the depth image; Information processing methods.

[0089] (Appendix 9) The information processing device, Further, the projection image and the depth image are used to correct an error in the depth image through a depth image correction process. 10. An information processing method as described in Appendix 8.

[0090] (Appendix 10) The depth image correction process includes: A machine learning model that inputs the projection image and the depth image and outputs the modified depth image with noise removed, or both the modified depth image and the projection image. 10. An information processing method as described in Supplementary Note 9.

[0091] (Appendix 11) The machine learning model is Input training data with noise added to the depth image and projection image to obtain a corrected depth image; The corrected depth image is input to the camera position correction unit, and machine learning is performed using the noise-free camera position correction value as output training data. 11. The information processing method according to claim 10.

[0092] (Appendix 12) The information processing device, further detecting a first image feature point included in the query image and a second image feature point included in the projected image, and detecting a first corresponding point where the first image feature point and the second image feature point correspond to each other; 10. An information processing method as described in Appendix 8.

[0093] (Appendix 13) The information processing device, Further, by using the corrected depth image and a first corresponding point where the first image feature point and the second image feature point correspond to each other, a second corresponding point between the query image and the projection image is obtained; Calculating a correction value by solving a perspective-n-point (PnP) problem using the second corresponding points; If the correction value is greater than a preset threshold, the correction value is added to the initial camera position or the current camera position; If the correction value is equal to or less than the threshold value, adopt the corrected camera position. 10. An information processing method as described in Appendix 8.

[0094] (Appendix 14) The three-dimensional information is information that represents a three-dimensional space using a nonlinear implicit function of the scene. 12. An information processing method according to any one of appendix 8 to 11.

[0095] (Appendix 15) On the computer, Based on a camera position for a query image obtained by photographing a scene from a camera position and three-dimensional information about a previous scene, a projection image corresponding to the photographing from the camera position and a depth image are generated; correcting the camera position based on the correspondence between the query image and the projection image and the depth image; program.

[0096] (Appendix 16) The program causes the computer to correcting an error in the depth image by a depth image correction process using the projection image and the depth image; 16. The program according to claim 15.

[0097] (Appendix 17) The depth image correction process includes: A machine learning model that inputs the projection image and the depth image and outputs the modified depth image with noise removed, or both the modified depth image and the projection image. 17. The program according to claim 16.

[0098] (Appendix 18) The machine learning model is Input training data with noise added to the depth image and projection image to obtain a corrected depth image; The corrected depth image is input to the camera position correction unit, and machine learning is performed using the noise-free camera position correction value as output training data. 18. The program according to claim 17.

[0099] (Appendix 19) The program causes the computer to detecting a first image feature point included in the query image and a second image feature point included in the projected image, and detecting a first corresponding point where the first image feature point and the second image feature point correspond to each other; 16. The program according to claim 15.

[0100] (Appendix 20) The program causes the computer to obtaining second corresponding points between the query image and the projection image using the corrected depth image and first corresponding points between the first image feature points and the second image feature points; Calculating a correction value by solving a perspective-n-point (PnP) problem using the second corresponding points; If the correction value is greater than a preset threshold, the correction value is added to the initial camera position or the current camera position; If the correction value is equal to or smaller than the threshold value, the corrected camera position is adopted. 16. The program according to claim 15.

[0101] (Appendix 21) The three-dimensional information is information that represents a three-dimensional space using a nonlinear implicit function of the scene. 19. The program according to any one of appendixes 15 to 18.

[0102] The invention has been described above with reference to the embodiment and modified example 1, but the invention is not limited to the above-mentioned embodiment and modified example 1. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the invention within the scope of the invention. [Industrial Applicability]

[0103] According to the above description, the camera position can be calculated with high accuracy, and is useful in fields where the calculation of the camera position is required. [Explanation of symbols]

[0104] 10. Information processing device 11 Image Generation Unit 12 Camera position correction section 13 Depth image correction section 14 Detection section 15 Output information generation unit 20 Camera 30 Storage device 40 Input Devices 50 Output Device 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display device 120 Recording media 121 Bus

Claims

1. an image generating means for generating a projection image corresponding to the shooting from the camera position and a depth image based on a camera position for a query image obtained by shooting a scene from the camera position and three-dimensional information about a front scene; a camera position correction means for correcting the camera position based on the correspondence between the query image and the projection image and the depth image; An information processing device having the above configuration.

2. The apparatus further includes a depth image correction means for correcting an error in the depth image by a depth image correction process using the projection image and the depth image. The information processing device according to claim 1 .

3. The depth image correction process includes: A machine learning model that inputs the projection image and the depth image and outputs the modified depth image with noise removed, or both the modified depth image and the projection image. The information processing device according to claim 2 .

4. The machine learning model is Input training data with noise added to the depth image and projection image to obtain a corrected depth image; The corrected depth image is input to the camera position correction means, and machine learning is performed using the correction value of the camera position that does not include noise as training data for output. The information processing device according to claim 3 .

5. The image processing apparatus further includes a detection means for detecting a first image feature point included in the query image and a second image feature point included in the projection image, and detecting a first corresponding point where the first image feature point and the second image feature point correspond to each other. The information processing device according to claim 1 .

6. The camera position correction means obtaining second corresponding points between the query image and the projection image using the corrected depth image and first corresponding points between the first image feature points and the second image feature points; Calculating a correction value by solving a perspective-n-point (PnP) problem using the second corresponding points; If the correction value is greater than a preset threshold, the correction value is added to the initial camera position or the current camera position; If the correction value is equal to or less than the threshold value, adopt the corrected camera position. The information processing device according to claim 1 .

7. The three-dimensional information is information that represents a three-dimensional space using a nonlinear implicit function of the scene.

5. The information processing device according to claim 1.

8. An information processing device, Based on a camera position for a query image obtained by photographing a scene from a camera position and three-dimensional information about a previous scene, a projection image corresponding to the photographing from the camera position and a depth image are generated; modifying the camera position based on the correspondence between the query image and the projection image and the depth image; Information processing methods.

9. On the computer, Based on a camera position for a query image obtained by photographing a scene from a camera position and three-dimensional information about a previous scene, a projection image corresponding to the photographing from the camera position and a depth image are generated; correcting the camera position based on the correspondence between the query image and the projection image and the depth image; program.

Images