Learning method and camera parameter estimation method

Abstract

To improve the accuracy of the SCR (Scene Coordinate Regression) mapping function. [Solution] The SCR mapping function is configured to perform SCR mapping, which estimates the 3D coordinates in 3D space corresponding to the 2D coordinates of points on an input image. The learning method includes training the SCR mapping function using a test image and obtaining a confidence map that shows the confidence level of the SCR mapping for each point on the test image. The learning method further includes extracting high-confidence points from the confidence map, selecting a first region containing the extracted high-confidence points, and generating a mask that excludes the second region other than the first region. The learning method further includes using the mask to select the first region as the learning target region of the test image, excluding the second region from the learning target region, and training the SCR mapping function again using the learning target region of the test image.

Description

Technical Field

[0001] The present disclosure relates to camera parameter estimation using SCR (Scene Coordinate Regression). The present disclosure also relates to the learning of the SCR mapping function.

Background Art

[0002] Patent Document 1 discloses a camera pose estimation device. The camera pose estimation device estimates the camera position with respect to a sphere using a camera image of the sphere and the radius information of the sphere. Further, the camera pose estimation device estimates the camera direction with respect to the sphere using the estimated camera position and the feature points of the sphere.

[0003] Non-Patent Document 1 and Non-Patent Document 2 disclose techniques related to SCR (Scene Coordinate Regression). SCR is a technique for efficiently estimating the 3D coordinates in 3D space corresponding to the 2D coordinates of points on an image of a certain scene. In SCR, a machine learning model such as a Regression Forest is used. If the correspondence between the 2D coordinates of points on the input image and the 3D coordinates in 3D space is obtained, the camera parameters can be estimated by the PnP (Perspective-n-Point) method using that correspondence.

[0004] [[ID=]19] Non-Patent Document 3 discloses a technique related to SAM (Segment Anything Model). SAM is a model that can segment any object in an image. A machine learning model is used in SAM. By inputting a prompt including a point or a box in the image to SAM, the object related to the prompt can be segmented with high accuracy.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2013-140547 [Non-patent literature]

[0006] [Non-Patent Document 1] Eric Brachmann et al., "Accelerated Coordinate Encoding: Learning to Relocalize in Minutes using RGB and Poses," in Proc. IEEE Conf. on Computer Vision and Recognition (CVPR), 2023. [Non-Patent Document 2] J. Shotton et al., "Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images," In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2013. [Non-Patent Document 3] Alexander Kirillov et al. "Segment Anything," arXiv:2304.02643, 5 Apr 2023. [Overview of the project] [Problems that the invention aims to solve]

[0007] The SCR mapping function is a machine learning model for estimating the 3D coordinates in 3D space corresponding to the 2D coordinates of points on an input image. Traditionally, SCR mapping functions have been trained using points from the entire region of a test image. However, test images may contain low-confidence regions that are unsuitable for training. Examples of low-confidence regions include areas with moving objects (e.g., vehicles, pedestrians) and areas with little texture (e.g., the sky). If an SCR mapping function is trained using points from the entire region, including such low-confidence regions, the accuracy of the SCR mapping function may remain at a certain level. There is room for improvement in training SCR mapping functions.

[0008] One objective of this disclosure is to provide a technique that can improve the accuracy of SCR mapping functions.

[0009] Another object of this disclosure is to provide a technique that can improve the accuracy of camera parameter estimation using SCR. [Means for solving the problem]

[0010] The first point concerns a learning method for training SCR (Scene Coordinate Regression) mapping functions using a computer. The SCR mapping function is configured to perform SCR mapping, which estimates the 3D coordinates in 3D space corresponding to the 2D coordinates of points on the input image. The learning method is, The SCR mapping function is trained using test images, and a confidence map is obtained showing the confidence level of the SCR mapping for each point on the test images. Extracting high-confidence points from the confidence map where the confidence level is above the first threshold, This involves generating a mask that selects the first region containing the extracted high-confidence points and excludes the second region, which is not the first region. By using a mask, we can select the first region as the training region of the test image and exclude the second region from the training region. By using the training target region of the test image, the SCR mapping function can be trained again. Includes.

[0011] The second point concerns a method for estimating camera parameters using a computer to estimate the camera parameters of a camera that has captured an image. The camera parameter estimation method is: Obtain an SCR (Scene Coordinate Regression) mapping function configured to perform SCR mapping, which estimates the 3D coordinates in 3D space corresponding to the 2D coordinates of points on the input image. By inputting an image into the SCR mapping function, the 3D coordinates in 3D space corresponding to the 2D coordinates of points on the image are estimated. Estimating camera parameters based on the correspondence between 2D coordinates and estimated 3D coordinates. Includes. To obtain the SCR mapping function, The SCR mapping function is trained using test images, and a confidence map is obtained showing the confidence level of the SCR mapping for each point on the test images. Extracting high-confidence points from the confidence map where the confidence level is above the first threshold, This involves generating a mask that selects the first region containing the extracted high-confidence points and excludes the second region, which is not the first region. By using a mask, we can select the first region as the training region of the test image and exclude the second region from the training region. By using the training target region of the test image, the SCR mapping function can be trained again. Includes. [Effects of the Invention]

[0012] According to the present disclosure, high-confidence points are automatically extracted from a confidence map indicating the reliability of SCR mapping. Further, a mask is automatically generated that selects a first region including the high-confidence points and excludes a second region outside the first region. The first region is a high-confidence region (robust region) suitable for learning the SCR mapping function. By performing learning of the SCR mapping function using such a high-confidence region (robust region), the accuracy of the SCR mapping function is improved. By improving the accuracy of the SCR mapping function, the accuracy of the camera parameter estimation process using the same is also improved.

Brief Description of the Drawings

[0013] [Figure 1] It is a block diagram for explaining the outline of an information processing apparatus. [Figure 2] It is a block diagram for explaining the outline of a camera parameter estimation process using SCR. [Figure 3] It is a block diagram for explaining the outline of learning of the SCR mapping function. [Figure 4] It is a block diagram for explaining the learning of the SCR mapping function. [Figure 5] It is a conceptual diagram for explaining the learning of the SCR mapping function. [Figure 6] It is a conceptual diagram showing an example of the evolution of the mask.

Embodiments of the Invention

[0014] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0015] 1. Information Processing Apparatus Figure 1 is a block diagram illustrating the overview of the information processing device 1 according to this embodiment. The information processing device 1 acquires an image (RGB image) captured by a camera as an input image IMG. Based on the input image IMG, the information processing device 1 estimates the camera parameters CPA of the camera that captured the input image IMG. The camera parameters CPA include intrinsic parameters and extrinsic parameters. Intrinsic parameters include distortion parameters, focal length, etc. Extrinsic parameters include the camera's position and orientation in the world coordinate system. If the intrinsic parameters are known, the estimation of the camera parameters CPA includes the estimation of the camera's position and orientation.

[0016] The information processing device 1 may consist of a single node or multiple nodes. More specifically, the information processing device 1 includes one or more processors 10 (hereinafter simply referred to as "processor 10"), one or more storage devices 20 (hereinafter simply referred to as "storage devices 20"), and one or more interfaces 30 (hereinafter simply referred to as "interface 30").

[0017] Processor 10 performs various processes. Examples of processor 10 include general-purpose processors, application-specific processors, CPUs (Central Processing Units), GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), etc. Processor 10 can also be called processing circuitry.

[0018] The storage device 20 stores various types of information 40 necessary for processing. Examples of storage devices 20 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc.

[0019] Interface 30 receives various data from the outside and outputs various data to the outside. For example, interface 30 includes a communication interface. Interface 30 may also include a user interface that provides information to the user and accepts input from the user. Examples of user interfaces include touch panels and displays.

[0020] The processor 10 receives an input image (IMG) via the interface 30. Based on the input image (IMG), the processor 10 performs a "camera parameter estimation process" to estimate the camera parameters (CPA) of the camera that captured the input image (IMG). The processor 10 may output the camera parameter CPA information via the interface 30. The processor 10 may also present the camera parameter CPA information to the user via a user interface.

[0021] Furthermore, processor 10 also performs the training of the machine learning model, which will be described later.

[0022] The processor 10 may execute the computer program 50. In this case, the functions of the information processing device 1 are realized through the cooperation of the processor 10 executing the computer program 50 and the storage device 20. The computer program 50 is stored in the storage device 20. The computer program 50 may be recorded on a computer-readable recording medium.

[0023] 2. Camera parameter estimation process using SCR Figure 2 is a block diagram illustrating the overview of the camera parameter estimation process by the information processing device 1 according to this embodiment. In this embodiment, SCR (Scene Coordinate Regression) is used for camera parameter estimation. SCR is a technique for efficiently estimating the 3D coordinates in 3D space corresponding to the 2D coordinates of points on an image of a given scene. The SCR algorithm is disclosed, for example, in Non-Patent Document 1, Non-Patent Document 2, etc.

[0024] The information processing device 1 includes an SCR unit 100. The SCR unit 100 receives an input image IMG of a specific scene captured by a camera. Pi represents the 2D coordinate in the image coordinate system of a point (i) on the input image IMG. The SCR unit 100 calculates the feature quantities of the point on the input image IMG and estimates the 3D coordinate Yi in 3D space corresponding to the 2D coordinate Pi of the point on the input image IMG using the SCR algorithm. Estimating the 3D coordinate Yi in 3D space corresponding to the 2D coordinate Pi of the point on the input image IMG using the SCR algorithm is hereinafter referred to as "SCR mapping". SCR mapping is performed for each pixel.

[0025] More specifically, the SCR unit 100 includes an SCR mapping function f configured to perform SCR mapping (Yi = f(Pi)). The SCR mapping function f can also be called a transformation function. A machine learning model such as Regression Forest is used as the SCR mapping function f. The SCR mapping function f is generated in advance through training to perform SCR mapping for a specific scene. Details of the training of the SCR mapping function f will be described later. Note that SCR mapping is possible with just one input image IMG. Furthermore, SCR mapping is superior to SfM (Structure from Motion) in terms of low processing load and high processing speed.

[0026] The SCR mapping described above provides a correspondence between the 2D coordinates Pi of a point on the input image IMG and its 3D coordinates Yi in 3D space. This correspondence can be used to estimate the camera parameter CPA using the PnP (Perspective-n-Point) method. The PnP method uses a point cloud consisting of n points, where n is an integer greater than or equal to 2.

[0027] More specifically, the information processing device 1 includes a camera parameter estimation unit 200. The camera parameter estimation unit 200 acquires information about the correspondence between the 2D coordinates Pi of a point on the input image IMG and the 3D coordinates Yi in 3D space. Based on this correspondence, the camera parameter estimation unit 200 estimates (calculates) the camera parameter CPA of the camera that captured the input image IMG by solving a PnP problem.

[0028] Furthermore, the SCR unit 100 may be configured to perform SCR mapping and generate (output) a confidence map CON indicating the confidence level of the SCR mapping for each point on the input image IMG. By using the confidence map CON, only high-confidence points, whose confidence level is above a predetermined threshold, may be selected from the input image IMG. In this case, the camera parameter estimation unit 200 may estimate (calculate) the camera parameter CPA based on the correspondence between the 2D coordinates Pi and 3D coordinates Yi of the selected high-confidence points. This further improves the estimation accuracy of the camera parameter CPA.

[0029] 3. Learning the SCR mapping function Figure 3 is a block diagram illustrating the overview of the learning process for the SCR mapping function f by the information processing device 1. For convenience, the input image IMG used to learn the SCR mapping function f is referred to as the "test image IMG-T". During the learning phase, the camera parameters of the camera that captured the test image IMG-T are given as the ground truth camera parameters.

[0030] During training, the information processing device 1 inputs the test image IMG-T to the SCR mapping function f. The SCR mapping function f estimates the 3D coordinate Yi in 3D space corresponding to the 2D coordinate Pi of point (i) on the test image IMG-T. Subsequently, the information processing device 1 reprojects the estimated 3D coordinate Yi onto the image coordinate system of the test image IMG-T using the ground truth camera parameters. The reprojected 2D coordinate Ri is the 2D coordinate obtained by reprojecting the 3D coordinate Yi onto the image coordinate system of the test image IMG-T. The reprojection error Ei is the error between the 2D coordinate Pi and the reprojected 2D coordinate Ri of the same point (i). For example, the reprojection error Ei is the distance between the 2D coordinate Pi and the reprojected 2D coordinate Ri. The information processing device 1 calculates the reprojection error Ei for each point on the test image IMG-T and generates a reprojection error map ERR. The reprojection error map ERR is a map that shows the reprojection error Ei for each point on the test image IMG-T. The information processing device 1 then learns the SCR mapping function f to optimize the reprojection error Ei.

[0031] 4. Improving the accuracy of the SCR mapping function Traditionally, SCR mapping functions f have been trained using points from the entire region of the test image IMG-T. However, the test image IMG-T may contain low-confidence regions that are unsuitable for training. One example of a low-confidence region is a region containing moving objects (e.g., vehicles, pedestrians). In regions containing moving objects, the features change over time, so the confidence of the estimated 3D coordinate Yi is not necessarily high. Another example of a low-confidence region is a region with little texture (e.g., the sky). In regions with little texture, feature extraction is difficult, so the confidence of the estimated 3D coordinate Yi is not necessarily high. It can also be said that low-confidence regions are regions where the reprojection error Ei inevitably becomes large. If the SCR mapping function f is trained using points from the entire region, including such low-confidence regions, the accuracy of the SCR mapping function f may remain at a certain level. There is room for improvement in the training of the SCR mapping function f.

[0032] To improve the accuracy of the SCR mapping function f, it is conceivable to exclude low-confidence regions during the training of the SCR mapping function f. However, the inventors of this application have found that low-confidence regions vary from scene to scene and are not uniform. For example, it was found that even in regions where objects of the same class are displayed, the reprojection error Ei will differ if the scene is different. This means that it is not possible to pre-define low-confidence regions based on the object class, etc. Therefore, it is desirable to "automatically" exclude low-confidence regions that are unsuitable for training the SCR mapping function f for a given scene. In other words, it is desirable to "automatically" identify and select high-confidence regions (robust regions) that are suitable for training the SCR mapping function f for a given scene.

[0033] From the above perspective, this disclosure proposes a technique that enables the training of an SCR mapping function f while automatically selecting a high-confidence region (robust region) suitable for training the SCR mapping function f for a given scene. By training the SCR mapping function f using the high-confidence region (robust region), the accuracy of the SCR mapping function f is improved. As the accuracy of the SCR mapping function f improves, the accuracy of the camera parameter estimation process using it also improves.

[0034] Figure 4 is a block diagram illustrating the learning of the SCR mapping function f in the information processing device 1. Figure 5 is a conceptual diagram illustrating the learning of the SCR mapping function f.

[0035] As described above, the test image IMG-T is used to train the SCR mapping function f. The information processing device 1 trains the SCR mapping function f using the test image IMG-T. More specifically, the information processing device 1 (SCR unit 100) performs SCR mapping using the SCR mapping function f. The information processing device 1 also obtains a reprojection error map ERR that shows the reprojection error Ei for each point on the test image IMG-T (see Figure 5).

[0036] The reprojection error map ERR obtained during the learning process can be described as a "confidence map" that shows the confidence level of the SCR mapping for each point on the test image IMG-T. The smaller the reprojection error Ei, the higher the confidence level, and the larger the reprojection error Ei, the lower the confidence level. The information processing device 1 extracts "high confidence points" from the reprojection error map ERR (confidence map) where the confidence level is above the first threshold. Low error points where the reprojection error Ei is below the third threshold correspond to high confidence points where the confidence level is above the first threshold. The information processing device 1 extracts low error points where the reprojection error Ei is below the third threshold from the reprojection error map ERR as "high confidence points" (see Figure 5).

[0037] As a variation, if the SCR unit 100 is configured to generate (output) the confidence map CON described above, the confidence map CON may be used as a "confidence map". The confidence map shows the confidence level of the SCR mapping for each point on the test image IMG-T. High confidence points with a confidence level of 2 or higher correspond to high confidence points with a confidence level of 1 or higher. The information processing device 1 may extract high confidence points with a confidence level of 2 or higher from the confidence map CON as "high confidence points".

[0038] Next, the information processing device 1 generates a mask MSK to be applied to the test image IMG-T based on the extracted high-confidence points. More specifically, the test image IMG-T is divided into a first region containing the extracted high-confidence points and a second region excluding the first region. The first region can be said to be a high-confidence region, and the second region a low-confidence region. The information processing device 1 generates a mask MSK that selects only the first region (high-confidence region) and excludes the second region (low-confidence region).

[0039] For example, the Segment Anything Model (SAM) disclosed in Non-Patent Document 3 is used to generate a mask MSK. SAM is a model that can segment any object in an image. SAM uses a machine learning model. By inputting a prompt containing a point or box in the image into SAM, objects related to the prompt can be segmented with high accuracy. Therefore, the information processing device 1 can easily generate a mask MSK that selects a first region containing the extracted high-confidence points and excludes the second region by inputting the extracted high-confidence points as prompts into SAM. For convenience, the mask MSK generated by SAM is called a "semantic mask MSK-S". In a semantic mask MSK-S, the first region tends to contain objects of the same semantic class. In the example of a semantic mask MSK-S shown in Figure 5, the white region represents the first region (high-confidence region), and the black region represents the second region (low-confidence region).

[0040] The information processing device 1 generates a mask MSK based on the semantic mask MSK-S. For example, the information processing device 1 may generate a mask MSK by combining the semantic mask MSK-S with the confidence map CON described above. In the semantic mask MSK-S, the first region tends to contain objects of the same semantic class, and there is a possibility that low-confidence regions are mixed in with a part of the first region. Therefore, the information processing device 1 may improve the semantic mask MSK-S by referring to the confidence map CON. More specifically, the information processing device 1 recognizes low-confidence points whose confidence level is below the second threshold based on the confidence map CON. Then, the information processing device 1 generates the final mask MSK by excluding the low-confidence points from the first region of the semantic mask MSK-S and moving them to the second region. In the example shown in Figure 5, the confidence level of the window region of the building is low. That window region is included in the first region (white region) of the semantic mask MSK-S. By excluding the window region from the first region of the semantic mask MSK-S and changing it to the second region, the mask MSK is obtained.

[0041] As a variation, the semantic mask MSK-S may be used as is, as the mask MSK.

[0042] In this way, a mask MSK is generated that selects the first region (high confidence region) and excludes the second region (low confidence region). In the example mask MSK shown in Figure 5, the white region represents the first region (high confidence region), and the black region represents the second region (low confidence region). A mask MSK generated in this way can also be called an EGFS (Error-Guided Feature Selection) mask.

[0043] As shown in Figure 4, the generated mask MSK is fed back to the SCR unit 100. The SCR unit 100 includes a selection unit 110. The selection unit 110 uses the mask MSK to select the training area of ​​the test image IMG-T. More specifically, the selection unit 110 selects a first area as the training area of ​​the test image IMG-T and excludes a second area from the training area. For example, the SCR unit 100 extracts the features of each point in the test image IMG-T and generates a feature map. Then, the selection unit 110 applies the mask MSK to the feature map to select a first area as the training area and excludes a second area from the training area.

[0044] Then, the information processing device 1 retrains the SCR mapping function f using the training area of ​​the test image IMG-T. This improves the accuracy of the SCR mapping function f.

[0045] The information processing device 1 may repeatedly execute the series of processes described above. That is, the information processing device 1 may repeatedly execute the series of processes: [1] learning the SCR mapping function f using the mask MSK, [2] obtaining the confidence map, [3] extracting high confidence points, and [4] generating (updating) the mask MSK. The series of processes may be repeatedly executed until the accuracy of the SCR mapping function f reaches or exceeds the desired level.

[0046] Figure 6 shows the evolution of the mask MSK through the repetition of a series of processes. Each iteration contains five epochs. As the series of processes are repeated, the accuracy of the mask MSK improves. As the accuracy of the mask MSK improves, the accuracy of the SCR mapping function f further improves.

[0047] As described above, according to this embodiment, high-confidence points are automatically extracted from the confidence map showing the confidence level of the SCR mapping. Furthermore, a mask is automatically generated that selects the first region containing the high-confidence points and excludes the second region other than the first region. The first region is a high-confidence region (robust region) suitable for training the SCR mapping function f. By training the SCR mapping function f using such a high-confidence region (robust region), the accuracy of the SCR mapping function f is improved. As the accuracy of the SCR mapping function f improves, the accuracy of the camera parameter estimation process using it also improves. [Explanation of Symbols]

[0048] 1. Information Processing Device 100 SCR section 200 Camera parameter estimation unit MSK Mask

Claims

1. A learning method for learning SCR (Scene Coordinate Regression) mapping functions using a computer, The SCR mapping function is configured to perform SCR mapping, which estimates the three-dimensional coordinates in three-dimensional space corresponding to the two-dimensional coordinates of points on the input image. The aforementioned learning method is The SCR mapping function is trained using test images, and a confidence map is obtained that shows the confidence level of the SCR mapping for each point on the test images. Extracting high-confidence points from the confidence map where the confidence level is above the first threshold, A mask is generated that selects a first region containing the extracted high-confidence points and excludes a second region other than the first region. By using the aforementioned mask, the first region is selected as the training target region of the test image, and the second region is excluded from the training target region. The SCR mapping function is retrained by using the training target region of the test image. including Learning methods.

2. A learning method according to claim 1, The process of generating the aforementioned mask is to By inputting the extracted high-confidence points as prompts into the SAM (Segment Anything Model), a semantic mask is generated that selects the first region containing the extracted high-confidence points and excludes the second region. To generate the mask based on the semantic mask. including Learning methods.

3. A learning method according to claim 2, The method further includes performing the SCR mapping and generating a confidence map indicating the confidence level of the SCR mapping for each point on the test image, Generating the mask based on the semantic mask includes generating the mask by excluding low confidence points whose confidence level is below the second threshold from the first region of the semantic mask. Learning methods.

4. A learning method according to any one of claims 1 to 3, The confidence map is a reprojection error map obtained during the training of the SCR mapping function, which shows the reprojection error for each point on the test image. The aforementioned high-reliability points include points where the reprojection error is less than or equal to the third threshold. Learning methods.

5. A camera parameter estimation method that uses a computer to estimate the camera parameters of a camera that has taken an image, Obtain an SCR (Scene Coordinate Regression) mapping function configured to perform SCR mapping, which estimates the 3D coordinates in 3D space corresponding to the 2D coordinates of points on the input image. By inputting the aforementioned image into the SCR mapping function, the three-dimensional coordinates in three-dimensional space corresponding to the two-dimensional coordinates of points on the image are estimated. The camera parameters are estimated based on the correspondence between the two-dimensional coordinates and the estimated three-dimensional coordinates. Includes, Obtaining the aforementioned SCR mapping function means The SCR mapping function is trained using test images, and a confidence map is obtained that shows the confidence level of the SCR mapping for each point on the test images. Extracting high-confidence points from the confidence map where the confidence level is above the first threshold, A mask is generated that selects a first region containing the extracted high-confidence points and excludes a second region other than the first region. By using the aforementioned mask, the first region is selected as the training target region of the test image, and the second region is excluded from the training target region. The SCR mapping function is retrained by using the training target region of the test image. including Camera parameter estimation method.

Images