A dense matching method for two-frame images based on a block area transfer model

Abstract

This invention discloses a method for dense matching of two images, belonging to the field of computer vision. The two images to be matched (a reference frame and a target frame) are first segmented into multiple equal-sized blocks. Due to differences in shooting angle and distance, the blocks in the two frames may have many-to-many relationships; that is, a small block in the reference frame may correspond to multiple small blocks in the target frame, and vice versa, or other many-to-many subordinate relationships. This invention uses a neural network to simultaneously predict the relative area relationships and correspondences of these blocks, and then further subdivides the corresponding blocks into different numbers of smaller blocks based on their area size. The subdivided blocks are then used again by the neural network to predict even more detailed correspondences. Through this iterative process, this invention ultimately obtains dense and accurate pixel matching results between the two images.

Description

Technical Field

[0001] This invention relates to a two-frame dense matching estimation method in the fields of computer vision, 3D reconstruction, and mixed reality, and in particular to a two-frame dense matching method based on a block area transfer model. Background Technology

[0002] The goal of two-frame image matching is to find the coordinates of corresponding points from two input image frames. It is a fundamental technique in computer vision, with important areas such as optical flow estimation, 3D reconstruction, and visual localization all relying on the output of two-frame image matching. Existing two-frame matching methods can be broadly categorized into sparse matching and dense matching based on the density of the matching pairs. Sparse matching prioritizes extracting accurate and repeatable correspondences, with the requirement for completeness limited to avoiding overly concentrated distributions. Dense matching, on the other hand, prioritizes the completeness of the matching relationships, aiming to obtain the corresponding output wherever a match exists.

[0003] Traditional dense matching methods often rely on high image similarity, leading to significant errors when there are large differences in viewpoint or position between two images. While sparse matching methods based on extracted feature points can handle larger deviations in viewpoint or position better, the feature points themselves are sparse, making it difficult to find complete matching point locations. Furthermore, because sparse matching methods rely on pre-extracted feature points, the accuracy of matching point coordinates rarely exceeds the pixel level, limiting its application. Finally, feature point extraction faces two additional challenges when there are significant scale differences between two images: first, larger-scale images often yield fewer feature points, potentially failing to meet application requirements; second, feature points extracted from larger-scale images tend to have greater relative errors, which also poses a challenge in practical applications.

[0004] Currently, there is no method that can simultaneously ensure robustness against differences in position and viewpoint between two images, as well as the integrity and density of image matching pairs. If both advantages could be achieved simultaneously, then outstanding performance could be obtained in practical applications. Summary of the Invention

[0005] To address the problems in existing technologies, this invention provides a two-frame dense matching method based on a block area transfer model. This invention includes several iterations, where the input for each iteration is a set of matching images segmented into numerous small blocks. After block area transfer and projection estimation of reference frame blocks, each reference frame block and its projection are output as a set of even smaller and more numerous matching images, scaled to the same resolution. By incorporating image area information into the matching algorithm using the block area transfer model, this method combines the completeness of dense matching methods with the robustness of sparse matching methods, while also providing more accurate sub-pixel matching coordinate estimation than both of these types of methods.

[0006] This invention first provides a method for dense matching of two frames of images based on a block area transmission model, which includes the following steps:

[0007] (1) Divide the two frames of images to be matched into multiple small blocks of equal size in a grid pattern and extract the attributes of the small blocks; the attributes of the small blocks include the position, area and image features of the small blocks; wherein, the position of the small blocks is recorded as the coordinates of the center point of the small blocks, and the area and image features are predicted by establishing a neural network structure;

[0008] (2) Calculate the area transfer matrix from the reference frame block to the target frame block by constructing an optimal transmission problem using the attributes of the small blocks; that is, take each small block as a node of the optimal transmission problem, take the area attribute of the small block as the sum of the node transmission content, take the negative similarity between the image feature attributes of the small blocks as the transfer cost function, and set trash can nodes for the reference frame and the target frame respectively to correspond to those small blocks that do not have a matching relationship; solve the optimal area transfer matrix for this optimal transmission problem so that it can complete the transfer of node content with the minimum total transfer cost; this process of constructing the area transfer matrix between the area attributes of small blocks is the block area transmission model, and each area transfer value in the area transfer matrix represents the area transfer relationship between a reference frame block and another target frame block;

[0009] (3) Using the prior continuity of the image, erroneous area transfer relationships are eliminated, and a bounding box along the XY axis of the image is determined, where the X axis is horizontal to the right and the Y axis is vertical to the down. Within this bounding box, the previously calculated area transfer matrix is ​​used to estimate the midpoint projection position of the small block in the reference frame and the relative projection area of ​​the small block in the target frame.

[0010] (4) Using the midpoint projection position of the reference frame block in the target frame and the relative projection area of ​​the block obtained in step (3), establish the bounding box corresponding to the reference frame block in the target frame, and then extract rectangular images from the original reference frame and the target frame; wherein, the extracted rectangular image is centered on the position of the reference frame block and its midpoint projection position in the target frame, and has a side length that is a fixed multiple of the side length of the bounding box of the reference frame block and the target frame; then, scale the image extracted from the target frame to the same resolution as the image extracted from the reference frame, and use it as the new reference frame and target frame image for the next iteration input;

[0011] (5) Repeat the iterative process of steps (1) to (4) a set number of times. In the last iteration, the position attributes of the reference frame block and the projection position of the reference frame block in the target frame calculated in step (3) are the corresponding coordinates of the dense matching.

[0012] As a preferred embodiment of the present invention, in step (1), the neural network structure includes: a wide-area feature encoding module, a basic feature extraction module, a position information encoding module, a visual self-attention module, and an area regression module; the wide-area feature encoding module uses a multilayer perceptron to encode the feature vector calculated in the previous iteration step, but this module is not used in the first iteration step; the basic feature extraction module uses a convolutional neural network to extract the feature vector corresponding to each small block, and combines it with the output of the wide-area feature encoding module, but no combination operation is performed in the first iteration step; the position encoding module encodes the position information of the small block into the feature vector; the visual self-attention module globally combines the feature vectors of all small blocks on the reference frame and the target frame, and encodes them into the feature vector of each small block, thereby obtaining the image feature attributes of each small block; then the image feature attributes of the small block are input into the area regression module to obtain the area attributes of the small block.

[0013] As a preferred embodiment of the present invention, an entropy regularization term is added to the optimization function of the optimal transport problem in step (2), and the area transfer matrix is ​​solved iteratively using the sinking angle algorithm.

[0014] As a preferred embodiment of the present invention, in step (3), the steps for determining the bounding box are as follows: first, find the small block with the largest area transfer on the receiving reference frame in the target frame; then, based on the spatial continuity and the value of the area transfer matrix, use the flood filling algorithm to iteratively expand it into a bounding box.

[0015] In step (3), an area transfer relationship that satisfies the following conditions is considered an incorrect area transfer relationship: a) the small block on the target frame is outside the bounding box; b) the area transfer value from the small block on the reference frame to the small block on the target frame is greater than zero.

[0016] As a preferred embodiment of the present invention, the step (3) of estimating the midpoint projection position of the small block in the reference frame and the relative projection area of ​​the small block in the target frame is as follows: first, calculate the ratio of the area transfer value of the small block in the reference frame to the target frame relative to the area attribute of the small block in each target frame, and use this ratio as the weight W of the small block in the target frame; then, calculate the midpoint projection position of the small block in the reference frame in the target frame using the weighted average of the center point coordinates of the small block in the target frame and the weight W, and calculate the relative projection area of ​​the small block using the weighted average of the area attribute of the small block in the target frame and the weight W.

[0017] As a preferred embodiment of the present invention, in step (4), a plurality of new reference frames and target frames are generated, and there are overlapping parts in these new matching images; in the next iteration step (4), for the small blocks of the reference frames that have overlap, only the small block with the largest area transfer value is retained to construct a new matching image by comparing the size of their area transfer values.

[0018] As a preferred embodiment of the present invention, in step (5), the size of the cut-out small pieces is reduced in each iteration, while the number of matching images and cut-out small pieces is increased. The resulting matching density becomes denser as the iteration round increases, while always maintaining that each point on the reference frame corresponds to only one point on the target frame.

[0019] This invention matches two frames of images based on a block area transfer model. This matching technique replaces feature points in sparse matching methods with small blocks as the basic unit of matching. By utilizing the unique area properties of these small blocks, the matching process is gradually transformed from matching between two large frames to matching between a large number of small frames. This achieves the robustness of dense matching methods while maintaining similarities to sparse matching methods. Furthermore, traditional sparse matching methods require feature point extraction before matching, meaning that the positional accuracy of feature points cannot be further improved using information obtained during the matching process. This invention avoids this problem, reducing the estimated matching coordinate error to the sub-pixel level. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the present invention, which visually illustrates one cycle of the present invention;

[0021] Figure 2 This is a flowchart of the present invention;

[0022] Figure 3 This is a schematic diagram of the overall framework of the neural network structure and post-processing part of the present invention;

[0023] Figure 4This is an actual matching effect diagram of the present invention. For ease of visualization, only one matching point is visualized within each 32*32 range of the matching effect diagram.

[0024] Figure 5 This invention utilizes from Figure 4 The 3D reconstruction results are calculated from the dense matching extracted from the model, including the effect diagrams of the 3D model viewed from viewpoint 1 and viewpoint 2. Detailed Implementation

[0025] The representative embodiments shown in the accompanying drawings will now be further refined. It should be understood that the following description is not intended to limit the embodiments to a single preferred embodiment. Rather, it is intended to cover alternatives, modifications, and equivalents that may be included within the substance and scope of the embodiments defined by the appended claims.

[0026] like Figure 1 , Figure 2 As shown, the method for estimating dense matching between two frames based on the block area transmission model of the present invention includes the following steps:

[0027] (1) The two frames of images to be matched are grid-divided into multiple equal-sized blocks and the block attributes are extracted; the block attributes include the block's position, area, and image features; wherein, the block's position is recorded as the coordinates of its center point, and the area and image features are predicted by establishing a neural network structure. The neural network structure of this invention is as follows: Figure 3 As shown, it includes a wide-area feature encoding module, a basic feature extraction module, a position information encoding module, a visual self-attention module, and an area regression module. The wide-area feature encoding module uses a multilayer perceptron to encode the feature vector calculated in the previous iteration step, but this module is not used in the initial iteration step. The basic feature extraction module uses a convolutional neural network to extract the feature vector corresponding to each small block and combines it with the output of the wide-area feature encoding module; this combination operation is not performed in the initial iteration step. The position encoding module encodes the position information of the small blocks into the feature vector. The visual self-attention module globally combines the feature vectors of all small blocks on the reference frame and the target frame, encoding them into the feature vector of each small block, thereby obtaining the image feature attributes of each small block. Then, the image feature attributes of the small blocks are input into the area regression module, i.e., a convolutional neural network with a 3*3 kernel, to obtain the area attributes of the small blocks.

[0028] (2) The area transfer matrix from the reference frame block to the target frame block is calculated by constructing an optimal transmission problem using the attributes of the small blocks. That is, each small block is used as a node in the optimal transmission problem, the area attribute of the small block is used as the sum of the node's transmitted content, and the negative similarity between the image feature attributes of the small blocks is used as the transfer cost function. At the same time, trash can nodes are set for the reference frame and the target frame respectively to correspond to those small blocks that do not have a matching relationship. The optimal area transfer matrix is ​​solved for this optimal transmission problem so that it can complete the transfer of node content with the minimum total transfer cost. This process of constructing an area transfer matrix between the area attributes of small blocks is the block area transmission model. Each item in the area transfer matrix represents an area transfer relationship. In this embodiment, an entropy regularization term is added to the optimization function of the optimal transmission problem in step (2), and the area transfer matrix is ​​solved iteratively using the sinking angle algorithm.

[0029] (3) Estimate the corresponding position of the midpoint of the reference frame patch in the target frame, and the projected area of ​​the reference frame patch in the target frame. Utilize the prior continuity of the image to eliminate erroneous area transfer relationships, and determine a bounding box along the xy-axis. i The area transfer value P calculated earlier is used in the middle. i,j Estimate the midpoint projection position of block i in the reference frame in the target frame. Relative projected area of ​​small pieces The formula is as follows:

[0030]

[0031]

[0032]

[0033] (4) Using the midpoint projection position of the reference frame block in the target frame and the relative projection area of ​​the block obtained in step (3), establish the bounding box corresponding to the reference frame block in the target frame, and then extract rectangular images from the original reference frame and the target frame; wherein, the extracted rectangular images are centered on the position of the reference frame block and its midpoint projection position in the target frame, and have a side length that is a fixed multiple of the side length of the bounding box of the reference frame block and the target frame; then, the image extracted from the target frame is scaled to the same resolution as the image extracted from the reference frame, and used as the new reference frame and target frame images for the next iteration. It should be noted that multiple new reference frames and target frames will be generated in step (4), and there are overlapping parts in these new matching images; in the next iteration step (4), for the overlapping reference frame blocks, only the block with the largest area transfer value is retained to construct a new matching image by comparing the size of their area transfer values.

[0034] (5) Repeat steps (1) to (4) a set number of times. In the last iteration, the positional attributes of the reference frame block and the projection position of the reference frame block in the target frame calculated in step (3) are the corresponding coordinates of the dense match. Each iteration reduces the size of the cut-out block while increasing the number of matching images and cut-out blocks. The resulting matching density becomes denser as the number of iterations increases, while always maintaining that each point on the reference frame corresponds to only one point on the target frame.

[0035] In a specific embodiment of the present invention, the steps for determining the bounding box in step (3) are as follows: first, find the small block with the largest area transfer on the receiving reference frame in the target frame; then, based on the spatial continuity and the value of the area transfer matrix, iteratively expand it into a bounding box using the flood filling algorithm.

[0036] In step (3), an area transfer relationship that satisfies the following conditions is considered an incorrect area transfer relationship: a) the small block on the target frame is outside the bounding box; b) the area transfer value from the small block on the reference frame to the small block on the target frame is greater than zero.

[0037] In a specific embodiment of the present invention, the step (3) of estimating the midpoint projection position of the small block in the reference frame and the relative projected area of ​​the small block in the target frame is as follows: First, calculate the ratio of the area transfer value of the small block in the reference frame to the target frame relative to the area attribute of the small block in each target frame, and use this ratio as the weight W of the small block in the target frame; then, calculate the midpoint projection position of the small block in the reference frame in the target frame using the weighted average of the center point coordinates of the small block in the target frame and the weight W, and calculate the relative projected area of ​​the small block using the weighted average of the area attribute of the small block in the target frame and the weight W.

[0038] A schematic diagram of the overall framework of the neural network structure and post-processing part of this invention is shown below. Figure 3 As shown, for convenience, we will refer to the small image feature attributes as small patch feature attributes here. Figure 3 In this context, the optimal transmission module is the module that executes step (2) of the present invention. Figure 3 The deduplication module is used to remove matching images with smaller area transfer values. Specifically, considering that multiple new reference frames and target frames will be generated in step (4), there are overlapping parts in these new matching images; in the next iteration step (4), for the small blocks of the reference frames that have overlap, only the small block with the largest area transfer value is retained by comparing the size of their area transfer values ​​to construct a new matching image. Figure 3The cutting module corresponds to step (4) of the present invention, which involves cropping a rectangular image from the reference frame and the target frame; the scaling module corresponds to step (4), which involves scaling the cropped target frame rectangular image to the same resolution as the cropped reference frame rectangular image.

[0039] Figure 3 The framework first extracts the image feature attributes and area attributes of each small patch in the image pair, then inputs them into the optimal transfer module to calculate the area transfer matrix corresponding to each small patch. Based on the area transfer matrix, a flood fill algorithm is used to calculate the bounding box corresponding to each small patch within the reference frame. Within each bounding box, the corresponding midpoint projection coordinates and area projection values ​​are calculated, and duplicate point matching pairs in different subdivided images are removed. If the method terminates at this level, the corresponding point matching pair is directly output; if the method does not terminate, a rectangular image is cropped from the reference frame and the target frame, centered on the position of the small patch in the reference frame and its midpoint projection position in the target frame, with a side length that is a fixed multiple of the side length of the bounding box of the small patch in the reference frame and the target frame. The cropped image on the target frame is scaled to the same scale and resolution as the cropped image in the reference frame, thus obtaining a new image pair, and then the entire process is repeated.

[0040] In one specific embodiment of the present invention, the entire process is repeated three times. In each of the three iterations, the small block is cut into smaller blocks of 32*32, 8*8, and 2*2 resolution, respectively, with corresponding feature dimensions of 448, 256, and 64 dimensions. In the second and third iterations, the number of blocks is 12*12 and 8*8, respectively.

[0041] This invention uses a residual neural network as the basic feature extraction module.

[0042] In the second loop of this invention, the output of the wide-area feature coding module is obtained by concatenating the features of the previous layer and passing them through the multilayer perceptron; in the third loop of this invention, the output of the wide-area feature coding module is obtained by performing deconvolution on the features of the previous layer.

[0043] In the second loop, this invention uses the middle 10*10 block of all 12*12 blocks as input to the deduplication module. In the third loop, it uses the middle 4*4 block of all 8*8 blocks as input to the deduplication module.

[0044] This invention skips the scaling module after the second loop and directly uses the output of the cutting module as the matching pair for the third loop.

[0045] In the first and second loops, the image resolutions after cropping and scaling are 96*96 and 12*12, respectively.

[0046] Preferably, the present invention uses the Megadepth or ScanNet dataset as the data for training the neural network in step (1) of the present invention.

[0047] The final matching result using the above method is as follows: Figure 4 As shown in the diagram. The gray lines represent the corresponding matching points found using this method. For better visualization, we only retained one pair of matching points in each 32*32 block of the reference frame. Using the matching results, 3D reconstruction can be completed, based on... Figure 4 The effect of completing the 3D reconstruction using two densely matched frames is as follows: Figure 5 As shown, this includes reconstruction results viewed from two different perspectives. It can be seen that observing the 3D reconstruction results from these two significantly different perspectives reveals images with a high degree of coverage and a 3D structure that accurately reflects the scene. Traditional dense matching methods often require many more frames to achieve this effect, while this invention achieves the same result with only two frames – a groundbreaking advancement.

[0048] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes to the present invention within the spirit and scope of the claims fall within the protection scope of the present invention.

Claims

1. A method for dense matching of two frames of images based on a block area transfer model, characterized in that, The method includes the following steps: (1) Divide the two frames of images to be matched into multiple small blocks of equal size in a grid pattern and extract the attributes of the small blocks; the attributes of the small blocks include the position, area and image features of the small blocks; wherein, the position of the small blocks is recorded as the coordinates of the center point of the small blocks, and the area and image features are predicted by establishing a neural network structure; (2) Calculate the area transfer matrix from the reference frame block to the target frame block by constructing an optimal transmission problem using the attributes of the small blocks; take each small block as a node of the optimal transmission problem, take the area attribute of the small block as the sum of the node transmission content, take the negative similarity between the image feature attributes of the small blocks as the transfer cost function, and set trash can nodes for the reference frame and the target frame respectively to correspond to those small blocks that do not have a matching relationship; solve the optimal area transfer matrix for this optimal transmission problem so that it can complete the transfer of node content with the minimum total transfer cost; this process of constructing an area transfer matrix between the area attributes of small blocks is the block area transmission model, and each area transfer value in the area transfer matrix represents the area transfer relationship between a reference frame block and another target frame block; (3) Using the prior continuity of the image, eliminate erroneous area transfer relationships and determine a bounding box along the XY axis of the image, where the X axis is horizontal to the right and the Y axis is vertical to the down. In this bounding box, use the previously calculated area transfer matrix to estimate the midpoint projection position of the small block in the reference frame and the relative projection area of ​​the small block in the target frame. (4) Using the midpoint projection position of the reference frame block in the target frame and the relative projection area of ​​the block obtained in step (3), establish the bounding box corresponding to the reference frame block in the target frame, and then extract rectangular images from the original reference frame and the target frame; wherein, the extracted rectangular image is centered on the position of the reference frame block and its midpoint projection position in the target frame, and has a side length that is a fixed multiple of the side length of the bounding box of the reference frame block and the target frame; then, scale the image extracted from the target frame to the same resolution as the image extracted from the reference frame, and use it as the new reference frame and target frame image for the next iteration input; (5) Repeat the iterative process of steps (1) to (4) a set number of times. In the last iteration, the position attributes of the reference frame block and the projection position of the reference frame block in the target frame calculated in step (3) are the corresponding coordinates of the dense matching.

2. The method for dense matching of two frames of images based on the block area transmission model according to claim 1, characterized in that, In step (1), the neural network structure includes: a wide-area feature encoding module, a basic feature extraction module, a position information encoding module, a visual self-attention module, and an area regression module. The wide-area feature encoding module uses a multilayer perceptron to encode the feature vector calculated in the previous iteration step, but this module is not used in the first iteration step. The basic feature extraction module uses a convolutional neural network to extract the feature vector corresponding to each small block and combines it with the output of the wide-area feature encoding module. In the first iteration step, no combination operation is performed. The position encoding module encodes the position information of the small block into the feature vector. The visual self-attention module globally combines the feature vectors of all small blocks on the reference frame and the target frame and encodes them into the feature vector of each small block, thereby obtaining the image feature attributes of each small block. Then, the image feature attributes of the small block are input into the area regression module to obtain the area attributes of the small block. The area regression module is a convolutional neural network with a 3*3 convolutional kernel.

3. The method for dense matching of two frames of images based on the block area transmission model according to claim 1, characterized in that, In step (2), an entropy regularization term was added to the optimization function of the optimal transport problem, and the area transfer matrix was solved iteratively using the sinking angle algorithm.

4. The method for dense matching of two frames of images based on the block area transmission model according to claim 1, characterized in that, In step (3), the steps for determining the bounding box are as follows: first, find the small block with the largest area transfer on the receiving reference frame in the target frame; then, based on the spatial continuity and the value of the area transfer matrix, use the flood filling algorithm to iteratively expand it into a bounding box. In step (3), an area transfer relationship that satisfies the following conditions is considered an incorrect area transfer relationship: a) the small block on the target frame is outside the bounding box; b) the area transfer value from the small block on the reference frame to the small block on the target frame is greater than zero.

5. The method for dense matching of two frames of images based on the block area transmission model according to claim 1, characterized in that, The steps in step (3) of estimating the midpoint projection position of the small block in the reference frame and the relative projected area of ​​the small block in the target frame are as follows: First, calculate the ratio of the area transfer value of the small block in the reference frame to the target frame relative to the area attribute of the small block in each target frame, and use this ratio as the weight W of the small block in the target frame; then, calculate the midpoint projection position of the small block in the reference frame in the target frame using the weighted average of the center point coordinates of the small block in the target frame and the weight W, and calculate the relative projected area of ​​the small block using the weighted average of the area attribute of the small block in the target frame and the weight W.

6. The method for dense matching of two frames of images based on the block area transmission model according to claim 1, characterized in that, In step (4), a plurality of new reference frames and target frames are generated, and there are overlapping parts in these new matching images. In the next iteration step (4), for the small blocks of the reference frames that have overlap, only the small block with the largest area transfer value is retained to construct a new matching image by comparing the size of their area transfer values.

7. The method for dense matching of two frames of images based on the block area transmission model according to claim 1, characterized in that, In step (5), the size of the cut-out pieces is reduced in each iteration, while the number of matching images and cut-out pieces is increased. The resulting matching density becomes denser as the number of iterations increases, while always maintaining that each point on the reference frame corresponds to only one point on the target frame.

Images