Multi-modal semi-dense image feature matching method in large-view-angle scene

Abstract

The invention discloses a multi-modal semi-dense image feature matching method in a large-view-angle scene, and the method comprises the steps: firstly building and training a multi-modal overlapping region detection feature matching network, and then predicting an overlapping region frame in a to-be-detected image pair through the trained multi-modal overlapping region detection feature matching network, and finally, extracting an overlapping region frame and sending the overlapping region frame into the feature matching network to obtain a final feature matching result, thereby solving the problem of inaccurate matching precision in a large-view-angle scene in the prior art.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and more specifically relates to a multi-modal semi-dense image feature matching method under a large view angle scene. BACKGROUND

[0002] Among the numerous branches of image processing technology, image feature detection and matching play a central role. By extracting and describing representative features in images, it can achieve the purposes of image comparison, recognition and classification, and lay the foundation for subsequent image analysis and understanding. In the task of three-dimensional reconstruction, image feature matching technology is one of the core means to achieve this goal. By taking images of objects or scenes from multiple angles and matching the features in the images, the positional relationship of feature points in 3D space can be determined, and a complete 3D model can be constructed. In the above task, the camera captures the object from different angles, which will make the same object in two images show different poses, and the pose transformation caused by the angle of view will affect the effect of feature point extraction and the matching result of the algorithm.

[0003] Image feature matching based on the SIFT (Scale-invariant feature transform) algorithm has high usability, but the generation of its descriptor depends on the local information of the image. As a whole, the global information of the image contains important clues such as the layout of the scene and the spatial relationship between objects. The lack of description of global information by manual operators makes traditional feature point extraction and matching algorithms show insufficient robustness when facing complex scenes or large geometric or angle transformations of images. For example, in the image stitching task, when the images to be stitched have large view angles, scale changes and multiple similar objects in the scene, the descriptor that only relies on local gray-scale information may not be able to accurately find the corresponding feature points, resulting in failure to stitch.

[0004] The COTR (Correspondence transformer for matching across images) algorithm is the earliest to apply the Transformer architecture to the image matching task, and uses the attention mechanism to alleviate the matching inaccuracy problem caused by large angle transformation. The attention mechanism in the architecture can focus on local or global information according to the correlation of the image region. When facing a scene containing multiple moving objects, it can capture the local motion information of each object and the global structure information of the scene at the same time. However, it still has some shortcomings. The algorithm needs to stack multiple Transformer modules to maintain the perception ability for large angle transformation. Although the Transformer has certain advantages in capturing global image information, it is limited by its high demand for video memory, so it cannot use a large number of Transformer attention modules to improve the matching accuracy. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a multi-modal semi-dense image feature matching method in a large angle scene, which predicts the overlap region frame by fusing point cloud features and image features, and then performs subsequent feature matching, thereby solving the problem of inaccurate matching precision in the prior art in a large angle scene.

[0006] To achieve the above-mentioned purpose of the application, the multi-modal semi-dense image feature matching method in a large angle scene comprises the following steps:

[0007] (1) Image download and preprocessing to obtain a training data set composed of image pairs and an overlap region frame of the image pairs;

[0008] (2) Extracting the point cloud feature map of the overlap region frame;

[0009] (3) Constructing a multi-modal overlap region detection feature matching network;

[0010] (4) Training the multi-modal overlap region detection feature matching network;

[0011] (5) Using the trained multi-modal overlap region detection feature matching network to realize multi-modal semi-dense image feature matching detection.

[0012] The purpose of the present application is achieved as follows:

[0013] The application is a multi-modal semi-dense image feature matching method under a large visual angle scene, which first builds and trains a multi-modal overlapping region detection feature matching network, then the trained multi-modal overlapping region detection feature matching network predicts the overlapping region frame in the image pair to be detected, and finally extracts the overlapping region frame and sends it into the feature matching network to obtain the final feature matching result, solving the problem of inaccurate matching precision in the prior art under a large visual angle scene.

[0014] Meanwhile, the multi-modal semi-dense image feature matching method under a large visual angle scene also has the following beneficial effects:

[0015] (1) The application fuses image features and point cloud features to improve the accuracy of image matching detection, uses the overlapping frame region pixels of two images for coordinate back projection, greatly reduces the data volume by limiting the processing region, thereby effectively reducing the GPU memory occupation, ensuring that the model can run smoothly within the range of hardware resources allowed; at the same time, the interference of non-overlapping regions on overlapping region features is avoided, so that the model can focus on feature extraction and analysis of overlapping regions, reducing the influence of irrelevant information, thereby improving the accuracy and effectiveness of feature extraction of overlapping regions by the model, laying a good foundation for subsequent feature matching tasks.

[0016] (2) The application designs a multi-modal feature alignment module, which is based on low-dimensional physical expressions and is projected into high dimensions for modeling, and can effectively obtain the relationship between 3D and 2D in the feature level, so that the model can better fuse 3D and 2D information when processing multi-modal data, further improving the performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of the multi-modal semi-dense image feature matching method under a large visual angle scene of the application;

[0018] Figure 2 is a visualization schematic diagram of the overlapping region between the image pair;

[0019] Figure 3 is a multi-modal overlapping region detection feature matching network structure diagram;

[0020] Figure 4 is a PointNet network structure diagram;

[0021] Figure 5 is a ResNet50 network structure diagram;

[0022] Figure 6 is a feature fusion module structure diagram;

[0023] Figure 7This is a diagram showing the visualization results of multimodal semi-dense image feature matching. DETAILED DESCRIPTION

[0024] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0025] Example

[0026] In this embodiment, if Figure 1 As shown, the present invention provides a multimodal semi-dense image feature matching method in a wide viewing angle scene, comprising the following steps:

[0027] (1) Image download and preprocessing;

[0028] (1.1), image download;

[0029] Download multiple sets of image pairs containing overlapping areas and the corresponding depth maps of each set of image pairs from the public dataset MegaDepth; let the i-th set of image pairs be The corresponding depth map is i=1,2,…,N, where N represents the number of image pairs;

[0030] Download multiple sets of image pairs containing overlapping areas and the corresponding depth maps of each set of image pairs from the public dataset MegaDepth; let the i-th set of image pairs be The corresponding depth map is i=1,2,…,N, where N represents the number of image pairs;

[0031] In this example, a total of 10,000 image pairs were downloaded, and the visualization results of the overlapping areas between the image pairs are as follows: Figure 2 As shown;

[0032] (1.2), image preprocessing;

[0033] (1.2.1), traverse the depth map D i , add the pixel points with pixel depth values ​​greater than 0 and the corresponding coordinates to the set V i ; Then traverse the set V i The coordinate values ​​of each pixel in the horizontal and vertical directions are selected as the minimum and maximum values ​​respectively. The four vertices of the overlapping area are obtained by combining the minimum and maximum values ​​in the horizontal and vertical directions respectively. The area surrounded by the vertices is image I. i Overlapping area box in It can be expressed as: Finally, record the overlapping area box Central location

[0034] (1.2.2) Use the back-projection formula to estimate the image The depth value of each pixel in

[0035]

[0036] Among them, (u j ,v j ) represents image I i The coordinates of the j-th pixel in , Representing an image The coordinates of the jth pixel in , d j Depth map D i The depth value of the j-th pixel in , They represent the internal and external parameters of the camera for obtaining the i-th group of image pairs;

[0037] Traversing the depth map The depth value of each pixel satisfies: The pixel points and corresponding coordinates are added to the collection in, Represents a depth map The depth value of the j-th pixel in ;

[0038] Iterating over a collection The coordinate values ​​of each pixel in the horizontal and vertical directions are selected as the minimum and maximum values ​​respectively. The four vertices of the overlapping area are obtained by combining the minimum and maximum values ​​in the horizontal and vertical directions respectively. The area surrounded by the vertices is the image. Overlapping area box in It can be expressed as: Finally, record the overlapping area box Central location

[0039] (2) Extract overlapping area boxes Point cloud feature map;

[0040] (2.1) According to the overlapping area frame In image I i Intercept the corresponding overlapping area, and according to the overlapping area frame In the image Intercept the corresponding overlapping area;

[0041] (2.2) Convert the intercepted overlapping area into a point cloud feature map;

[0042] Use the point cloud coordinate formula to calculate the point cloud coordinates of each pixel in the point cloud feature map;

[0043]

[0044] Among them, (u j' ,v j' ) indicates the overlapping area The coordinates of the j'th pixel in , Indicates overlapping areas The coordinates of the j'th pixel in d j' 、 Represent overlapping areas The depth value of the j'th pixel in (x j' ,y j' ,z j' ) represents the point cloud feature map P i The three-dimensional coordinates of the j'th pixel point in, Represents point cloud feature map The three-dimensional coordinates of the j'th pixel point in;

[0045] (3) Construct a multimodal overlapping region detection feature matching network;

[0046] In this embodiment, if Figure 3 As shown in the figure, the multimodal overlapping region detection feature matching network includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module, an overlapping box regression module, and a center regression module;

[0047] In this embodiment, if Figure 4 As shown, the point cloud feature extraction network uses the PointNet network to extract point cloud features from the point cloud feature map and obtain point cloud features. The PointNet network is an existing network, and its specific structure and processing process will not be described here in detail.

[0048] In this embodiment, if Figure 5 As shown in the figure, the image feature extraction network uses the ResNet50 network to extract the image features. Perform image feature extraction to obtain paired image features The ResNet50 network is an existing network, and its specific structure and processing process will not be described here;

[0049] In this embodiment, if Figure 6 As shown, the feature fusion module includes: a linear transformation layer, a feature aligner, a splicing module and multiple feature extraction modules connected in series;

[0050] Among them, the feature aligner consists of multiple sets of serially connected multi-layer perceptron layers, batch normalization layers, and ReLU activation function layers;

[0051] Each feature extraction module is composed of multiple sets of convolutional layers, batch normalization layers, and ReLU activation function layers connected in series;

[0052] The input of the feature fusion module includes camera internal and external parameters, point cloud features and image features, and the output is multimodal fusion features;

[0053] First, the product FE of the internal and external parameters of the camera is input into the linear transformation layer, and the input parameters FE are mapped into a high-dimensional space to obtain the high-dimensional feature F trans ; Then, the feature aligner first aligns the high-dimensional features F trans Perform alignment processing to obtain the spatial transformation matrix T, and then perform spatial projection transformation on the point cloud features according to the spatial transformation matrix T to obtain the projection feature F project ;Finally, the projection feature F project After being spliced ​​with the image features, it is input into multiple feature extraction modules in series. After multiple feature extractions, the fusion feature F is obtained. fusion In this embodiment, the fused multimodal features are

[0054] The overlapping box regression module is based on the multimodal features of the input Predict the coordinate values ​​of the four vertices of the overlapping area box

[0055] In this embodiment, the overlapping regression module is composed of a multi-layer perceptron with an embedding dimension of 256, a ReLU activation function, and a multi-layer perceptron with an embedding dimension of 4;

[0056] The center regression module consists of a single convolutional layer, which is based on the multimodal features of the input. Predict the center coordinates of the overlapping area box

[0057] (4) Training multimodal overlapping region detection feature matching network;

[0058] (4.1) Randomly select n groups of image pairs from the downloaded image pairs And the corresponding point cloud feature map In this embodiment, n=10;

[0059] (4.2), the point cloud feature map Input to the point cloud feature extraction network to extract point cloud features

[0060] (4.3), the image Input into the image feature extraction network to extract image features

[0061] (4.4) Through the feature fusion module, point cloud features and image features Perform multimodal fusion, the fusion process is:

[0062]

[0063]

[0064] Among them, Linear represents a multi-layer perceptron, Conv represents a convolutional layer, and Concat represents a concatenation operation; F trans Represents the high-dimensional mapping representation of the input parameter FE, and T represents the spatial transformation matrix;

[0065] (4.5) Multimodal features Input to the overlapping box regression module and the center regression module, and the overlapping area box is predicted by the overlapping box regression module And the coordinate values ​​of the four vertices Indicates the overlapping area box The coordinate values ​​of the upper left, lower left, upper right and lower right vertices; Indicates the overlapping area box The coordinate values ​​of the upper left, lower left, upper right, and lower right vertices of the overlapping area are predicted by the center regression module.

[0066] (4.5) Calculate the loss function value L after this round of training;

[0067]

[0068] in, express The symmetry consistency center position, L iou Indicates the calculation of IOU loss, λ con ,λ loc ,λ iou 、 They are all hyperparameters;

[0069] (4.6) Based on the loss function value L, the gradient descent method is used to update the parameters of the multimodal overlapping region detection feature matching network, and then return to step (4.1) for the next round of training until the multimodal overlapping region detection feature matching network converges;

[0070] (5) Multimodal semi-dense image feature matching;

[0071] Download the unknown image pair containing overlapping areas, input it into the trained multimodal overlapping area detection feature matching network, so as to predict the overlapping area box in the image pair, and finally extract the overlapping area box and send it into the feature matching network to obtain the final feature matching result.

[0072] In this embodiment, the feature matching network uses the SuperPoint network or the Superglue network, and the feature matching results are as follows: Figure 7 As shown, the thin lines in the figure represent the correct matching feature points.

[0073] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A multimodal semi-dense image feature matching method in a wide viewing angle scene, characterized by: The following steps are involved: (1) Image download and preprocessing; (1.1), image download; Download multiple sets of image pairs containing overlapping areas and the corresponding depth maps of each set of image pairs from the public dataset; the i-th set of image pairs is denoted as The corresponding depth map is N represents the number of image pairs; (1.2), image preprocessing; (1.2.1), traverse the depth map D i , add the pixel points with pixel depth values ​​greater than 0 and the corresponding coordinates to the set V i ; Then traverse the set V i The coordinate values ​​of each pixel point in the image are selected, and the minimum and maximum values ​​are selected in the horizontal and vertical directions respectively. The four vertices of the overlapping area are obtained by combining the minimum and maximum values ​​in the horizontal and vertical directions respectively. The area surrounded by the vertices is the image I i Overlapping area box in Finally, record the overlapping area box Central location (1.2.2) Use the back-projection formula to estimate the image The depth value of each pixel in Among them, (u j ,v j ) represents image I i The coordinates of the j-th pixel in , Representing an image The coordinates of the j-th pixel in , d j Depth map D i The depth value of the j-th pixel in , They represent the internal and external parameters of the camera for obtaining the i-th group of image pairs; Traversing the depth map The depth value of each pixel satisfies: The pixel points and corresponding coordinates are added to the collection in, Represents a depth map The depth value of the j-th pixel in ; Iterating over a collection The coordinate values ​​of each pixel point in the image are obtained by selecting the minimum and maximum values ​​in the horizontal and vertical directions respectively. The four vertices of the overlapping area are obtained by combining the minimum and maximum values ​​in the horizontal and vertical directions respectively. The area surrounded by the vertices is the image. Overlapping area box in Finally, record the overlapping area box Central location (2) Extract overlapping area boxes Point cloud feature map; (2.1), according to the overlapping area frame In image I i Intercept the corresponding overlapping area, and according to the overlapping area frame In the image Intercept the corresponding overlapping area; (2.2) Convert the intercepted overlapping area into a point cloud feature map; Use the point cloud coordinate formula to calculate the point cloud coordinates of each pixel in the point cloud feature map; Among them, (u j' ,v j' ) indicates the overlapping area The coordinates of the j'th pixel in , Indicates overlapping areas The coordinates of the j'th pixel in d j' 、 Represent overlapping areas The depth value of the j'th pixel in (x j' ,y j' ,z j' ) represents the point cloud feature map P i The three-dimensional coordinates of the j'th pixel point in, Represents point cloud feature map The three-dimensional coordinates of the j'th pixel point in; (3) Construct a multimodal overlapping region detection feature matching network; The multimodal overlapping region detection feature matching network includes a point cloud feature extraction network, an image feature extraction network, a feature fusion module, an overlapping box regression module, and a center regression module; The point cloud feature extraction network uses the PointNet network to extract point cloud features from the point cloud feature map to obtain point cloud features. The image feature extraction network uses the ResNet50 network to extract the image features. Perform image feature extraction to obtain paired image features The feature fusion module is used to align and fuse the features of different modalities. The fused multimodal features The overlapping frame regression module is based on the multimodal features of the input Predict the coordinate values ​​of the four vertices of the overlapping area box The center regression module is based on the multimodal features of the input Predict the center coordinates of the overlapping area box (4) Training multimodal overlapping region detection feature matching network; (4.1) Randomly select n groups of image pairs from the downloaded image pairs And the corresponding point cloud feature map (4.2), the point cloud feature map Input to the point cloud feature extraction network to extract point cloud features (4.3), the image Input into the image feature extraction network to extract image features (4.4) Through the feature fusion module, point cloud features and image features Perform multimodal fusion, the fusion process is: Among them, Linear represents a multi-layer perceptron, Conv represents a convolutional layer, and Concat represents a concatenation operation; F trans Represents the high-dimensional mapping representation of the input parameter FE, and T represents the spatial transformation matrix; (4.5) Multimodal features Input to the overlapping box regression module and the center regression module, and the overlapping area box is predicted by the overlapping box regression module And the coordinate values ​​of the four vertices Indicates the overlapping area box The coordinate values ​​of the upper left, lower left, upper right and lower right vertices; Indicates the overlapping area box The coordinate values ​​of the upper left, lower left, upper right, and lower right vertices of the overlapping area are predicted by the center regression module. (4.5) Calculate the loss function value L after this round of training; in, express The symmetry consistency center position, L iou Indicates the calculation of IOU loss, λ con ,λ loc ,λ iou 、 They are all hyperparameters; (4.6) Based on the loss function value L, the gradient descent method is used to update the parameters of the multimodal overlapping region detection feature matching network, and then return to step (4.1) for the next round of training until the multimodal overlapping region detection feature matching network converges; (5) Multimodal semi-dense image feature matching; Download the unknown image pair containing overlapping areas, input it into the trained multimodal overlapping area detection feature matching network, so as to predict the overlapping area box in the image pair, and finally extract the overlapping area box and send it into the feature matching network to obtain the final feature matching result.

2. The multimodal semi-dense image feature matching method in a wide viewing angle scene according to claim 1, characterized in that: The feature fusion module includes: a linear transformation layer, a feature aligner, a splicing module and multiple feature extraction modules connected in series; Among them, the feature aligner consists of multiple sets of serially connected multi-layer perceptron layers, batch normalization layers, and ReLU activation function layers; Each feature extraction module is composed of multiple sets of convolutional layers, batch normalization layers, and ReLU activation function layers connected in series; The input of the feature fusion module includes camera internal and external parameters, point cloud features and image features, and the output is multimodal fusion features; First, the product FE of the internal and external parameters of the camera is input into the linear transformation layer, and the input parameters FE are mapped into a high-dimensional space to obtain the high-dimensional feature F trans ; Then, the feature aligner first aligns the high-dimensional features F trans Perform alignment processing to obtain the spatial transformation matrix T, and then perform spatial projection transformation on the point cloud features according to the spatial transformation matrix T to obtain the projection feature F project ;Finally, the projection feature F project After being spliced ​​with the image features, it is input into multiple feature extraction modules in series. After multiple feature extractions, the fusion feature F is obtained. fusion .

Images