Method and system for constructing homography estimation model based on cross Mangban dynamic feature fusion optimization

Abstract

The invention discloses a method and system for constructing a homography estimation model based on cross Mangbar dynamic feature fusion optimization, and the method comprises the following steps: building an overall framework of a network which is composed of neural network layers and comprises an input module feature pyramid, a rough matching module and a fine matching module; setting a structure of a crossed Mangbar model; setting the number, the connection mode and the type of crossed Mangbar models in the rough matching module; training the network to obtain a homography estimation network and a Mangbar network for multi-modal image fusion, better exploring intra-modal and inter-modal features, and dynamically enhancing detail texture information of a source image and specific information of each modal; for a feature extraction and reconstruction part of the network, a Mangbar model suitable for a fusion task is designed by combining a visual state space model with dynamic convolution and channel attention, so that the performance and the global modeling capability of the Mangbar model are kept, the channel redundancy is reduced, and the local feature extraction capability is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence image registration, and specifically relates to a method and system for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization. Background Art

[0002] With the development of computer vision and the sharp increase in the demand for robot positioning slam, homography estimation of image pairs has also become a hot topic, especially for the homography estimation problem in complex scenes, which has broad application prospects. Traditional methods of homography estimation usually solve the problem through the following series of processes: feature extraction, corresponding feature matching for homography estimation. At present, the more popular methods are mainly divided into two categories, matching-based methods and unsupervised deep homography methods. However, the former lacks discriminative features when dealing with textureless or blurred images, and they may be sensitive to errors or ambiguities in the data, which makes them difficult to handle image pairs with large baselines; the latter has high learning costs, and attention tends to spread over a large area instead of focusing on the corresponding area, so noise may be introduced during the feature fusion update process, resulting in incorrect matching.

[0003] Moreover, these methods cannot fully capture the cross-modal correlation between the two modalities and have high computational complexity. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a method for constructing a homography estimation model based on CrossMamba dynamic feature fusion optimization, which utilizes the image feature extraction capability of feature pyramid (FPN), Mamba global information modeling capability and CrossMamba dynamic feature fusion capability to achieve end-to-end homography estimation.

[0005] In order to achieve the above object, in a first aspect, the present invention provides a method for constructing a homography estimation model based on CrossMamba dynamic feature fusion optimization, comprising the following steps: The network model is constructed including input module, coarse matching module and fine matching module; the input module is used to extract input features of different scales of the image, the coarse matching module is used to generate coarse-level matching, and the fine matching module is used to generate the final fine matching array; Set the extracted feature size of the input module, set the structure of the coarse matching module, and use the CrossMamba model inside the coarse matching module; set the algorithm of the fine matching module, and use the random sampling consistency geometry method; Optimize the internal weight connection parameters of the network model to obtain the homography estimation network model.

[0006] Furthermore, in the overall framework of the network model: The input module is a feature pyramid network. For a given pair of input images, a coarse feature map Cl with a size of one eighth of the original image dimension and a fine feature map Fl with a size of one half of the original image dimension are extracted. The coarse feature map Cl and the fine feature map Fl are input into the coarse matching module and the fine matching module respectively. The coarse matching module inputs the two coarse features C0 and C1 into the CrossMamba module group for intra-modal and inter-modal learning, generates a pixel-to-pixel confidence matrix Pc, performs threshold processing on Pc, performs mutual nearest neighbor search, and determines the coarse level matching array Mc; The fine matching module uses the classic random sampling consistency geometry method to locate the coarse matching position on the fine feature map given the coarse matching array Mc and the previous fine features F0 and F1, performs cropping transformation to calculate the matching probability, and finally generates the sub-pixel matching array Mf.

[0007] Furthermore, the coarse matching module adopts the dynamic feature fusion model CrossMamba based on the Mamba model, as follows: The same two features x1 are input into the self CrossMamba model for intra-modal learning, and the same two features x2 are input into the self CrossMamba model for intra-modal learning. After obtaining new features x1 and x2, the two are cross-combined (x1, x2), (x2, x1); input into CrossMamba for inter-modal learning; The above process is repeated 4 times, i.e. 4 [self, cross] modules, to obtain the detail texture information of the dynamically enhanced source image and the information of each modality.

[0008] Furthermore, CrossMamba utilizes the cross-scanning scanning mechanism and adopts a four-way scanning strategy, so that each element in the feature integrates information from all other positions in different directions to form a global receptive field, integrates the HomoAttention local attention component, and establishes connections between input features from different modalities.

[0009] Furthermore, HomoAttention uses a two-stage method to fuse feature information, learns image features through convolution and activation functions, and finally generates feature weight values. Specifically, in the first stage, the image features (x1, x2) of different modalities are first learned through the visual state space module to learn the internal information of each image, and the output (y1, y2) is passed through the HomoAttention module to calculate the local attention weight U, and the initial (y1, y2) features are dynamically weighted and combined according to the importance of the features; In the second stage, the mixed features are further input into the visual state space module to enhance the learned fusion feature information, and the original feature information (x1, x2) is dynamically weighted and combined.

[0010] Further, traversing the image along the four scanning paths includes the following steps: Flatten a 2D feature into a one-dimensional vector along 4 different directions; The S6 block independently sends the four one-dimensional vectors obtained in the previous step to the S6 model for processing; The one-dimensional vectors processed by the S6 model are fused into a 2D feature output.

[0011] In a second aspect, based on the above method, the present invention provides a system for constructing a homography estimation model based on CrossMamba dynamic feature fusion optimization, including a network structure construction module, a network parameter preset module and a training module; The network structure building module is used to build the overall framework of the network model, wherein the network model is composed of a neural network layer, including an input module, a coarse matching module and a fine matching module; the input module is used to extract input features of different scales of the image, the coarse matching module is used to generate coarse-level matching, and the fine matching module is used to generate the final fine matching array; The network parameter preset module is used to set the extraction feature size of the input module feature pyramid (FPN); set the coarse matching module structure, and use a specially connected CrossMamba model inside the coarse matching module; set the fine matching module algorithm, and use the random sampling consistency geometry method; The training module is used to optimize the internal weight connection parameters of the network model and obtain the homography estimation network model.

[0012] In the third aspect, based on the model obtained by the above method, the present invention provides a method for homography estimation based on CrossMamba dynamic feature fusion optimization, comprising the following steps: obtaining a pair of (grayscale) input images, passing them through an end-to-end homography estimation network model, and outputting the homography relationship between the pair of images, that is, the homography matrix; the end-to-end homography estimation network model is obtained by the above-mentioned method for constructing a homography estimation model based on CrossMamba dynamic feature fusion optimization.

[0013] In a fourth aspect, the present invention also provides a computer device, comprising a processor and a memory, the memory being used to store a computer executable program, the processor reading part or all of the computer executable program from the memory and executing it, and when the processor executes part or all of the computer executable program, it can implement the method for constructing a homography estimation model based on CrossMamba dynamic feature fusion optimization or the method for homography estimation based on CrossMamba dynamic feature fusion optimization described in the present invention.

[0014] At the same time, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the method for constructing a homography estimation model based on CrossMamba dynamic feature fusion optimization or the method for homography estimation based on CrossMamba dynamic feature fusion optimization described in the present invention can be implemented.

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: the method of the present invention focuses on feature fusion to enhance the model. The CrossMamba model is introduced, which is a homography estimation network using Mamba as the core module; CrossMamba uses the inherent cross-modal relationship between the two to achieve fusion, filter redundant modal features and obtain fine fusion results, extract global information from two views and perform efficient feature fusion. Due to its efficient feature extraction and fusion capabilities, the model has surpassed the current advanced methods and achieved excellent fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall architecture of the network in the present invention.

[0017] Figure 2 It is a structural schematic diagram of the coarse matching module in the present invention.

[0018] Figure 3 It is a schematic diagram of the structure of the CrossMamba module in the present invention.

[0019] Figure 4Schematic diagram of the visual Mamba module scanning method used in the present invention. DETAILED DESCRIPTION

[0020] The problem to be solved by the present invention is to perform homography estimation on images of different viewing angles. A pair of input images is obtained, and through an end-to-end homography estimation network model, the homography relationship between the pair of images, that is, the homography matrix, is output.

[0021] For example, when the camera is moving, the pictures taken at two different positions are encoded and input into the network model, and the homography relationship between the two images is output.

[0022] The input end of the network model can be various image collection devices, such as cameras, and the output end can be a specific monitoring indication device, a task execution device or a more complex computing model for further analysis and processing.

[0023] The network model is completely composed of neural network layers, which are divided into three modules in the order of information transmission: input module feature pyramid (FPN), coarse matching module and fine matching module; the original pair of images with different viewpoints is input into the network model, and the homography relationship between the two images is obtained at the output. The overall architecture of the network model is as follows: Figure 1 shown.

[0024] The network model and modules are as follows: The input module needs to be able to effectively extract features of different sizes of images. The present invention adopts a common feature pyramid (FPN) feature pyramid network. For a given pair of input images, a coarse-level feature map Cl with an eighth of the original image dimension and a fine-level feature map Fl with a half of the original image dimension are extracted, and the coarse matching module and the fine matching module are input respectively. The feature pyramid (FPN) includes three parts, a bottom-up circuit, a top-down circuit and a lateral connection. The lateral connection 1*1 convolution kernel is mainly used to adjust the output channels (channel) of different feature maps to 256, because fusion is required, and only by making them have the same shape can fusion be achieved, so that the features of other layers after upsampling can be added, and the size of the feature map is not changed. Bottom-up: In fact, it is the forward process of the network. In the forward process, selection is performed according to an integer multiple of 2. The size of the feature map will change after passing through certain layers, but will not change when passing through other layers. The layers that do not change the size of the feature map are classified as one stage, so that the features extracted each time are the last layer output of each stage, so that the feature pyramid can be formed. The backbone of the feature selection network is scaled by an integer multiple of 2. Top-down: Upsampling is used in the process, and 2x upsampling is achieved through a simple neighboring interpolation algorithm. The horizontal connection is to fuse the upsampling result with the feature map of the same size generated from the bottom up. After fusion, a 3*3 convolution kernel is used to convolve each fusion result to eliminate the aliasing effect of upsampling. It is assumed that the generated feature map results are P2, P3, P4, P5, which correspond to the original bottom-up convolution results C2, C3, C4, C5.

[0025] Coarse matching module (coarse matching) such as Figure 2 As shown, it is used to generate coarse-level matching and consists of multiple CrossMamba modules. The two coarse-level features output by the feature pyramid network (FPN) are input into the innovative CrossMamba module group for intra-modal and inter-modal learning to generate a pixel-to-pixel confidence matrix Pc. The coarse-level matching array Mc is determined by thresholding the confidence matrix Pc and then performing a mutual nearest neighbor search.

[0026] Since the subsequent modules use the random sampling consistency geometry method to significantly reduce the image area involved in the attention calculation, they are more dependent on the output of the upstream operation rough matching module, and the accuracy of the output seriously affects the results of the subsequent random sampling consistency algorithm. Therefore, the focus of the improvement of the present invention is on rough matching. First, the same two x1 are input into the CrossMamba model for intra-modal learning. The same is true for x2. After obtaining new x1 and x2, the two are combined (x1, x2), (x2, x1), and then input into CrossMamba again for inter-modal learning. The above process is repeated 4 times, that is, 4 [self, cross] modules.

[0027] CrossMamba module Figure 3 As shown in the figure, based on the improvement of the visual Mamba model, the visual state space model is combined with dynamic convolution and channel attention to design a Mamba model suitable for fusion tasks, which not only maintains the performance and global modeling ability of Mamba, but also reduces channel redundancy and enhances local feature extraction capabilities. The key to the success of V Mamba lies in the use of the selective scanning space state sequence model (S6 model), which was originally designed to solve natural language processing (NLP) tasks. Unlike the attention mechanism in the ViT model, the S6 model interacts each element in a one-dimensional vector (such as a text sequence) with the information scanned before, effectively reducing the quadratic complexity to linear.

[0028] However, since visual signals (such as images) do not have natural order like text sequences, the data scanning method in the S6 model cannot be directly applied to visual signals. For this reason, a cross scanning mechanism is designed. The cross scanning module (CSM) adopts a four-way scanning strategy, that is, scanning from the four corners of the feature map at the same time, such as Figure 4 As shown in Figure 2. This strategy ensures that each element in the feature integrates information from all other positions in different directions to form a global receptive field without increasing linear computational complexity. Based on CSM, the Visual Mamba Network proposes a 2D selection scanning module SS2D module, which constructs a context information acquisition method with linear time complexity by traversing the image along four scanning paths. The module processing process includes three steps: 1) Flatten a 2D feature into a one-dimensional vector along 4 different directions (upper left, lower right, lower left, upper right).

[0029] 2) S6 block independently sends the four one-dimensional vectors obtained in the previous step to the S6 model operation.

[0030] 3) The four one-dimensional vectors obtained in the first step are fused into a 2D feature output.

[0031] In addition to the visual Mamba module, the present invention innovatively introduces the homography attention (HomoAttention) module. The input of the attention module, the initial integration quality may greatly affect the final fusion weight. Since it is still a feature fusion problem, an intuitive approach is to use another attention module to fuse the input features. HomoAttention goes through a series of convolutional layers and activation functions, and finally assigns the obtained weights to different modalities, establishes connections between input features from different modalities, and adopts a two-stage method to fuse feature information. This module can effectively mine the correlation information between modalities and promote better information interaction between modalities.

[0032] Here is the calculation process of the CrossMamba model: In the first stage, the image features of different modalities (x1, x2) are first used to learn the internal information of each image through the visual state space module.

[0033]

[0034]

[0035] The obtained output (y1, y2) is passed through the HomoAttention module to calculate the local attention weight U. W1 and W2 are point-by-point convolutions.

[0036]

[0037] The initial (y1, y2) features are dynamically weighted and combined according to their importance, thereby improving the model's ability to extract relevant information from the input signal.

[0038]

[0039] Entering the second stage, the mixed features are further input into the visual state space module to enhance the learned fusion feature information.

[0040]

[0041] Dynamically weight and combine the original feature information (x1, x2).

[0042]

[0043] The network construction and training steps of the present invention are as follows: 1. Build an overall framework based on the CrossMamba dynamic feature fusion network. Based on the CrossMamba dynamic feature fusion network, the overall architecture includes the input module feature pyramid (FPN), the coarse matching module and the fine matching module to form an end-to-end homography estimation network model. The one-eighth feature map extracted from the input module feature pyramid (FPN) is directly input into the coarse matching network; the CrossMamba module of the coarse matching network combines the visual state space model with dynamic convolution and channel attention for multimodal image fusion; the one-half feature map extracted from the input feature pyramid (FPN) plus the coarse matching matrix output by the coarse matching network are input into the fine matching network, and finally the fine matching network outputs the final homography matrix.

[0044] 2. Train the network. We use PyTorch to implement the network. Based on our GPU resources (4 NVIDIA GeForce RTX 3090 cards), the maximum size of the training image is set to 640 pixels. The optimizer uses Adam, β = (0.9, 0.999), and the initial learning rate is 0.001. Each mini-batch contains a pair of images. The maximum number of training cycles is 20. In the inference phase, given the fine match Mf, cv2.findHomography and random sampling consistency are used as robust estimators for homography fitting.

[0045] The training results are as follows. Compared with the Transformer Local Feature Matching Model (LoFTR), GeoFormer Model, and GeoFormer with quadtree, the CrossMamba model has much higher accuracy than other models at all scales. We also verified the effectiveness of the HomoAtttion module:

[0046] Embodiment 2, the present invention provides a method for homography estimation based on CrossMamba dynamic feature fusion optimization, comprising the following steps: A pair of input images is obtained, and through an end-to-end homography estimation network model, the homography relationship between the pair of images, that is, the homography matrix, is output; the input image is a grayscale image; In the end-to-end homography estimation network model: image features are extracted based on a feature pyramid (FPN) feature pyramid network, for example, a coarse feature map Cl with a size of one eighth of the original image dimension and a fine feature map Fl with a size of one half of the original image dimension can be extracted; Perform intra-modal and inter-modal learning on a set of coarse-level feature maps to generate a pixel-to-pixel confidence matrix Pc. Then, perform threshold processing on the confidence matrix Pc and perform mutual nearest neighbor search to determine the coarse-level matching array Mc. Using the classic random sampling consistency geometry method, given the coarse-level matching array Mc and the previous fine features F0 and F1, locate the coarse matching position on the fine feature map, perform cropping transformation to calculate the matching probability, and finally generate the sub-pixel matching array Mf; The same two features x1 are learned within the modality, and the same two features x2 are learned within the modality to obtain new features x1 and x2, which are then cross-combined to form (x1, x2), (x2, x1); inter-modality learning is performed; the above process is repeated 4 times to obtain the detailed texture information of the dynamically enhanced source image and the information of each modality; a cross-scan scanning mechanism is used with a four-way scanning strategy so that each element in the feature integrates information from all other positions in different directions to form a global receptive field, and based on the homography attention (HomoAttention) local attention component, image features are learned through convolution and activation functions to finally generate feature weight values ​​and establish connections between input features from different modalities.

[0047] The present invention can also provide a computer device, including a processor and a memory, the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and when the processor executes part or all of the computer executable program, it can implement the method for constructing a homography estimation model based on CrossMamba dynamic feature fusion optimization according to the present invention.

[0048] On the other hand, the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it can implement the method for constructing a homography estimation model based on CrossMamba dynamic feature fusion optimization and the method for homography estimation based on CrossMamba dynamic feature fusion optimization described in the present invention.

[0049] The computer device may be a notebook computer, a desktop computer, a vehicle-mounted computer or a workstation.

[0050] The memory described in the present invention may be an internal storage unit of a laptop computer, a desktop computer, a vehicle-mounted computer or a workstation, such as a memory or a hard disk; or an external storage unit, such as a mobile hard disk or a flash memory card.

[0051] Computer-readable storage media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media may include: read-only memory (ROM), random access memory (RAM), solid-state drive (SSD) or optical disk, etc. Among them, random access memory may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

Claims

1. A method for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization, characterized in that: The following steps are involved: The network model is constructed including input module, coarse matching module and fine matching module; the input module is used to extract input features of different scales of the image, the coarse matching module is used to generate coarse-level matching, and the fine matching module is used to generate the final fine matching array; Set the extraction feature size of the input module, set the structure of the coarse matching module, and use the cross-Mamba model inside the coarse matching module; set the algorithm of the fine matching module, and use the random sampling consistency geometry method; Optimize the internal weight connection parameters of the network model to obtain the homography estimation network model.

2. According to claim 1, the method for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization is characterized in that: In the overall framework of the network model: The input module is a feature pyramid network. For a given pair of input images, a coarse feature map Cl with a size of one eighth of the original image dimension and a fine feature map Fl with a size of one half of the original image dimension are extracted. The coarse feature map Cl and the fine feature map Fl are input into the coarse matching module and the fine matching module respectively. The coarse matching module inputs the two coarse features C0 and C1 into the cross-mamba module group for intra-modal and inter-modal learning, generates a pixel-to-pixel confidence matrix Pc, performs threshold processing on Pc, performs mutual nearest neighbor search, and determines the coarse-level matching array Mc; The fine matching module uses the classic random sampling consistency geometry method to locate the coarse matching position on the fine feature map given the coarse matching array Mc and the previous fine features F0 and F1, performs cropping transformation to calculate the matching probability, and finally generates the sub-pixel matching array Mf.

3. According to claim 1, the method for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization is characterized in that: The coarse matching module uses the dynamic feature fusion model cross-Mamba based on the Mamba model, as follows: The same two features x1 are input into the self-crossed Mamba model for intra-modal learning. The same two features x2 are input into the self-crossed Mamba model for intra-modal learning. After obtaining new features x1 and x2, the two are cross-combined (x1, x2), (x2, x1); input into the cross-crossed Mamba model for inter-modal learning. The above process is repeated multiple times, i.e., multiple [self, cross] modules, to obtain the detailed texture information of the dynamically enhanced source image and the information of each modality.

4. According to claim 3, the method for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization is characterized in that: Cross Mamba uses a cross-scanning mechanism and a four-way scanning strategy to enable each element in the feature to integrate information from all other positions in different directions to form a global receptive field, incorporate the homography attention local attention component, and establish connections between input features from different modalities.

5. According to claim 4, the method for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization is characterized in that: Homography attention uses a two-stage method to fuse feature information, learns image features through convolution and activation functions, and finally generates feature weight values. Specifically, in the first stage, the image features (x1, x2) of different modalities are first learned through the visual state space module to learn the internal information of each image, and the output (y1, y2) is passed through the homography attention module to calculate the local attention weight U, and the initial (y1, y2) features are dynamically weighted and combined according to the importance of the features; In the second stage, the mixed features are input into the visual state space module to enhance the learned fusion feature information, and the original feature information (x1, x2) is dynamically weighted and combined.

6. According to claim 1, the method for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization is characterized in that: Traversing the image along the four scan paths involves the following steps: Flatten a 2D feature into a one-dimensional vector along 4 different directions; The S6 block independently sends the four one-dimensional vectors obtained in the previous step to the S6 model for processing; The one-dimensional vectors processed by the S6 model are fused into a 2D feature output.

7. A system for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization, characterized in that: It includes a network structure building module, a network parameter preset module and a training module; The network structure building module is used to build the overall framework of the network model, wherein the network model is composed of a neural network layer, including an input module, a coarse matching module and a fine matching module; the input module is used to extract input features of different scales of the image, the coarse matching module is used to generate coarse-level matching, and the fine matching module is used to generate the final fine matching array; The network parameter preset module is used to set the extraction feature size of the input module feature pyramid FPN; set the coarse matching module structure, and use a specially connected cross-Mamba model inside the coarse matching module; set the fine matching module algorithm, and use a random sampling consistency geometry method; The training module is used to optimize the internal weight connection parameters of the network model and obtain the homography estimation network model.

8. A homography estimation method based on cross-Mamba dynamic feature fusion optimization, characterized in that: The method comprises the following steps: obtaining a pair of grayscale input images, passing them through an end-to-end homography estimation network model, and outputting the homography relationship between the pair of images, that is, the homography matrix; the end-to-end homography estimation network model is obtained by adopting the method for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization as described in any one of claims 1 to 6.

9. A computer device, characterized in that: It includes a processor and a memory, the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and when the processor executes part or all of the computer executable program, it can implement the method for constructing a homography estimation model based on cross-Mamba dynamic feature fusion optimization as described in any one of claims 1 to 6 or the method for homography estimation based on cross-Mamba dynamic feature fusion optimization as described in claim 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the method for constructing a homography estimation model based on cross-mamba dynamic feature fusion optimization as described in any one of claims 1 to 6 or the method for homography estimation based on cross-mamba dynamic feature fusion optimization as described in claim 7.

Images