A depth homography estimation method based on dense scene matching

Abstract

This invention discloses a depth homography estimation method based on dense scene matching, belonging to the field of computer vision. The method proposes a coarse-to-fine depth homography estimation network consisting of three sub-models, combining linear and nonlinear homography estimation. The first two sub-models perform linear homography estimation for coarse alignment of two images, while the last sub-model performs nonlinear homography estimation for fine alignment through mesh deformation. This coarse-to-fine estimation approach further improves image alignment accuracy within a limited network capacity. Furthermore, the method explicitly estimates corner offsets through a dense scene matching module, considering the correlation between image feature points and incorporating matching information. This more reasonably establishes the connection between the image and corner offsets, avoiding network overfitting and improving the network's ability to process image details.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and more specifically, relates to a depth homography estimation method based on dense scene matching. Background Technology

[0002] A homography matrix represents the projective mapping relationship between two planes. For two images taken from different angles, the homography matrix can project one image onto the other to align them. The homography principle is widely used in many computer vision tasks such as image registration, panoramic stitching, robot localization SLAM, and AR (Augmented Reality). Traditional homography estimation methods typically extract image feature points for registration first, and then calculate the homography matrix using RANSAC and direct linear transformation. The homography matrix can only perfectly align two images of purely rotated or planar scenes, but real-world images often cannot meet these requirements, so some methods have proposed grid deformation strategies. These methods divide the image into multiple grids using a grid, and then estimate a homography matrix for each grid. This multi-homography matrix describes the nonlinear mapping relationship between the two images, and can better align image details. However, all these traditional homography estimation methods are highly dependent on feature matching, making them very unreliable in weakly textured scenes. Some existing deep homography estimation methods estimate the homography transformation between two images by training neural networks. These methods transform homography estimation into estimating corner offsets via implicit regression networks, avoiding reliance on feature matching and thus achieving better results than traditional methods. However, the implicit regression networks in these methods are designed to be very simple, typically using only multiple cascaded convolutional layers to regress corner offsets, which limits further performance improvements. However, this type of implicit regression network has two problems: 1) Corner offset is essentially a coordinate system expressing a matching relationship, but the implicit network completely ignores matching and directly establishes a connection between the image and the corner offset, easily leading to overfitting. 2) The expressive power of implicit regression networks is limited, resulting in poor performance in processing image details. Summary of the Invention

[0003] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a depth homography estimation method based on dense scene matching. This solves the problems of existing methods failing to perfectly align image details and easily leading to network overfitting, thereby further improving the alignment accuracy of homography estimation and enhancing its application performance in various visual tasks.

[0004] To achieve the above objectives, according to a first aspect of the present invention, a depth homography estimation method based on dense scene matching is provided, comprising:

[0005] Image I to be aligned T and reference image I RInput to a pre-trained deep monopitch estimation network to convert I T with I R Alignment;

[0006] The depth homography estimation network includes:

[0007] The first linear homography estimation model is used for I R I T Linear homography estimation yields I T Compared to I R The homography transformation matrix H1;

[0008] The coarse alignment module is used to align I according to H1. T Transform to I R Image obtained on the image plane

[0009] The second linear homography estimation model is used for I R and Linear homography estimation is obtained Compared to I R The homography transformation matrix H2;

[0010] The fine alignment module is used to align I according to H1*H2. T Transform to I R Image obtained on the image plane

[0011] Nonlinear homography estimation model, used for I R and Nonlinear homography estimation is performed to obtain Compared to I R Mesh homography transformation diagram H map ;

[0012] The detail alignment module is used to align H1*H2*H map to I T Perform mesh deformation projection to transform it to I R The aligned image is obtained on the image plane.

[0013] The depth homography estimation network uses I R `and The loss function is obtained by training with photometric consistency loss and edge consistency loss as the loss functions; where, I R `As a reference image sample, To make I R `and sample image I T The aligned image is input into the depth homography estimation network.

[0014] According to a second aspect of the present invention, an electronic device is provided, comprising: a computer-readable storage medium and a processor;

[0015] The computer-readable storage medium is used to store executable instructions;

[0016] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0017] According to a third aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first aspect.

[0018] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0019] The method provided by this invention designs an end-to-end depth homography estimation network, which includes two linear homography estimation models and one nonlinear multihomography estimation model. These three sub-models progressively align two input images at different resolutions. Using two images taken from different perspectives as input, the network predicts the homography transformation relationship between the two images. The homography matrix is ​​then applied to project one image onto the other to achieve image alignment. A coarse-to-fine depth homography estimation method is employed, with the three sub-models progressively aligning the two images at different resolutions. Combining linear homography estimation and nonlinear multihomography estimation image alignment strategies, linear homography estimation achieves coarse alignment, while multihomography estimation further refines the alignment of local details. Features are re-extracted from the projected image in each sub-model to meet the different property requirements of image features at different alignment stages, resulting in stronger invariance in the coarse alignment stage and stronger specificity in the fine alignment stage. This invention incorporates edge consistency loss during network training, making the network pay more attention to image edge information, improving the network's ability to process local image details, and further enhancing image alignment accuracy. The method provided by this invention can be used in multiple fields such as panoramic cameras, video surveillance, and autonomous driving.

[0020] Furthermore, the method provided by this invention explicitly estimates the corner point offset through a dense scene matching module. Compared with traditional implicit regression networks, the explicit method of this invention considers the correlation between image feature points when calculating the matching probability between image feature points during the calculation process, thereby incorporating matching information and establishing a more reasonable connection between the image and the corner point offset. This avoids network overfitting and improves the network's ability to process image details. Attached Figure Description

[0021] Figure 1 A schematic diagram illustrating the training process of a depth homography estimation network provided in an embodiment of the present invention;

[0022] Figure 2 X provided for embodiments of the present invention grid and Y grid A cutting diagram. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0024] This invention provides a depth homography estimation method based on dense scene matching, comprising:

[0025] Image I to be aligned T and reference image I R Input to a pre-trained deep monopitch estimation network to convert I T with I R Alignment;

[0026] The depth homography estimation network includes:

[0027] The first linear homography estimation model is used for I R I T Linear homography estimation yields I T Compared to I R The homography transformation matrix H1;

[0028] The coarse alignment module is used to align I according to H1. T Transform to I R Image obtained on the image plane

[0029] The second linear homography estimation model is used for I R and Linear homography estimation is obtained Compared to I R The homography transformation matrix H2;

[0030] The fine alignment module is used to align I according to H1*H2. T Transform to I R Image obtained on the image plane

[0031] Nonlinear homography estimation model, used for I R and Nonlinear homography estimation is performed to obtain Compared to I R Mesh homography transformation diagram H map ;

[0032] The detail alignment module is used to align H1*H2*H map to I T Perform mesh deformation projection to transform it to I R The aligned image is obtained on the image plane.

[0033] The depth homography estimation network uses I R `and The loss function is obtained by training with photometric consistency loss and edge consistency loss as the loss functions; where, I R `As a reference image sample, To make I R `and sample image I T The aligned image is input into the depth homography estimation network.

[0034] Specifically, such as Figure 1 As shown, the training process of the deep homography estimation network provided by this invention is as follows:

[0035] Step S1. Read two input images I R I T These two images were taken from different perspectives and have some overlapping content. The goal is to combine image I. T Projected onto image I R Align the two images on a flat surface;

[0036] Step S2. Perform preprocessing operations on the images, scaling both input images to a preset resolution (e.g., 128*128, 256*256, etc.) to meet the input requirements of the depth homography estimation network;

[0037] Step S3. Transfer the preprocessed image I R and I T The first linear homography estimation is performed by inputting sub-model 1 of the deep homography estimation network (i.e., the first linear homography estimation model of the deep homography estimation network), and I is obtained through neural network estimation. T Compared to I R The homography transformation matrix H1;

[0038] Step S4. Apply homography matrix H1 to I T Transform to I R Image obtained on the image plane This achieves coarse alignment between the two images;

[0039] Step S5. Align the coarsely aligned image I R and Sub-model 2 of the deep homography estimation network (i.e., the nonlinear homography estimation model of the deep homography estimation network) is input for a second linear homography estimation, which is then obtained through neural network estimation. Compared to I R The homography transformation matrix H2;

[0040] Step S6. Apply the homography transformation matrix H1*H2 to I T Transform to I R Image obtained on the image plane This achieves precise alignment of the two images;

[0041] Step S7. Align the finely aligned image I R and Sub-model 3 of the deep homography estimation network (i.e., the first linear homography estimation model of the deep homography estimation network) is input for nonlinear homography estimation, and the result is obtained through neural network estimation. Compared to I R Mesh homography transformation diagram H map ;

[0042] Step S8. Apply the homography transformation matrix H1*H2*H map to I T Perform mesh deformation projection and transform it to I. R Image obtained on the image plane This achieves detail alignment between the two images;

[0043] Step S9. During algorithm testing, steps S1-S8 constitute all steps. When jointly training sub-models 1-3, the final transformed image needs to be... With Image I R We calculate the photometric consistency loss and edge consistency loss together, update the convolutional network parameters through the backpropagation of the gradients of these two loss functions, and train the network model to converge to the optimal alignment position of the two images.

[0044] It is worth noting that the aforementioned first linear homography estimation, second linear homography estimation, or nonlinear homography estimation can all employ existing homography estimation methods. Furthermore, to improve the accuracy of homography estimation, preferably, the first linear homography estimation model provided by this invention is used to perform linear homography estimation to obtain H1. The first linear homography estimation model includes:

[0045] A multi-scale feature encoder is used to extract I separately. R and I T Feature map and

[0046] The global correlation calculation module is used to calculate... Each feature point in the data and The global correlation matrix is ​​obtained by calculating the cosine similarity of all feature points in the matrix.

[0047] A first U-shaped network is used to perform contextual information interaction based on the global correlation matrix to update the global correlation matrix;

[0048] The first matching probability calculation module is used to perform softmax regularization on the updated global correlation matrix along the channel direction to obtain a first probability matrix; each element in the first probability matrix represents I. R and I T The matching probability of each pair of feature points in the dataset;

[0049] The first dense coordinate graph calculation module is used to calculate the first probability matrix and the first horizontal and vertical coordinate sequence X. Index1 and Y Index1 Multiplying them yields the corresponding dense coordinate map X. map1 and Y map1 ;in, k1 is the number of channels in the first probability matrix. It is a feature map The width of , % is for integer division and remainder calculation, / / is for integer division calculation;

[0050] The first fully connected layer is used to process X respectively. map1 and Y map1 I is obtained by performing coordinate sampling R Compared to I T The four corner points are offset;

[0051] The first linear transformation module is used to transform the I... R Compared to I T H1 is obtained by directly linearly transforming the offsets of the four corner points.

[0052] That is, step S3 includes:

[0053] S301. For image I R and I T The first preset size feature map is extracted using a multi-scale feature encoder with shared weights (i.e., sub-models 1-3 share the same feature encoder). and Perform intensive scene matching;

[0054] S302. For For each feature point in the dataset, calculate its relationship with... The cosine similarity of all feature points yields a global correlation cost volume (i.e., a global correlation matrix);

[0055] S303. Input the global correlation cost volume into a U-shaped network with preferred triple upsampling for context information interaction to obtain a cost volume with the same size as the original cost volume but more accurate;

[0056] S304. Perform softmax regularization on the cost volume along the channel direction to obtain a probability volume (i.e., the first probability matrix), where each element in the probability volume represents the matching probability of a pair of feature points in two images;

[0057] S305. Multiply the probability volume by the corresponding x and y coordinate sequence X of each point. Index1 and Y Index1 Two dense coordinate graphs X were obtained. map and Y map The sequence of horizontal and vertical coordinates can be calculated using the following formula:

[0058]

[0059] Where k1 is the number of channels of the probability volume. It is a feature map The width of the variable; % is for integer division and remainder calculation, / / is for integer division calculation.

[0060] S306. The two dense coordinate maps are respectively processed by a sampling module composed of fully connected layers to obtain image I. R Relative to image I T The four corner points are offset;

[0061] S307. The estimated corner point offsets are used to calculate image I through a direct linear transformation. T Relative to image I R The homography transformation matrix H1.

[0062] To improve the accuracy of homography estimation, preferably, the second linear homography estimation model provided by the present invention is used to perform linear homography estimation to obtain H2. The second linear homography estimation model includes:

[0063] A multi-scale feature encoder is used to extract I separately. R and Feature map and

[0064] The local correlation calculation module is used to calculate... Each feature point in the data and The cosine similarity of all feature points within a preset region is used to obtain a local correlation matrix; wherein, the preset region is a rectangular region of size (2R+1)*(2R+1), and R is the radius of the rectangular region;

[0065] The second U-shaped network is used to perform contextual information interaction based on the global correlation matrix to update the local correlation matrix;

[0066] The second matching probability calculation module is used to perform softmax regularization on the updated local correlation matrix along the channel direction to obtain a second probability matrix; each element in the probability matrix represents I. R and The matching probability of each pair of feature points in the dataset;

[0067] The second dense coordinate graph calculation module is used to combine the second probability matrix with the second horizontal and vertical coordinate sequence X. Index2 and Y Index2 Multiplying them yields the corresponding dense coordinate map X. map2 and Y map2 ; where X Index (k)=k2%(2R+1)-R,Y Index2 (k) = k2 / / (2R+1) - R, where k2 is the number of channels in the second probability matrix, % is the integer division modulo operation, and / / is the integer division operation.

[0068] The second fully connected layer is used to process X separately. map2 and Y map2 I is obtained by performing coordinate sampling R Compared to The four corner points are offset;

[0069] The second linear transformation module is used to transform the I... R Compared to H2 is obtained by directly linearly transforming the offsets of the four corner points.

[0070] That is, step S5 includes:

[0071] S501. For image I R and A multi-scale feature encoder with shared weights is used to extract feature maps of the second preset size. and Perform intensive scene matching;

[0072] S502. For For each feature point in the graph, we calculate its relationship with... The cosine similarity of all feature points within a rectangular region of size (2R+1)*(2R+1) corresponding to the position yields a local correlation cost body (i.e., a local correlation matrix); R is the radius of the rectangular region. When the image resolution of the input network is 128*128, R can be 3.

[0073] S503. Input the local correlation cost volume into a U-shaped network with preferred triple upsampling for context information interaction to obtain a cost volume with the same size as the original cost volume but more accurate;

[0074] S504. Perform softmax regularization on the cost volume along the channel direction to obtain a probability volume, where each element in the probability volume represents the matching probability of a pair of feature points in two images;

[0075] S505. Multiply the probability volume by the corresponding x and y coordinate sequence X of each point. Index2 and Y Index2 Two dense coordinate graphs X were obtained. map and Y map The sequence of horizontal and vertical coordinates can be calculated using the following formula:

[0076] X Index2 (k2)=k2%(2R+1)-R,Y Index2 (k)=k2 / / (2R+1)-R,

[0077] Where k2 is the number of channels in the probability volume;

[0078] S506. The two dense coordinate maps are respectively processed by a sampling module composed of fully connected layers to obtain image I. R Relative to image The four corner points are offset;

[0079] S507. The estimated corner point offsets are used to calculate image I through a direct linear transformation. T Relative to image The homography transformation matrix H2.

[0080] To improve the accuracy of homography estimation, preferably, the nonlinear homography estimation model provided by this invention is used to perform nonlinear homography estimation to obtain H. map The nonlinear homography estimation model includes:

[0081] A multi-scale feature encoder is used to extract I separately. R and Feature map and

[0082] The correlation calculation module is used to calculate... Each feature point in the data and The local correlation matrix of the target is obtained by the cosine similarity of all feature points within the preset region; wherein, the preset region is a rectangular region of size (2R+1)*(2R+1), and R is the radius of the rectangular region;

[0083] A U-shaped network is used to perform contextual information interaction based on the target local correlation matrix in order to update the target local correlation matrix;

[0084] The matching probability calculation module is used to perform softmax regularization on the updated target local correlation matrix along the channel direction to obtain the target probability matrix; each element in the target probability matrix represents I. R and The matching probability of each pair of feature points in the dataset;

[0085] The dense coordinate graph calculation module is used to calculate the target probability matrix and the x and y coordinate sequences X. Index and Y Index Corresponding dense coordinate map X map and Y map ; where X Index (k)=k%(2R+1)-R,Y Index (k) = k / / (2R+1)-R, where k is the number of channels in the target probability matrix, % is the integer division modulo operation, and / / is the integer division operation.

[0086] Fully connected layers are used to process X separately. map and Y map I is obtained by performing coordinate sampling R Compared to Grid point coordinate offset map X grid and Y grid It can be understood that the grid point coordinate offset map of an image is a map composed of the horizontal and vertical coordinate offsets of each grid point after the image is gridded.

[0087] The offset calculation module is used to calculate the offsets of X starting from (0,0), (0,1), (1,0), and (1,1), respectively. grid and Y grid The coordinate offset maps of all the cut maps are overlaid to form a mesh; the eight-channel tensor of each grid position in the mesh is the offset of the four corner points of that grid position; where X grid and Y grid All dimensions are (grid) h +1)*(grid w +1), the dimensions of the grid point coordinate offset map after cutting are all grid. h *grid w grid h The number of rows in the grid, grid w This represents the number of columns in the grid.

[0088] like Figure 2 As shown, X grid and Y grid The size is (grid)h +1)*(grid w +1), corresponding to the x and y coordinate offsets of all grid cells. Where, grid h The grid represents the number of rows. w This indicates the number of columns in the grid. Starting from (0,0), (0,1), (1,0), and (1,1) respectively, X is divided into... grid Cut into four grid-sized sheets h *grid w The coordinate offset diagrams represent the top-left, top-right, bottom-left, and bottom-right horizontal coordinate offsets for all grid cells. Similarly, starting from (0,0), (0,1), (1,0), and (1,1), the Y-axis is shifted... grid Cut into four grid-sized sheets h *grid w The coordinate offset maps represent the top-left, top-right, bottom-left, and bottom-right ordinate offsets of all grid points. These eight grid point coordinate offset maps are overlaid to form a grid volume. The eight-channel tensor of each grid point's location represents the offset of its four corner points.

[0089] The linear transformation module is used to perform a direct linear transformation calculation on the offsets of the four corner points of the mesh to obtain H. map .

[0090] That is, step S7 includes:

[0091] S701. For image I R and A multi-scale feature encoder with shared weights is used to extract feature maps of a third preset size. and Perform intensive scene matching;

[0092] S702. For For each feature point in the graph, we calculate its relationship with... The cosine similarity of all feature points within a rectangular region of size (2R+1)*(2R+1) at the corresponding position yields a local correlation cost volume (i.e., the target correlation matrix);

[0093] S703. Input the local correlation cost volume into a U-shaped network with preferred triple upsampling for context information interaction to obtain a cost volume with the same size as the original cost volume but more accurate;

[0094] S704. Perform softmax regularization on the cost volume along the channel direction to obtain a probability volume, where each element represents the matching probability of a pair of feature points in two images.

[0095] S705. Multiply the probability volume by the corresponding x and y coordinate sequence X of each point. Index and Y Index Two dense coordinate graphs X were obtained. map and Y map The sequence of horizontal and vertical coordinates can be calculated using the following formula:

[0096] X Index (k)=k%(2R+1)-R,Y Index (k)=k / / (2R+1)-R,

[0097] Where k is the number of channels in the probability volume;

[0098] S706. The two dense coordinate maps are respectively processed by a sampling module composed of fully connected layers to obtain image I. R Relative to image Grid point coordinate offset map X grid and Y grid ;

[0099] S707. Starting from (0,0), (0,1), (1,0), (1,1), respectively, X grid and Y grid After being cut and overlapped, they form a mesh; the eight-channel tensor of each mesh location represents the offset of the four corner points of that mesh location;

[0100] S708. The estimated corner point offsets of the mesh are calculated to obtain image I through a direct linear transformation. T Relative to image Mesh homography transformation diagram H map .

[0101] and Size A1 and Size A2 and The dimensions A3 increase sequentially. For example, the first to third preset dimensions are 1 / 8, 1 / 4, and 1 / 2 respectively.

[0102] In summary, the method provided by this invention proposes a coarse-to-fine deep homography estimation network, which consists of three sub-models combining linear and nonlinear homography estimation. The first two sub-models perform linear homography estimation for coarse alignment of two images, while the last sub-model performs nonlinear homography estimation for fine alignment through mesh deformation. This coarse-to-fine estimation approach can further improve image alignment accuracy within a limited network capacity.

[0103] This invention provides an electronic device, including: a computer-readable storage medium and a processor;

[0104] The computer-readable storage medium is used to store executable instructions;

[0105] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0106] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0107] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A depth homography estimation method based on dense scene matching, characterized in that, include: Image to be aligned I T and reference images I R Input into a pre-trained deep monoparticle estimation network to... I T and I R Alignment; The depth homography estimation network includes: The first linear homography estimation model is used for... I R , I T Linear homography estimation is obtained I T Compared to I R homography transformation matrix H 1; The coarse alignment module is used to align according to H 1 I T Transform to I R Image obtained on the image plane ; The second linear homography estimation model is used for... I R and Linear homography estimation is obtained Compared to I R homography transformation matrix H 2; The fine alignment module is used to align according to H 1 H 2 I T Transform to I R Image obtained on the image plane ; Nonlinear homography estimation model, used for... I R and Nonlinear homography estimation is performed to obtain Compared to I R Mesh Uniform Transformation Diagram H map ; The detail alignment module is used to align details according to... H 1 H 2 H map right I T Perform mesh deformation projection to transform it to I R Aligned image obtained on the image plane ; The depth homography estimation network uses I R and The loss function is obtained by training with photometric consistency loss and edge consistency loss as the loss functions; where... I R For reference image samples, To be I R and sample images I T The aligned image is input into the depth monotone estimation network. The nonlinear homography estimation model includes: A multi-scale feature encoder is used to extract features separately. I R and Feature map and ; The correlation calculation module is used to calculate... Each feature point in the data and The cosine similarity of all feature points within a preset region is used to obtain the target local correlation matrix; wherein, the size of the preset region is (2 R +1) (2 R A rectangular area of ​​+1), R The radius of the rectangular region; A U-shaped network is used to perform contextual information interaction based on the target local correlation matrix in order to update the target local correlation matrix; The matching probability calculation module is used to perform softmax regularization on the updated target local correlation matrix along the channel direction to obtain the target probability matrix; each element in the target probability matrix represents... I R and The matching probability of each pair of feature points in the dataset; The dense coordinate graph calculation module is used to combine the target probability matrix with the horizontal and vertical coordinate sequences. X Index and Y Index Corresponding dense coordinate graph X map and Y map ;in, , , k It is the number of channels in the target probability matrix. For division and remainder calculation, For divisibility calculations; Fully connected layers are used to separately handle... X map and Y map Coordinate sampling is performed to obtain I R Compared to Grid point coordinate offset map X grid and Y grid ; The offset calculation module is used to calculate the offsets starting from (0,0), (0,1), (1,0), and (1,1), respectively. X grid and Y grid The mesh is cut, and the coordinate offset maps of all the cut mesh points are superimposed to form a mesh body; the eight-channel tensor of each mesh position is the offset of the four corner points of that mesh position; where, X grid and Y grid The dimensions are all ( grid h +1) ( grid w +1), the dimensions of the offset map of the cut grid points are all... grid h grid w , grid h The number of rows in the grid. grid w The number of columns in the grid; The linear transformation module is used to perform direct linear transformation calculations on the offsets of the four corner points of the mesh. H map .

2. The method as described in claim 1, characterized in that, The first linear homography estimation model includes: A multi-scale feature encoder is used to extract features separately. I R and I T Feature map and ; The global correlation calculation module is used to calculate... Each feature point in the data and The global correlation matrix is ​​obtained by calculating the cosine similarity of all feature points in the matrix. A first U-shaped network is used to perform contextual information interaction based on the global correlation matrix to update the global correlation matrix; The first matching probability calculation module is used to perform softmax regularization on the updated global correlation matrix along the channel direction to obtain the first probability matrix; each element in the first probability matrix represents I R and I T The matching probability of each pair of feature points in the dataset; The first dense coordinate graph calculation module is used to combine the first probability matrix with the first horizontal and vertical coordinate sequence. X Index1 and Y Index1 Multiplication yields the corresponding dense coordinate graph X map1 and Y map1 ;in, It is the number of channels in the first probability matrix. It is a feature map width, For division and remainder calculation, For divisibility calculations; The first fully connected layer is used to respectively... X map1 and Y map1 Coordinate sampling is performed to obtain I R Compared to I T The four corner points are offset; The first linear transformation module is used to transform the... I R Compared to I T The offsets of the four corner points are calculated by direct linear transformation. H 1.

3. The method as described in claim 2, characterized in that, The second linear homography estimation model includes: A multi-scale feature encoder is used to extract features separately. I R and Feature map and ; The local correlation calculation module is used to calculate... Each feature point in the data and The cosine similarity of all feature points within a preset region is used to obtain a local correlation matrix; wherein, the size of the preset region is (2... R +1) (2 R A rectangular area of ​​+1), R The radius of the rectangular region; The second U-shaped network is used to perform contextual information interaction based on the global correlation matrix to update the local correlation matrix; The second matching probability calculation module is used to perform softmax regularization on the updated local correlation matrix along the channel direction to obtain a second probability matrix; each element in the probability matrix represents... I R and The matching probability of each pair of feature points in the dataset; The second dense coordinate graph calculation module is used to combine the second probability matrix with the second horizontal and vertical coordinate sequences. X Index2 and Y Index2 Multiplication yields the corresponding dense coordinate graph X map2 and Y map2 ;in, , , It is the number of channels in the second probability matrix. For division and remainder calculation, For divisibility calculations; The second fully connected layer is used to respectively... X map2 and Y map2 Coordinate sampling is performed to obtain I R Compared to The four corner points are offset; The second linear transformation module is used to transform the... I R Compared to The offsets of the four corner points are calculated by direct linear transformation. H 2.

4. The method as described in claim 1, characterized in that, and Size A1 and Size A2 and The dimensions A3 increase sequentially.

5. The method as described in claim 1, characterized in that, The image to be aligned I T and reference images I R Before inputting into the pre-trained deep monometry estimation network, the following is also included: right I T and I R Preprocessing is performed to make its resolution size meet the input requirements of the depth homography estimation network.

6. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in any one of claims 1-5.

Images