A method and system, device, medium for estimating a homography
By constructing a neural network model that includes overlap detection and homography estimation networks, the problem of accuracy in homography estimation for large baseline images is solved, and the image alignment effect is significantly improved, especially in weak texture scenes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2024-01-29
- Publication Date
- 2026-06-26
AI Technical Summary
Traditional methods suffer from reduced homography matrix estimation quality when dealing with large baseline image pairs, especially in weakly textured scenes, and deep learning-based methods perform poorly on large perspective images or images with low overlap.
A deep neural network is used to detect overlapping regions of large baseline image pairs. By constructing a neural network model that includes an overlap detection network and a homography matrix estimation network, and training it using the overlap detection network loss function, the homography matrix estimation loss function, and the end-to-end network training loss function, coarse-to-fine homography matrix estimation is achieved.
It improves the accuracy of homography matrix estimation, especially in weak texture and large perspective image scenes, and significantly improves image alignment.
Smart Images
Figure CN118172638B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computational graphics and computer vision, and in particular to a method, system, device, and medium for estimating homography matrix. Specifically, it relates to a technique for estimating homography matrix by detecting overlapping regions of large baseline image pairs using deep neural networks. Background Technology
[0002] A homography matrix is a 3×3 matrix containing 8 degrees of freedom (DOF), with 2 for each of the scaling, rotation, translation, and perspective components. Homography matrices are frequently used in image alignment and have wide applications in various fields, including but not limited to image and video stitching, video stabilization, SLAM, and multiframe image restoration.
[0003] Traditional homography estimation methods typically follow a standard procedure: 1) Detecting and matching image features, such as SIFT, SURF, and ORB as classic handcrafted features, or using BEBLID, LPM, GMS, SuperPoint, SOSNet, LIFT, SuperGlue, and LoFTR as learned features; 2) Solving for the homography matrix using outlier rejection methods such as RANSAC, IRLS, or MAGSAC through direct linear transformation (DLT) least squares. However, the quality of traditional homography estimation largely depends on the quality of the detected and matched image features. In cases where features cannot be detected, such as in scenes with insufficient texture, the quality of homography estimation drops sharply.
[0004] Deep learning-based homography estimation methods take two images as input and directly output the homography matrix. The network can extract depth features from the entire image. Deep learning-based homography estimation methods can be divided into supervised and unsupervised methods. For supervised methods, network training is guided by labels; however, since each training image pair and label is created from a single image without disparity, there is a disparity difference between the synthetic and real datasets. Unsupervised methods directly minimize the grayscale loss between real image pairs, where one image is transformed according to the estimated homography matrix and compared with the target image in the image domain or feature space. Unsupervised methods show good performance on real datasets. In addition, many mask-based loss calculation methods have been proposed in unsupervised methods to improve estimation accuracy. However, the above methods currently only address the problem of small baseline image pairs.
[0005] When dealing with large baselines, traditional methods rely directly on feature point matching, which can fail in situations with weak textures. Traditional image stitching methods apply a similar logic, focusing on utilizing overlapping regions to extract more image features for better stitching results. Currently, deep image stitching methods employ specialized network layers to handle low-overlap-rate images. One deep learning-based image stitching method proposes a context-sensitive layer that captures long-term correlations within feature maps, implicitly handling low-overlap-rate images. However, this still doesn't yield satisfactory results when dealing with large perspective images or images with low overlap rates.
[0006] This invention provides a method, system, device, and medium for estimating homography matrix, specifically relating to a technique for estimating homography matrix by detecting overlapping regions of large baseline image pairs using deep neural networks. Summary of the Invention
[0007] This invention provides a method, system, device, and medium for estimating homography matrix, specifically relating to a technique for estimating homography matrix by detecting overlapping regions of large baseline image pairs using deep neural networks.
[0008] This invention is achieved through the following technical solution: a method for estimating a homography matrix, comprising the following steps:
[0009] Step S1: Collect images with different overlap rates as training datasets;
[0010] Step S2: Construct a neural network model, which includes an overlap detection network and a homography matrix estimation network connected sequentially from front to back;
[0011] Step S3: Use the overlap detection network loss function, homography matrix estimation loss function, and end-to-end network training loss function to guide and optimize the neural network model, and calculate the loss value between the image transformation result of the network output homography matrix and the target image.
[0012] Step S4: Train the neural network model using the training dataset, pre-setting the number of iterations and the learning rate until the number of iterations equals the maximum number of iterations, then stop training and generate the trained neural network model.
[0013] Step S5: Collect two images with different overlap rates and input them into the trained neural network model. Use the overlap detection network module to identify the overlapping area between the two images and crop out the estimated common area between the two images.
[0014] Step S6: Use the deep homography matrix estimation network to perform coarse-to-fine homography matrix estimation.
[0015] To better realize the present invention, step S2 further includes:
[0016] The overlap detection network includes a ResNet backbone, a first fully connected layer, a second fully connected layer, and a third fully connected layer. The ResNet backbone includes a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block. The first convolutional block, the third convolutional block, and the first fully connected layer are connected. The second convolutional block, the fourth convolutional block, and the second fully connected layer are connected to each other. The fourth convolutional block, the third convolutional block, and the first fully connected layer are connected to each other. The first fully connected layer and the second fully connected layer are respectively connected to the third fully connected layer.
[0017] The homography matrix estimation network includes a coarse estimation part and a fine estimation part. The coarse estimation part includes a first residual network module, and the fine estimation part includes a second residual network module and a third residual network module. The first residual network module, the second residual network module, and the third residual network module are cascaded.
[0018] To better implement the present invention, step S3 further includes:
[0019] The loss function of the overlap detection network is expressed as:
[0020] Among them, L overlap p(x) represents the loss of the overlap detection network, and p(x) represents the coordinates of the overlapping region in the large baseline image output by the overlap detection network. GT (x)| represents the target coordinates of the large baseline image over the overlapping region;
[0021] The homography matrix estimation loss function is expressed as follows:
[0022] L n (I a ,I b )=||F a -F b ||1
[0023]
[0024]
[0025] Where H represents the estimated homography matrix, M represents the mask, F represents the feature, and the subscripts a and b represent the two output images respectively, L n (I a I b F represents the L1 loss of features between two input images. a F represents the features of image a. bL represents the features of image b. n (I′ a I b I′ represents the feature loss between the image transformation result obtained by estimating the homography matrix using a network and the target image. a F′ represents the image after image a has undergone homography transformation by the network output. a M′ represents the features of image a after homography transformation by the network output matrix. a Min represents the mask of image a after homography transformation by the network output. m，f，h H represents minimizing the triple loss function. Coarse This represents the homography matrix of the first coarse estimate of the output, where λ and v represent the training weight parameters;
[0026] The loss function for training the end-to-end network is the same as the loss function for homography matrix estimation. The image transformation process of the end-to-end network is represented as follows:
[0027]
[0028] Where warp represents the homography matrix transformation of the image, Let represent the homography matrix, which represents the transformation from the large baseline image pair to the small baseline image pair output by the overlap detection network for image a. H represents the homography matrix, which represents the transformation from large baseline image pairs to small baseline image pairs output by the image overlap detection network for image b. Fine This represents the homography matrix of the aligned image output by the homography matrix estimation network, from coarse to fine.
[0029] To better realize the present invention, step S4 further includes:
[0030] When training the neural network model for the first time using the training dataset, the overlap detection network and the homography matrix estimation network need to be trained separately. After the overlap detection network and the homography matrix estimation network are initialized and trained, the end-to-end training and fine-tuning of the neural network model can be performed.
[0031] To better realize the present invention, step S4 further includes:
[0032] When training with the training dataset for the first time, training needs to be performed separately. After the upstream and downstream are initialized and trained, end-to-end training and network fine-tuning are then performed.
[0033] To better implement the present invention, step S5 further includes:
[0034] The overlap detection network uses the ResNet backbone framework to extract dual-scale features from the input image. The features extracted at different scales contain different image feature information. Large-scale features are used to estimate the global overlap region of the input image, while small-scale features are used to locate the overlap region.
[0035] Features extracted at different scales are input into the first fully connected layer and the second fully connected layer to estimate the overlapping region at each scale.
[0036] Subsequently, the overlap estimation results at different scales are merged and input into the third fully connected layer to perform complete overlap detection from global to local.
[0037] The overlap detection network outputs a matrix containing the coordinates of the overlapping regions of two large baseline image pairs. This coordinate matrix is used to crop out the overlapping regions of the large baseline image pairs and transform the overlapping regions of the two images to the same size, which are then used as input to the homography matrix estimation network.
[0038] To better implement the present invention, step S6 further includes:
[0039] The coarse estimation part of the deep homography matrix estimation network first extracts the features of the image and extracts the mask of the image. It then uses the product of the features and the mask to exclude outliers in the image and uses the first residual network to estimate the homography matrix only in the regions that need to be aligned.
[0040] The fine estimation part of the deep homography matrix estimation network reuses the features and mask extracted by the coarse estimation part. The product of the mask and the features is input into the second residual network, and the result of the first correction is output. The second correction uses the third residual network. The second correction has the same structure as the first correction. After the two corrections, the homography matrix estimation effect is obtained.
[0041] This invention also provides a homography matrix estimation system, comprising a data acquisition unit, a model building unit, a model training module, and a homography matrix estimation module, wherein:
[0042] The acquisition unit is used to acquire images with different overlap rates as training datasets.
[0043] A model building unit is used to build a neural network model, which includes an overlap detection network and a homography matrix estimation network connected sequentially from front to back.
[0044] The model training module is used to guide and optimize the neural network model using the overlap detection network loss function, the homography matrix estimation loss function, and the end-to-end network training loss function; calculate the loss value between the image transformation result of the network output homography matrix and the target image; and train the neural network model using the training dataset, presetting the number of iterations and the learning rate until the number of iterations equals the maximum number of iterations, then stopping the training and generating the trained neural network model.
[0045] The homography matrix estimation module is used to collect two images with different overlap rates and input them into a trained neural network model. The overlap detection network module is used to identify the overlapping region between the two images and to crop out the estimated common region of the two images. It is used to implement coarse-to-fine homography matrix estimation using the deep homography matrix estimation network.
[0046] The present invention also provides an electronic device comprising a processor and a memory; the processor includes the homography matrix estimation system described in the second aspect above.
[0047] The present invention also provides a computer-readable storage medium comprising instructions that, when executed on an electronic device described in the third aspect, cause the electronic device to perform the method described in the first aspect.
[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0049] (1) This invention provides a method, system, device and medium for estimating homography matrix. In the homography matrix estimation module, we propose a coarse-to-fine homography matrix estimation method, which improves the accuracy of homography matrix estimation. Attached Figure Description
[0050] The present invention will be further described in conjunction with the following drawings and embodiments. All inventive concepts of the present invention should be considered as disclosed content and within the scope of protection of the present invention.
[0051] Figure 1 A schematic diagram of the structure of a neural network model in a homography matrix estimation method, system, device, and medium provided in this application embodiment;
[0052] Figure 2 The illustration shows the effect of using a homography matrix estimation method, system, device, and medium provided in the embodiments of this application.
[0053] Figure 3 The following is a demonstration image showing the effect of testing a homography matrix estimation method, system, device, and medium provided in this application embodiment in a scene with weak texture;
[0054] Figure 4 The following diagram illustrates the effect of a homography matrix estimation method, system, device, and medium on the overlap detection network in the embodiment of this application.
[0055] Figure 5 The diagram illustrates the effect of a homography matrix estimation network in a method, system, device, and medium provided in this application embodiment to verify the effectiveness of the homography matrix estimation network. Detailed Implementation
[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be regarded as a limitation on the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0057] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0058] Example 1:
[0059] This embodiment provides a method, system, device, and medium for estimating a homography matrix, such as... Figure 1 As shown, we employ a deep homography matrix estimation method for large baseline image pairs based on overlap region detection, primarily to address the homography matrix estimation problem arising from the limited overlap between two images in a large baseline image pair. To solve this problem, we first use an overlap detection network module to identify the overlapping regions between the two images. Next, we crop out the estimated common regions of the two images and use a deep homography matrix estimation module to perform coarse-to-fine homography matrix estimation. Specifically, the overlap detection network aims to identify the common regions between the two images, rather than locating specific object types as in traditional detection networks. Furthermore, to improve the accuracy of homography matrix estimation, we propose a coarse-to-fine homography matrix estimation method within the homography matrix estimation module, thereby enhancing the accuracy of the estimation.
[0060] Example 2:
[0061] This embodiment further optimizes upon Embodiment 1 by introducing a large baseline image homography matrix estimation dataset based on content overlap rate classification. This dataset contains images with varying overlap rates and includes challenging weak-texture images, which are used to train our network, enabling it to adapt to different overlap rates and challenging scenarios. Since our network is divided into upstream and downstream components, they need to be trained separately during the initial training. After both upstream and downstream components have been initialized and trained, end-to-end training and network fine-tuning are then performed.
[0062] The other parts of this embodiment are the same as those in Embodiment 1, so they will not be described again.
[0063] Example 3:
[0064] This embodiment further optimizes upon Embodiment 1 or 2 described above. An overlap detection network is used to detect regions where the content of two input images is similar. We input two large baseline image pairs into the overlap detection network, and the network outputs the coordinates of the overlapping regions between these two large baseline image pairs. For the overlap detection network, we use a ResNet backbone framework to extract dual-scale features from the input images. Features extracted at different scales contain different image feature information: larger-scale features are used to estimate the global overlap region of the input images, and smaller-scale features are used to locate the overlap region. We input the features extracted at different scales into a fully connected layer to estimate the overlap region at each scale. Subsequently, the overlap estimation results at different scales are merged and input into another fully connected layer to perform complete overlap detection from global to local.
[0065] The overlap detection network outputs a matrix containing the coordinates of the overlapping regions of two large baseline image pairs. We use this coordinate matrix to crop the overlapping regions of the large baseline image pairs and transform the overlapping areas of the two images to the same size, which is then used as input to the downstream network. We use Spatial Transformer Net to perform differentiable cropping and image scaling within the network, achieving a differentiable transformation from large baseline image pairs to small baseline image pairs.
[0066] The other parts of this embodiment are the same as any one of the embodiments 1-2 above, so they will not be described again.
[0067] Example 4:
[0068] This embodiment further optimizes any one of embodiments 1-3 above. For homography estimation, we propose a coarse-to-fine estimation strategy using a cascaded isomorphic network. The coarse estimation part first uses a CNN to extract image features and a mask. Outliers are excluded by multiplying the features and mask, and homography estimation is performed only in the regions requiring alignment. The coarse estimation network uses ResNet34 (the first residual network) as its skeleton to estimate the homography matrix. After coarse estimation, we obtain a homography matrix that can roughly align two images. The fine estimation part uses a cascaded homography correction network with ResNet34 as its skeleton to make subtle corrections to the coarse estimation result, achieving a more accurate alignment. The fine estimation part reuses the features extracted during coarse estimation, inputting the product of the mask and features into the first fine estimation correction network, still using ResNet34 as the network skeleton, and outputting the result of the first correction. The second correction has the exact same structure as the first correction. After two corrections, a better homography estimation result can be obtained. The three residual networks have the same structure, all being ResNet34. The first residual network is used for coarse estimation, while the second and third residual networks are used for fine estimation.
[0069] The other parts of this embodiment are the same as any one of the embodiments 1-3 above, so they will not be described again.
[0070] Example 5:
[0071] This embodiment further optimizes any one of embodiments 1-4 above. The loss function used in training the above model is mainly as follows:
[0072] The loss function of the overlap detection network is expressed as follows:
[0073]
[0074] The loss function for homography matrix estimation is expressed as follows:
[0075] L n (I a ,I b )=||F a -F b ||1
[0076]
[0077]
[0078] Where H represents the estimated homography matrix, M represents the mask, and F represents the feature. We use the loss function described above, calculating only the loss for the portion that needs to be aligned, ignoring outliers.
[0079] The end-to-end network training loss function is the same as the homography matrix estimation loss function. The end-to-end image transformation is represented as follows:
[0080]
[0081] Where warp represents the homography matrix transformation of the image.
[0082] The other parts of this embodiment are the same as any one of the embodiments 1-4 above, so they will not be described again.
[0083] Example 6:
[0084] like Figure 2 The image shown demonstrates the effectiveness of this method. The first and second columns represent the inputs, and the third column shows the alignment effect of the homography matrix estimated by the network. The first row contains images with an overlap rate below 40%, the second row contains images with an overlap rate of 40%-50%, the third row contains images with an overlap rate of 50%-60%, and the fourth row contains images with an overlap rate greater than 60%.
[0085] like Figure 3 As shown, we also tested our method in weakly textured scenes to verify its effectiveness in challenging scenarios. The first and second columns are the inputs, and the third column is the alignment effect of the homography matrix estimated by the network.
[0086] like Figure 4 As shown, we also validated the performance of the overlap detection network and the coarse-to-fine homography matrix estimation network, where the boxes represent the ground truth (GT) of the overlap detection network and the output of the overlap detection network, respectively.
[0087] like Figure 5 As shown, we validated the performance of the coarse-to-fine homography matrix estimation network. The first column represents the input of the coarse-to-fine homography matrix estimation network, the second column represents the alignment effect of the first coarse estimation, the third column represents the alignment effect after the first correction, and the fourth column represents the final alignment effect after two corrections.
[0088] The present invention also provides a homography matrix estimation system that matches the method, including a data acquisition unit, a model building unit, a model training module, and a homography matrix estimation module.
[0089] The present invention also provides an electronic device comprising a processor and a memory; the processor includes the homography matrix estimation system described above.
[0090] Example 7:
[0091] The present invention also provides a computer-readable storage medium comprising instructions; when the instructions are executed on the electronic device described in the above embodiments, the electronic device causes the electronic device to perform the methods described in the above embodiments. Optionally, the computer-readable storage medium may be a memory.
[0092] The processor involved in the embodiments of this application can be a chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.
[0093] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0094] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0095] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0096] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0097] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.
[0098] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0099] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.
[0100] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.
Claims
1. A method for estimating a homography matrix, characterized in that, Includes the following steps: Step S1: Collect images with different overlap rates as training datasets; Step S2: Construct a neural network model, which includes an overlap detection network and a homography matrix estimation network connected sequentially from front to back; The overlap detection network includes a ResNet backbone, a first fully connected layer, a second fully connected layer, and a third fully connected layer. The ResNet backbone includes a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block. The first convolutional block, the third convolutional block, and the first fully connected layer are connected. The second convolutional block, the fourth convolutional block, and the second fully connected layer are connected to each other. The fourth convolutional block, the third convolutional block, and the first fully connected layer are connected to each other. The first fully connected layer and the second fully connected layer are respectively connected to the third fully connected layer. The homography matrix estimation network includes a coarse estimation part and a fine estimation part. The coarse estimation part includes a first residual network module, and the fine estimation part includes a second residual network module and a third residual network module. The first residual network module, the second residual network module, and the third residual network module are cascaded. Step S3: Use the overlap detection network loss function, homography matrix estimation loss function, and end-to-end network training loss function to guide and optimize the neural network model, and calculate the loss value between the image transformation result of the network output homography matrix and the target image. Step S4: Train the neural network model using the training dataset, pre-setting the number of iterations and the learning rate until the number of iterations equals the maximum number of iterations, then terminate the training and generate the trained neural network model. Step S5: Collect two images with different overlap rates and input them into the trained neural network model. Use the overlap detection network to identify the overlapping area between the two images and crop out the estimated common area between the two images. Step S6: Use the homography matrix estimation network to perform coarse-to-fine homography matrix estimation; The coarse estimation part of the homography matrix estimation network first extracts the features of the image and extracts the mask of the image. It then uses the product of the features and the mask to exclude outliers in the image and uses the first residual network to estimate the homography matrix only in the regions that need to be aligned. The fine estimation part of the homography matrix estimation network reuses the features and mask extracted by the coarse estimation part, and inputs the product of the mask and the features into the second residual network to output the result of the first correction. The second correction uses the third residual network. The second correction has the same structure as the first correction. After the two corrections, the homography matrix estimation effect is obtained.
2. The method for estimating a homography matrix according to claim 1, characterized in that, Step S3 includes: the loss function of the overlap detection network is expressed as: ,in, This represents the loss of the overlap detection network. This indicates that the overlap detection network outputs the coordinates of the overlapping region in the large baseline image. Indicates the coordinates of the target in the overlapping region of the large baseline image; The homography matrix estimation loss function is expressed as follows: Where H represents the estimated homography matrix, M represents the mask, F represents the feature, and the subscripts a and b represent the two output images respectively. The L1 loss represents the feature difference between two input images. Representing the features of image a, Representing the features of image b, This represents the feature loss between the image transformation result obtained by estimating the homography matrix using a network and the target image. This represents the image after image a has undergone homography transformation by the network output. This represents the features of image a after the homography matrix transformation output by the network. This represents the mask of image 'a' after the homography matrix transformation at the network output. This represents minimizing the triple loss function. This represents the homography matrix of the first coarse estimate of the output, where λ and v represent the training weight parameters; The loss function for training the end-to-end network is the same as the loss function for homography matrix estimation. The image transformation process of the end-to-end network is represented as follows: Where warp represents the homography matrix transformation of the image, Let represent the homography matrix, which represents the transformation from the large baseline image pair to the small baseline image pair output by the overlap detection network for image a. Let represent the homography matrix, which represents the transformation from the large baseline image pair to the small baseline image pair output by the image overlap detection network for image b. This represents the homography matrix of the aligned image output by the homography matrix estimation network, from coarse to fine.
3. The method for estimating a homography matrix according to claim 1, characterized in that, Step S4 includes: when training the neural network model for the first time using the training dataset, training the overlap detection network and the homography matrix estimation network respectively; after the overlap detection network and the homography matrix estimation network have been initialized and trained, end-to-end training and fine-tuning of the neural network model are performed.
4. The method for estimating a homography matrix according to claim 1, characterized in that, Step S5 includes: the overlap detection network uses the ResNet backbone framework to extract dual-scale features from the input image. The features extracted at different scales contain different image feature information. The large-scale features are used to estimate the global overlapping region of the input image, and the small-scale features are used to locate the overlapping region. Features extracted at different scales are input into the first and second fully connected layers to estimate the overlapping region at each scale. Then, the overlap estimation results at different scales are merged and input into the third fully connected layer to perform complete overlap detection from global to local. The overlap detection network outputs a matrix containing the coordinates of the overlapping regions of two large baseline image pairs. This coordinate matrix is used to crop out the overlapping regions of the large baseline image pairs and transform the overlapping regions of the two images to the same size, which are then used as input to the homography matrix estimation network.
5. A system for estimating a homography matrix, characterized in that, It includes a data acquisition unit, a model building unit, a model training module, and a homography matrix estimation module, wherein: The acquisition unit is used to acquire images with different overlap rates as training datasets. A model building unit is used to build a neural network model, which includes an overlap detection network and a homography matrix estimation network connected sequentially from front to back. The homography matrix estimation network includes a coarse estimation part and a fine estimation part. The coarse estimation part includes a first residual network module, and the fine estimation part includes a second residual network module and a third residual network module. The first residual network module, the second residual network module, and the third residual network module are cascaded. The model training module is used to guide and optimize the neural network model using the overlap detection network loss function, the homography matrix estimation loss function, and the end-to-end network training loss function. It calculates the loss value between the image transformation result of the network output homography matrix and the target image. It also trains the neural network model using the training dataset, presetting the iteration count and learning rate until the iteration count equals the maximum iteration count, at which point training terminates, generating a trained neural network model. The overlap detection network includes a ResNet backbone framework, a first fully connected layer, a second fully connected layer, and a third fully connected layer. The ResNet backbone framework includes a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block. The first and third convolutional blocks are connected to the first fully connected layer; the second and fourth convolutional blocks are connected to the second fully connected layer; the first and second convolutional blocks and the second fully connected layer are interconnected; the fourth and third convolutional blocks are interconnected with the first fully connected layer; and the first and second fully connected layers are respectively connected to the third fully connected layer. The homography matrix estimation module is used to collect two images with different overlap rates and input them into a trained neural network model. The overlap detection network is used to identify the overlapping region between the two images and to crop out the estimated common region between the two images. It is also used to implement coarse-to-fine homography matrix estimation using the homography matrix estimation network. The coarse estimation part of the homography matrix estimation network first extracts the features of the image and extracts the mask of the image. It then uses the product of the features and the mask to exclude outliers in the image and uses the first residual network to estimate the homography matrix only in the regions that need to be aligned. The fine estimation part of the homography matrix estimation network reuses the features and mask extracted by the coarse estimation part. The product of the mask and the features is input into the second residual network, and the result of the first correction is output. The second correction uses the third residual network. The second correction has the same structure as the first correction. After the two corrections, the homography matrix estimation effect is obtained.
6. An electronic device, characterized in that, It includes a processor and a memory; the processor is used to run the homography matrix estimation system as described in claim 5.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on the electronic device as claimed in claim 6, cause the electronic device to perform the method as claimed in any one of claims 1-4.