A point prompt-based image registration method

Abstract

The application discloses a point prompt-based image registration method. The application is established on the basis of a cost volume-based image registration method, encodes prompt points with rich information meanings into prompt tokens, and then integrates the encoded prompt token information into a cost volume by using self and cross attention mechanisms, and optimizes the cost volume. At the same time, the application designs a method for automatically selecting prompt points, and improves the usability and use efficiency of the application. Experimental results prove that the average corner point error performance of the application on many public data sets is in the forefront of the field.

Description

Technical Field

[0001] This invention relates to an image registration method in the field of computer vision technology, and in particular to an image registration method that optimizes cost volume based on point cues. Background Technology

[0002] Homography estimation is widely used to align single-modal or multimodal images taken from different perspectives, and is suitable for scenes close to planar surfaces or when objects are far from the observer. Therefore, it plays a crucial role in image registration in rigid scenes and has found wide application in various computer vision tasks, including image fusion, image stitching, remote sensing, and robot navigation.

[0003] Specifically, this task focuses on single-modal or multi-modal image registration in rigid scenes, and presents the following challenges:

[0004] (1) There are large nonlinear distortions in luminance or color between different modal images.

[0005] (2) Non-planar scenes, moving objects, and occlusions create out-of-points, which violate the homography assumption.

[0006] Deep learning methods have been widely applied in homography estimation tasks. Among these methods, DeTone et al.'s DHN and Le et al.'s MHN use convolutional neural networks (CNNs) to regress the homography matrix, while Shao et al.'s LocalTrans uses a transformer (Transformer) to regress the homography matrix. Chang et al.'s CLKN designed cascaded networks and recurrent architectures to improve estimation accuracy. To address violations of the homography assumption, previous methods such as Le et al.'s MHN and Zhang et al.'s UDHN typically rely on using learnable masks to exclude unreliable regions. Recently, Cao et al.'s IHN and RHWF, based on cost-volume architectures, iteratively estimate the homography transformation, achieving state-of-the-art performance. Summary of the Invention

[0007] To address the problems existing in the background art, this invention provides a depth homography estimation method based on point hints to optimize cost volume. This invention optimizes cost volume based on point hints, guiding homography estimation towards more reliable regions while ignoring unreliable regions. Compared to other disclosed methods, it is superior in key error metrics. This invention optimizes cost volume based on point hints, guiding homography estimation towards more reliable regions while ignoring unreliable regions.

[0008] The technical solution adopted in this invention is:

[0009] I. An Image Registration Method Based on Point Cueing

[0010] 1) Train the baseline network model using unregistered image pairs from the image registration training set to obtain a trained baseline network model;

[0011] 2) Construct a cue point selection module using the convolutional neural network in the trained baseline network model, and then construct a point-cue-based image registration network model using the cue point selection module. Train the confidence network of the point-cue-based image registration network model using the already registered image pairs in the image registration training set. After training, fix the parameters of the confidence network and record it as the pre-trained image registration network model.

[0012] 3) Use the unregistered images in the image registration training set to train the pre-trained image registration network model to obtain a well-trained image registration network model;

[0013] 4) Input the image to be matched into the trained image registration network model, and the model outputs the image registration result.

[0014] The point-cue-based image registration network model includes a convolutional neural network, a region-of-interest transformer, a feature similarity layer, a cue point selection module, a cue point encoder, a cost volume optimizer, an iterative homography estimator, and a homography transformation module. The target image I of the image pair... T The target image feature map is input to the transformer of the region of interest transformation after passing through a convolutional neural network. The source image I of the image pair S The source image feature map is input to the region of interest transform transformer after passing through a convolutional neural network. The region of interest transform transformer sequentially passes through a feature similarity layer, a cost volume optimizer, and an iterative homography estimator before being connected to the homography transformation module. The homography transformation module is connected to the source image I of the image pair. S Connected, target image I T As input to the cue point selection module, the cue point selection module is connected to the cost volume optimizer after passing through the cue point encoder.

[0015] The prompt point selection module includes a confidence network, a convolutional neural network in a trained baseline network model, a pseudo-label map calculation layer, and a Top-K random sampler; target image I T The input is fed into a confidence network, which outputs a confidence graph. This serves as the input to a Top-K random sampler, which is connected to a cue point encoder; the convolutional neural network in the trained baseline network model includes the target image I. T The corresponding convolutional neural network and source image I S The corresponding convolutional neural networks are denoted as the first convolutional neural network and the second convolutional neural network, respectively. The target image I T It is also input into the first convolutional neural network, from the source image IS Registered to target image I T Image I T←S The input is fed into the second convolutional neural network. The outputs of the first and second convolutional neural networks are fed together into the pseudo-label map computation layer. The pseudo-label map computation layer outputs the pseudo-label map M. During the training of the confidence network, the confidence map is calculated. The loss function between the pseudo-label map M and the pseudo-label map M.

[0016] The confidence network consists of 4 downsamplers and 4 upsamplers, and adopts a U-Net network structure.

[0017] In the pseudo-label graph computation layer, the initial pseudo-label graph M is first calculated based on the outputs of the first convolutional neural network and the second convolutional neural network. F The formula is as follows:

[0018]

[0019] Among them, M F (x) represents the initial pseudo-label image M. F The label value of pixel x, F T←S (x) and F T←S (y) represent the feature values ​​of pixels x and y in the output of the second convolutional neural network, respectively. F T (x) represents the feature value of pixel x in the output of the first convolutional neural network, T represents transpose, and τ is the temperature hyperparameter;

[0020] Then, the initial pseudo-label image M is processed. F Upsampling is performed to obtain a pseudo-label image M with the same size as the input image.

[0021] The confidence map The formula for calculating the loss function between the pseudo-label map M and the pseudo-label map M is as follows:

[0022]

[0023] in, For the set of positive categories, For the set of negative categories, For confidence plot The label value of pixel x, || represents the number of elements in the set, and μ and σ are the mean and standard deviation of the pseudo-label map M.

[0024] In the aforementioned prompt point encoder, for N tokGiven a prompt point and its coordinate position p, first calculate the token position code of the prompt point to obtain the global two-dimensional position code Pos. Then, index each prompt point's coordinate position p into the global two-dimensional position code Pos to obtain the token position code Pos(p). Finally, use the following formula to calculate the prompt tokens corresponding to positive and negative prompt points, thus obtaining the prompt token Tok. The formula is as follows:

[0025]

[0026] Among them, Tok P and Tok N The positive and negative embeddings of the prompt point are respectively represented by Tok(p), which is the prompt token Tok at coordinate position p.

[0027] The cost volume optimizer is a decoder composed of two interconnected Transformer layers. Each Transformer layer contains a self-attention layer, a token-to-volume attention layer, a multi-layer encoder, and a volume-to-token attention layer. The point hint token Tok is input to the self-attention layer, which is then connected in sequence to the token-to-volume attention layer, the multi-layer encoder, and the volume-to-token attention layer. The cost volume Vol is input to both the token-to-volume attention layer and the volume-to-token attention layer. The output of the volume-to-token attention layer is used as the output of each Transformer layer.

[0028] II. A computer device

[0029] The computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method.

[0030] III. A computer-readable storage medium

[0031] The computer-readable storage medium stores a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. This invention proposes a point-based cueing method that uses information-rich points to guide the optimization of cost volume, thereby improving the accuracy and robustness of depth homography estimation.

[0034] 2. This invention designs a method for automatically generating information points, which does not rely on point-to-point truth monitoring, making our framework practical and efficient.

[0035] 3. This invention demonstrates the potential of cue learning in deep homography estimation tasks. Attached Figure Description

[0036] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the baseline network model in an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of a point-based prompting network model in an embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram of the prompt point selection module designed in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of the confidence network designed in an embodiment of the present invention.

[0041] Figure 6 This is a schematic diagram of the cost volume optimizer designed in an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0043] The embodiments and specific implementation processes of the present invention are as follows:

[0044] like Figure 1 As shown, the method includes the following steps:

[0045] 1) Train the baseline network model using unregistered images from the image registration training set to obtain a trained baseline network model. The network structure diagram of the baseline network model is shown below. Figure 2 As shown;

[0046] The baseline image registration network model includes a first convolutional neural network, a first region-of-interest transformer, a first feature similarity layer, a first iterative homography estimator, and a first homography transformation module. The unregistered image is then compared to the source image I. S and target image I T The input is fed into the first convolutional neural network to calculate the initial source image feature map. and initial target image feature map The transformer input to the first region of interest transform is the feature map of the initial source image. and initial target image feature map The first region of interest undergoes alternating self-attention and cross-attention calculations with the Transformer, which changes with each iteration, to obtain the advanced source image feature map. and advanced target image feature maps

[0047] The operation of the first feature similarity layer is as follows: After calculating the similarity between the two advanced feature maps output by the first convolutional neural network, the first feature similarity layer calculates the cost volume of the image pair, as shown in the following formula:

[0048]

[0049] Where x = (u, v) are the two-dimensional coordinates of the target image feature map, and r represents the displacement of the source image coordinates relative to the target image coordinates. Let represent the value vectors of the two feature maps at corresponding coordinates, ReLU represent the ReLU activation function, and T represent the transpose operation. C represents the similarity map calculated from the two feature maps, and R represents the similarity map calculated from the two feature maps. C This indicates the radius range considered in calculating the similarity map.

[0050] Next, the similarity graph C is encoded by a convolution to obtain the cost volume Vol.

[0051] Finally, the cost volume Vol is input into the first iterative homography estimator, which outputs the homography transformation result of the image pair. The updated homography transformation is used in the next iteration to obtain the transformed image from the source image. In iteration k, the residual homography transformation ΔH for the current iteration is estimated by the transformation of the four corner points of the image. k And through H k =H k-1 ΔH k Update the overall homography transformation matrix, use the first homography transformation module to perform homography transformation on the source image, and perform the next round of registration.

[0052] The baseline image registration network model is a two-scale form, that is, based on the above operations, an additional network structure with roughly the same structure is built, and an additional homography estimation is performed using feature maps with doubled resolution to obtain more accurate homography estimation results.

[0053] The loss function for training the baseline image registration network model is:

[0054]

[0055] in, This represents the corner displacement result estimated in the nth round, Iter represents the total number of iteration rounds within one scale, and α is a weighting factor. For accurate estimation at two scales, the loss functions of the two scales are summed to obtain the final loss function.

[0056] 2) Construct a cue point selection module using the convolutional neural network in the trained baseline network model. The network structure diagram of the image registration network model based on cue points is as follows: Figure 4As shown, a point-based image registration network model is then constructed using the prompt point selection module. The network structure diagram of the point-based image registration network model is shown below. Figure 3 As shown, the confidence network of the image registration network model based on point cue is trained using the registered images in the image registration training set. After training, the parameters of the confidence network are fixed and denoted as the pre-trained image registration network model.

[0057] The point-cue-based image registration network model includes a convolutional neural network, a region-of-interest transformer, a feature similarity layer, a cue point selection module, a cue point encoder, a cost volume optimizer, an iterative homography estimator, and a homography transformation module. The target image I of the image pair... T The target image feature map is input to the transformer of the region of interest transformation after passing through a convolutional neural network. The source image I of the image pair S The source image feature map is input to the transformer of the region of interest after passing through the convolutional neural network, and the target image I is input. T and source image I S The convolutional neural network uses weight sharing. The domain-of-interest transformer passes sequentially through a feature similarity layer, a cost volume optimizer, and an iterative homography estimator before being connected to the homography transformation module. The homography transformation module is connected to the source image I of the image pair. S Connected, target image I T As input to the cue point selection module, the cue point selection module is connected to the cost volume optimizer after passing through the cue point encoder. Cue points are selected based on the confidence map, and the cost volume optimizer is used to optimize the calculated cost volume of the image to be registered, resulting in a more accurate image registration result.

[0058] The cue point selection module includes a confidence network, a convolutional neural network in a pre-trained baseline network model, a pseudo-label map computation layer, and a Top-K random sampler; target image I T The input is fed into a confidence network, which outputs a confidence graph. This serves as input to a Top-K random sampler, which is connected to a cue encoder. The Top-K random sampler selects cue points based on the confidence map. The convolutional neural network in the trained baseline network model includes the target image I. T The corresponding convolutional neural network and source image I S The corresponding convolutional neural networks are denoted as the first convolutional neural network and the second convolutional neural network, respectively. The target image I T It is also input into the first convolutional neural network, from the source image I S Registered to target image I T Image I T←SThe inputs are fed into the second convolutional neural network. The outputs of the first and second convolutional neural networks are then fed into the pseudo-label map computation layer. The pseudo-label map computation layer outputs a pseudo-label map M, which is used to supervise the confidence network. During the training of the confidence network, the confidence map is calculated. The loss function between the pseudo-label map M and the pseudo-label map M.

[0059] like Figure 5 As shown, the confidence network comprises four downsamplers, four upsamplers, one intermediate convolutional layer, and one output convolutional layer, using the UNet network architecture. Each downsampler contains two 3×3 convolutional modules. These convolutional modules include a ReLU activation function and a 2×2 max-pooling layer. Each upsampler contains one upsampled convolutional layer (deconvolutional layer), one feature concatenation module, and two 3×3 convolutional modules. These convolutional modules also include a ReLU activation function.

[0060] In the pseudo-label graph computation layer, the initial pseudo-label graph M is first calculated based on the outputs of the first and second convolutional neural networks. F The formula is as follows:

[0061]

[0062] Among them, M F (x) represents the initial pseudo-label image M. F The label value of pixel x, F T←S (x) and F T←S (y) represents the output of the second convolutional neural network (i.e., feature map F). T←S The feature values ​​of pixels x and y in F T (x) is the output of the first convolutional neural network (i.e., feature map F). T The feature value of pixel x in ) is T, which represents transpose, and τ is the temperature hyperparameter;

[0063] Then, the initial pseudo-label image M is processed. F Upsampling is performed to obtain a pseudo-label image M with the same size as the input image.

[0064] Confidence plot The formula for calculating the loss function between the pseudo-label map M and the pseudo-label map M is as follows:

[0065]

[0066] in, For the set of positive categories, For the set of negative categories, For confidence plot The label value of pixel x, || represents the number of elements in the set, and μ and σ are the mean and standard deviation of the pseudo-label map M.

[0067] In the Top-K random sampler, after obtaining the confidence graph, the k largest points in the confidence graph are selected to construct the positive sampling pool, and the k smallest points are selected to construct the negative sampling pool. k is set to 10 in subsequent experiments.

[0068] 3) Use unregistered images from the image registration training set to train the pre-trained image registration network model. That is, train the image registration network model using all parameters except the confidence network, and use the loss function of the training benchmark image registration network model to obtain the trained image registration network model.

[0069] When training and testing the image registration network model, 1-3 positive cue points and 1-3 negative cue points are randomly selected from the positive sampling pool and the negative sampling pool, respectively, as the final extracted cue points.

[0070] In the cue point encoder, for N tok Given a prompt point and its coordinate position p, first calculate the token position code of the prompt point to obtain the global two-dimensional position code Pos. Then, index each prompt point's coordinate position p into the global two-dimensional position code Pos to obtain the token position code Pos(p). Finally, use the following formula to calculate the prompt tokens corresponding to positive and negative prompt points, thus obtaining the prompt token Tok. The formula is as follows:

[0071]

[0072] Among them, Tok P and Tok N The positive and negative embeddings corresponding to the prompt points are superimposed on the token position encoding, and Tok(p) is the prompt token Tok at coordinate position p.

[0073] Next, the cost volume Vol and the point hint token ToK are input into the cost volume optimizer, and the optimized cost volume is input into the second iterative homography estimator.

[0074] like Figure 6As shown, the cost volume optimizer is a decoder composed of two interconnected Transformer layers. Each Transformer layer contains a self-attention layer, a token-to-volume attention layer, a multi-layer encoder, and a volume-to-token attention layer. The point hint token Tok is input to the self-attention layer, which is sequentially connected to the token-to-volume attention layer, the multi-layer encoder, and the volume-to-token attention layer. The cost volume Vol is input to both the token-to-volume and volume-to-token attention layers. The output of the volume-to-token attention layer serves as the output of each Transformer layer. The cost volume input to the second Transformer layer is the cost volume output by the first Transformer layer. The output Vol' of the volume-to-token attention layer of the second Transformer layer is the final output of the cost volume optimizer.

[0075] The cost volume optimizer is primarily based on the Transformer attention algorithm. The formula for the Transformer attention mechanism is as follows:

[0076]

[0077] Where Q, K, and V represent the query, key, and value after mapping through a fully connected network, respectively.

[0078] Define a trainable global two-dimensional positional encoding as:

[0079] Pos=sin(2π·x·Pm)⊕cos(2π·x·Pm)

[0080] Where x represents a two-dimensional coordinate normalized to [-1,1], ⊕ represents the concatenation operation, and Pm is a trainable matrix initialized with a Gaussian distribution. Based on this expression, for the cost volume input Vol, the positional encoding Pos is calculated according to the above formula.

[0081] In each Transformer layer, the token is first input into the self-attention layer, and its expression is:

[0082]

[0083] Here, Att() represents the self-attention layer. For the input of the second layer, the corresponding positional encoding Pos(p) needs to be superimposed on the query and key positions.

[0084] Next, the output of the self-attention layer and the cost volume are input into the token-to-volume attention layer, expressed as follows:

[0085] Tok_to_Vol=Att(Tok+Pos(p),Vol+Pos,Vol)

[0086] Here, Tok_to_Vol represents the output of the attention layer from token to volume.

[0087] Next, the output of the token-to-volume attention layer is input into the multilayer encoder, and then the output of the multilayer encoder and the cost volume are input into the volume-to-token attention layer, expressed as follows:

[0088] Vol_to_Tok=Att(Vol+Pos,Tok+Pos(p),Tok)

[0089] Here, Vol_to_Tok represents the output of the attention layer from volume to token.

[0090] This invention optimizes the cost volume at scale 1 and the cost volume at scale 2 using two cost volume optimizers with identical structures but different scales.

[0091] 4) Input the image to be matched into the trained image registration network model, and the model outputs the image registration result.

[0092] To verify the effectiveness of this invention, it was validated on publicly available multimodal image datasets Google Earth and Google Maps, as well as the same-modal dataset MSCOCO, and compared with the most advanced image registration methods currently available:

[0093] GoogleEarth is a multimodal dataset consisting of cross-seasonal satellite imagery, containing 8750 training image pairs and 850 test image pairs. GoogleMap is a multimodal dataset consisting of Google Maps and Google satellite imagery, containing 8822 training image pairs and 888 test image pairs.

[0094] MSCOCO is a commonly used dataset in computer vision problems. The experiments conducted in this invention simulate homography transformation on the same image from this dataset to obtain image pairs to be registered in the same modality. By default, all images in these datasets are trained and tested at a size of 256×256.

[0095] This invention uses the average corner error (ACE) as the evaluation metric. To achieve optimal network performance, the parameter details are as follows:

[0096] The initial parameters of the first and second convolutional neural networks are obtained from pre-training on ImageNet. The number of iterations (Iter) within one scale is 6. The focus radius of FocusFormer is [G,4,2,1,1,1] for the 6 iterations at scale 1 and [2,2,1,1,1,1] for the 6 iterations at scale 2, where G represents global focus. For the two scales, the radius for calculating the cost volume similarity is R. C =8 and R C =4, N=96 for image features extracted by the CNN feature extractor, and the dimension of the similarity map encoded as the cost volume is 128. In the loss function, α=0.85 is set, and τ=1 is set for the calculation of the pseudo-label map.

[0097] For the sampling strategy, this invention randomly selects 1-3 positive cue points and 1-3 negative cue points from both the positive and negative sampling pools, each with a size of k=10. Furthermore, for the training of the confidence network of the cue point selection module, the AdamW optimizer is used, with a learning rate of 0.0004, a batch size of 32, and 1000 training iterations. For the Google Earth and Google Maps datasets, this invention uses the loss function L... cls Training the confidence network. For the MSCOCO dataset, this invention directly inputs two identical target images into the pre-trained network to calculate pseudo-label maps, and uses these pseudo-label maps directly as confidence maps in the cue point selection module. This is because the MSCOCO dataset corresponds to the same-modality image registration task, and the source image, based on the ground truth transformation, is completely consistent with the target image content. For training the main network, the AdamW optimizer is used with a learning rate of 0.0004, a batch size of 16, and 120,000 training iterations, during which the parameters of the confidence network are fixed.

[0098] The experiment mainly consists of two parts. The first part is a comparative experiment between the method of this invention and the most advanced image registration methods currently available. The second part is a controlled variable experiment of each module in the network of this invention, which is to illustrate the effectiveness of each module in this invention.

[0099] Part 1: Comparative experiments with the most advanced image registration methods to demonstrate the superior performance of the network in this invention.

[0100] Tables 1, 2, and 3 respectively show the comparison results of the complete model of the present invention (denoted by PromptDHE) with other state-of-the-art methods on the multimodal image datasets GoogleEarth, GoogleMap, and the graph modal image dataset MSCOCO.

[0101] As shown in Table 1, the full model on Google Earth outperforms other published or backup paper methods in terms of ACE percentage (less than 0.5%, 1%, 3%, 5%, 10%) and average ACE (mACE).

[0102] Table 1 shows the performance metrics of this invention compared to published state-of-the-art (SOTA) methods on Google Earth.

[0103]

[0104]

[0105] As shown in Table 2, the complete model on Google Maps has ACE less than 0.5%, ACE less than 1%, ACE less than 3%, ACE less than 5%, ACE less than 10%, and average ACE (mACE) performance that is better than other methods in published or backup papers.

[0106] Table 2 shows the comparison of the metrics of this invention with published state-of-the-art (SOTA) methods on Google Maps.

[0107]

[0108] As shown in Table 3, the complete model on MSCOCO outperforms other published or backup paper methods in terms of ACE less than 0.1%, ACE less than 0.5%, ACE less than 1%, ACE less than 3%, ACE less than 5%, ACE less than 10%, and average ACE (mACE).

[0109] Table 3 shows the performance comparison of this invention with published SOTA methods on MSCOCO.

[0110]

[0111] Part Two: The controlled variable experiments for each module in this invention are to illustrate the effectiveness of each module in this invention. The statistics are the percentage of image pairs with an ACE error of less than 1 and the average ACE (mACE) performance on the Google Earth dataset.

[0112] Effectiveness of selecting cue points based on confidence network: Two variations are obtained: randomly selecting cue points based on the score map output by the SuperPoint network, and selecting cue points based on the confidence network. The comparison results are listed in Table 4. The results show that the method of selecting cue points based on the confidence network outperforms the method of selecting cue points based on SuperPoint.

[0113] Table 4 is a comparison table of prompt point selection methods.

[0114]

[0115] Effectiveness of selecting a random number of cue points: Three variations can be obtained by selecting different numbers of cue points. The comparison results are listed in Table 5. The results show that the method of using 1-3 random positive cue points and 1-3 random negative cue points performs better than the method of selecting a fixed number of 1 positive cue point and 1 negative cue point or 2 positive cue points and 2 negative cue points.

[0116] Table 5 is a comparison table of different numbers of prompt points.

[0117]

[0118] Effectiveness of combining positive and negative cue points: Three variations can be obtained based on different cue point selection types. The comparison results are listed in Table 6. The results show that the method of combining positive and negative cue points performs better than the method of using only positive cue points or only negative cue points.

[0119] Table 6 is a comparison table of different types of prompt points.

[0120]

[0121] The above description is merely a specific embodiment of the present invention and is not intended to limit the present invention in any way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above examples based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An image registration method based on point cues, characterized in that, Includes the following steps: 1) Train the baseline network model using unregistered images from the image registration training set to obtain a trained baseline network model; 2) Construct a cue point selection module using the convolutional neural network in the trained baseline network model, and then construct a point-cue-based image registration network model using the cue point selection module. Train the confidence network of the point-cue-based image registration network model using the already registered image pairs in the image registration training set. After training, fix the parameters of the confidence network and record it as the pre-trained image registration network model. The point-cue-based image registration network model includes a convolutional neural network, a region-of-interest transformer, a feature similarity layer, a cue point selection module, a cue point encoder, a cost volume optimizer, an iterative homography estimator, and a homography transformation module. The target image I of the image pair... T The target image feature map is input to the transformer of the region of interest transformation after passing through a convolutional neural network. The source image I of the image pair S The source image feature map is input to the region of interest transform transformer after passing through a convolutional neural network. The region of interest transform transformer sequentially passes through a feature similarity layer, a cost volume optimizer, and an iterative homography estimator before being connected to the homography transformation module. The homography transformation module is connected to the source image I of the image pair. S Connected, target image I T As input to the cue point selection module, the cue point selection module is connected to the cost volume optimizer after passing through the cue point encoder; The cost volume optimizer is a decoder composed of two connected Transformer layers. Each Transformer layer contains a self-attention layer, a token-to-volume attention layer, a multi-layer encoder, and a volume-to-token attention layer. The point hint token Tok is input to the self-attention layer, which is then connected in sequence to the token-to-volume attention layer, the multi-layer encoder, and the volume-to-token attention layer. The cost volume Vol is input to the token-to-volume attention layer and the volume-to-token attention layer. The output of the volume-to-token attention layer is used as the output of each Transformer layer. The prompt point selection module includes a confidence network, a convolutional neural network in a trained baseline network model, a pseudo-label map calculation layer, and a Top-K random sampler; target image I T The input is fed into a confidence network, which outputs a confidence graph. This serves as the input to a Top-K random sampler, which is connected to a cue point encoder; the convolutional neural network in the trained baseline network model includes the target image I. T The corresponding convolutional neural network and source image I S The corresponding convolutional neural networks are denoted as the first convolutional neural network and the second convolutional neural network, respectively. The target image I T It is also input into the first convolutional neural network, from the source image I S Registered to target image I T Image The inputs are fed into the second convolutional neural network. The outputs of the first and second convolutional neural networks are then fed together into the pseudo-label map computation layer, which outputs the pseudo-label map. During the training of the confidence network, the confidence map is calculated. and pseudo-label images Between loss functions; 3) Use the unregistered images in the image registration training set to train the pre-trained image registration network model to obtain a trained image registration network model; 4) Input the image to be matched into the trained image registration network model, and the model outputs the image registration result.

2. The image registration method based on point cues according to claim 1, characterized in that, The confidence network consists of 4 downsamplers and 4 upsamplers, and adopts a U-Net network structure.

3. The image registration method based on point cues according to claim 1, characterized in that, In the pseudo-label graph calculation layer, the initial pseudo-label graph is first calculated based on the outputs of the first convolutional neural network and the second convolutional neural network. The formula is as follows: in, For the initial pseudo-label image medium pixel The tag value, and These are the pixels in the output of the second convolutional neural network. and eigenvalues, For the pixels in the output of the first convolutional neural network eigenvalues, Indicates transpose. It is a temperature hyperparameter; Then, the initial pseudo-label image is processed. Upsampling is performed to obtain a pseudo-label image of the same size as the input image. .

4. The image registration method based on point cues according to claim 1, characterized in that, The confidence map and pseudo-label images The formula for calculating the loss function is as follows: in, , , For the set of positive categories, For the set of negative categories, For confidence plot medium pixel The tag value, Indicates the number of elements in the set. and Pseudo-label image The mean and standard deviation.

5. The image registration method based on point cues according to claim 1, characterized in that, In the aforementioned prompt point encoder, for Each prompt point input and its coordinate position. First, calculate the token location code of the point prompt to obtain the global two-dimensional location code. Then, based on the coordinates of each prompt point... Global two-dimensional positional encoding The token location code is obtained by indexing the middle index. Then, the following formula is used to calculate the point hint tokens corresponding to positive and negative hint points, thereby obtaining the point hint tokens. The formula is as follows: in, and These correspond to the positive and negative embedding of the cue points, respectively. coordinate position Dot prompt token .

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images