Method and apparatus for generating matching point pair

Abstract

Embodiments of the present specification provide a method and apparatus for generating a matching point pair. The method comprises: obtaining a respective first feature map and second feature map of a first size and a respective third feature map and fourth feature map of a second size for a first image and a second image at different angles of a same scene; using a matching feature point pair corresponding to the first feature map and the second feature map having a small size to determine a feature block pair corresponding to the matching feature point pair from the third feature map and the fourth feature map having a large size; on the basis of the similarity between at least a portion of feature points in a first feature block and at least a portion of feature points in a second feature block in the feature block pair, updating a current homography matrix corresponding to the feature block pair, to obtain a target homography matrix corresponding to the feature block pair; and, on the basis of the target homography matrix, determining a matching point pair between the first image and the second image, to implement determination of a matching point pair between images.

Description

A method and apparatus for generating matching point pairs

[0001] This application claims priority to Chinese Patent Application No. 2024114300109, filed on October 14, 2024, entitled “A Method and Apparatus for Generating Matching Point Pairs”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This specification relates to the field of image processing technology, and in particular to a method and apparatus for generating matching point pairs. Background Technology

[0003] In scenarios such as image stitching and 3D reconstruction, it is first necessary to determine the matching point pairs between at least two image frames. These matching point pairs are pixels corresponding to the same spatial point in the at least two image frames. Then, based on the determined matching point pairs, image stitching or 3D reconstruction is performed. In the above process, the effect of the image stitching result or the effect of the 3D reconstruction depends to a certain extent on the determination of the matching point pairs between the at least two image frames.

[0004] Therefore, how to provide a method for generating matching point pairs has become an urgent problem to be solved. Summary of the Invention

[0005] This specification provides one or more embodiments of a matching point pair generation method and apparatus to determine matching point pairs between images.

[0006] According to the first aspect, a method for generating matching point pairs is provided, including:

[0007] First feature map and second feature map of first size, and third feature map and fourth feature map of second size are extracted from the first image and the second image respectively, wherein the first image and the second image are images of the same scene from different angles, and the second size is larger than the first size;

[0008] Using the matching feature point pairs corresponding to the first feature map and the second feature map, feature block pairs corresponding to the matching feature point pairs are determined from the third feature map and the fourth feature map, wherein the feature block pairs include a first feature block determined from the third feature map and a second feature block determined from the fourth feature map;

[0009] Based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, update the current homography matrix corresponding to the feature block pair and determine the target homography matrix corresponding to the feature block pair;

[0010] Based on the target homography matrix, matching point pairs between the first image and the second image are determined.

[0011] According to the second aspect, a matching point pair generation apparatus is provided, comprising:

[0012] The first obtaining module is configured to extract a first feature map and a second feature map of a first size, and a third feature map and a fourth feature map of a second size from the first image and the second image, respectively, wherein the first image and the second image are images of the same scene from different angles, and the second size is larger than the first size;

[0013] The first determining module is configured to use the matching feature point pairs corresponding to the first feature map and the second feature map to determine the feature block pairs corresponding to the matching feature point pairs from the third feature map and the fourth feature map, wherein the feature block pairs include a first feature block determined from the third feature map and a second feature block determined from the fourth feature map;

[0014] The update module is configured to update the current homography matrix corresponding to the feature block pair based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block, and determine the target homography matrix corresponding to the feature block pair.

[0015] The second determining module is configured to determine the matching point pair between the first image and the second image based on the target homography matrix.

[0016] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0017] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0018] According to the matching point pair generation method and apparatus provided in the embodiments of this specification, coarse-grained feature maps (first feature map and second feature map) and fine-grained feature maps (third feature map and fourth feature map) are extracted from the first image and the second image to be matched, respectively. Then, using the matching feature point pairs corresponding to the two coarse-grained feature maps, feature block pairs corresponding to the matching feature point pairs are determined from the two fine-grained feature maps. Next, based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair. In the above process, the current homography matrix corresponding to the feature block pair can be updated based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair to obtain the target homography matrix corresponding to the feature block pair. Such a target homography matrix can reflect the homography relationship between all positions of the two feature blocks in the feature block pair. Furthermore, based on the aforementioned target homography matrix, more matching point pairs existing between the first image and the second image can be better determined. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 is a schematic diagram of the implementation framework of an embodiment disclosed in this specification;

[0021] Figure 2 is a flowchart illustrating a matching point pair generation method provided in the embodiment;

[0022] Figure 3A is a schematic diagram of a fifth feature map (or a sixth feature map) provided in the embodiment;

[0023] Figure 3B is a schematic diagram of one structure of the first similarity matrix provided in the embodiment;

[0024] Figure 4 is a schematic block diagram of a matching point pair generation device provided in the embodiment. Detailed Implementation

[0025] The technical solutions of the embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0026] This specification discloses a method and apparatus for generating matching point pairs. The application scenarios and technical concepts of the method are described below:

[0027] As mentioned earlier, in scenarios such as image stitching and 3D reconstruction, the quality of the image stitching or 3D reconstruction results depends to a certain extent on the determination of matching point pairs between at least two frames of images. Therefore, how to provide a method for generating matching point pairs has become an urgent problem to be solved.

[0028] Currently, a semi-dense matching method has been proposed to determine matching point pairs between two image frames. The process generally involves: acquiring two images A and B to be matched; extracting features from images A and B to obtain coarse-grained feature map A1 and fine-grained feature map A2 corresponding to image A, and coarse-grained feature map B1 and fine-grained feature map B2 corresponding to image B. Feature map A1 and feature map B1 have the same size, feature map A2 and feature map B2 have the same size, and the size of the fine-grained feature map is larger than the size of the coarse-grained feature map.

[0029] Next, the similarity values ​​between each feature point in feature map A1 and each feature point in feature map B1 are calculated to obtain the similarity matrix 1 between the two. Then, based on the similarity matrix 1, the matching result 1 between feature map A1 and feature map B1 is determined. This matching result 1 indicates the matching point pair between feature map A1 and feature map B1 (a coarse-grained matching point pair between feature maps), that is, it indicates which feature point in feature map A1 and which feature point in feature map B1 are a matching point pair. For example, feature point (a1, b1) in feature map A1 and feature point (a2, b2) in feature map B1 are a matching point pair. Among them, the feature points that are matching point pairs have a higher similarity.

[0030] After obtaining matching result 1, feature block pairs corresponding to each matching point pair are extracted from feature map A2 and feature map B2 based on matching result 1. Taking a pair of matching point pairs (a1,b1) and (a2,b2) in matching result 1 as an example, feature block t1 corresponding to feature point (a1,b1) is determined from feature map A2, and feature block t2 corresponding to feature point (a2,b2) is determined from feature map B2. Feature blocks t1 and t2 form the feature block pairs corresponding to matching point pairs (a1,b1) and (a2,b2).

[0031] Next, for each pair of feature blocks, a fine-grained pair of matching points is determined. Taking the feature block pairs (a1, b1) and (a2, b2) as examples, the central feature point t11 ​​of feature block t1 is determined; the similarity values ​​between the central feature point t11 ​​and each feature point in feature block t2 are calculated; based on the similarity values ​​between the central feature point t11 ​​and each feature point in feature block t2, a corresponding probability expectation distribution is constructed using the expected probability calculation method. The feature point t21 in feature block t2 corresponding to the position with the highest expected probability in the probability expectation distribution is taken as the matching point corresponding to the central feature point t11, thus obtaining a fine-grained pair of matching points (central feature point t11, feature point t21 in feature block t2).

[0032] Then, based on the fine-grained matching point pairs corresponding to each feature block pair, the matching point pairs between image A and image B are determined. This determination process is similar to the aforementioned process of determining a pair of fine-grained matching points from each feature block pair, and will not be repeated here.

[0033] In the above process, the number of matching point pairs between image A and image B determined by the above method is small and not dense enough (for example, when the coarse-grained feature map is 1 / 8 of the original image size, at most one matching point pair can be determined from image B in every 8*8 area in image A), which affects the effect of subsequent tasks. For example, due to the small number of matching point pairs, the image stitching result is not good enough, and the 3D reconstruction effect is not good enough.

[0034] Furthermore, in the process of determining matching point pairs in fine-grained feature block pairs, the fine-grained matching point pairs are determined by calculating the expected probability, that is, based on all regions between the two feature blocks in the feature block pair (including irrelevant regions, i.e., regions with low similarity values ​​between the two feature blocks and relevant regions, i.e., regions with high similarity values ​​between the two feature blocks). This will be affected by the spatial variance brought about by the irrelevant regions in the feature block pair, which will lead to a decrease in the accuracy of the determined fine-grained matching point pairs.

[0035] Furthermore, referring to the process of determining matching point pairs in fine-grained feature block pairs, it can be seen that it can only determine the corresponding matching point for the feature point (central feature point) at the center position of the source feature block (e.g., feature block t1, the feature block determined from feature map A2). The matching point corresponding to the central feature point is not guaranteed to be the center point of the target feature block (the feature block determined from feature map B2). When image B is used as the source image and is matched with other images (e.g., image C) to determine the matching point pairs between them, it is necessary to use the feature points at the center positions of each feature block determined from each fine-grained feature map extracted from image B to determine the corresponding matching point from the corresponding feature block of image C. This can easily lead to discontinuous matching points.

[0036] To address the aforementioned problems, the inventors propose a matching point pair generation method. Figure 1 illustrates an implementation scenario according to an embodiment disclosed in this specification. In this implementation scenario, two images to be matched are acquired, namely a first image and a second image, which are images of the same scene from different angles. Then, feature extraction is performed on the first image and the second image respectively, extracting their respective coarse-grained feature maps (a first feature map of the first size corresponding to the first image and a second feature map of the first size corresponding to the second image) and their respective fine-grained feature maps (i.e., a third feature map of the second size corresponding to the first image and a fourth feature map of the second size corresponding to the second image), wherein the second size is larger than the first size.

[0037] Next, using the matching feature point pairs corresponding to the first and second feature maps, feature block pairs corresponding to the matching feature point pairs are determined from the third and fourth feature maps. The feature block pair includes a first feature block determined from the third feature map (determined based on the feature points in the first feature map in the feature block pair and the relationship between the first and third feature maps) and a second feature block determined from the fourth feature map (determined based on the feature points in the second feature map in the feature block pair and the relationship between the second and fourth feature maps).

[0038] After identifying the feature block pairs, a homography matrix is ​​constructed based on the feature block pairs to obtain the homography relationship of all positions that can be found between the feature block pairs. Specifically, the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair is determined. This similarity can characterize which feature points in the second feature block are more matched (or more correlated) with which feature points in the first feature block. Then, based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair. Finally, based on the target homography matrix, the matching point pairs between the first image and the second image are determined.

[0039] In the above process, based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair. Such a target homography matrix can reflect the homography relationship (i.e., perspective mapping relationship) between all positions of the two feature blocks in the feature block pair. Furthermore, based on the target homography matrix, more matching point pairs existing between the first image and the second image can be better determined.

[0040] The matching point pair generation method provided in this specification will be described in detail below with reference to specific embodiments.

[0041] Figure 2 shows a flowchart of a matching point pair generation method in one embodiment of this specification. This method is executed by an electronic device, which can be any device, equipment, platform, device cluster, etc., with computing and processing capabilities. In the process of determining the network model's energy consumption, as shown in Figure 2, the method includes the following steps S210-S240:

[0042] In step S210, a first feature map and a second feature map of a first size, as well as a third feature map and a fourth feature map of a second size, are extracted from the first image and the second image respectively. The first image and the second image are images of the same scene from different angles, and the second size is larger than the first size.

[0043] In some possible examples, the electronic device acquires a first image and a second image to be matched. The first image and the second image can be a sequence of images captured for the same scene. The first image and the second image are captured from different angles for the scene. There is an overlapping area between the capture angles of the first image and the second image, that is, at least pixel 1 of the first image and pixel 2 of the second image correspond to the same spatial point (or physical point) in the scene.

[0044] After obtaining the first and second images to be matched, any feature extraction algorithm from related technologies can be used to extract features from the first and second images respectively. This yields a first feature map of a first size and a third feature map of a second size from the first image, and a second feature map of a first size and a fourth feature map of a second size from the second image. Since the second size is larger than the first size, the first and second feature maps can be called coarse-grained feature maps, while the third and fourth feature maps are called fine-grained feature maps. The first and second images can have the same size.

[0045] The first feature map and the second feature map corresponding to the first image are used as examples for explanation. Each feature point in the first feature map and the second feature map can represent a more abstract local feature of the local area corresponding to each feature point in the first image (such as structural features, i.e., shape contours in the local area of ​​the first image, as well as color and material features). The feature map can better help determine the matching point pair between the first image and the second image.

[0046] In some possible examples, features can be extracted from the first and second images using a neural network (hereinafter referred to as the extraction network). Specifically, the electronic device can input the first and second images into the extraction network, respectively, and obtain a first feature map and a third feature map corresponding to the first image, as well as a second feature map and a fourth feature map corresponding to the second image.

[0047] In some possible examples, the extraction network can be any type of neural network capable of extracting image features, such as FPN (Feature Pyramid Networks) or other convolutional neural networks with different structures. In some examples, the first size can be 1 / 8 of the original image size (i.e., the first image or the second image), and the second size can be 1 / 2 of the original image size. For example, if the first image and the second image both have dimensions (N, M), then the first size can be (N / 8, M / 8), and the second size can be (N / 2, M / 2).

[0048] When using an FPN network for feature extraction, taking the third feature map of the second size as an example, this third feature map can aggregate the features of the multi-scale feature maps extracted from the first image by the FPN network, thus better representing the features of the first image. Similarly, the fourth feature map can better represent the features of the second image, helping to improve the accuracy of determining matching point pairs between the first and second images.

[0049] In some other possible examples, feature extraction can be performed on the first image and the second image separately using a preset downsampling method, to extract a first feature map and a second feature map of a first size, as well as a third feature map and a fourth feature map of a second size, respectively, from the first image and the second image. For example, the first image (or the second image) is mean-sampled according to the first downsampling size to obtain the third feature map corresponding to the first image; then, the third feature map is mean-sampled again according to the second downsampling size to obtain the first feature map corresponding to the first image.

[0050] After obtaining the coarse-grained first feature map and second feature map, in step S220, using the matching feature point pairs corresponding to the first feature map and second feature map, the feature block pairs corresponding to the matching feature point pairs are determined from the third feature map and fourth feature map, wherein the feature block pairs include the first feature block determined from the third feature map and the second feature block determined from the fourth feature map.

[0051] In this step, after the electronic device obtains the coarse-grained first and second feature maps, it first determines coarse-grained matching feature point pairs based on the first and second feature maps. That is, it determines the matching feature point pairs corresponding to the first and second feature maps, where each matching feature point pair includes one feature point from the first feature map and one feature point from the second feature map. Next, using the coarse-grained matching feature point pairs, it determines the corresponding feature block pairs from the third and fourth feature maps. These feature block pairs include a first feature block determined from the third feature map (determined based on the feature points in the first feature map within the feature block pair and the relationship between the third and first feature maps) and a second feature block determined from the fourth feature map (determined based on the feature points in the second feature map within the feature block pair and the relationship between the fourth and second feature maps).

[0052] Understandably, the matching feature point pairs corresponding to the first and second feature maps can be one or more pairs. When there are multiple pairs of matching feature point pairs, the corresponding feature block pairs can be determined from the third and fourth feature maps for each pair of matching feature point pairs.

[0053] In some possible examples, step 220 may include steps 11-12:

[0054] In step 11, a first matching result is determined using the first feature map and the second feature map, wherein the first matching result indicates the matching feature point pair corresponding to the first feature map and the second feature map.

[0055] In some implementations, step 11 may include steps 111-113:

[0056] In step 111, based on a preset aggregation operation, the first feature map and the second feature map are processed to obtain a fifth feature map corresponding to the first image and a sixth feature map corresponding to the second image. In this step, the electronic device processes the first feature map and the second feature map based on a preset aggregation operation to obtain a fifth feature map that can better represent the features of the first image and a sixth feature map that can better represent the features of the second image.

[0057] In some possible examples, the electronic device can use a pre-trained attention-based aggregation network to process the first and second feature maps according to the preset aggregation operation to obtain a fifth and a sixth feature map. Specifically, the electronic device inputs the first and second feature maps into the aggregation network to aggregate them, obtaining a fifth feature map corresponding to the first image and a sixth feature map corresponding to the second image. In some examples, to further improve the accuracy of the determined matching results, the aggregation network can be a Transformer-based cross-attention mechanism network.

[0058] For example, step 111 may include steps 1111-1112:

[0059] In step 1111, the first feature map and the second feature map are respectively input into the first sub-network of the aggregation network based on the self-attention mechanism, to obtain the first aggregated feature map corresponding to the first feature map and the second aggregated feature map corresponding to the second feature map. Taking the first feature map as an example, the electronic device inputs the first feature map into the aforementioned first sub-network to obtain the first query Q matrix, the first key K matrix, and the first value V matrix corresponding to the first feature map. Then, using a preset attention weight formula, based on the first Q matrix, the first K matrix, and the number of dimensions of the first K matrix, the first attention weight is obtained. Then, using the first attention weight and the first V matrix, the first aggregated feature map is obtained, realizing the self-attention aggregation of the first feature map. Similarly, the second aggregated feature map corresponding to the second feature map is obtained.

[0060] Next, in step 1112, the first aggregated feature map and the second aggregated feature map are input into the second sub-network of the aggregation network based on the cross-attention mechanism to obtain the fifth feature map and the sixth feature map. In this step, the first aggregated feature map and the second aggregated feature map are input into the aforementioned second sub-network to process the first aggregated feature map and the second aggregated feature map respectively, to obtain the second Q matrix, the second K matrix and the second V matrix corresponding to the first aggregated feature map, as well as the third Q matrix, the third K matrix and the third V matrix corresponding to the second aggregated feature map; then, through the second sub-network, using a preset attention weight formula, based on the second Q matrix, the third K matrix and the number of dimensions of the third K matrix, the second attention weight is obtained; then, using the second attention weight and the third V matrix, the cross-attention output of the first aggregated feature map with respect to the second aggregated feature map is obtained, that is, the fifth feature map is obtained. This fifth feature map has stronger representational ability and helps to obtain more accurate matching results.

[0061] Furthermore, through the second sub-network, using a preset attention weight formula, a third attention weight is obtained based on the third Q matrix, the second K matrix, and the number of dimensions of the second K matrix. Then, using the third attention weight and the second V matrix, the cross-attention output of the second aggregated feature map with respect to the third aggregated feature map is obtained, resulting in a sixth feature map. This sixth feature map has stronger representational capabilities and helps to obtain more accurate matching results.

[0062] For example, the aforementioned aggregation network may also include only the aforementioned second sub-network based on the cross-attention mechanism. Accordingly, the electronic device may directly input the first feature map and the second feature map into the aforementioned second sub-network to perform cross-attention aggregation on the first feature map and the second feature map through the second sub-network to obtain the fifth feature map and the sixth feature map.

[0063] In some possible examples, the dimensions of the fifth and sixth feature maps can be the first dimension or not the first dimension.

[0064] Then, in step 112, a first similarity matrix is ​​determined using the fifth feature map and the sixth feature map, wherein the contents of each element in the first similarity matrix indicate the similarity between each feature point in the first feature map and each feature point in the second feature map.

[0065] For example, an electronic device can calculate the similarity values ​​between each feature point in the fifth feature map and each feature point in the sixth feature map to obtain a first similarity matrix, wherein the content of each element in the first similarity matrix indicates the similarity between each feature point in the first feature map and each feature point in the second feature map. In some possible examples, the larger the similarity value between each feature point in the fifth feature map and each feature point in the sixth feature map, the greater the probability that they are a matching feature point pair.

[0066] In some possible implementations, the similarity value can be determined based on the inner product between each feature point in the fifth feature map and each feature point in the sixth feature map. In other possible implementations, the similarity value can be determined based on the distance (e.g., Euclidean distance or cosine distance) between each feature point in the fifth feature map and each feature point in the sixth feature map; this is also acceptable.

[0067] The following example illustrates the process of determining the first similarity matrix, using the dot product between feature points in the fifth and sixth feature maps as an example. For instance, both the fifth and sixth feature maps are feature maps with L channels and a size of H (height) * W (width). As shown in Figure 3A, H * W can represent a two-dimensional coordinate space. Each point in the two-dimensional coordinate space represented by H * W can be represented by a vector with L elements (as shown in Figure 3A, the values ​​of the black positions in the L channels form the vector of the corresponding point), where each element represents the value at the corresponding position (two-dimensional coordinates) of the feature point in the feature map. Accordingly, each feature point in the fifth and sixth feature maps can be represented by a vector with L elements. The electronic device can calculate the dot product between the vectors of each feature point in the fifth feature map and the vectors of each feature point in the sixth feature map to obtain the similarity value between the feature points in the fifth and sixth feature maps, thus obtaining the first similarity matrix.

[0068] In this model, for any feature point in the fifth feature map, there are H*W similarity values ​​between it and each feature point in the sixth feature map, which can form a sub-matrix of size H*W. The sub-matrix of size H*W corresponding to each feature point in the fifth feature map can be arranged according to the positional relationship of each feature point in the fifth feature map to obtain the first similarity matrix. Figure 3B shows a schematic diagram of one structure of the first similarity matrix.

[0069] After obtaining the first similarity matrix, in step 113, the first matching result is determined based on the first similarity matrix.

[0070] In some implementations, the electronic device can, based on the first similarity matrix, determine the largest similarity value from the H*W similarity values ​​corresponding to each feature point in the fifth feature map (taking feature point i as an example), and determine the feature point j in the sixth feature map corresponding to the largest similarity value; then, it is determined whether the largest similarity value is greater than the first similarity threshold. If the largest similarity value is greater than the first similarity threshold, feature point i and feature point j are determined to be a pair of matching feature points. If the largest similarity value is not greater than the first similarity threshold, feature point i and feature point j are determined not to be a pair of matching feature points. This process is repeated to determine all matching feature point pairs and obtain the first matching result.

[0071] In some other implementations, step 113 may include steps 1131-1132:

[0072] In step 1131, the first similarity matrix is ​​processed to obtain the probability matrix. In this step, the electronic device can use a preset activation function to process the first similarity matrix to normalize it and obtain the probability matrix. In some examples, the preset activation function can be the Softmax function. For example, the electronic device can use the preset activation function to process the first similarity matrix along both its row and column dimensions to obtain the probability matrix.

[0073] Next, in step 1132, a first matching result is determined based on the nearest neighbor algorithm and the probability matrix. In this step, the electronic device can use the nearest neighbor algorithm to determine the nearest neighbor feature point pairs between the fifth and sixth feature maps based on each feature point in the fifth feature map and each feature point in the sixth feature map. Each pair of nearest neighbor feature point pairs includes one feature point in the fifth feature map and one feature point in the sixth feature map. Then, based on the probability matrix and the second similarity threshold, it is determined whether each pair of nearest neighbor feature point pairs is a matching feature point pair to obtain the first matching result. For example, if the probability value corresponding to the nearest neighbor feature point pair in the probability matrix is ​​not less than the second similarity threshold, it can be determined that it is a matching feature point pair; otherwise, it is not a matching feature point pair. It can be understood that the first matching result includes one or more pairs of matching feature point pairs (i.e., one or more pairs of matching feature point pairs corresponding to the first and second feature maps).

[0074] In some possible examples, the electronic device can also directly determine a similarity matrix based on the similarity between feature points in the first feature map and feature points in the second feature map, and then determine a first matching result based on the similarity matrix. The process of determining the similarity matrix can be referred to the aforementioned process of determining the first similarity matrix, and the process of determining the first matching result based on the similarity matrix can be referred to the aforementioned process of determining the first matching result based on the first similarity matrix, and will not be repeated here.

[0075] After the electronic device obtains the first matching result, namely the matching feature point pair corresponding to the first feature map and the second feature map, in step 12, it uses the aforementioned matching feature point pair and the dimensional relationship between the first size and the second size to determine the feature block pair corresponding to the aforementioned matching feature point pair from the third feature map and the fourth feature map.

[0076] In some implementations, the electronic device can determine the corresponding feature block pairs from the third and fourth feature maps based on each pair of matching feature points and the dimensional relationship between the first and second dimensions. Understandably, the process of determining the corresponding feature block pairs for any of the aforementioned pairs of matching feature points is similar. The following explanation uses any pair of matching feature points (e.g., matching feature point pair P) as an example to illustrate the process of determining its corresponding feature block pairs.

[0077] Suppose that the matching feature point pair P includes a feature point i (coordinates denoted as (ai, bi)) from the aforementioned fifth feature map and a feature point j (coordinates denoted as (aj, bj)) from the aforementioned sixth feature map, which are mutually matching points. In some examples, when the fifth and sixth feature maps have a first dimension, the feature point i (ai, bi) can be mapped to the third feature map based on the dimensional relationship between the first and second dimensions. After mapping, the corresponding feature point i in the third feature map is obtained. ， The coordinates of the second feature point j (ai', bi') in the third feature map are (ai', bi'), where ai' equals Q times ai, and bi' equals Q times bi, where Q represents the ratio of the second dimension to the first dimension. Similarly, based on the dimensional relationship between the first and second dimensions, the feature point j (aj, bj) is mapped to the fourth feature map. After mapping, the corresponding feature point j' in the fourth feature map is obtained, and its coordinates in the third feature map are (aj', bj'), where aj' equals Q times aj, and bj' equals Q times bj.

[0078] Next, in the third feature map, with feature point i'(ai',bi') as the center point, a feature block of specified size (w*w) corresponding to feature point i is determined; similarly, in the fourth feature map, with feature point j'(aj',bj') as the center point, a feature block of specified size (w*w) corresponding to feature point j is determined, thus obtaining the feature block pair corresponding to the matching feature point pair P. This process is repeated to obtain the feature block pair corresponding to each matching feature point pair.

[0079] In some examples, the value of w can be set according to the dimensional relationship between the first and second dimensions, for example, the value can be set to be greater than or equal to the ratio Q of the second dimension to the first dimension.

[0080] In some other examples, when the dimensions of the fifth and sixth feature maps are not the first dimension, the feature block pairs corresponding to each matching feature point pair can be determined from the third and fourth feature maps based on the dimensional relationship between the dimensions of the fifth (or sixth) feature map and the first dimension, as well as the dimensional relationship between the first and second dimensions, using each matching feature point pair. For details, please refer to the aforementioned process for determining feature block pairs, which will not be elaborated here.

[0081] After obtaining the feature block pair corresponding to the matching feature point pair, in step S230, based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair.

[0082] Understandably, a homography matrix is ​​a 3x3 matrix that describes the perspective (mapping) transformation relationship between two planes. A homography matrix can map points on one plane to another while preserving the linear relationship between the points. This linear relationship means that if points on the planes are collinear, they will remain collinear on the mapped plane. Based on this principle, in this embodiment, for each feature block pair, the electronic device obtains the initial homography matrix corresponding to the feature block pair as the current homography matrix. Then, combining the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, with the goal of aligning the correlated regions between the first and second feature blocks (i.e., the regions where feature points with high similarity values ​​are located), the current homography matrix corresponding to the feature block pair is updated and optimized to obtain a target homography matrix that can better align the correlated regions between the first and second feature blocks. The current homography matrix can be iteratively updated multiple times to obtain the target homography matrix.

[0083] In some possible examples, step S230 can be set as follows: based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, the current homography matrix corresponding to the feature block pair is updated through a pre-trained homography optimization network to obtain the target homography matrix corresponding to the feature block pair.

[0084] The single-image optimization network is trained based on several sample image groups and the corresponding label data of each sample image group. A single sample image group includes two sample images from different angles of the same sample scene. The label data is used to indicate the label matching point pair between the two sample images in the corresponding sample image group.

[0085] In this example, for each feature block pair, the electronic device obtains the initial homography matrix corresponding to the feature block pair as the current homography matrix. Then, combining the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, and leveraging the ability of the homography optimization network to align the correlated regions (i.e., the regions where feature points with high similarity values ​​are located) between the two feature blocks, it estimates the homography residual between the two feature blocks (i.e., the subsequent optimization matrix). Furthermore, for a certain feature point in the first feature block and a feature point in the second feature block... If a feature point is unrelated to two other features (i.e., mismatched), the similarity value between them will be very low. Consequently, its impact on the output of the homography optimization network will be smaller, allowing the homography optimization network to focus more on the higher similarity values ​​(i.e., the correlated regions between the two feature blocks in the feature block pair). This results in a better homography residual (i.e., the subsequent optimization matrix). Then, based on the estimated homography residual between the two feature blocks, the current homography matrix is ​​updated and optimized to obtain the target homography matrix that can align the correlated regions between the first and second feature blocks in the feature block pair.

[0086] For ease of calculation, the coordinates of feature points in the two feature blocks of a feature block pair can be set to lie in the same two-dimensional coordinate space (e.g., a subsequently specified two-dimensional coordinate system). In some examples, the initial homography matrix obtained during initialization can be the identity matrix.

[0087] In some examples, step S230 may include steps 21-24:

[0088] In step 21, a second similarity matrix is ​​determined, wherein each element of the second similarity matrix indicates the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block.

[0089] In the step, the electronic device can determine the similarity value between at least some feature points in the first feature block and at least some feature points in the second feature block (e.g., Euclidean distance or cosine distance), or inner product, to obtain a second similarity matrix.

[0090] The process of determining the second similarity matrix is ​​illustrated below, taking the determination of the similarity value between at least some feature points in the first feature block and at least some feature points in the second feature block as an example.

[0091] For example, both the first and second feature blocks are feature maps of size w (height) * w (width) with D channels. w * w can represent a two-dimensional coordinate space. Each point in the two-dimensional coordinate space represented by w * w can be represented by a vector of D elements, where each element represents the value at the corresponding position (two-dimensional coordinates) of the feature point in the feature block. Accordingly, each feature point in the first and second feature blocks can be represented by a vector of D elements. The electronic device can calculate the inner product between the vectors of at least some feature points in the first feature block and the vectors of at least some feature points in the second feature block to obtain the similarity values ​​between the at least some feature points in the first and second feature blocks, thereby obtaining a second similarity matrix.

[0092] Taking the first feature block as an example, at least some of the feature points therein may include all the feature points of the first feature block, or some of the feature points therein (e.g., feature points at every other position); similarly, at least some of the feature points in the second feature block may include all the feature points of the second feature block, or some of the feature points therein (e.g., feature points at every other position).

[0093] In this system, for any feature point in the first feature block, there are w*w similarity values ​​between it and at least some feature points in the second feature block, which can form a matrix sub-block of size w*w. The matrix sub-blocks of size w*w corresponding to at least some feature points in the first feature block can be arranged according to the positional relationship of at least some feature points in the first feature block to obtain a second similarity matrix, which includes w*w matrix sub-blocks of size w*w. In some cases, the second similarity matrix can be considered a w*w dimensional matrix, where each element includes a matrix sub-block corresponding to each feature point in the first feature block (i.e., subsequent first feature points), and each matrix sub-block has a size of w*w, representing the similarity value between the corresponding feature point in the first feature block and each feature point in the second feature block (i.e., subsequent second feature points). For ease of calculation, the second similarity matrix can be represented as a 4-dimensional tensor, i.e., a w*w*w*w tensor. Its schematic structure can be found in the aforementioned schematic structure of the first similarity matrix.

[0094] For a feature point in the first feature block where no similarity value has been calculated, the content of each element in its corresponding w*w matrix sub-block can be set to a specified value (e.g., 0). For a feature point in the first feature block, if its corresponding w*w matrix sub-block has not had a similarity value calculated with a feature point in the second feature block, the content of the element at the corresponding position can be set to a specified value (e.g., 0).

[0095] After obtaining the second similarity matrix, in step 22, based on the aforementioned current homography matrix, the second feature points corresponding to the first feature points in the first feature block are determined from the second feature block.

[0096] In this step, the two-dimensional coordinates (u, v) of each first feature point in the first feature block in a specified two-dimensional coordinate system can be set. For example, u and v can both be integers between [0, w-1]. In this case, the two-dimensional coordinates of the first feature point at the top left corner of the first feature block can be (0, 0). And the two-dimensional coordinates (u', v') of each second feature point in the second feature block in a specified two-dimensional coordinate system can be set. Here, u' and v' can both be integers between [0, w-1]. In this case, the two-dimensional coordinates of the second feature point at the top left corner of the second feature block can be (0, 0).

[0097] Next, the electronic device constructs the corresponding homogeneous coordinates of each first feature point in the first feature block based on the two-dimensional coordinates (u, v). Then, based on the homogeneous coordinates of each first feature point in the first feature block and the aforementioned homography matrix, it determines the two-dimensional coordinates (u', v') of each second feature point in the second feature block. For example, this can be represented by the following formula (1):

[0098] in, This represents the current homography matrix. This represents the homogeneous coordinates corresponding to the two-dimensional coordinates of each first feature point in the first feature block. The homogeneous coordinates represent the two-dimensional coordinates of the second feature points corresponding to each first feature point.

[0099] Next, in step 23, input data is constructed based on each first feature point and its corresponding second feature point, as well as the second similarity matrix.

[0100] In some possible examples, step 23 may include steps 231-232:

[0101] In step 231, based on each first feature point and its corresponding second feature point, each matrix sub-block in the second similarity matrix corresponding to each first feature point in the first feature block is sampled to determine the first sampling matrix.

[0102] In this step, the electronic device locates the matrix sub-block (i.e., the matrix sub-block of size w*w) corresponding to the first feature point (u1, v1) in the second similarity matrix based on the two-dimensional coordinates of each first feature point (for example, taking the first feature point (u1, v1) as an example). Based on the two-dimensional coordinates (u1', v1') of the second feature point corresponding to the first feature point (u1, v1), the electronic device finds the position corresponding to the second feature point (u1', v1') in the matrix sub-block corresponding to the first feature point (u1, v1), and then samples based on this position to obtain the sampling result corresponding to the first feature point (u1, v1). Similarly, based on the second feature point corresponding to each first feature point, the electronic device samples each matrix sub-block corresponding to each first feature point to obtain the sampling result corresponding to each first feature point. Based on the sampling results corresponding to each first feature point, the first sampling matrix is ​​determined.

[0103] Specifically, in some possible implementations, step 231, determining the first sampling matrix, may include:

[0104] For each matrix sub-block in the second similarity matrix corresponding to each first feature point, the matrix sub-block is sampled with a preset size as the sampling size and the position corresponding to each second feature point of each first feature point as the center point to obtain a first sampling matrix. The content of each element of the matrix sub-block indicates the similarity between the corresponding first feature point and each second feature point in the second feature block.

[0105] Continuing with the example of the first feature point (u1, v1), based on the two-dimensional coordinates (u1', v1') of the second feature point corresponding to the first feature point (u1, v1), the position corresponding to the second feature point (u1', v1') is found in the matrix sub-block corresponding to the first feature point (u1, v1) in the second similarity matrix. Using a preset size (e.g., (2r+1)*(2r+1)) as the sampling size and the position corresponding to the second feature point (u1', v1') as the center point, the matrix sub-block corresponding to the first feature point (u1, v1) is sampled to obtain the sampling result corresponding to the first feature point (u1, v1). This sampling result includes a region (matrix) of size (2r+1)*(2r+1) in the matrix sub-block corresponding to the first feature point (u1, v1), centered at the position corresponding to the second feature point (u1', v1'). This process is repeated to obtain the sampling results corresponding to each first feature point. Based on the sampling results corresponding to each first feature point, the first sampling matrix is ​​obtained. Here, r is a preset exploration radius, which can be set according to actual conditions and needs.

[0106] The first sampling matrix can be considered a w*w dimensional matrix, where each element includes the sampling result corresponding to each first feature point in the first feature block. The size of each sampling result is (2r+1)*(2r+1), representing the similarity value between the corresponding first feature point (i.e., the subsequent first feature point) and each second feature point within a range of (2r+1)*(2r+1) centered at the second feature point (u1', v1'). In some cases, the first sampling matrix can be represented as a 4-dimensional tensor, i.e., a w*w*(2r+1)*(2r+1) tensor.

[0107] After obtaining the first sampling matrix, in step 232, input data is constructed based on the first sampling matrix.

[0108] In some possible examples, the electronic device can flatten the sampling results corresponding to each first feature point in the first sampling matrix (that is, flatten the (2r+1)*(2r+1) elements in each sampling result from matrix form to vector form) to obtain a vector containing (2r+1)*(2r+1) elements corresponding to each first feature point. Accordingly, based on the positional relationship of each first feature point in the first feature map, and based on the vector containing (2r+1)*(2r+1) elements corresponding to each first feature point, a first feature data with (2r+1)*(2r+1) channels and each channel including data of size w*w is obtained; then, this first feature data is used as input data. The input data includes the similarity values ​​between each first feature point and the corresponding second feature point and its surrounding second feature points. This is used as input to the homography optimization network, allowing the network to explore second feature points that better match each first feature point based on the similarity values ​​between the first feature point and its corresponding second feature point and its surrounding second feature points. This leads to the corresponding optimized offset information (i.e., the subsequent optimization matrix), helping to better align the correlated regions between the first and second feature blocks.

[0109] In some other possible implementations, step 23 may include steps 233-234:

[0110] In step 233, based on each first feature point and its corresponding second feature point, the contents of each element in the second similarity matrix are sampled to determine the first sampling matrix. The implementation principle of step 233 is similar to that of step 231 described above, and its implementation process can be found in the implementation process of step 231 described above, which will not be repeated here.

[0111] Then, in step 234, input data is constructed based on each first feature point and its corresponding second feature point and first sampling matrix.

[0112] In this step, the electronic device can flatten the sampling results corresponding to each first feature point in the first sampling matrix to obtain first feature data with (2r+1)*(2r+1) channels, and each channel including data of size w*w; and flatten the two-dimensional coordinates of each first feature point to obtain second feature data with 2 channels and each channel having a size of w*w; similarly, flatten the two-dimensional coordinates of the second feature points corresponding to each first feature point to obtain third feature data with 2 channels and each channel having a size of w*w; then, the first feature data, second feature data and third feature data are concatenated in a specified dimension (channel dimension) to obtain input data.

[0113] This type of input data includes the position coordinates of each first feature point and its corresponding second feature point, which can better help obtain accurate matching results.

[0114] In some possible implementations, step 23 may include step 235: directly constructing input data based on the second similarity matrix and each first feature point and its corresponding second feature point. The implementation principle of step 235 is similar to that of step 234 described above, and its implementation process can be found in the implementation process of step 234 described above, so it will not be repeated here.

[0115] In some possible scenarios, the imaging content sizes of the first image and the second image may differ (for example, the imaging content area of ​​object A in the scene of the first image is larger than the imaging content area of ​​object A in the scene of the second image). To ensure the accurate determination of matching point pairs in the first image and the second image, in some possible implementations, step 23 may include steps 236-237:

[0116] Step 236: For the matrix sub-blocks corresponding to each first feature point in the second similarity matrix, pooling processing is performed on the matrix sub-blocks to obtain the third similarity matrix. In this step, the electronic device can perform pooling processing on the matrix sub-blocks corresponding to each first feature point in the second similarity matrix (taking the first feature point (u1, v1) as an example). For the matrix sub-block corresponding to the first feature point (u1, v1) in the second similarity matrix, which is a matrix sub-block of size w*w, pooling processing is performed on the matrix sub-block of size w*w to obtain the pooling result corresponding to the first feature point (u1, v1). This process is repeated to obtain the pooling results corresponding to each first feature point, thus obtaining the third similarity matrix. The third similarity matrix includes the pooling results corresponding to each first feature point arranged according to its position in the first feature block. Each pooling result can be a sub-matrix (for example, a sub-matrix of size w'*w'). For ease of subsequent processing, the third similarity matrix can be considered a w*w matrix, where each element includes a submatrix of size w'*w'. The third similarity matrix can be represented as a 4-dimensional tensor, for example, a tensor of size w*w*w'*w'. For instance, w' can be equal to w / 2.

[0117] Next, in step 237: based on each first feature point and its corresponding second feature point, the second similarity matrix and the third similarity matrix, the input data is constructed.

[0118] In this step, during the sampling of each sub-matrix corresponding to each first feature point in the third similarity matrix based on each first feature point and its corresponding second feature point, the sub-matrixes corresponding to each first feature point in the third similarity matrix can first be subtracted using bilinear interpolation to obtain the sub-matrixes corresponding to each first feature point in the third similarity matrix after the difference, wherein the size of each sub-matrix after the difference is restored to w*w; then, referring to the aforementioned sampling process for the second similarity matrix, the sub-matrixes corresponding to each first feature point in the third similarity matrix after the difference are sampled based on each first feature point and its corresponding second feature point to obtain the second sampling matrix. Referring to the aforementioned method, the second similarity matrix is ​​sampled based on each first feature point and its corresponding second feature point to obtain the first sampling matrix; in another implementation, the input data is constructed by combining at least the second sampling matrix and the first sampling matrix, as well as the two-dimensional coordinates of each first feature point and its corresponding two-dimensional coordinates of the second feature point. The process of constructing the input data can be found in the implementation process of steps 232 or 234 mentioned above, and will not be repeated here.

[0119] This type of input data allows for a better optimization matrix to be obtained from the homography optimization matrix, leading to a more accurate target homography matrix.

[0120] After obtaining the input data using the above method, in step 24, the current homography matrix is ​​updated based on the input data using a homography optimization network to obtain the target homography matrix. In this step, after the electronic device obtains the input data, it inputs the input data into the homography optimization network to process the input data and obtain the optimization matrix corresponding to the current homography matrix. This optimization matrix is ​​used to adjust the current homography matrix so that the adjusted homography matrix can better align the correlated regions in the first and second feature blocks. The current homography matrix is ​​then optimized using the optimization matrix to obtain the target homography matrix. For example, the aforementioned optimization of the current homography matrix using the optimization matrix can be achieved by performing matrix multiplication on the current homography matrix and the optimization matrix.

[0121] In some possible examples, the single-application optimization network can be implemented using a decoder based on a CNN (Convolutional Neural Network) network, or it can be implemented using other types of decoders.

[0122] In some possible examples, to obtain a target homography matrix that is more accurate and better represents the perspective (mapping) transformation relationship between two feature blocks in a feature block pair, multiple iterations can be performed using a homography optimization network to obtain the target homography matrix; correspondingly, after step 24, which updates the current homography matrix based on the input data using the homography optimization network, the following steps 25-26 can also be included:

[0123] In step 25, it is determined whether the number of times the current homography matrix has been updated has reached a preset number. This preset number can be set according to actual conditions; for example, it can be set based on the aforementioned sampling size w and / or exploration radius r, to better and more comprehensively explore the input data, obtain a higher-quality target homography matrix, and better reflect the perspective transformation relationship between the first and second feature blocks.

[0124] Subsequently, in step 26, if the determination is negative, the updated current homography matrix is ​​used as the new current homography matrix, and the step of determining the second feature points corresponding to the first feature points in the first feature block from the second feature block based on the current homography matrix (i.e., the aforementioned step 21) continues.

[0125] In the above implementation, after obtaining the updated current homography matrix using the homography optimization network, it is then determined whether the number of times the current homography matrix has been updated has reached a preset number. If the determination is no, meaning the number of times the current homography matrix has been updated has not reached the preset number, the updated current homography matrix is ​​used as the new current homography matrix. Based on this new current homography matrix, the second feature points corresponding to each first feature point are determined from the second feature block. Then, based on the newly determined first feature points and their corresponding second feature points, as well as the second similarity matrix, input data is constructed, and the current homography matrix is ​​updated again using the homography optimization network based on the input data. The determination of whether the number of times the current homography matrix has been updated has been reached is repeated, and so on, until the number of times the current homography matrix has been updated has reached the preset number. The updated current homography matrix at this point is then used as the target homography matrix.

[0126] After obtaining the target homography matrix, in step S240, based on the target homography matrix, the matching point pairs between the first image and the second image are determined.

[0127] It is understood that the aforementioned feature block pairs can be one or more. When there are multiple feature block pairs, it is necessary to determine the matching feature point pairs corresponding to each feature block pair between the first and second images based on the target homography matrix corresponding to each feature block pair. Then, based on the sum of the matching feature point pairs corresponding to each feature block pair, the matching point pairs between the first and second images are determined. The process of determining the matching point pairs between the first and second images based on the target homography matrix corresponding to each feature block pair is similar. The following explanation uses the target homography matrix corresponding to one pair of feature block pairs as an example to illustrate the process of determining the matching point pairs between the first and second images. In this step, the electronic device can determine the matching point pairs between the first and second images based on the target homography matrix and its corresponding feature block pairs.

[0128] In one possible example, the electronic device can obtain a second matching result based on the target homography matrix, the two-dimensional coordinates (u, v) of each first feature point in the first feature block of the feature block pair in the aforementioned specified two-dimensional coordinate system, and the two-dimensional coordinates (u', v') of each second feature point in the second feature block in the aforementioned specified two-dimensional coordinate system. The second matching result includes at least one pair of fine-grained matching feature point pairs. Then, for each pair of fine-grained matching feature point pairs, the electronic device, based on the coordinates of the upper left corner feature point of the first feature block in the third feature map, transforms the two-dimensional coordinates of the first feature point in the fine-grained matching feature point pair to the coordinate system of the third feature map (for example, adding the two-dimensional coordinates of the first feature point in the fine-grained matching feature point pair to the coordinates of the upper left corner feature point of the first feature block in the third feature map to transform the two-dimensional coordinates of the first feature point in the fine-grained matching feature point pair to the coordinate system of the third feature map). In the coordinate system of the fine-grained matching feature point pair, the coordinates (position) of the first feature point in the third feature map are obtained. Based on the coordinates of the upper left corner feature point of the second feature block in the fine-grained matching feature point pair in the fourth feature map, the two-dimensional coordinates of the second feature point in the fine-grained matching feature point pair are transformed to the coordinate system of the fourth feature map (for example, the two-dimensional coordinates of the second feature point in the fine-grained matching feature point pair are added to the coordinates of the upper left corner feature point of the second feature block in the fourth feature map to transform the two-dimensional coordinates of the second feature point in the fine-grained matching feature point pair to the coordinate system of the fourth feature map), thus obtaining the coordinates (position) of the second feature point in the fine-grained matching feature point pair in the fourth feature map. Similarly, the coordinates (positions) of the first feature point in the third feature map and the coordinates (positions) of the second feature point in the fourth feature map of each pair of fine-grained matching feature point pairs are obtained.

[0129] Subsequently, based on the coordinates (positions) of the first feature point in the third feature map and the coordinates (positions) of the second feature point in the fourth feature map of each pair of fine-grained matching feature points, as well as the size relationship between the third feature map and the first image (or the size relationship between the fourth feature map and the second image), matching point pairs between the first and second images are obtained. Specifically, based on the size relationship between the third feature map and the first image, the coordinates (positions) of the first feature point in the third feature map of each pair of fine-grained matching feature points are transformed to the coordinate system of the first image to obtain its coordinates (positions) in the first image; based on the size relationship between the third feature map and the first image, the coordinates (positions) of the second feature point in the fourth feature map of each pair of fine-grained matching feature points are transformed to the coordinate system of the second image to obtain its coordinates (positions) in the second image, thus obtaining matching point pairs between the first and second images. At this point, semi-dense matching point pairs between the first and second images can be obtained.

[0130] For example, the aforementioned process of transforming the coordinates (positions) of the first feature point in each pair of fine-grained matching feature points in the third feature map to the coordinate system of the first image based on the size relationship between the third feature map and the first image can be as follows: Based on the size of the first image and the size of the third feature map, calculate the size ratio between the first image and the third feature map; multiply the coordinates (positions) of the first feature point in each pair of fine-grained matching feature points in the third feature map by the size ratio between the first image and the third feature map to transform it to the coordinate system of the first image. The process of transforming the coordinates (positions) of the second feature point in each pair of fine-grained matching feature points in the fourth feature map to the coordinate system of the second image is similar to the process of transforming the coordinates (positions) of the first feature point in each pair of fine-grained matching feature points in the third feature map to the coordinate system of the first image, and will not be elaborated upon here.

[0131] In some possible examples, step S240 includes: determining a pair of matching points between the first image and the second image based on the target homography matrix and preset densification information, wherein the densification radius is used to increase the feature point density corresponding to the first feature block.

[0132] For example, an electronic device can determine a dense grid coordinate set Ge based on preset dense information, and use the target homography matrix, the dense grid coordinate set Ge, and each first feature point in the first feature block and each second feature point in the second feature block to determine the matching point pair between the first image and the second image.

[0133] The preset densification information can be set according to actual needs. For example, the preset densification information may include a preset densification radius and a preset step size. The preset step size is used to limit the interval of the generated grid coordinates. The preset densification radius and preset step size can be set according to actual needs, for example, based on the size relationship between the third feature map and the first image (or the size relationship between the fourth feature map and the second image) and / or the required density of matching point pairs between the first and second images. As another example, the preset densification information includes the aforementioned densified grid coordinate set Ge, which can be parsed from the preset densification information.

[0134] In some implementations, step S240 may include steps 31-33:

[0135] In step 31, based on the preset densification information, multiple densified coordinates corresponding to each first feature point in the first feature block and multiple densified coordinates corresponding to each second feature point in the second feature block are obtained based on the two-dimensional coordinates of each first feature point in the first feature block and the two-dimensional coordinates of each second feature point in the second feature block, respectively.

[0136] Let's take the example of preset densification information including preset densification radius and preset step size. Taking a third feature map size that is half the size of the first image as an example, the size of the third feature map needs to be increased by a factor of 2 to be reconstructed to the size of the first image. To obtain more matching point pairs between the first and second images (for example, to ensure that for every 4 pixels in each 2*2 region of the first image, at most 4 matching points can be found in the second image), we can set the preset step size to 0.5 and the preset densification radius Re = 1. Correspondingly, we can obtain the densified grid coordinate set Ge:

[0137] For example, if it is necessary to obtain a denser (more numerous) number of matching point pairs between the first image and the second image, a larger preset densification radius (e.g., 2 or 3) and / or a smaller preset step size (e.g., 0.25) can be set to obtain the corresponding densified grid coordinate set Ge.

[0138] After obtaining the densed grid coordinate set Ge, based on the two-dimensional coordinates of each first feature point in the first feature block (the two-dimensional coordinates in the aforementioned specified two-dimensional coordinate system) and the densed grid coordinate set Ge, the two-dimensional coordinates of each first feature point in the first feature block are dilated to obtain multiple densed coordinates corresponding to each first feature point in the specified two-dimensional coordinate system. For example, for each first feature point (for example, taking the first feature point O as an example), each coordinate in the densed grid coordinate set Ge is added to the two-dimensional coordinates of the first feature point O in the aforementioned specified two-dimensional coordinate system to obtain multiple densed coordinates corresponding to the first feature point O in the specified two-dimensional coordinate system. This process is repeated to obtain multiple densed coordinates corresponding to each first feature point in the specified two-dimensional coordinate system.

[0139] Similarly, based on the two-dimensional coordinates of each second feature point in the second feature block (which are two-dimensional coordinates in the aforementioned specified two-dimensional coordinate system) and the densed grid coordinate set Ge, multiple densed coordinates corresponding to each second feature point in the specified two-dimensional coordinate system are obtained.

[0140] Next, in step 32, based on the target homography matrix and multiple densed coordinates corresponding to each first feature point, the matching densed coordinates that match each densed coordinate corresponding to each first feature point are determined from the multiple densed coordinates corresponding to each second feature point, so as to determine the densed matching result.

[0141] In this step, the electronic device constructs homogeneous coordinates corresponding to each densed coordinate (coordinates under the aforementioned specified two-dimensional coordinates) of each first feature point (taking the first feature point O as an example). Using the aforementioned formula (1), and utilizing the homogeneous coordinates corresponding to each densed coordinate of the first feature point O and the target homography matrix, it determines the matching densed coordinates that match each densed coordinate of the first feature point O from the multiple densed coordinates corresponding to each second feature point. This process is repeated to obtain the matching densed coordinates that match each densed coordinate of the first feature point, thus obtaining the densed matching result. The densed matching result may include multiple pairs of densed matching feature points. A single densed matching feature point pair includes a densed coordinate corresponding to the first feature point, and a matching densed coordinate that matches the densed coordinate of the first feature point (this matching densed coordinate belongs to the multiple densed coordinates corresponding to each second feature point in the second feature block).

[0142] Then, in step 33, based on the aforementioned dense matching results, a pair of matching points between the first image and the second image is determined.

[0143] The electronic device can map the dense matching results back to the corresponding images (that is, map the dense coordinates corresponding to the first feature point in the dense matching result to the coordinate system of the first image based on the first mapping relationship, and map the matching dense coordinates that match the dense coordinates corresponding to the first feature point in the dense matching result to the coordinate system of the second image based on the second mapping relationship) to obtain matching point pairs between the first and second images. The first mapping relationship can be determined based on the relationship between the first feature block and the first image (e.g., the positional relationship between the first feature block and the third feature map, and the size relationship between the third feature map and the first image). The second mapping relationship in the above process can be determined based on the relationship between the second feature block and the second image (e.g., the positional relationship between the second feature block and the fourth feature map, and the size relationship between the fourth feature map and the second image).

[0144] In some specific examples, the electronic device can, at least based on the coordinates (positions) of the first feature point at the top left corner of the first feature block in the third feature map, transform the densed coordinates corresponding to each first feature point to the coordinate system of the third feature map (for example, adding the densed coordinates corresponding to each first feature point to the coordinates (positions) of the first feature point at the top left corner of the first feature block in the third feature map), to obtain the coordinates of the densed coordinates corresponding to each first feature point in the coordinate system of the third feature map.

[0145] Similarly, based at least on the coordinates (positions) of the second feature point at the top left corner of the second feature block in the fourth feature map, the matching dense coordinates that correspond to each first feature point can be transformed to the coordinate system of the fourth feature map (for example, adding the matching dense coordinates that correspond to each first feature point to the coordinates (positions) of the second feature point at the top left corner of the second feature block in the fourth feature map), to obtain the coordinates of the matching dense coordinates that correspond to each first feature point in the coordinate system of the fourth feature map.

[0146] Subsequently, based on the coordinates of the densification coordinates corresponding to each first feature point in the coordinate system of the third feature map, and the size relationship between the third feature map and the first image, the coordinates of the densification coordinates corresponding to each first feature point in the coordinate system of the first image are determined.

[0147] Similarly, based on the coordinates of each matching densed coordinate corresponding to each first feature point in the coordinate system of the fourth feature map, and the size relationship between the fourth feature map and the second image, the coordinates of each matching densed coordinate corresponding to each first feature point in the coordinate system of the second image are determined to obtain matching point pairs between the first image and the second image. For example, each matching point pair between the first image and the second image may include the coordinates of a densed coordinate corresponding to a first feature point in the coordinate system of the first image, and the coordinates of the matching densed coordinate corresponding to that first feature point in the coordinate system of the second image.

[0148] In some examples, there can be multiple pairs of matching feature points (coarse-grained matching feature points) corresponding to the first and second feature maps. Correspondingly, using these coarse-grained matching feature point pairs, multiple pairs of feature blocks are determined from the third and fourth feature maps (each pair of coarse-grained matching feature points corresponds to a pair of feature blocks). For example, the matching feature point pairs corresponding to the first and second feature maps include a first matching feature point pair and a second matching feature point pair, where the first matching feature point pair corresponds to a first feature block pair, and the second matching feature point pair corresponds to a second feature block pair. By densifying the two-dimensional coordinates of each first feature point in the first feature block of the first feature block pair, we obtain... After obtaining multiple densed coordinates corresponding to each first feature point in the first feature block of the first feature block pair (hereinafter referred to as multiple densed coordinates corresponding to the first feature block in the first feature block pair); and after performing densed processing on the two-dimensional coordinates of each first feature point in the first feature block of the second feature block pair to obtain multiple densed coordinates corresponding to each first feature point in the first feature block of the second feature block pair (hereinafter referred to as multiple densed coordinates corresponding to the first feature block in the second feature block pair), it is inevitable that there will be overlapping coordinates between the multiple densed coordinates corresponding to the first feature block of the first feature block of the first feature block pair and the multiple densed coordinates corresponding to the first feature block of the second feature block pair.

[0149] For a specific example, the multiple densed coordinates corresponding to the first feature point a in the first feature block A1 of the first feature block pair include the first densed coordinates, and the multiple densed coordinates corresponding to the first feature point b in the first feature block B1 of the second feature block pair also include the first densed coordinates.

[0150] In the above situation, the target homography matrices corresponding to the first feature block pair and the second feature block pair may be different. In this case, different matching densed coordinates that match the first densed coordinates may be determined from the multiple densed coordinates corresponding to each second feature point in the second feature block A2 of the first feature block pair and from the multiple densed coordinates corresponding to each second feature point in the second feature block B2 of the second feature block pair, respectively. (For example, among the multiple densed coordinates corresponding to each second feature point in the second feature block A2, there exists a matching densed coordinate 1 that matches the first densed coordinate (for example, it is the subsequent first candidate coordinate); among the multiple densed coordinates corresponding to each second feature point in the second feature block B2, there exists a matching densed coordinate 2 that matches the first densed coordinate (for example, it is the subsequent second candidate coordinate)).

[0151] Since the matching point pairs between two image frames (or two feature maps) are generally one-to-one (for example, there is generally a pixel in the second image that is a matching point with a pixel in the first image), the final matching denser coordinates that match the first denser coordinates can be determined from matching denser coordinates 1 and matching denser coordinates 2 by using the distance between the first denser coordinates and the denser center coordinates corresponding to the first feature block A1 in the first feature block pair, and the distance between the first denser coordinates and the denser center coordinates corresponding to the first feature block B1 in the second feature block pair. (For example, if the distance between the first denser coordinates and the denser center coordinates corresponding to the first feature block A1 in the first feature block pair is closer, then matching denser coordinates 1 will be used as the final matching denser coordinates that match the first denser coordinates.)

[0152] Accordingly, in some specific examples, the aforementioned matching feature point pairs include a first matching feature point pair and a second matching feature point pair, the first matching feature point pair corresponds to a first feature block pair, and the second matching feature point pair corresponds to a second feature block pair; the multiple densed coordinates corresponding to the first feature point a in the first feature block in the first feature block pair include the first densed coordinates, and the multiple densed coordinates corresponding to the first feature point b in the first feature block in the second feature block pair include the first densed coordinates.

[0153] Accordingly, step 32 may include steps 321-323:

[0154] In step 321, based on the target homography matrix corresponding to the first feature block pair and the first densed coordinates, the first candidate coordinates corresponding to the first densed coordinates are determined from the multiple densed coordinates corresponding to the second feature points of the second feature block in the first feature block pair.

[0155] In step 322, based on the target homography matrix corresponding to the second feature block pair and the first densed coordinates, a second alternative coordinate corresponding to the first densed coordinates is determined from multiple densed coordinates corresponding to the second feature points of the second feature block in the second feature block pair.

[0156] In step 323, based on the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the first feature block pair, and the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the second feature block pair, a matching densed coordinate that matches the first densed coordinate is determined from the first candidate coordinate and the second candidate coordinate.

[0157] The density center coordinates of the first feature block in the first feature block pair are determined based on multiple density coordinates corresponding to each first feature point in the first feature block of the first feature block pair, and the density center coordinates of the first feature block in the second feature block pair are determined based on multiple density coordinates corresponding to each first feature point in the first feature block of the second feature block pair.

[0158] For example, if the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the first feature block pair is less than the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the second feature block pair, then the first candidate coordinate is determined as the matching densed coordinate that matches the first densed coordinate. If the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the first feature block pair is greater than the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the second feature block pair, then the second candidate coordinate is determined as the matching densed coordinate that matches the first densed coordinate, so as to obtain a densed matching point pair between the first image and the second image. If the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the first feature block pair is equal to the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the second feature block pair, either the first candidate coordinate or the second candidate coordinate can be randomly determined as the matching densed coordinate that matches the first densed coordinate.

[0159] In this embodiment, a pre-trained homography optimization network can optimize the current homography matrix corresponding to a feature block pair based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block. This optimization yields a target homography matrix for the feature block pair. This target homography matrix allows us to obtain the homography relationships between all positions within the feature block pair. Furthermore, based on the target homography matrix, matching point pairs between the first and second images can be determined. In this process, based on the coarse-grained matching results, corresponding feature block pairs are determined from the fine-grained feature map. Then, using the homography optimization matrix and the similarity values ​​between the feature points of the two feature blocks in the feature block pair, the corresponding target homography matrix is ​​determined, thereby identifying matching point pairs between the images. This approach is highly efficient.

[0160] In some examples, for the first image and the second image, the pixels that are identified as matching points can be called key points of the images. For example, pixel 1 in the first image and pixel 2 in the second image are a pair of matching points, that is, pixel 1 is a key point in the first image and pixel 2 is a key point in the second image.

[0161] In the above process, the first, second, and third images in the sequence are images of the same scene from different angles. In determining the matching points of pixels in the first image from the second image, the target homography matrix, which characterizes the perspective transformation relationship between two feature blocks, can be used as a basis. Based on the target homography matrix corresponding to the feature block pair, the corresponding matches of all positions on the first feature block within the second feature block can be obtained. Correspondingly, all matching point pairs existing between the first and second images can be obtained (i.e., a pair of matching points between the first and second images, including pixels corresponding to the same physical point in both; such pixels can be called keypoints in the first or second image). Similarly, in determining the matching points of pixels in the second image from the third image, all matching point pairs existing between the second and third images can also be determined.

[0162] In view of the above, if there is an overlapping area between the first image and the third image, the matching point pair generation process provided in this embodiment can realize the continuity of key points between the sequence images (also known as realizing the repetition of key points between the sequence images. For example, assuming that the first image, the second image and the third image all include pixels corresponding to spatial point PO, based on the matching point pair generation process provided in this embodiment, for pixel PO1 corresponding to spatial point PO in the first image, the matching point PO2 (i.e., the pixel corresponding to spatial point PO in the second image) that matches pixel PO1 can be determined from the second image. Furthermore, the matching point PO3 (i.e., the pixel corresponding to spatial point PO in the third image) that matches pixel PO2 can also be determined from the third image.

[0163] In some examples, the aforementioned extraction network, aggregation network, and homography optimization network can be pre-trained jointly based on several sample image groups and the corresponding label data for each sample image group. During training, a training set can be obtained, which includes several sample image groups and the corresponding label data for each sample image group. Each sample image group includes two sample images from different angles of the same sample scene. The label data is used to indicate the label matching point pairs between the two sample images in the corresponding sample image group.

[0164] After obtaining the training set, for each sample image group, the current matching point pair between two sample images in each sample image group can be obtained based on the aforementioned matching point pair generation process. Then, a preset loss function (such as the L2 loss function) can be used to construct a loss value based on the difference between the current matching point pair and the label matching point pair between two sample images in the label data. With the goal of minimizing the loss value, the parameters of the extraction network, the aforementioned aggregation network, and the homography optimization network are adjusted until the preset convergence condition is reached. Once the training of the extraction network, the aforementioned aggregation network, and the homography optimization network is completed, the aforementioned matching point pair generation process can be executed.

[0165] In some examples, the aforementioned current matching point pairs may include dense matching point pairs between two sample images in each sample image group, and semi-dense matching point pairs between two sample images in each sample image group. The process for determining dense matching point pairs between two sample images in each sample image group, and the process for determining semi-dense matching point pairs between two sample images in each sample image group, can be found in the aforementioned processes for determining dense matching point pairs and semi-dense matching point pairs between the first and second images, respectively, and will not be repeated here. In this case, the label matching point pairs between two sample images in the label data may include semi-dense label matching point pairs and dense label matching point pairs between the two sample images.

[0166] The preset convergence conditions may include, but are not limited to: the loss value is lower than a preset threshold, the number of parameter adjustments exceeds a preset threshold, or the training time exceeds a preset time.

[0167] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0168] Corresponding to the above method embodiments, this specification provides a matching point pair generation device 400, the schematic block diagram of which is shown in FIG4, including: a first obtaining module 410, configured to extract a first feature map and a second feature map of a first size, and a third feature map and a fourth feature map of a second size, respectively, from a first image and a second image, wherein the first image and the second image are images of the same scene from different angles, and the second size is larger than the first size; a first determining module 420, configured to use the matching feature point pairs corresponding to the first feature map and the second feature map to obtain a first feature map and a second feature map of a second size from the third feature map and the second feature map. In the feature map and the fourth feature map, feature block pairs corresponding to the matching feature point pairs are determined, wherein the feature block pairs include a first feature block determined from the third feature map and a second feature block determined from the fourth feature map; the update module 430 is configured to update the current homography matrix corresponding to the feature block pairs based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block, to obtain the target homography matrix corresponding to the feature block pairs; the second determination module 440 is configured to determine the matching point pairs between the first image and the second image based on the target homography matrix.

[0169] In some possible examples, the first determining module 420 includes: a first determining unit (not shown in the figure), configured to determine a first matching result using the first feature map and the second feature map, wherein the first matching result indicates a pair of matching feature points corresponding to the first feature map and the second feature map; and a second determining unit (not shown in the figure), configured to determine a pair of feature blocks corresponding to the pair of matching feature points from the third feature map and the fourth feature map using the pair of matching feature points and the dimensional relationship between the first size and the second size.

[0170] In some possible examples, the first determining unit includes: an input obtaining submodule (not shown in the figure), configured to process the first feature map and the second feature map based on a preset aggregation operation to obtain a fifth feature map corresponding to the first image and a sixth feature map corresponding to the second image; a first determining submodule (not shown in the figure), configured to determine a first similarity matrix using the fifth feature map and the sixth feature map, wherein the content of each element in the first similarity matrix indicates the similarity between each feature point in the first feature map and each feature point in the second feature map; and a second determining submodule (not shown in the figure), configured to determine the first matching result based on the first similarity matrix.

[0171] In some possible examples, the input obtaining sub-module is specifically configured to input the first feature map and the second feature map into the first sub-network of the aggregation network based on the self-attention mechanism, respectively, to obtain the first aggregated feature map corresponding to the first feature map and the second aggregated feature map corresponding to the second feature map; and input the first aggregated feature map and the second aggregated feature map into the second sub-network of the aggregation network based on the cross-attention mechanism to obtain the fifth feature map and the sixth feature map.

[0172] In some possible examples, the second determining submodule is specifically configured to process the first similarity matrix to obtain a probability matrix; and determine the first matching result based on the mutual nearest neighbor algorithm and the probability matrix.

[0173] In some possible examples, the second determining module 440 is specifically configured to determine a pair of matching points between the first image and the second image based on the target homography matrix and preset densification information, wherein the densification radius is used to increase the feature point density corresponding to the first feature block.

[0174] In some possible examples, the second determining module 440 includes: a obtaining unit (not shown in the figure), configured to obtain multiple densed coordinates corresponding to each first feature point in the first feature block and multiple densed coordinates corresponding to each second feature point in the second feature block based on preset densed information and the two-dimensional coordinates of each first feature point in the first feature block and the two-dimensional coordinates of each second feature point in the second feature block; a third determining unit (not shown in the figure), configured to determine, based on the target homography matrix and the multiple densed coordinates corresponding to each first feature point, each matching densed coordinate that matches each densed coordinate corresponding to each first feature point from the multiple densed coordinates corresponding to each second feature point, so as to determine the densed matching result; and a fourth determining unit (not shown in the figure), configured to determine the matching point pair between the first image and the second image based on the densed matching result.

[0175] In some possible examples, the matching feature point pair includes a first matching feature point pair and a second matching feature point pair, the first matching feature point pair corresponding to a first feature block pair, and the second matching feature point pair corresponding to a second feature block pair; the plurality of densed coordinates corresponding to the first feature point a in the first feature block in the first feature block pair includes the first densed coordinates, and the plurality of densed coordinates corresponding to the first feature point b in the first feature block in the second feature block pair includes the first densed coordinates;

[0176] The third determining unit is specifically configured to: based on the target homography matrix corresponding to the first feature block pair and the first densed coordinates, determine a first candidate coordinate corresponding to the first densed coordinates from multiple densed coordinates corresponding to the second feature points of the second feature block in the first feature block pair; based on the target homography matrix corresponding to the second feature block pair and the first densed coordinates, determine a second candidate coordinate corresponding to the first densed coordinates from multiple densed coordinates corresponding to the second feature points of the second feature block in the second feature block pair; based on the target homography matrix corresponding to the second feature block pair and the first densed coordinates, determine a second candidate coordinate corresponding to the first densed coordinates from multiple densed coordinates corresponding to the second feature points of the second feature block in the second feature block pair; and based on the target homography matrix corresponding to the first feature block in the first feature block pair and the first densed coordinates corresponding to the second feature points of the second feature block in the first feature block pair, determine a second candidate coordinate corresponding to the first densed coordinates; and based on the target homography matrix corresponding to the second ... The distance between the densification center coordinates and the distance between the first densification coordinates and the densification center coordinates corresponding to the first feature block in the second feature block pair are used to determine the matching densification coordinates that match the first densification coordinates from the first candidate coordinates and the second candidate coordinates. The densification center coordinates corresponding to the first feature block in the first feature block pair are determined based on multiple densification coordinates corresponding to each first feature point in the first feature block in the first feature block pair, and the densification center coordinates corresponding to the first feature block in the second feature block pair are determined based on multiple densification coordinates corresponding to each first feature point in the first feature block in the second feature block pair.

[0177] In some possible examples, the update module 430 is specifically configured to update the current homography matrix corresponding to the feature block pair through a pre-trained homography optimization network based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, thereby obtaining the target homography matrix corresponding to the feature block pair.

[0178] In some possible examples, the homography optimization network is trained based on several sample image groups and the corresponding label data for each sample image group, wherein a single sample image group includes two sample images from different angles of the same sample scene, and the label data is used to indicate the label matching point pairs between the two sample images in the corresponding sample image group.

[0179] In some possible examples, the update module 430 includes: a fifth determining unit (not shown in the figure), configured to determine a second similarity matrix, wherein the contents of each element in the second similarity matrix indicate the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block; a sixth determining unit (not shown in the figure), configured to determine, based on the current homography matrix, each second feature point corresponding to each first feature point in the first feature block from the second feature block; a construction unit (not shown in the figure), configured to construct input data based on each first feature point and its corresponding second feature point, and the second similarity matrix; and an update unit (not shown in the figure), configured to update the current homography matrix based on the input data through the homography optimization network to obtain the target homography matrix.

[0180] In some possible examples, the construction unit includes: a third determining submodule (not shown in the figure), configured to sample each matrix sub-block in the second similarity matrix corresponding to each first feature point based on each first feature point and its corresponding second feature point, to determine a first sampling matrix; and a construction submodule (not shown in the figure), configured to construct the input data based on the first sampling matrix.

[0181] In some possible examples, the construction submodule is specifically configured to construct the input data based on each first feature point and its corresponding second feature point and the first sampling matrix.

[0182] In some possible examples, the construction unit is specifically configured to perform pooling processing on each matrix sub-block in the second similarity matrix corresponding to each first feature point to obtain a third similarity matrix; and to construct input data based on each first feature point and its corresponding second feature point, the second similarity matrix and the third similarity matrix.

[0183] In some possible examples, the third determining submodule is specifically configured to sample each matrix sub-block in the second similarity matrix corresponding to each first feature point in the first feature block, with a preset size as the sampling size and the position corresponding to the second feature point of each first feature point as the center point, to obtain the first sampling matrix, wherein the content of each element in the matrix sub-block indicates the similarity between the corresponding first feature point and each second feature point in the second feature block.

[0184] In some possible examples, the update module 430 further includes: a judgment unit (not shown in the figure), configured to determine whether the number of times the current homography matrix has been updated has reached a preset number after the current homography matrix has been updated based on the input data through the homography optimization network; and a determination unit (not shown in the figure), configured to, if the determination is not true, use the updated current homography matrix as the new current homography matrix and continue to trigger the sixth determination unit.

[0185] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0186] This specification also provides a computer-readable storage medium storing a computer program that, when executed in a computer, causes the computer to perform the matching point pair generation method provided in this specification.

[0187] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the matching point pair generation method provided in this specification.

[0188] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0189] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0190] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating matching point pairs, comprising: First feature map and second feature map of first size, and third feature map and fourth feature map of second size are extracted from the first image and the second image respectively, wherein the first image and the second image are images of the same scene from different angles, and the second size is larger than the first size; Using the matching feature point pairs corresponding to the first feature map and the second feature map, feature block pairs corresponding to the matching feature point pairs are determined from the third feature map and the fourth feature map, wherein the feature block pairs include a first feature block determined from the third feature map and a second feature block determined from the fourth feature map; Based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair. Based on the target homography matrix, matching point pairs between the first image and the second image are determined.

2. The method of claim 1, wherein, Determining the feature block pair corresponding to the matching feature point pair includes: Using the first feature map and the second feature map, a first matching result is determined, wherein the first matching result indicates the matching feature point pair corresponding to the first feature map and the second feature map; Using the matching feature point pairs and the dimensional relationship between the first size and the second size, the feature block pairs corresponding to the matching feature point pairs are determined from the third feature map and the fourth feature map.

3. The method of claim 2, wherein, Determining the first matching result includes: Based on a preset aggregation operation, the first feature map and the second feature map are processed to obtain the fifth feature map corresponding to the first image and the sixth feature map corresponding to the second image. Using the fifth feature map and the sixth feature map, a first similarity matrix is ​​determined, wherein the content of each element in the first similarity matrix indicates the similarity between each feature point in the first feature map and each feature point in the second feature map; Based on the first similarity matrix, the first matching result is determined.

4. The method of claim 3, wherein, Obtaining the fifth feature map corresponding to the first image and the sixth feature map corresponding to the second image includes: The first feature map and the second feature map are respectively input into the first sub-network of the aggregation network based on the self-attention mechanism to obtain the first aggregated feature map corresponding to the first feature map and the second aggregated feature map corresponding to the second feature map. The first aggregated feature map and the second aggregated feature map are input into the second sub-network of the aggregated network based on the cross-attention mechanism to obtain the fifth feature map and the sixth feature map.

5. The method of claim 3, wherein, Determining the first matching result based on the first similarity matrix includes: Process the first similarity matrix to obtain the probability matrix; The first matching result is determined based on the nearest neighbor algorithm and the probability matrix.

6. The method of claim 1, wherein, Determining the matching point pair between the first image and the second image includes: Based on the target homography matrix and the preset densification information, matching point pairs between the first image and the second image are determined, wherein the densification radius is used to increase the feature point density corresponding to the first feature block.

7. The method of claim 6, wherein, Determining the matching point pair between the first image and the second image includes: Based on the preset densification information, and based on the two-dimensional coordinates of each first feature point in the first feature block and the two-dimensional coordinates of each second feature point in the second feature block, multiple densified coordinates corresponding to each first feature point in the first feature block and multiple densified coordinates corresponding to each second feature point in the second feature block are obtained. Based on the target homography matrix and multiple densed coordinates corresponding to each first feature point, the matching densed coordinates that match each densed coordinate corresponding to each first feature point are determined from the multiple densed coordinates corresponding to each second feature point, so as to determine the densed matching result. Based on the dense matching results, matching point pairs between the first image and the second image are determined.

8. The method of claim 7, wherein, The matching feature point pair includes a first matching feature point pair and a second matching feature point pair. The first matching feature point pair corresponds to a first feature block pair, and the second matching feature point pair corresponds to a second feature block pair. The multiple densed coordinates corresponding to the first feature point a in the first feature block in the first feature block pair include the first densed coordinates. The multiple densed coordinates corresponding to the first feature point b in the first feature block in the second feature block pair include the first densed coordinates. The determination of the matching denser coordinates that correspond to each of the first feature points includes: Based on the target homography matrix corresponding to the first feature block pair and the first densed coordinates, a first candidate coordinate corresponding to the first densed coordinates is determined from multiple densed coordinates corresponding to each second feature point of the second feature block in the first feature block pair. Based on the target homography matrix corresponding to the second feature block pair and the first densed coordinates, a second alternative coordinate corresponding to the first densed coordinates is determined from multiple densed coordinates corresponding to each second feature point of the second feature block in the second feature block pair. Based on the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the first feature block pair, and the distance between the first densed coordinate and the densed center coordinate corresponding to the first feature block in the second feature block pair, a matching densed coordinate that matches the first densed coordinate is determined from the first candidate coordinate and the second candidate coordinate. The densed center coordinate corresponding to the first feature block in the first feature block pair is determined based on multiple densed coordinates corresponding to each first feature point in the first feature block of the first feature block pair, and the densed center coordinate corresponding to the first feature block in the second feature block pair is determined based on multiple densed coordinates corresponding to each first feature point in the first feature block of the second feature block pair.

9. The method of any one of claims 1-8, wherein, Determining the target homography matrix corresponding to the feature block includes: Based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block of the feature block pair, the current homography matrix corresponding to the feature block pair is updated through a pre-trained homography optimization network to obtain the target homography matrix corresponding to the feature block pair.

10. The method of claim 9, wherein, The homography optimization network is trained based on several sample image groups and the corresponding label data of each sample image group. A single sample image group includes two sample images from different angles of the same sample scene. The label data is used to indicate the label matching point pairs between the two sample images in the corresponding sample image group.

11. The method of claim 9, wherein, Determining the target homography matrix corresponding to the feature block includes: A second similarity matrix is ​​determined, wherein the contents of each element in the second similarity matrix indicate the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block; Based on the current homography matrix, determine the second feature points corresponding to each first feature point in the first feature block from the second feature block; Input data is constructed based on each first feature point and its corresponding second feature point, as well as the second similarity matrix; The homography optimization network updates the current homography matrix based on the input data to obtain the target homography matrix.

12. The method of claim 11, wherein, The input data for construction includes: Based on each first feature point and its corresponding second feature point, sample each matrix sub-block in the second similarity matrix corresponding to each first feature point to determine the first sampling matrix; The input data is constructed based on the first sampling matrix.

13. The method of claim 12, wherein, The construction of the input data includes: The input data is constructed based on each first feature point, its corresponding second feature point, and the first sampling matrix.

14. The method of claim 11, wherein constructing the input data comprises: For each matrix sub-block in the second similarity matrix that corresponds to each first feature point, pooling is performed on the matrix sub-block to obtain the third similarity matrix; Input data is constructed based on each first feature point and its corresponding second feature point, the second similarity matrix, and the third similarity matrix.

15. The method of claim 12, wherein, Determining the first sampling matrix includes: For each matrix sub-block in the second similarity matrix that corresponds to each first feature point in the first feature block, the matrix sub-block is sampled with a preset size as the sampling size and the position corresponding to the second feature point of each first feature point as the center point to obtain the first sampling matrix. The content of each element in the matrix sub-block indicates the similarity between the corresponding first feature point and each second feature point in the second feature block.

16. The method of claim 11, further comprising, after updating the current homography matrix based on the input data through the homography optimization network: Determine whether the number of times the current homography matrix has been updated has reached the preset number; If the determination is negative, the updated current homography matrix is ​​used as the new current homography matrix, and the step of determining the second feature points corresponding to the first feature points in the first feature block from the second feature block based on the current homography matrix continues.

17. A matching point pair generation apparatus, comprising: The first obtaining module is configured to extract a first feature map and a second feature map of a first size, and a third feature map and a fourth feature map of a second size from the first image and the second image, respectively, wherein the first image and the second image are images of the same scene from different angles, and the second size is larger than the first size; The first determining module is configured to use the matching feature point pairs corresponding to the first feature map and the second feature map to determine the feature block pairs corresponding to the matching feature point pairs from the third feature map and the fourth feature map, wherein the feature block pairs include a first feature block determined from the third feature map and a second feature block determined from the fourth feature map; The update module is configured to update the current homography matrix corresponding to the feature block pair based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block, thereby obtaining the target homography matrix corresponding to the feature block pair. The second determining module is configured to determine the matching point pair between the first image and the second image based on the target homography matrix.

18. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-16.

Images