Matching point pair generation method and device

Abstract

The embodiment of the invention provides a matching point pair generation method and device. The method comprises the following steps: obtaining a first feature map and a second feature map of a first size and a third feature map and a fourth feature map of a second size for a first image and a second image of different angles of the same scene; determining a feature block pair corresponding to a matching feature point pair from a third feature map and a fourth feature map which are large in size by using the matching feature point pair corresponding to the first feature map and the second feature map which are small in size; based on the similarity between at least part of feature points in a first feature block and at least part of feature points in a second feature block in the feature block pair, updating a current homography matrix corresponding to the feature block pair to obtain a target homography matrix corresponding to the feature block pair; and determining a matching point pair between the first image and the second image based on the target homography matrix, thereby realizing determination of the matching point pair between the images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a matching point pair generation method and device. Background Art

[0002] In scenes such as image stitching and 3D reconstruction, it is first necessary to determine the matching point pairs between at least two frames of images. The matching point pairs are pixel points corresponding to the same spatial point in at least two frames of images. Then, image stitching or 3D reconstruction is performed based on the determined matching point pairs. In the above process, the effect of the image stitching result or the effect of the 3D reconstruction depends to a certain extent on the determination result of the matching point pairs between at least two frames of images.

[0003] Then, how to provide a matching point pair generation method becomes an urgent problem to be solved. Summary of the invention

[0004] One or more embodiments of the present specification provide a matching point pair generation method and device to determine matching point pairs between images.

[0005] According to a first aspect, a matching point pair generation method is provided, comprising:

[0006] Extracting, from the first image and the second image, respectively, a first feature map and a second feature map of a first size, and a third feature map and a fourth feature map of a second size, respectively, wherein the first image and the second image are images of the same scene at different angles, and the second size is larger than the first size;

[0007] Using the matching feature point pairs corresponding to the first feature map and the second feature map, determining a feature block pair corresponding to the matching feature point pair from the third feature map and the fourth feature map, wherein the feature block pair includes a first feature block determined from the third feature map and a second feature block determined from the fourth feature map;

[0008] Based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block in the feature block pair, updating the current homography matrix corresponding to the feature block pair, and determining the target homography matrix corresponding to the feature block pair;

[0009] Based on the target homography matrix, matching point pairs between the first image and the second image are determined.

[0010] According to a second aspect, a matching point pair generating device is provided, comprising:

[0011] A first obtaining module is configured to extract, from the first image and the second image, respective first feature maps and second feature maps of first sizes, and respective third feature maps and fourth feature maps of second sizes, respectively, wherein the first image and the second image are images of the same scene at different angles, and the second size is larger than the first size;

[0012] A first determination module is configured to use the matching feature point pairs corresponding to the first feature map and the second feature map to determine, from the third feature map and the fourth feature map, a feature block pair corresponding to the matching feature point pair, wherein the feature block pair includes a first feature block determined from the third feature map and a second feature block determined from the fourth feature map;

[0013] An updating module, configured to update a current homography matrix corresponding to the feature block pair based on similarities between at least some feature points in a first feature block and at least some feature points in a second feature block in the feature block pair, and determine a target homography matrix corresponding to the feature block pair;

[0014] The second determination module is configured to determine matching point pairs between the first image and the second image based on the target homography matrix.

[0015] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0016] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0017] According to the matching point pair generation method and device provided by the embodiment of this specification, the coarse-grained feature map (the first feature map and the second feature map) and the fine-grained feature map (the third feature map and the fourth feature map) of the first image and the second image to be matched are respectively extracted, and then the matching feature point pairs corresponding to the two coarse-grained feature maps are used to determine the feature block pairs corresponding to the matching feature point pairs from the two fine-grained feature maps; then based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block in the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair. In the above process, the current homography matrix corresponding to the feature block pair can be updated based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block in the feature block pair to obtain the target homography matrix corresponding to the feature block pair, and the homography relationship between all positions of the two feature blocks in the feature block pair can be reflected through such a target homography matrix, and further, based on the aforementioned target homography matrix, more matching point pairs between the first image and the second image can be better determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in this specification;

[0020] Figure 2 A schematic diagram of a flow chart of a matching point pair generation method provided in an embodiment;

[0021] Figure 3A A schematic diagram of the fifth characteristic diagram (or the sixth characteristic diagram) provided in the embodiment;

[0022] Figure 3B A schematic diagram of a structure of a first similarity matrix provided in an embodiment;

[0023] Figure 4 A schematic block diagram of a matching point pair generating device provided in an embodiment. DETAILED DESCRIPTION

[0024] The technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0025] The embodiments of this specification disclose a matching point pair generation method and device. The application scenarios and technical concepts of the method are first introduced as follows:

[0026] As mentioned above, in scenes such as image stitching and 3D reconstruction, the effect of the image stitching result or the 3D reconstruction result depends to a certain extent on the determination result of the matching point pairs between at least two frames of images. Therefore, how to provide a matching point pair generation method becomes an urgent problem to be solved.

[0027] At present, a semi-dense matching method is proposed to determine the matching point pairs between two frames of images. The process is generally as follows: obtain two frames of images A and B to be matched, perform feature extraction on images A and B, and obtain a coarse-grained feature map A1 and a fine-grained feature map A2 corresponding to image A, and a coarse-grained feature map B1 and a fine-grained feature map B2 corresponding to image B, wherein the feature map A1 has the same size as the feature map B1, the feature map A2 has the same size as the feature map B2, and the size of the fine-grained feature map is larger than the size of the coarse-grained feature map.

[0028] Next, the similarity value between each feature point in feature graph A1 and each feature point in feature graph B1 is calculated to obtain a similarity matrix 1 between the two. Then, based on the similarity matrix 1, a matching result 1 between feature graph A1 and feature graph B1 is determined. The matching result 1 indicates a matching point pair between feature graph A1 and feature graph B1 (a matching point pair between coarse-grained feature graphs), that is, it indicates which position of the feature point in feature graph A1 and which position of the feature point in feature graph B1 are a matching point pair. For example, the feature point (a1, b1) in feature graph A1 and the feature point (a2, b2) in feature graph B1 are a pair of matching point pairs, wherein the similarity between the feature points that are matching point pairs is higher.

[0029] After obtaining the matching result 1, the feature block pairs corresponding to each matching point pair are extracted from the feature map A2 and the feature map B2 based on the matching result 1. Taking a pair of matching point pairs (a1, b1) and (a2, b2) in the matching result 1 as an example, based on the feature point (a1, b1), the corresponding feature block t1 is determined from the feature map A2, and based on the feature point (a2, b2), the corresponding feature block t2 is determined from the feature map B2. The feature block t1 and the feature block t2 constitute the feature block pair corresponding to the matching point pair (a1, b1) and (a2, b2).

[0030] Next, for each feature block pair, a pair of fine-grained matching point pairs is determined from them. The feature block pairs corresponding to the matching point pairs (a1, b1) and (a2, b2) are used as an example for explanation. The central feature point t11 ​​of feature block t1 is determined; the similarity values ​​between the central feature point t11 ​​and each feature point in feature block t2 are calculated; based on the similarity values ​​between the central feature point t11 ​​and each feature point in feature block t2, the corresponding probability expectation distribution is constructed in a manner of calculating probability expectation, and the feature point t21 in feature block t2 corresponding to the position with the maximum expectation in the probability expectation distribution is used as the matching point corresponding to the central feature point t11, so as to obtain a pair of fine-grained matching point pairs (central feature point t11, feature point t21 in feature block t2).

[0031] Then, based on the fine-grained matching point pairs corresponding to each feature block pair, the matching point pairs between image A and image B are determined. The determination process can refer to the aforementioned process of determining a pair of fine-grained matching point pairs from each feature block pair, and will not be repeated here.

[0032] In the above process, the number of matching point pairs between image A and image B determined by the above method is small and not dense enough (for example, when the coarse-grained feature map is 1 / 8 of the size of the original image, in each 8*8 area in image A, at most one pair of matching point pairs is determined from image B), which affects the effects of subsequent tasks. For example, due to the small number of matching point pairs, the image stitching result is not good enough, and due to the small number of matching point pairs, the three-dimensional reconstruction effect is not good enough.

[0033] In addition, in the process of determining the matching point pairs in the fine-grained feature block pairs, the fine-grained matching point pairs are determined in a manner of calculating the probability expectation, that is, they are determined based on all areas between the two feature blocks in the feature block pair (including irrelevant areas, i.e., areas where the similarity values ​​between the two feature blocks are low, and relevant areas, i.e., areas where the similarity values ​​between the two feature blocks are high). This will be affected by the spatial variance brought by the irrelevant areas in the feature block pair, which will result in a reduction in the accuracy of the determined fine-grained matching point pairs.

[0034] Moreover, referring to the process of determining the matching point pairs in the fine-grained feature block pairs, it can be known that it can only determine the corresponding matching point for the feature point (central feature point) at the center position of the source feature block (for example, feature block t1, the feature block determined from the feature map A2), and the matching point corresponding to the central feature point is not guaranteed to be the center point of the target feature block (the feature block determined from the feature map B2); when image B is used as the source image and matched with other images (for example, image C) to determine the matching point pairs therebetween, it is necessary to use the feature points at the center position of each feature block determined in each fine-grained feature map extracted from image B, and determine the corresponding matching points from the corresponding feature blocks corresponding to image C, which can easily lead to discontinuity in the determined matching points.

[0035] In order to solve the above problems, the inventor proposes a matching point pair generation method. Figure 1 A schematic diagram of an implementation scenario according to an embodiment disclosed in this specification is shown. In this implementation scenario, two images to be matched are obtained, namely, a first image and a second image, which are images of the same scene at different angles; then, feature extraction is performed on the first image and the second image, respectively, and their respective coarse-grained feature maps (a first feature map of a first size corresponding to the first image and a second feature map of a first size corresponding to the second image) and their respective fine-grained feature maps (i.e., a third feature map of a second size corresponding to the first image and a fourth feature map of a second size corresponding to the second image) are extracted from the first image and the second image, respectively, wherein the second size is larger than the first size.

[0036] Next, using the matching feature point pairs corresponding to the first feature map and the second feature map, a feature block pair corresponding to the matching feature point pairs is determined from the third feature map and the fourth feature map, wherein the feature block pair includes a first feature block determined from the third feature map (determined based on the feature points in the first feature map in the feature block pair and the relationship between the first feature map and the third feature map) and a second feature block determined from the fourth feature map (determined based on the feature points in the second feature map in the feature block pair and the relationship between the second feature map and the fourth feature map).

[0037] After determining the feature block pairs, a homography matrix between the feature block pairs is constructed based on the feature block pairs, so as to obtain the homography relationship of all positions that can be found between the feature block pairs. Specifically, the similarity between at least part of the feature points in the first feature block and at least part of the feature points in the second feature block in the feature block pair is determined, and the similarity can characterize which feature points in the second feature block are more matched with which feature points in the first feature block (or are more correlated); then, based on the similarity between at least part of the feature points in the first feature block and at least part of the feature points in the second feature block in the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair; then, based on the target homography matrix, the matching point pairs between the first image and the second image are determined.

[0038] In the above process, based on the similarity between at least part of the feature points in the first feature block of the feature block pair and at least part of the feature points in the second feature block of the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair. Such a target homography matrix can reflect the homography relationship (i.e., perspective mapping relationship) between all positions of the two feature blocks in the feature block pair. Furthermore, based on the target homography matrix, more matching point pairs between the first image and the second image can be better determined.

[0039] The matching point pair generation method provided in this specification is described in detail below in conjunction with specific embodiments.

[0040] Figure 2 The flowchart of the matching point pair generation method in one embodiment of the present specification is shown. The method is executed by an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. In the process of determining the energy consumption of the network model, such as Figure 2 As shown, the method includes the following steps S210-S240:

[0041] In step S210, a first feature map and a second feature map of a first size, and a third feature map and a fourth feature map of a second size are extracted from the first image and the second image, respectively, wherein the first image and the second image are images of the same scene at different angles, and the second size is larger than the first size.

[0042] In some possible examples, an electronic device obtains a first image and a second image to be matched, and the first image and the second image may be a sequence of images captured for the same scene. The first image and the second image have different capture angles for the scene, and there is an overlapping area between the capture viewing angles of the first image and the second image, that is, at least pixel point 1 of the first image and pixel point 2 of the second image correspond to the same spatial point (or physical point) in the scene.

[0043] After obtaining the first image and the second image to be matched, any feature extraction algorithm in the related art can be used to extract features from the first image and the second image, respectively, so as to extract the first feature map of the first size and the third feature map of the second size corresponding to the first image, and extract the second feature map of the first size and the fourth feature map of the second size corresponding to the second image. Wherein, the second size is larger than the first size, and the first feature map and the second feature map can be called coarse-grained feature maps, and the third feature map and the fourth feature map can be called fine-grained feature maps. Wherein, the sizes of the first image and the second image can be the same.

[0044] Among them, the first feature map and the second feature map corresponding to the first image are taken as examples for explanation. Each feature point in the first feature map and the second feature map can represent a more abstract local feature of the local area corresponding to each feature point in the first image (for example, including structural features, i.e., shape contours in the local area of ​​the first image, as well as color and material features, etc.). The feature map can better help determine the matching point pairs between the first image and the second image.

[0045] In some possible examples, features of the first image and the second image may be extracted by a neural network for extracting image features (hereinafter referred to as an extraction network). Specifically, the electronic device may input the first image and the second image into the extraction network respectively, and obtain a first feature map and a third feature map corresponding to the first image, and obtain a second feature map and a fourth feature map corresponding to the second image through the extraction network.

[0046] In some possible examples, the extraction network may be any type of neural network that can extract image features, such as FPN (Feature Pyramid Networks) or convolutional neural networks of other structures. In some examples, the first size may be 1 / 8 of the size of the original image (i.e., the first image or the second image), and the second size may be 1 / 2 of the size of the original image. For example, if the sizes of the first image and the second image are both (N, M), then the first size may be (N / 8, M / 8), and the second size may be (N / 2, M / 2).

[0047] When the extraction network is an FPN network, taking the third feature map of the second size as an example, the third feature map can aggregate the features of the multi-scale feature map extracted by the FPN network of the first image, and can better characterize the features of the first image. Similarly, the fourth feature map can better characterize the features of the second image, which helps to improve the accuracy of determining the matching point pairs between the first image and the second image.

[0048] In some other possible examples, the first image and the second image may be feature extracted by a preset downsampling method, so as to extract the first feature map and the second feature map of the first size, and the third feature map and the fourth feature map of the second size, respectively, from the first image and the second image. Exemplarily, the first image (or the second image) is mean sampled according to the first downsampling size to obtain the third feature map corresponding to the first image, and then the third feature map is further mean sampled according to the second downsampling size to obtain the first feature map corresponding to the first image.

[0049] After obtaining the coarse-grained first feature map and the second feature map, in step S220, the matching feature point pairs corresponding to the first feature map and the second feature map are used to determine the feature block pairs corresponding to the matching feature point pairs from the third feature map and the fourth feature map, wherein the feature block pairs include the first feature block determined from the third feature map and the second feature block determined from the fourth feature map.

[0050] In this step, after the electronic device obtains the coarse-grained first feature map and the second feature map, it first determines the coarse-grained matching feature point pairs based on the first feature map and the second feature map, that is, determines the matching feature point pairs corresponding to the first feature map and the second feature map, wherein each pair of matching feature point pairs includes a feature point in the first feature map and a feature point in the second feature map. Then, using the coarse-grained matching feature point pairs, the feature block pairs corresponding to the matching feature point pairs are determined from the third feature map and the fourth feature map. The feature block pair includes a first feature block determined from the third feature map (determined based on the feature points in the first feature map in the feature block pair, and the relationship between the third feature map and the first feature map) and a second feature block determined from the fourth feature map (determined based on the feature points in the second feature map in the feature block pair, and the relationship between the fourth feature map and the second feature map).

[0051] It is understandable that the matching feature point pairs corresponding to the first feature map and the second feature map may be one pair or multiple pairs. When there are multiple pairs of matching feature point pairs, for each pair of matching feature point pairs, the feature block pairs corresponding to each matching feature point pair may be determined from the third feature map and the fourth feature map.

[0052] In some possible examples, step 220 may include steps 11-12:

[0053] In step 11, a first matching result is determined using the first feature map and the second feature map, wherein the first matching result indicates a pair of matching feature points corresponding to the first feature map and the second feature map.

[0054] In some implementations, step 11 may include steps 111-113:

[0055] In step 111, based on a preset aggregation operation, the first feature map and the second feature map are processed to obtain a fifth feature map corresponding to the first image and a sixth feature map corresponding to the second image. In this step, the electronic device processes the first feature map and the second feature map based on a preset aggregation operation to obtain a fifth feature map that can better characterize the features of the first image and a sixth feature map that can better characterize the features of the second image.

[0056] In some possible examples, the electronic device may perform processing corresponding to the preset aggregation operation on the first feature map and the second feature map through a pre-trained aggregation network based on the attention mechanism to obtain the fifth feature map and the sixth feature map. Specifically, the electronic device inputs the first feature map and the second feature map into the aggregation network to aggregate the first feature map and the second feature map through the aggregation network to obtain the fifth feature map corresponding to the first image and the sixth feature map corresponding to the second image. In some examples, in order to better improve the accuracy of the determined matching results, the aggregation network can be a cross-attention mechanism network based on Transformer.

[0057] Exemplarily, step 111 may include steps 1111-1112:

[0058] In step 1111, the first feature map and the second feature map are respectively input into the first sub-network based on the self-attention mechanism of the aggregation network, and the first aggregated feature map corresponding to the first feature map and the second aggregated feature map corresponding to the second feature map are respectively obtained. Taking the first feature map as an example, the electronic device inputs the first feature map into the aforementioned first sub-network to obtain the first query Q matrix, the first key K matrix and the first value V matrix corresponding to the first feature map, and then uses the preset attention weight formula to obtain the first attention weight based on the first Q matrix, the first K matrix and the number of dimensions of the first K matrix; then uses the first attention weight and the first V matrix to obtain the first aggregated feature map, and realizes the self-attention aggregation of the first feature map. Similarly, the second aggregated feature map corresponding to the second feature map is obtained.

[0059] Next, in step 1112, the first aggregated feature map and the second aggregated feature map are input into the second sub-network based on the cross-attention mechanism of the aggregation network to obtain the fifth feature map and the sixth feature map. In this step, the first aggregated feature map and the second aggregated feature map are input into the aforementioned second sub-network, so that the first aggregated feature map and the second aggregated feature map are processed respectively by the second sub-network to obtain the second Q matrix, the second K matrix and the second V matrix corresponding to the first aggregated feature map, and the third Q matrix, the third K matrix and the third V matrix corresponding to the second aggregated feature map; then, through the second sub-network, using the preset attention weight formula, based on the second Q matrix, the third K matrix and the number of dimensions of the third K matrix, the second attention weight is obtained; then, using the second attention weight and the third V matrix, the cross-attention output of the first aggregated feature map with respect to the second aggregated feature map is obtained, that is, the fifth feature map is obtained, and the fifth feature map has a stronger representation ability and helps to obtain a more accurate matching result.

[0060] In addition, through the second sub-network, using the preset attention weight formula, based on the third Q matrix, the second K matrix and the number of dimensions of the second K matrix, a third attention weight is obtained; then, using the third attention weight and the second V matrix, the cross-attention output of the second aggregated feature map with respect to the third aggregated feature map is obtained to obtain a sixth feature map, which has a stronger representation ability and helps to obtain more accurate matching results.

[0061] Also exemplarily, the aforementioned aggregation network may also only include the aforementioned second sub-network based on the cross-attention mechanism. Accordingly, the electronic device can directly input the first feature map and the second feature map into the aforementioned second sub-network, so as to perform cross-attention aggregation on the first feature map and the second feature map through the second sub-network to obtain the fifth feature map and the sixth feature map.

[0062] In some possible examples, the sizes of the fifth characteristic map and the sixth characteristic map may be the first size or may not be the first size.

[0063] Then, in step 112, a first similarity matrix is ​​determined using the fifth feature map and the sixth feature map, wherein the content of each element in the first similarity matrix indicates the similarity between each feature point in the first feature map and each feature point in the second feature map.

[0064] Exemplarily, the electronic device may calculate the similarity value between each feature point in the fifth feature map and each feature point in the sixth feature map to obtain a first similarity matrix, wherein the content of each element in the first similarity matrix indicates the similarity between each feature point in the first feature map and each feature point in the second feature map. In some possible examples, the greater the similarity value between each feature point in the fifth feature map and each feature point in the sixth feature map, the greater the possibility that they are matching feature point pairs.

[0065] In some possible implementations, the similarity value between each feature point in the fifth feature map and each feature point in the sixth feature map may be determined based on the inner product between them. In some other possible implementations, the similarity value between each feature point in the fifth feature map and each feature point in the sixth feature map may be determined based on the distance (e.g., Euclidean distance or cosine distance) between them, which is also possible.

[0066] The following is an example of determining the similarity value between the feature points in the fifth feature map and the feature points in the sixth feature map based on the inner product between them to illustrate the process of determining the first similarity matrix. For example, the fifth feature map and the sixth feature map are both feature maps with L channels and a size of H (height) * W (width). Figure 3A As shown, H*W can represent a two-dimensional coordinate space. Each point in the two-dimensional coordinate space represented by H*W can be represented by a vector with L elements (such as Figure 3A As shown, the values ​​of each black position in the L channels constitute the vector of the corresponding point), and the value of each element is the value at the corresponding position (two-dimensional coordinate) of the feature point in the feature map. Accordingly, each feature point in the fifth feature map and the sixth feature map can be represented by a vector with L elements. The electronic device can calculate the inner product with the vector of each feature point in the sixth feature map for the vector of each feature point in the fifth feature map, respectively, to obtain the similarity value between each feature point in the fifth feature map and each feature point in the sixth feature map, thereby obtaining a first similarity matrix.

[0067] Among them, for any feature point in the fifth feature map, the similarity value between it and each feature point in the sixth feature map includes H*W, which can form a sub-matrix with a size of H*W; the sub-matrix with a size of H*W corresponding to each feature point in the fifth feature map can be arranged according to the position relationship of each feature point in the fifth feature map to obtain a first similarity matrix. Figure 3B The figure shows a structural diagram of the first similarity matrix.

[0068] After the first similarity matrix is ​​obtained, in step 113, a first matching result is determined based on the first similarity matrix.

[0069] In some implementations, the electronic device can determine, based on the first similarity matrix, for each feature point in the fifth feature map (taking feature point i as an example), from the corresponding H*W similarity values, the similarity value with the largest numerical value, and determine the feature point j in the sixth feature map corresponding to the similarity value with the largest numerical value; then determine whether the similarity value with the largest numerical value is greater than the first similarity threshold; if the similarity value with the largest numerical value is greater than the first similarity threshold, then determine that feature point i and feature point j are a pair of matching feature point pairs; if the similarity value with the largest numerical value is not greater than the first similarity threshold, then determine that feature point i and feature point j are not a pair of matching feature point pairs; and so on, determine all matching feature point pairs to obtain a first matching result.

[0070] In some further implementations, step 113 may include steps 1131-1132:

[0071] In step 1131, the first similarity matrix is ​​processed to obtain a probability matrix. In this step, the electronic device may process the first similarity matrix using a preset activation function to normalize the first similarity matrix to obtain a probability matrix. In some examples, the preset activation function may be a Softmax function. Exemplarily, the electronic device may process the first similarity matrix using a preset activation function in two dimensions, namely, rows and columns, to obtain a probability matrix.

[0072] Next, in step 1132, based on the mutual nearest neighbor algorithm and the probability matrix, the first matching result is determined. In this step, the electronic device can use the mutual nearest neighbor algorithm to determine the nearest neighbor feature point pairs between the fifth feature map and the sixth feature map based on the feature points in the fifth feature map and the feature points in the sixth feature map, each pair of nearest neighbor feature point pairs includes a feature point in the fifth feature map and a feature point in the sixth feature map; then, based on the probability matrix and the second similarity threshold, determine whether each pair of nearest neighbor feature point pairs is a matching feature point pair to obtain the first matching result. For example, if the probability value corresponding to the nearest neighbor feature point pair in the probability matrix is ​​not less than the second similarity threshold, it can be determined that it is a matching feature point pair, otherwise, it is not a matching feature point pair. It can be understood that the first matching result includes one or more pairs of matching feature point pairs (i.e., one or more pairs of matching feature point pairs corresponding to the first feature map and the second feature map).

[0073] In some possible examples, the electronic device may also directly determine a similarity matrix based on the similarity between each feature point in the first feature map and each feature point in the second feature map, and then determine the first matching result based on the similarity matrix. The process of determining the similarity matrix can refer to the aforementioned process of determining the first similarity matrix, and the process of determining the first matching result based on the similarity matrix can refer to the aforementioned process of determining the first matching result based on the first similarity matrix, which will not be described in detail here.

[0074] After the electronic device obtains the first matching result, i.e., the matching feature point pairs corresponding to the first feature map and the second feature map, in step 12, the feature block pairs corresponding to the matching feature point pairs are determined from the third feature map and the fourth feature map using the matching feature point pairs and the size relationship between the first size and the second size.

[0075] In some implementations, the electronic device may determine, from the third feature map and the fourth feature map, the feature block pairs corresponding to each pair of matching feature point pairs and the size relationship between the first size and the second size. It can be understood that for any of the aforementioned matching feature point pairs, the process of determining the corresponding feature block pairs is similar. The following takes any of the matching feature point pairs (e.g., matching feature point pair P) as an example to illustrate the process of determining the corresponding feature block pairs.

[0076] Assume that the matching feature point pair P includes a feature point i in the aforementioned fifth feature map (coordinates are represented by (ai, bi)) and a feature point j in the aforementioned sixth feature map, whose coordinates are represented by (aj, bj). In some examples, when the sizes of the fifth feature map and the sixth feature map are the first size, the feature point i (ai, bi) can be mapped to the third feature map based on the size relationship between the first size and the second size, and the corresponding feature point i in the third feature map is obtained after mapping. ， , its coordinates in the third feature map are (ai', bi'), where ai' is equal to Q times ai, bi' is equal to Q times bi, and Q represents the ratio of the second size to the first size. Similarly, based on the size relationship between the first size and the second size, the feature point j (aj, bj) is mapped to the fourth feature map, and the corresponding feature point j' in the fourth feature map is obtained after mapping, and its coordinates in the third feature map are (aj', bj'), where aj' is equal to Q times aj, and bj' is equal to Q times bj.

[0077] After that, in the third feature map, with feature point i'(ai',bi') as the center point, determine the feature block with a specified size of (w*w) corresponding to feature point i; similarly, in the fourth feature map, with feature point j'(aj',bj') as the center point, determine the feature block with a specified size of (w*w) corresponding to feature point j, and obtain the feature block pair corresponding to the matching feature point pair P. And so on, obtain the feature block pair corresponding to each matching feature point pair.

[0078] In some examples, the value of w may be set according to the size relationship between the first size and the second size, for example, the value may be set to be greater than or equal to the ratio Q of the second size to the first size.

[0079] In some other examples, when the size of the fifth feature map and the sixth feature map is not the first size, based on the size relationship between the size of the fifth feature map (or the sixth feature map) and the first size, and the size relationship between the first size and the second size, the feature block pairs corresponding to each matching feature point pair can be determined from the third feature map and the fourth feature map using each matching feature point pair. For details, please refer to the aforementioned process of determining the feature block pairs, which will not be repeated here.

[0080] After obtaining the feature block pair corresponding to the matching feature point pair, in step S230, based on the similarity between at least part of the feature points in the first feature block and at least part of the feature points in the second feature block in the feature block pair, the current homography matrix corresponding to the feature block pair is updated to obtain the target homography matrix corresponding to the feature block pair.

[0081] It can be understood that the homography matrix is ​​a 3*3 matrix, which can describe the perspective (mapping) transformation relationship between two planes. The homography matrix can map points on one plane to another plane while maintaining the linear relationship of the points, wherein the linear relationship of the points means that if the points on the plane are collinear, these points are still collinear on the mapped plane. In view of the above principle, in this embodiment, for each feature block pair, the electronic device obtains the initial homography matrix corresponding to the feature block pair as the current homography matrix, and then combines the similarity between at least part of the feature points in the first feature block and at least part of the feature points in the second feature block in the feature block pair, with the goal of aligning the area with correlation between the first feature block and the second feature block (i.e., the area where the feature points with higher similarity values ​​between the first feature block and the second feature block are located), and updates and optimizes the current homography matrix corresponding to the feature block pair, so as to obtain a target homography matrix that can better align the area with correlation between the first feature block and the second feature block. Among them, the current homography matrix can be iteratively updated multiple times to obtain the target homography matrix.

[0082] In some possible examples, in step S230, it can be set as follows: based on the similarity between at least part of the feature points in the first feature block in the feature block pair and at least part of the feature points in the second feature block, a pre-trained homography optimization network is used to update the current homography matrix corresponding to the feature block pair to obtain a target homography matrix corresponding to the feature block pair.

[0083] The homography optimization network is trained based on several sample image groups and label data corresponding to each sample image group, wherein a single sample image group includes two frames of sample images at different angles for the same sample scene, and the label data is used to indicate: label matching point pairs between two frames of sample images in the corresponding sample image group.

[0084] In this example, for each feature block pair, the electronic device obtains an initial homography matrix corresponding to the feature block pair as the current homography matrix, and then combines the similarity between at least some feature points in the first feature block of the feature block pair and at least some feature points in the second feature block, and estimates the homography residual (that is, the subsequent optimization matrix) between the two feature blocks through the ability of the homography optimization network to align the correlated areas between the two feature blocks in the feature block pair (that is, the areas where the feature points with higher similarity values ​​are located), and for a feature point in the first feature block and a feature point in the second feature block, If two feature points are unrelated (i.e., do not match), the similarity value between them will be very low. Correspondingly, its impact on the output of the homography optimization network will be smaller, so that the homography optimization network can pay more attention to similarity values ​​with higher values ​​(i.e., pay attention to the correlated areas between the two feature blocks in the feature block pair), thereby obtaining a better homography residual (i.e., the subsequent optimization matrix). After that, based on the estimated homography residual between the two feature blocks, the current homography matrix is ​​updated and optimized to obtain the target homography matrix that can align the correlated areas between the first feature block and the second feature block in the feature block pair.

[0085] For ease of calculation, the coordinates of the feature points of the two feature blocks in the feature block pair may be set in the same two-dimensional coordinate space (eg, in a subsequently specified two-dimensional coordinate system). In some examples, the initial homography matrix obtained by initialization may be a unit matrix.

[0086] In some examples, step S230 may include steps 21-24:

[0087] In step 21, a second similarity matrix is ​​determined, wherein the content of each element in the second similarity matrix indicates the similarity between at least part of the feature points in the first feature block and at least part of the feature points in the second feature block.

[0088] In this step, the electronic device can determine the similarity value between at least some feature points in the first feature block and at least some feature points in the second feature block based on the distance (such as Euclidean distance or cosine distance), or the inner product between them to obtain a second similarity matrix.

[0089] The following describes the process of determining the second similarity matrix by taking the example of determining the similarity value between at least some feature points in the first feature block and at least some feature points in the second feature block based on the inner product between them.

[0090] Exemplarily, the first feature block and the second feature block are both feature maps with a size of w (height) * w (width) and D channels. w*w can represent a two-dimensional coordinate space. Each point in the two-dimensional coordinate space represented by w*w can be represented by a vector with D elements, and the value of each element is the value at the corresponding position (two-dimensional coordinate) of the feature point in the feature block. Correspondingly, each feature point in the first feature block and the second feature block can be represented by a vector with D elements. The electronic device can calculate the inner product of the vectors of at least some of the feature points in the first feature block and the vectors of at least some of the feature points in the second feature block, respectively, to obtain the similarity values ​​between at least some of the feature points in the first feature block and at least some of the feature points in the second feature block, thereby obtaining a second similarity matrix.

[0091] Among them, taking the first feature block as an example, at least part of the feature points therein may include: all the feature points of the first feature block, and may also include part of the feature points therein (for example, including the feature points at every other position); similarly, at least part of the feature points in the second feature block may include: all the feature points of the second feature block, and may also include part of the feature points therein (for example, including the feature points at every other position).

[0092] Among them, for any feature point in the first feature block, the similarity value between it and at least part of the feature points in the second feature block includes w*w, which can form a matrix sub-block of size w*w; the matrix sub-blocks of size w*w corresponding to at least part of the feature points in the first feature block can be arranged according to the positional relationship of at least part of the feature points in the first feature block in the first feature block, to obtain a second similarity matrix, which includes w*w matrix sub-blocks of size w*w. In some cases, the second similarity matrix can be considered as a w*w-dimensional matrix, in which each element includes a matrix sub-block corresponding to each feature point in the first feature block (i.e., the subsequent first feature point), and the size of each matrix sub-block is w*w, which represents the similarity value between the corresponding feature point in the first feature block and each feature point in the second feature block (i.e., the subsequent second feature point). For the convenience of calculation, the second similarity matrix can be represented as a 4-dimensional tensor, that is, a w*w*w*w tensor. Its schematic structure can refer to the schematic structure of the aforementioned first similarity matrix.

[0093] For feature points in the first feature block whose similarity values ​​are not calculated, the content of each element in the corresponding matrix sub-block of size w*w can be set to a specified value (e.g., 0). For a feature point in the first feature block, if the similarity value is not calculated with a feature point in the second feature block, the content of the element at the corresponding position in the matrix sub-block of size w*w can be set to a specified value (e.g., 0).

[0094] After the second similarity matrix is ​​obtained, in step 22, second feature points corresponding to first feature points in the first feature block are determined from the second feature block based on the current homography matrix.

[0095] In this step, the two-dimensional coordinates (u, v) of each first feature point in the first feature block in the specified two-dimensional coordinate system can be set. Exemplarily, u and v can both be integers between [0, w-1]. At this time, the two-dimensional coordinates of the first feature point in the upper left corner of the first feature block can be (0, 0); and the two-dimensional coordinates (u', v') of each second feature point in the second feature block in the specified two-dimensional coordinate system are set, wherein u', v' can both be integers between [0, w-1]. At this time, the two-dimensional coordinates of the second feature point in the upper left corner of the second feature block can be (0, 0).

[0096] Next, the electronic device constructs the corresponding homogeneous coordinates based on the two-dimensional coordinates (u, v) of each first feature point in the first feature block, and then determines the two-dimensional coordinates (u', v') of each second feature point corresponding to each first feature point in the second feature block based on the homogeneous coordinates corresponding to each first feature point in the first feature block and the aforementioned current homography matrix. Exemplarily, it can be expressed by the following formula (1):

[0097]

[0098] in, represents the current homography matrix, represents the homogeneous coordinates corresponding to the two-dimensional coordinates of each first feature point in the first feature block, The homogeneous coordinates corresponding to the two-dimensional coordinates of each second feature point corresponding to each first feature point.

[0099] Next, in step 23, input data is constructed based on each first feature point and its corresponding second feature point, and a second similarity matrix.

[0100] In some possible examples, step 23 may include steps 231-232:

[0101] In step 231, based on each first feature point and its corresponding second feature point, each matrix sub-block corresponding to each first feature point in the first feature block in the second similarity matrix is ​​sampled to determine a first sampling matrix.

[0102] In this step, the electronic device locates the matrix sub-block (i.e., the matrix sub-block with a size of w*w) corresponding to the first feature point (u1, v1) in the second similarity matrix based on the two-dimensional coordinates of each first feature point (for example, taking the first feature point (u1, v1) as an example), and finds the position corresponding to the second feature point (u1', v1') from the matrix sub-block corresponding to the first feature point (u1, v1) based on the two-dimensional coordinates (u1', v1') of the second feature point corresponding to the first feature point (u1, v1), and then samples based on the position to obtain the sampling result corresponding to the first feature point (u1, v1). By analogy, based on the second feature point corresponding to each first feature point, sample each matrix sub-block corresponding to each first feature point to obtain the sampling result corresponding to each first feature point; based on the sampling result corresponding to each first feature point, determine to obtain the first sampling matrix.

[0103] Specifically, in some possible implementations, in step 231, determining the first sampling matrix may include:

[0104] For the matrix sub-blocks corresponding to each first feature point in the second similarity matrix, the matrix sub-blocks are sampled with a preset size as the sampling size and with the position corresponding to the second feature point corresponding to each first feature point as the center point to obtain a first sampling matrix, wherein the content of each element of the matrix sub-block indicates the similarity between the corresponding first feature point and each second feature point in the second feature block.

[0105] Continuing to take the first feature point (u1, v1) as an example for explanation, based on the two-dimensional coordinates (u1', v1') of the second feature point corresponding to the first feature point (u1, v1), the position corresponding to the second feature point (u1', v1') is found from the matrix sub-block corresponding to the first feature point (u1, v1) in the second similarity matrix, and the matrix sub-block corresponding to the first feature point (u1, v1) is sampled with a preset size (for example, (2r+1)*(2r+1)) as the sampling size and the position corresponding to the second feature point (u1', v1') as the center point, to obtain the sampling result corresponding to the first feature point (u1, v1); the sampling result includes the area (matrix) with the position corresponding to the second feature point (u1', v1') as the center point and the size of (2r+1)*(2r+1) in the matrix sub-block corresponding to the first feature point (u1, v1). By analogy, the sampling results corresponding to each first feature point are obtained, and the first sampling matrix is ​​obtained based on the sampling results corresponding to each first feature point. Among them, r is a preset exploration radius, which can be set according to actual conditions and needs.

[0106] The first sampling matrix can be considered as a w*w dimensional matrix, in which each element includes the sampling results corresponding to each first feature point in the first feature block, and the size of each sampling result is (2r+1)*(2r+1), which represents the similarity value between the corresponding first feature point (i.e., the subsequent first feature point) and each second feature point within the range of (2r+1)*(2r+1) centered on the second feature point (u1', v1'). In some cases, the first sampling matrix can be represented as a 4-dimensional tensor, that is, a w*w*(2r+1)*(2r+1) tensor.

[0107] After the first sampling matrix is ​​obtained, in step 232, input data is constructed based on the first sampling matrix.

[0108] In some possible examples, the electronic device may flatten each sampling result corresponding to each first feature point in the first sampling matrix (i.e., flatten the (2r+1)*(2r+1) elements in each sampling result from matrix form to vector form) to obtain a vector including (2r+1)*(2r+1) elements corresponding to each first feature point. Accordingly, according to the positional relationship of each first feature point in the first feature map, based on the vector including (2r+1)*(2r+1) elements corresponding to each first feature point, first feature data with a channel number of (2r+1)*(2r+1) and each channel including data of size w*w is obtained; and then the first feature data is used as input data. The input data includes similarity values ​​between each first feature point and the second feature point corresponding to each first feature point and the second feature points around it; the similarity values ​​are used as input to the homography optimization network, so that the homography optimization network can explore the second feature points that are more matched with each first feature point from the second feature points corresponding to each first feature point and the second feature points around it based on the similarity values ​​between each first feature point and the second feature points corresponding to each first feature point and the second feature points around it, and then obtain corresponding optimization offset information (i.e., the subsequent optimization matrix), which helps to better align the correlated areas between the first feature blocks and the second feature blocks.

[0109] In some other possible implementations, step 23 may include steps 233-234:

[0110] In step 233, based on each first feature point and its corresponding second feature point, each element content in the second similarity matrix is ​​sampled to determine a first sampling matrix. The implementation principle of step 233 is similar to that of the aforementioned step 231, and its implementation process can refer to the implementation process of the aforementioned step 231, which will not be described in detail here.

[0111] Thereafter, in step 234, input data is constructed based on each first feature point and its corresponding second feature point and the first sampling matrix.

[0112] In this step, the electronic device can flatten the sampling results corresponding to each first feature point in the first sampling matrix to obtain first feature data with a channel number of (2r+1)*(2r+1), and each channel includes data of a size of w*w; and for each first feature point, the two-dimensional coordinates of each first feature point are flattened to obtain second feature data with 2 channels and a size of w*w for each channel; similarly, for the second feature points corresponding to each first feature point, the two-dimensional coordinates of the second feature points corresponding to each first feature point are flattened to obtain third feature data with 2 channels and a size of w*w for each channel; thereafter, the first feature data, the second feature data and the third feature data are spliced ​​on the specified dimension (channel dimension) to obtain input data.

[0113] This type of input data also includes the position coordinate features of each first feature point and its corresponding second feature point, which can better help obtain accurate matching results.

[0114] In some possible implementations, step 23 may include step 235: constructing input data directly based on the second similarity matrix and each first feature point and its corresponding second feature point. The implementation principle of step 235 is similar to that of the aforementioned step 234, and its implementation process can refer to the implementation process of the aforementioned step 234, which will not be described in detail here.

[0115] In some possible scenarios, the imaging content sizes between the first image and the second image may be different (for example, the imaging content area of ​​the object A in the scene in the first image is larger than the imaging content area of ​​the object A in the scene in the second image). In order to ensure accurate determination of the matching point pairs in the first image and the second image, in some further possible implementations, step 23 may include steps 236-237:

[0116] In step 236: for the matrix sub-blocks corresponding to each first feature point in the second similarity matrix, the matrix sub-blocks are pooled to obtain a third similarity matrix. In this step, the electronic device can perform pooling on the matrix sub-blocks corresponding to each first feature point in the second similarity matrix (taking the first feature point (u1, v1) as an example), and for the matrix sub-block corresponding to the first feature point (u1, v1) in the second similarity matrix, which is a matrix sub-block of size w*w, to obtain the pooling result corresponding to the first feature point (u1, v1), and so on, to obtain the pooling result corresponding to each first feature point, that is, to obtain the third similarity matrix. The third similarity matrix includes the pooling results corresponding to each first feature point arranged according to the arrangement position of each first feature point in the first feature block, and each pooling result can be a sub-matrix (for example, a sub-matrix of size w'*w'). For the convenience of subsequent processing, the third similarity matrix can be considered as a w*w matrix, in which each element includes a submatrix of size w'*w', and the third similarity matrix can be represented as a 4-dimensional tensor, for example, a tensor of w*w*w'*w'. Exemplarily, the w' can be equal to w / 2.

[0117] Next, in step 237 : based on each first feature point and its corresponding second feature point, the second similarity matrix and the third similarity matrix, construct input data.

[0118] In this step, in the process of sampling each submatrix corresponding to each first feature point in the third similarity matrix based on each first feature point and its corresponding second feature point, the submatrix corresponding to each first feature point in the third similarity matrix can be firstly interpolated based on the bilinear interpolation method to obtain each submatrix corresponding to each first feature point in the third similarity matrix after the interpolation, wherein the size of each submatrix after the interpolation is restored to w*w; then, referring to the aforementioned sampling process of the second similarity matrix, based on each first feature point and its corresponding second feature point, each submatrix corresponding to each first feature point in the third similarity matrix after the interpolation is sampled to obtain a second sampling matrix. Referring to the aforementioned method, based on each first feature point and its corresponding second feature point, the second similarity matrix is ​​sampled to obtain a first sampling matrix; in the latter implementation method, at least the second sampling matrix and the first sampling matrix are combined to construct input data; in another implementation method, the second sampling matrix and the first sampling matrix are combined, as well as the two-dimensional coordinates of each first feature point and the two-dimensional coordinates of the corresponding second feature point to construct input data. The process of constructing the input data may refer to the implementation process of the aforementioned step 232 or step 234, which will not be described in detail here.

[0119] This type of input data can make the homography optimization matrix obtain a better optimization matrix, so as to obtain a target homography matrix with better accuracy.

[0120] After obtaining the input data in the above manner, in step 24, the current homography matrix is ​​updated based on the input data through the homography optimization network to obtain the target homography matrix. In this step, after the electronic device obtains the input data, the input data is input into the homography optimization network to process the input data through the homography optimization network to obtain the optimization matrix corresponding to the current homography matrix, and the optimization matrix is ​​used to adjust the current homography matrix so that the adjusted homography matrix can better align the areas with correlation in the first feature block and the second feature block; the current homography matrix is ​​optimized using the optimization matrix to obtain the target homography matrix. Exemplarily, the aforementioned optimization of the current homography matrix using the optimization matrix can be matrix multiplication of the current homography matrix and the optimization matrix.

[0121] In some possible examples, the homography optimization network can be implemented by a decoder based on a CNN (convolutional neural network) network, or by other types of decoders.

[0122] In some possible examples, in order to obtain a target homography matrix with better accuracy and better characterization of the perspective (mapping) transformation relationship between two feature blocks in the feature block pair, multiple iterations can be performed through the homography optimization network to obtain the target homography matrix; accordingly, after the step of updating the current homography matrix based on the input data through the homography optimization network in step 24, the following steps 25-26 can also be included:

[0123] In step 25, it is determined whether the number of times the current homography matrix is ​​updated reaches a preset number of times. The preset number of times can be set according to actual conditions, and can be set according to the aforementioned sampling size w and / or exploration radius r, so as to better and more comprehensively explore the input data, obtain a higher quality target homography matrix, and better reflect the perspective transformation relationship between the first feature block and the second feature block.

[0124] Subsequently, in step 26, if the judgment is no, the updated current homography matrix is ​​used as the new current homography matrix, and the step of determining the second feature points corresponding to the first feature points in the first feature block from the second feature block based on the current homography matrix (i.e., the aforementioned step 21) is continued.

[0125] In the above implementation, after using the homography optimization network to obtain the updated current homography matrix, it is then determined whether the number of times the current homography matrix is ​​updated reaches the preset number. If it is determined to be no, that is, the number of times the current homography matrix is ​​adjusted to update does not reach the preset number, the updated current homography matrix is ​​continued to be used as the new current homography matrix, and based on the new current homography matrix, the second feature points corresponding to the first feature points are determined from the second feature block; then, based on the newly determined first feature points and their corresponding second feature points, and the second similarity matrix, input data is constructed, and through the homography optimization network, based on the input data, the current homography matrix is ​​continued to be updated, and then it is continued to be determined whether the number of times the current homography matrix is ​​updated reaches the preset number, and so on, until it is determined that the number of times the current homography matrix is ​​updated reaches the preset number, and the current homography matrix updated at this time is used as the target homography matrix.

[0126] After the target homography matrix is ​​obtained, in step S240 , matching point pairs between the first image and the second image are determined based on the target homography matrix.

[0127] It can be understood that the feature block pairs determined above may be one or more. In the case of multiple feature block pairs, it is necessary to determine the matching feature point pairs corresponding to each feature block pair between the first image and the second image based on the target homography matrix corresponding to each feature block pair, and then determine the matching point pairs between the first image and the second image based on the sum of the matching feature point pairs corresponding to each feature block pair. The process of determining the matching point pairs between the first image and the second image based on the target homography matrix corresponding to each feature block pair is similar. The process of determining the matching point pairs between the first image and the second image is described below by taking the target homography matrix corresponding to one pair of feature block pairs as an example. In this step, the electronic device can determine the matching point pairs between the first image and the second image based on the target homography matrix and its corresponding feature block pairs.

[0128] In a possible example, the electronic device may obtain a second matching result based on the target homography matrix, the two-dimensional coordinates (u, v) of each first feature point in the first feature block of the feature block pair in the aforementioned specified two-dimensional coordinate system, and the two-dimensional coordinates (u', v') of each second feature point in the second feature block in the aforementioned specified two-dimensional coordinate system, wherein the second matching result includes at least one pair of fine-grained matching feature point pairs; then, the electronic device converts the two-dimensional coordinates of the first feature point in the fine-grained matching feature point pair to the coordinate system of the third feature map based on the coordinates of the upper left corner feature point of the first feature block in the third feature map for each pair of fine-grained matching feature point pairs (for example, the two-dimensional coordinates of the first feature point in the fine-grained matching feature point pair are added to the coordinates of the upper left corner feature point of the first feature block in the third feature map to convert the two-dimensional coordinates of the first feature point in the fine-grained matching feature point pair to the coordinate system of the third feature map). The coordinate system where the first feature point in the fine-grained matching feature point pair is located is obtained by calculating the coordinates (position) of the first feature point in the fine-grained matching feature point pair in the third feature map; based on the coordinates of the upper-left corner feature point of the second feature block in the fine-grained matching feature point pair in the fourth feature map, the two-dimensional coordinates of the second feature point in the fine-grained matching feature point pair are converted to the coordinate system where the fourth feature map is located (for example, the two-dimensional coordinates of the second feature point in the fine-grained matching feature point pair are added to the coordinates of the upper-left corner feature point of the second feature block in the fourth feature map to convert the two-dimensional coordinates of the second feature point in the fine-grained matching feature point pair to the coordinate system where the fourth feature map is located), and the coordinates (position) of the second feature point in the fine-grained matching feature point pair in the fourth feature map is obtained; and so on, the coordinates (position) of the first feature point in the third feature map and the coordinates (position) of the second feature point in the fourth feature map of each pair of fine-grained matching feature point pairs are obtained.

[0129] Afterwards, based on the coordinates (position) of the first feature point in the third feature map, the coordinates (position) of the second feature point in the fourth feature map, and the size relationship between the third feature map and the first image (or the size relationship between the fourth feature map and the second image), the matching point pairs between the first image and the second image are obtained. Specifically, based on the size relationship between the third feature map and the first image, the coordinates (position) of the first feature point in the third feature map of each pair of fine-grained matching feature point pairs are converted to the coordinate system of the first image to obtain its coordinates (position) in the first image; based on the size relationship between the third feature map and the first image, the coordinates (position) of the second feature point in the fourth feature map of each pair of fine-grained matching feature point pairs are converted to the coordinate system of the second image to obtain its coordinates (position) in the second image, so as to obtain the matching point pairs between the first image and the second image. At this time, semi-dense matching point pairs between the first image and the second image can be obtained.

[0130] Exemplarily, the aforementioned process of converting the coordinates (position) of the first feature point in each pair of fine-grained matching feature points in the third feature map to the coordinate system of the first image based on the size relationship between the third feature map and the first image may be: based on the size of the first image and the size of the third feature map, calculating the size ratio between the first image and the third feature map, multiplying the coordinates (position) of the first feature point in each pair of fine-grained matching feature points in the third feature map by the size ratio between the first image and the third feature map to convert to the coordinate system of the first image. The process of converting the coordinates (position) of the second feature point in each pair of fine-grained matching feature points in the fourth feature map to the coordinate system of the second image is similar to the process of converting the coordinates (position) of the first feature point in each pair of fine-grained matching feature points in the third feature map to the coordinate system of the first image, and will not be repeated here.

[0131] In some possible examples, step S240 includes: determining matching point pairs between the first image and the second image based on the target homography matrix and preset densification information, wherein the densification radius is used to increase the density of feature points corresponding to the first feature block.

[0132] Exemplarily, the electronic device can determine a dense grid coordinate set Ge based on preset densification information, and use the target homography matrix and the dense grid coordinate set Ge, as well as each first feature point in the first feature block and each second feature point in the second feature block to determine the matching point pairs between the first image and the second image.

[0133] Among them, the preset densification information can be set according to actual needs. Exemplarily, the preset densification information may include a preset densification radius and a preset step size, and the preset step size is used to limit the interval of the generated grid coordinates. The preset densification radius and the preset step size can be set according to actual needs, for example, based on the size relationship between the third feature map and the first image (or the size relationship between the fourth feature map and the second image) and / or the required density of the matching point pairs between the first image and the second image. In another exemplary embodiment, the preset densification information includes the aforementioned dense grid coordinate set Ge, and the aforementioned dense grid coordinate set Ge can be parsed from the preset densification information.

[0134] In some implementations, the aforementioned step S240 may include steps 31-33:

[0135] In step 31, according to the preset densification information, based on the two-dimensional coordinates of each first feature point in the first feature block and the two-dimensional coordinates of each second feature point in the second feature block, multiple densified coordinates corresponding to each first feature point in the first feature block and multiple densified coordinates corresponding to each second feature point in the second feature block are obtained.

[0136] The preset densification information includes a preset densification radius and a preset step size. For example, the size of the third feature map is 1 / 2 of the size of the first image. At this time, the size of the third feature map needs to be doubled to restore it to the size of the first image. In order to obtain more matching point pairs between the first image and the second image (for example, in order to be able to find the matching points corresponding to each of the 4 pixel points in each 2*2 area in the first image, that is, to find 4 matching point pairs), at this time, the preset step size can be set to 0.5, and the preset densification radius Re=1. Correspondingly, the densified grid coordinate set Ge can be obtained:

[0137]

[0138] As another example, if it is necessary to obtain denser (larger) matching point pairs between the first image and the second image, a larger preset densification radius (for example, 2 or 3, etc.) and / or a smaller preset step size (for example, set to 0.25) can be set to obtain the corresponding densified grid coordinate set Ge.

[0139] After obtaining the dense grid coordinate set Ge, based on the two-dimensional coordinates of each first feature point in the first feature block (the two-dimensional coordinates in the aforementioned specified two-dimensional coordinate system) and the dense grid coordinate set Ge, the two-dimensional coordinates of each first feature point in the first feature block are expanded to obtain a plurality of densified coordinates in the specified two-dimensional coordinate system corresponding to each first feature point; for example, for each first feature point (for example, taking the first feature point O as an example), each coordinate in the dense grid coordinate set Ge is added to the two-dimensional coordinate of the first feature point O in the aforementioned specified two-dimensional coordinate system to obtain a plurality of densified coordinates in the specified two-dimensional coordinate system corresponding to the first feature point O. By analogy, a plurality of densified coordinates in the specified two-dimensional coordinate system corresponding to each first feature point are obtained.

[0140] Similarly, based on the two-dimensional coordinates of each second feature point in the second feature block (the two-dimensional coordinates in the aforementioned specified two-dimensional coordinate system) and the dense grid coordinate set Ge, multiple densified coordinates corresponding to each second feature point in the specified two-dimensional coordinate system are obtained.

[0141] Next, in step 32, based on the target homography matrix and the multiple densified coordinates corresponding to the first feature points, the matching densified coordinates that match the densified coordinates corresponding to the first feature points are determined from the multiple densified coordinates corresponding to the second feature points to determine the densified matching result.

[0142] In this step, the electronic device constructs the homogeneous coordinates corresponding to each densified coordinate (the coordinates under the aforementioned specified two-dimensional coordinates) corresponding to each first feature point O for each first feature point (taking the first feature point O as an example), and adopts the aforementioned formula (1), using the homogeneous coordinates corresponding to each densified coordinate corresponding to the first feature point O and the target homography matrix, to determine the matching densified coordinates that match the densified coordinates corresponding to the first feature point O from the multiple densified coordinates corresponding to each second feature point. By analogy, the matching densified coordinates that match the densified coordinates corresponding to each first feature point are obtained to obtain a densified matching result. The densified matching result may include multiple pairs of densified matching feature point pairs, and a single densified matching feature point pair includes a densified coordinate corresponding to the first feature point, and a matching densified coordinate that matches the one densified coordinate corresponding to the first feature point (the matching densified coordinate belongs to the multiple densified coordinates corresponding to each second feature point in the second feature block).

[0143] Then in step 33, based on the aforementioned densified matching result, matching point pairs between the first image and the second image are determined.

[0144] The electronic device may map the densified matching results back to the corresponding images (i.e., the densified coordinates corresponding to the first feature point in the densified matching results are mapped to the coordinate system of the first image based on the first mapping relationship, and the matching densified coordinates that match the densified coordinates corresponding to the first feature point in the densified matching results are mapped to the coordinate system of the second image based on the second mapping relationship) to obtain matching point pairs between the first image and the second image. The first mapping relationship may be determined based on the relationship between the first feature block and the first image (for example, including the positional relationship between the first feature block and the third feature map and the size relationship between the third feature map and the first image), and the second mapping relationship in the above process may be determined based on the relationship between the second feature block and the second image (for example, including the positional relationship between the second feature block and the fourth feature map and the size relationship between the fourth feature map and the second image).

[0145] In some specific examples, the electronic device can convert the densified coordinates corresponding to each first feature point to the coordinate system of the third feature map at least based on the coordinates (position) of the first feature point in the upper left corner of the first feature block in the third feature map (for example, adding the densified coordinates corresponding to each first feature point to the coordinates (position) of the first feature point in the upper left corner of the first feature block in the third feature map), and obtain the coordinates of the densified coordinates corresponding to each first feature point in the coordinate system of the third feature map.

[0146] Similarly, each matching densified coordinate that matches each densified coordinate corresponding to each first feature point can be converted to the coordinate system of the fourth feature map at least based on the coordinates (position) of the second feature point in the upper left corner of the second feature block in the fourth feature map (for example, each matching densified coordinate that matches each densified coordinate corresponding to each first feature point is added to the coordinates (position) of the second feature point in the upper left corner of the second feature block in the fourth feature map), and the coordinates of each matching densified coordinate that matches each densified coordinate corresponding to each first feature point in the coordinate system of the fourth feature map are obtained.

[0147] Afterwards, based on the coordinates of the densified coordinates corresponding to each first feature point in the coordinate system of the third feature map and the size relationship between the third feature map and the first image, the coordinates of the densified coordinates corresponding to each first feature point in the coordinate system of the first image are determined.

[0148] Similarly, based on the coordinates of each matched densified coordinate matched by each densified coordinate corresponding to each first feature point in the coordinate system of the fourth feature map, and the size relationship between the fourth feature map and the second image, the coordinates of each matched densified coordinate matched by each densified coordinate corresponding to each first feature point in the coordinate system of the second image are determined to obtain matching point pairs between the first image and the second image. Exemplarily, each matching point pair between the first image and the second image may include the coordinates of a densified coordinate corresponding to the first feature point in the coordinate system of the first image, and the coordinates of a matched densified coordinate matched by a densified coordinate corresponding to the first feature point in the coordinate system of the second image.

[0149] In some examples, there may be multiple matching feature point pairs (coarse-grained matching feature point pairs) corresponding to the first feature map and the second feature map. Correspondingly, using the coarse-grained matching feature point pairs, multiple feature block pairs are determined from the third feature map and the fourth feature map (each pair of coarse-grained matching feature point pairs corresponds to a pair of feature block pairs); for example, the matching feature point pairs corresponding to the first feature map and the second feature map include a first matching feature point pair and a second matching feature point pair, the first matching feature point pair corresponds to the first feature block pair, and the second matching feature point pair corresponds to the second feature block pair; after densifying the two-dimensional coordinates of each first feature point in the first feature block in the first feature block pair, After densifying the two-dimensional coordinates of each first feature point in the first feature block in the first feature block pair to obtain a plurality of densified coordinates corresponding to each first feature point in the first feature block in the second feature block pair (hereinafter referred to as the plurality of densified coordinates corresponding to the first feature block in the first feature block pair); and obtaining a plurality of densified coordinates corresponding to each first feature point in the first feature block in the second feature block pair (hereinafter referred to as the plurality of densified coordinates corresponding to the first feature block in the second feature block pair), it is inevitable that there will be overlapping coordinates between the plurality of densified coordinates corresponding to the first feature block in the first feature block pair and the plurality of densified coordinates corresponding to the first feature block in the second feature block pair.

[0150] Specifically, the multiple densified coordinates corresponding to the first feature point a in the first feature block A1 in the first feature block pair include the first densified coordinates, and the multiple densified coordinates corresponding to the first feature point b in the first feature block B1 in the second feature block pair also include the first densified coordinates.

[0151] For the above situation, the target homography matrices corresponding to the first feature block pair and the second feature block pair may be different. At this time, based on the target homography matrix corresponding to the first feature block pair and the target homography matrix corresponding to the second feature block pair, different matching densified coordinates matching the first densified coordinates can be determined from the multiple densified coordinates corresponding to each second feature point in the second feature block A2 in the first feature block pair and from the multiple densified coordinates corresponding to each second feature point in the second feature block B2 in the second feature block pair (for example, among the multiple densified coordinates corresponding to each second feature point in the second feature block A2, there is a matching densified coordinate 1 matching the first densified coordinate (for example, the subsequent first candidate coordinate); among the multiple densified coordinates corresponding to each second feature point in the second feature block B2, there is a matching densified coordinate 2 matching the first densified coordinate (for example, the subsequent second candidate coordinate)).

[0152] Since the matching point pairs generally existing between two frames of images (or two frames of feature maps) are one-to-one (for example, there is generally a pixel point in the second image that is a matching point with a pixel point in the first image), at this time, the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block A1 in the first feature block pair, and the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block B1 in the second feature block pair can be used to determine the final matching densified coordinate that matches the first densified coordinate from the matching densified coordinate 1 and the matching densified coordinate 2 (for example, if the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block A1 in the first feature block pair is closer, the matching densified coordinate 1 is used as the final matching densified coordinate that matches the first densified coordinate).

[0153] Correspondingly, in some specific examples, the aforementioned matching feature point pairs include a first matching feature point pair and a second matching feature point pair, the first matching feature point pair corresponds to a first feature block pair, and the second matching feature point pair corresponds to a second feature block pair; the multiple densified coordinates corresponding to the first feature point a in the first feature block in the first feature block pair include the first densified coordinates, and the multiple densified coordinates corresponding to the first feature point b in the first feature block in the second feature block pair include the first densified coordinates.

[0154] Accordingly, in step 32, steps 321-323 may be included:

[0155] In step 321, based on the target homography matrix corresponding to the first feature block pair and the first densified coordinates, a first candidate coordinate corresponding to the first densified coordinate is determined from a plurality of densified coordinates corresponding to each second feature point of the second feature block in the first feature block pair.

[0156] In step 322, based on the target homography matrix corresponding to the second feature block pair and the first densified coordinates, a second candidate coordinate corresponding to the first densified coordinate is determined from a plurality of densified coordinates corresponding to each second feature point of the second feature block in the second feature block pair.

[0157] In step 323, based on the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the first feature block pair, and the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the second feature block pair, a matching densified coordinate that matches the first densified coordinate is determined from the first alternative coordinates and the second alternative coordinates.

[0158] Among them, the densified center coordinates corresponding to the first feature block in the first feature block pair are determined based on the multiple densified coordinates corresponding to each first feature point in the first feature block in the first feature block pair, and the densified center coordinates corresponding to the first feature block in the second feature block pair are determined based on the multiple densified coordinates corresponding to each first feature point in the first feature block in the second feature block pair.

[0159] Wherein, illustratively, if the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the first feature block pair is less than the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the second feature block pair, the first candidate coordinate is determined as the matching densified coordinate matched with the first densified coordinate. If the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the first feature block pair is greater than the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the second feature block pair, the second candidate coordinate is determined as the matching densified coordinate matched with the first densified coordinate to obtain a densified matching point pair between the first image and the second image. If the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the first feature block pair is equal to the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the second feature block pair, the first candidate coordinate or the second candidate coordinate can be randomly determined as the matching densified coordinate matched with the first densified coordinate.

[0160] In this embodiment, the current homography matrix corresponding to the feature block pair can be optimized through a pre-trained homography optimization network based on the similarity between at least some feature points in the first feature block in the feature block pair and at least some feature points in the second feature block in the feature block pair to obtain a target homography matrix corresponding to the feature block pair. The homography relationship between all positions in the feature block pair can be obtained through such a homography matrix. Further, based on the target homography matrix, the matching point pairs between the first image and the second image can be determined. In addition, in the above process, based on the coarse-grained matching results, the corresponding feature block pairs are determined from the fine-grained feature map, and then the corresponding target homography matrix is ​​determined through the homography optimization matrix and the similarity value between each feature point of the two feature blocks in the feature block pair, and then the matching point pairs between the images are determined, which is highly efficient.

[0161] In some examples, for the first image and the second image, the pixel points determined to be matching points can be called key points of the images. For example, pixel 1 in the first image and pixel 2 in the second image are a pair of matching points, that is, pixel 1 is the key point in the first image, and pixel 2 is the key point in the second image.

[0162] In the above process, for the first image, the second image and the third image of the sequence of images, the three images are images of different angles for the same scene; in the process of determining the matching points of the pixels in the first image from the second image, based on the target homography matrix between the determined fine-grained feature block pairs that can characterize the perspective transformation relationship between the two feature blocks, based on the target homography matrix corresponding to the feature block pairs, the corresponding matches of all positions on the first feature block can be obtained from the second feature block, and accordingly, all the matching point pairs between the first image and the second image can be obtained (i.e., a pair of matching point pairs between the first image and the second image, including the pixel points in both corresponding to the same physical point, such pixel points can be called key points in the first image or the second image). Similarly, in the process of determining the matching points of the pixels in the second image from the third image, all the matching point pairs between the second image and the third image can also be determined.

[0163] In view of the above situation, if there is an overlapping area between the first image and the third image, the continuity of the key points between the sequence images can be achieved through the matching point pair generation process provided in this embodiment (it can also be called achieving the repeatability of the key points between the sequence images. For example, assuming that the first image, the second image and the third image all include pixel points corresponding to the spatial point PO, based on the matching point pair generation process provided in this embodiment, for the pixel point PO1 corresponding to the spatial point PO in the first image, the matching point matching the pixel point PO1, that is, the pixel point PO2 (that is, the pixel point corresponding to the spatial point PO in the second image) can be determined from the second image, and the matching point matching the pixel point PO2, that is, the pixel point PO3 (that is, the pixel point corresponding to the spatial point PO in the third image) can also be determined from the third image).

[0164] In some examples, the aforementioned extraction network, aggregation network, and homography optimization network may be obtained by joint training based on several sample image groups and label data corresponding to each sample image group in advance. During the training process, a training set may be obtained, the training set includes several sample image groups and label data corresponding to each sample image group, and a single sample image group includes two frames of sample images at different angles for the same sample scene, and the label data is used to indicate: label matching point pairs between two frames of sample images in the corresponding sample image group.

[0165] After obtaining the training set, for each sample image group, the current matching point pair between two frames of sample images in each sample image group can be obtained based on the aforementioned matching point pair generation process, and then a preset loss function (such as a two-norm loss function) can be used to construct a loss value based on the difference between the current matching point pair and the label matching point pair between the two frames of sample images in the label data; with the goal of minimizing the loss value, adjust the respective parameters of the extraction network, the aforementioned aggregation network, and the homography optimization network until the preset convergence conditions are reached, and determine that the extraction network, the aforementioned aggregation network, and the homography optimization network training are completed, then the aforementioned matching point pair generation process can be executed.

[0166] In some examples, the aforementioned current matching point pairs may include a densified matching point pair between two frames of sample images in each sample image group, and a semi-dense matching point pair between two frames of sample images in each sample image group. Among them, the process of determining the densified matching point pair between two frames of sample images in each sample image group, and the process of determining the semi-dense matching point pair between two frames of sample images in each sample image group, can refer to the aforementioned process of determining the densified matching point pair between the first image and the second image, and the process of determining the semi-dense matching point pair, respectively, which will not be repeated here. At this time, the label matching point pair between two frames of sample images in the label data may include a semi-dense label matching point pair between the two frames of sample images, and a densified label matching point pair.

[0167] Among them, the preset convergence conditions may include but are not limited to: the loss value is lower than a preset threshold, the number of parameter adjustments exceeds a preset number threshold, or the training time exceeds a preset time.

[0168] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0169] Corresponding to the above method embodiment, the present specification embodiment provides a matching point pair generating device 400, whose schematic block diagram is as follows: Figure 4 As shown, including:

[0170] A first obtaining module 410 is configured to extract, from the first image and the second image, respective first feature maps and second feature maps of first sizes, and respective third feature maps and fourth feature maps of second sizes, respectively, wherein the first image and the second image are images of the same scene at different angles, and the second size is larger than the first size;

[0171] A first determination module 420 is configured to use the matching feature point pairs corresponding to the first feature map and the second feature map to determine, from the third feature map and the fourth feature map, a feature block pair corresponding to the matching feature point pair, wherein the feature block pair includes a first feature block determined from the third feature map and a second feature block determined from the fourth feature map;

[0172] An updating module 430 is configured to update a current homography matrix corresponding to the feature block pair based on similarities between at least some feature points in the first feature block and at least some feature points in the second feature block in the feature block pair, so as to obtain a target homography matrix corresponding to the feature block pair;

[0173] The second determination module 440 is configured to determine matching point pairs between the first image and the second image based on the target homography matrix.

[0174] In some possible examples, the first determining module 420 includes:

[0175] A first determining unit (not shown in the figure) is configured to determine a first matching result using the first feature map and the second feature map, wherein the first matching result indicates a matching feature point pair corresponding to the first feature map and the second feature map;

[0176] The second determination unit (not shown in the figure) is configured to determine the feature block pair corresponding to the matching feature point pair from the third feature map and the fourth feature map by using the matching feature point pair and the size relationship between the first size and the second size.

[0177] In some possible examples, the first determining unit includes:

[0178] An input obtaining submodule (not shown in the figure) is configured to process the first feature map and the second feature map based on a preset aggregation operation to obtain a fifth feature map corresponding to the first image and a sixth feature map corresponding to the second image;

[0179] A first determination submodule (not shown in the figure) is configured to determine a first similarity matrix using the fifth feature map and the sixth feature map, wherein the content of each element in the first similarity matrix indicates the similarity between each feature point in the first feature map and each feature point in the second feature map;

[0180] The second determination submodule (not shown in the figure) is configured to determine the first matching result based on the first similarity matrix.

[0181] In some possible examples, the input obtaining submodule is specifically configured to input the first feature map and the second feature map into a first subnetwork based on a self-attention mechanism of an aggregation network, respectively, to obtain a first aggregated feature map corresponding to the first feature map and a second aggregated feature map corresponding to the second feature map, respectively;

[0182] The first aggregated feature map and the second aggregated feature map are input into a second sub-network based on a cross-attention mechanism of the aggregation network to obtain the fifth feature map and the sixth feature map.

[0183] In some possible examples, the second determination submodule is specifically configured to process the first similarity matrix to obtain a probability matrix; and determine the first matching result based on a mutual nearest neighbor algorithm and the probability matrix.

[0184] In some possible examples, the second determination module 440 is specifically configured to determine matching point pairs between the first image and the second image based on the target homography matrix and preset densification information, wherein the densification radius is used to increase the density of feature points corresponding to the first feature block.

[0185] In some possible examples, the second determining module 440 includes:

[0186] an obtaining unit (not shown in the figure), configured to obtain, according to preset densification information, a plurality of densified coordinates corresponding to each first feature point in the first feature block and a plurality of densified coordinates corresponding to each second feature point in the second feature block based on the two-dimensional coordinates of each first feature point in the first feature block and the two-dimensional coordinates of each second feature point in the second feature block;

[0187] A third determining unit (not shown in the figure) is configured to determine, from the multiple densified coordinates corresponding to the second feature points, the matching densified coordinates that match the densified coordinates corresponding to the first feature points based on the target homography matrix and the multiple densified coordinates corresponding to the first feature points, so as to determine a densified matching result;

[0188] The fourth determining unit (not shown in the figure) is configured to determine matching point pairs between the first image and the second image based on the densified matching result.

[0189] In some possible examples, the matching feature point pair includes a first matching feature point pair and a second matching feature point pair, the first matching feature point pair corresponds to a first feature block pair, and the second matching feature point pair corresponds to a second feature block pair; a plurality of densified coordinates corresponding to a first feature point a in a first feature block in the first feature block pair include a first densified coordinate, and a plurality of densified coordinates corresponding to a first feature point b in a first feature block in the second feature block pair include the first densified coordinate;

[0190] The third determining unit is specifically configured to: determine, based on the target homography matrix corresponding to the first feature block pair and the first densified coordinates, a first candidate coordinate corresponding to the first densified coordinate from a plurality of densified coordinates corresponding to each second feature point of the second feature block in the first feature block pair;

[0191] Based on the target homography matrix corresponding to the second feature block pair and the first densified coordinates, determining a second candidate coordinate corresponding to the first densified coordinate from a plurality of densified coordinates corresponding to each second feature point of the second feature block in the second feature block pair;

[0192] Based on the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the first feature block pair, and the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the second feature block pair, a matching densified coordinate matching the first densified coordinate is determined from the first candidate coordinates and the second candidate coordinates, wherein the densified center coordinate corresponding to the first feature block in the first feature block pair is determined based on a plurality of densified coordinates corresponding to each first feature point in the first feature block in the first feature block pair, and the densified center coordinate corresponding to the first feature block in the second feature block pair is determined based on a plurality of densified coordinates corresponding to each first feature point in the first feature block in the second feature block pair.

[0193] In some possible examples, the update module 430 is specifically configured to update the current homography matrix corresponding to the feature block pair based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block in the feature block pair through a pre-trained homography optimization network to obtain a target homography matrix corresponding to the feature block pair.

[0194] In some possible examples, the homography optimization network is trained based on several sample image groups and label data corresponding to each sample image group, wherein a single sample image group includes two frames of sample images from different angles of the same sample scene, and the label data is used to indicate: label matching point pairs between two frames of sample images in the corresponding sample image group.

[0195] In some possible examples, the updating module 430 includes:

[0196] a fifth determining unit (not shown in the figure), configured to determine a second similarity matrix, wherein the content of each element in the second similarity matrix indicates the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block;

[0197] a sixth determining unit (not shown in the figure), configured to determine, from the second feature block, each second feature point corresponding to each first feature point in the first feature block based on the current homography matrix;

[0198] a construction unit (not shown in the figure), configured to construct input data based on each first feature point and its corresponding second feature point, and the second similarity matrix;

[0199] An updating unit (not shown in the figure) is configured to update the current homography matrix based on the input data through the homography optimization network to obtain the target homography matrix.

[0200] In some possible examples, the construction unit includes:

[0201] A third determination submodule (not shown in the figure) is configured to sample each matrix subblock corresponding to each first feature point in the second similarity matrix based on each first feature point and its corresponding second feature point to determine a first sampling matrix;

[0202] A construction submodule (not shown in the figure) is configured to construct the input data based on the first sampling matrix.

[0203] In some possible examples, the construction submodule is specifically configured to construct the input data based on each first feature point and its corresponding second feature point and the first sampling matrix.

[0204] In some possible examples, the construction unit is specifically configured to perform pooling processing on each matrix sub-block corresponding to each first feature point in the second similarity matrix to obtain a third similarity matrix;

[0205] Input data is constructed based on each first feature point and its corresponding second feature point, the second similarity matrix and the third similarity matrix.

[0206] In some possible examples, the third determination submodule is specifically configured to sample each matrix subblock in the second similarity matrix corresponding to each first feature point in the first feature block, with a preset size as the sampling size and with the position corresponding to the second feature point corresponding to each first feature point as the center point, to obtain the first sampling matrix, wherein the content of each element in the matrix subblock indicates the similarity between the corresponding first feature point and each second feature point in the second feature block.

[0207] In some possible examples, the updating module 430 further includes:

[0208] a judging unit (not shown in the figure), configured to judge whether the number of times the current homography matrix is ​​updated reaches a preset number after the current homography matrix is ​​updated based on the input data through the homography optimization network;

[0209] As a unit (not shown in the figure), it is configured to, if the judgment is no, use the updated current homography matrix as the new current homography matrix and continue to trigger the sixth determination unit.

[0210] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0211] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the matching point pair generation method provided in the present specification.

[0212] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the matching point pair generation method provided in the present specification is implemented.

[0213] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0214] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0215] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for generating matching point pairs, comprising: Extracting, from the first image and the second image, respectively, a first feature map and a second feature map of a first size, and a third feature map and a fourth feature map of a second size, respectively, wherein the first image and the second image are images of the same scene at different angles, and the second size is larger than the first size; Using the matching feature point pairs corresponding to the first feature map and the second feature map, determining a feature block pair corresponding to the matching feature point pair from the third feature map and the fourth feature map, wherein the feature block pair includes a first feature block determined from the third feature map and a second feature block determined from the fourth feature map; Based on the similarity between at least some feature points in the first feature block and at least some feature points in the second feature block in the feature block pair, updating the current homography matrix corresponding to the feature block pair to obtain the target homography matrix corresponding to the feature block pair; Based on the target homography matrix, matching point pairs between the first image and the second image are determined.

2. The method of claim 1, wherein: The determining of the feature block pair corresponding to the matching feature point pair includes: Determine a first matching result using the first feature map and the second feature map, wherein the first matching result indicates a pair of matching feature points corresponding to the first feature map and the second feature map; The feature block pair corresponding to the matching feature point pair is determined from the third feature map and the fourth feature map by using the matching feature point pair and the size relationship between the first size and the second size.

3. The method of claim 2, wherein: The determining of the first matching result includes: Based on a preset aggregation operation, the first feature map and the second feature map are processed to obtain a fifth feature map corresponding to the first image and a sixth feature map corresponding to the second image; Determine a first similarity matrix using the fifth feature map and the sixth feature map, wherein the content of each element in the first similarity matrix indicates the similarity between each feature point in the first feature map and each feature point in the second feature map; Based on the first similarity matrix, the first matching result is determined.

4. The method of claim 3, wherein: The obtaining a fifth feature map corresponding to the first image and a sixth feature map corresponding to the second image includes: Inputting the first feature map and the second feature map into a first sub-network based on a self-attention mechanism of an aggregation network, respectively, to obtain a first aggregated feature map corresponding to the first feature map and a second aggregated feature map corresponding to the second feature map; The first aggregated feature map and the second aggregated feature map are input into a second sub-network based on a cross-attention mechanism of the aggregation network to obtain the fifth feature map and the sixth feature map.

5. The method of claim 3, wherein: The determining the first matching result based on the first similarity matrix includes: Processing the first similarity matrix to obtain a probability matrix; Based on a mutual nearest neighbor algorithm and the probability matrix, the first matching result is determined.

6. The method of claim 1, wherein: The determining of matching point pairs between the first image and the second image comprises: Based on the target homography matrix and preset densification information, matching point pairs between the first image and the second image are determined, wherein the densification radius is used to increase the density of feature points corresponding to the first feature block.

7. The method of claim 6, wherein: The determining of matching point pairs between the first image and the second image comprises: According to the preset densification information, based on the two-dimensional coordinates of each first feature point in the first feature block and the two-dimensional coordinates of each second feature point in the second feature block, a plurality of densified coordinates corresponding to each first feature point in the first feature block and a plurality of densified coordinates corresponding to each second feature point in the second feature block are obtained; Based on the target homography matrix and the multiple densified coordinates corresponding to the first feature points, determining, from the multiple densified coordinates corresponding to the second feature points, the matching densified coordinates that match the densified coordinates corresponding to the first feature points, so as to determine a densified matching result; Based on the densified matching result, matching point pairs between the first image and the second image are determined.

8. The method of claim 7, wherein: The matching feature point pair includes a first matching feature point pair and a second matching feature point pair, the first matching feature point pair corresponds to a first feature block pair, and the second matching feature point pair corresponds to a second feature block pair; a plurality of densified coordinates corresponding to a first feature point a in a first feature block in the first feature block pair include a first densified coordinate, and a plurality of densified coordinates corresponding to a first feature point b in a first feature block in the second feature block pair include the first densified coordinate; The determining of the matching densified coordinates that match the densified coordinates corresponding to the first feature points includes: Based on the target homography matrix corresponding to the first feature block pair and the first densified coordinates, determining a first candidate coordinate corresponding to the first densified coordinate from a plurality of densified coordinates corresponding to each second feature point of the second feature block in the first feature block pair; Based on the target homography matrix corresponding to the second feature block pair and the first densified coordinates, determining a second candidate coordinate corresponding to the first densified coordinate from a plurality of densified coordinates corresponding to each second feature point of the second feature block in the second feature block pair; Based on the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the first feature block pair, and the distance between the first densified coordinate and the densified center coordinate corresponding to the first feature block in the second feature block pair, a matching densified coordinate matching the first densified coordinate is determined from the first candidate coordinates and the second candidate coordinates, wherein the densified center coordinate corresponding to the first feature block in the first feature block pair is determined based on a plurality of densified coordinates corresponding to each first feature point in the first feature block in the first feature block pair, and the densified center coordinate corresponding to the first feature block in the second feature block pair is determined based on a plurality of densified coordinates corresponding to each first feature point in the first feature block in the second feature block pair.

9. The method according to any one of claims 1 to 8, wherein: The determining of the target homography matrix corresponding to the feature block pair includes: Based on the similarity between at least some feature points in the first feature block in the feature block pair and at least some feature points in the second feature block in the feature block pair, a current homography matrix corresponding to the feature block pair is updated through a pre-trained homography optimization network to obtain a target homography matrix corresponding to the feature block pair.

10. The method of claim 9, wherein: The homography optimization network is trained based on several sample image groups and label data corresponding to each sample image group, wherein a single sample image group includes two frames of sample images at different angles for the same sample scene, and the label data is used to indicate: label matching point pairs between two frames of sample images in the corresponding sample image group.

11. The method of claim 9, wherein: The determining of the target homography matrix corresponding to the feature block pair includes: Determine a second similarity matrix, wherein the content of each element in the second similarity matrix indicates the similarity between at least part of the feature points in the first feature block and at least part of the feature points in the second feature block; Based on the current homography matrix, determining, from the second feature block, second feature points corresponding to the first feature points in the first feature block; constructing input data based on each first feature point and its corresponding second feature point, and the second similarity matrix; The current homography matrix is ​​updated based on the input data through the homography optimization network to obtain the target homography matrix.

12. The method of claim 11, wherein: The construction input data includes: Based on each first feature point and its corresponding second feature point, sampling each matrix sub-block corresponding to each first feature point in the second similarity matrix to determine a first sampling matrix; The input data is constructed based on the first sampling matrix.

13. The method of claim 12, wherein: The constructing of the input data comprises: The input data is constructed based on each first feature point and its corresponding second feature point and the first sampling matrix.

14. The method according to claim 11, wherein constructing input data comprises: For each matrix sub-block corresponding to each first feature point in the second similarity matrix, perform pooling processing on the matrix sub-block to obtain a third similarity matrix; Input data is constructed based on each first feature point and its corresponding second feature point, the second similarity matrix and the third similarity matrix.

15. The method of claim 12, wherein: The determining of the first sampling matrix comprises: For each matrix sub-block in the second similarity matrix corresponding to each first feature point in the first feature block, the matrix sub-block is sampled with a preset size as the sampling size and with the position corresponding to the second feature point corresponding to each first feature point as the center point to obtain the first sampling matrix, wherein the content of each element in the matrix sub-block indicates the similarity between the corresponding first feature point and each second feature point in the second feature block.

16. The method of claim 11, after updating the current homography matrix based on the input data through the homography optimization network, further comprises: Determine whether the number of times the current homography matrix is ​​updated reaches a preset number; If the judgment is no, the updated current homography matrix is ​​used as the new current homography matrix, and the step of determining the second feature points corresponding to the first feature points in the first feature block from the second feature block based on the current homography matrix is ​​continued.

17. A matching point pair generating device, comprising: A first obtaining module is configured to extract, from the first image and the second image, respective first feature maps and second feature maps of first sizes, and respective third feature maps and fourth feature maps of second sizes, respectively, wherein the first image and the second image are images of the same scene at different angles, and the second size is larger than the first size; A first determination module is configured to use the matching feature point pairs corresponding to the first feature map and the second feature map to determine, from the third feature map and the fourth feature map, a feature block pair corresponding to the matching feature point pair, wherein the feature block pair includes a first feature block determined from the third feature map and a second feature block determined from the fourth feature map; An updating module, configured to update a current homography matrix corresponding to the feature block pair based on similarities between at least some feature points in a first feature block and at least some feature points in a second feature block in the feature block pair, to obtain a target homography matrix corresponding to the feature block pair; The second determination module is configured to determine matching point pairs between the first image and the second image based on the target homography matrix.

18. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 16 is implemented.

Images