Image descriptor generation method, model training method, and image matching method

Abstract

The application provides an image descriptor generation method, a model training method and an image matching method. The generation method comprises the following steps: inputting a target image carrying a key point detection result to be generated into a pre-trained descriptor generation model, and outputting a plurality of descriptors corresponding to a plurality of first key points in the key point detection result and an initial position of each descriptor from the model, wherein the position of each first key point in the key point detection result is used as the initial position of the corresponding descriptor, and the descriptor generation model comprises a multi-level image pyramid construction module, a key point supplementing module, a feature extraction module, a multi-level graph construction module and a multi-scale feature semantic matching module. The application can improve the accuracy of the generated descriptor, provide a more discriminative descriptor for image matching, and further improve the accuracy, calculation efficiency and robustness of image matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, matching and machine learning, and particularly relates to an image descriptor generation method, a descriptor generation model training method and an image matching method. BACKGROUND

[0002] Image matching is one of the basic tasks in computer vision, and the purpose is to find the same or similar regions or objects in two images. The descriptor generation method converts local image features such as corner points, edges and textures into high-dimensional vectors, i.e. descriptors, to find corresponding similar points or regions between images by comparing the similarity of these descriptors (such as Euclidean distance or cosine similarity), and provides data support for image matching technology. In the existing intelligent interpretation technology of remote sensing images, image matching takes key point slice pairs as input, and outputs the similarity result of the image pair after obtaining the feature description vector of the slice pair, and takes the similarity result as the input of the subsequent change matrix solving. The feature description vector of the slice pair is the descriptor of the slice pair. The purpose of the descriptor is to convert the key information in the image into a numerical representation that can be used for matching, classification and identification tasks.

[0003] In recent years, many deep learning-based image feature descriptor generation algorithms have been proposed, including L2-Net, HardNet (Hard Example Mining for Key point Matching), SOSNet, ContextDesc, etc. Detone et al. introduced attention mechanism into feature matching algorithm and proposed SuperGlue to construct matching descriptor through graph neural network, and regarded feature matching as an optimal transport problem, and used differentiable Sinkhorn algorithm for end-to-end training, combined with feature point spatial position and descriptor semantic information to realize high-precision and robust descriptor matching; LofTR, CoTR, ClusterGNN introduced Transformer into the feature matching process, realized long-distance interaction between features in space through attention mechanism, to a certain extent, broke through the limitation of limited local feature receptive field, and further optimized the feature extraction process that can be matched.

[0004] HardNet is an algorithm based on hard sample mining, aiming to improve the matching accuracy of feature descriptors by optimizing difficult samples in the feature matching process. The core idea of HardNet is to optimize the matching ability of the descriptor by finding difficult samples. However, the selection of difficult samples is not always reliable. Even in some cases, it is difficult to distinguish between positive and negative samples, or sometimes easy-to-match samples are mistakenly considered as difficult samples, resulting in a decrease in matching accuracy. In addition, HardNet mainly extracts features in a 32*32 local area for image matching. Although it performs well in processing feature points and local features, it may not meet the performance requirements for global feature or large-scale image processing. This is because it cannot effectively process global context information or large-scale changes.

[0005] Although LoFTR can capture long-range dependencies between features based on the self-attention mechanism of the Transformer, there are still problems as follows: first, the self-attention mechanism in the Transformer model needs to calculate the relationship between all positions, which becomes very intensive in computing when dealing with high-resolution images. Especially in the image matching task, as the image size and feature number increase, the computational complexity grows in a square level, resulting in a very high time cost for training and inference. Second, in local feature matching of images, it is often necessary to accurately capture the details of a specific area, and excessive attention to long-distance features will rely too much on global features, resulting in neglect or interference of local features. SUMMARY

[0006] In view of this, the embodiments of the present application provide an image descriptor generation method, a descriptor generation model training method and an image matching method to eliminate or improve one or more defects in the prior art.

[0007] The first aspect of the present application provides an image descriptor generation method, which comprises the following steps:

[0008] inputting a target image carrying a key point detection result to be generated into a pre-trained descriptor generation model, the key point detection result comprising a plurality of first key points and positions of each first key point, and the descriptor generation model comprising a multi-level image pyramid construction module, a key point supplementing module, a feature extraction module, a multi-level graph construction module and a multi-scale feature semantic matching module;

[0009] constructing a multi-level image pyramid corresponding to the first key point based on a plurality of multi-scale image slices corresponding to the target image and centered on the first key point by the multi-level image pyramid construction module;

[0010] The key point supplement module performs superpixel segmentation, key point rapid extraction and clustering on each level image of the multi-level image pyramid to obtain a plurality of superpixel regions and a plurality of key point clustering centers, and takes the center of each superpixel region and the key point clustering center as a second key point and a third key point respectively;

[0011] The feature extraction module extracts features of each level image in the multi-level image pyramid to obtain a corresponding feature map;

[0012] The multi-level graph construction module determines each level attention region based on each level image of the multi-level image pyramid, takes the first key point as a center node in each level image, takes a third key point in the respective level attention region and not belonging to the same superpixel region as the first key point, or a second key point and a third key point as a neighbor node of the center node, obtains a feature vector of each node and a similarity between nodes based on the corresponding feature map, and constructs an edge connecting the nodes based on the similarity, constructs each level graph based on the nodes and edges to form a multi-level graph;

[0013] The multi-scale feature semantic matching module updates each edge based on the feature vector of each node in each level graph using an attention mechanism, updates the feature vector of each node in a respective spatial region range based on the feature vector of each node distributed in the spatial region range in the multi-level graph, obtains updated each level graph based on the updated feature vector of the node and the edge to form an updated multi-level graph, and aggregates the updated multi-level graph to output a descriptor corresponding to the first key point and an initial position of the descriptor, thereby outputting a plurality of descriptors corresponding to a plurality of first key points and an initial position of each descriptor, and the initial position of the descriptor is the position of the corresponding first key point.

[0014] In some embodiments of the application, the descriptor generation model further comprises a descriptor position optimization module, and the method further comprises:

[0015] The key point supplement module also outputs the positions of the second key points and the third key points;

[0016] The descriptor position optimization module obtains a final matching score of each node based on the feature vector of each node of the lowest level graph in the updated multi-level graph, and outputs the position of the node with the highest final matching score as an optimized position of the corresponding descriptor.

[0017] In some embodiments of the application, updating each edge based on the feature vector of each node in each level graph using an attention mechanism comprises:

[0018] In each hierarchical graph, the feature vectors of each node are normalized based on the feature vectors of each node;

[0019] The residual features of each neighbor node are obtained based on the difference between the normalized feature vectors of the center node and each neighbor node, and the normalized feature vector of the center node is taken as the residual feature of the center node;

[0020] The attention weights between each node and each neighbor node of each node are obtained based on the normalized feature vectors of each node and each neighbor node of each node;

[0021] The edges between each node and each neighbor node of each node are updated according to the attention weights and the residual features of each neighbor node of each node.

[0022] In some embodiments of the present application, the feature vectors of each node in each respective spatial region range are updated based on the feature vectors of each node distributed in each spatial region range in the multi-hierarchical graph, including:

[0023] The multi-hierarchical graph is divided into multiple spatial region ranges;

[0024] The feature vectors of each node in each hierarchical graph in each respective spatial region range are updated based on the mean and standard deviation of the feature vectors of each node distributed in each spatial region range.

[0025] The second aspect of the present application provides a description of a generative model training method, which comprises the following steps:

[0026] Obtain a training set, the training set comprising a plurality of image matching pairs containing main images and auxiliary images carrying key point detection and matching results, and a plurality of positive samples and a plurality of negative samples corresponding to each image matching pair, the key point detection and matching results comprising a plurality of first key point pairs composed of matched first key points from the main images and the auxiliary images and the positions of each first key point pair, the positive sample being an image sample containing the first key point, and the negative sample being an image sample containing a non-first key point;

[0027] The preset descriptor generation model is pre-trained by minimizing a loss function based on the training set, to obtain a pre-trained descriptor generation model, wherein the descriptor generation model comprises a multi-level image pyramid construction module, a key point supplementing module, a feature extraction module, a multi-level graph construction module, and a multi-scale feature semantic matching module; the multi-level image pyramid construction module is configured to construct a multi-level image pyramid corresponding to the first key point or the non-first key point based on the main image and the auxiliary image, and multi-scale image slices corresponding to the first key point or the non-first key point obtained based on the positive sample and the negative sample; the key point supplementing module is configured to perform superpixel segmentation, key point rapid extraction and clustering on each level image of the multi-level image pyramid, to obtain a plurality of superpixel regions and a plurality of key point clustering centers, and take the center of the superpixel region and the key point clustering center as a second key point and a third key point, respectively; the feature extraction module is configured to extract features of each level image in the multi-level image pyramid, to obtain a corresponding feature map; the multi-level graph construction module is configured to determine a level attention region based on each level image of the multi-level image pyramid, and take the first key point or the non-first key point as a center node, and take a third key point in the level attention region and not belonging to the same superpixel region as the first key point or the non-first key point, or a second key point and a third key point as neighbor nodes of the center node, to obtain a feature vector of each node and a similarity between nodes based on the corresponding feature map, and to construct an edge connecting the nodes based on the similarity, to construct a level graph based on the nodes and the edges to form a multi-level graph; the multi-scale feature semantic matching module is configured to update each edge based on the feature vector of each node in each level graph by using an attention mechanism, to update the feature vector of each node in a respective spatial region range based on the feature vector of each node distributed in the respective spatial region range in the multi-level graph, to obtain updated level graphs based on the feature vector of the updated node and the edge to form an updated multi-level graph, and to aggregate the updated multi-level graph to output a descriptor corresponding to the first key point or the non-first key point and an initial position of the descriptor, so as to output a plurality of descriptor pairs corresponding to a plurality of first key point pairs, an initial position of each descriptor pair, a plurality of descriptors corresponding to a plurality of first key points corresponding to a plurality of positive samples, a plurality of descriptors corresponding to a plurality of non-first key points corresponding to a plurality of negative samples, and an initial position of each descriptor, wherein the initial position of the descriptor pair is the position of the corresponding first key point pair, and the initial position of the descriptor is the position of the corresponding first key point or non-first key point.

[0028] In some embodiments of the application, the key point supplementing module is further configured to output the positions of the second key point and the third key point.

[0029] The descriptor generation model further comprises a descriptor position optimization module, which comprises a feature aggregation module, a feature matching module, a spatial mapping module, and a node score calculation and output module,

[0030] The feature aggregation module is configured to update the feature vector of the center node by aggregating the feature vectors of the neighbor nodes in each hierarchical graph of the updated multi-level graph corresponding to the primary image and the secondary image to obtain a global feature;

[0031] The feature matching module is configured to sequentially match the feature vector of each node in each hierarchical graph corresponding to the primary image and the secondary image with the global feature of each hierarchical graph corresponding to the secondary image and the primary image to obtain a plurality of feature matching scores of each node in each hierarchical graph corresponding to the primary image and the secondary image;

[0032] The spatial mapping module is configured to map each node in the lowest hierarchical graph in each hierarchical graph corresponding to the primary image and the secondary image to each hierarchical graph of a higher level in each hierarchical graph corresponding to the secondary image and the primary image based on a preset spatial mapping relationship to obtain mapped nodes, and determine the mapping weight parameter between nodes based on whether the mapped nodes fall into the superpixel region to which the nodes in the corresponding higher level graph belong.

[0033] The node score calculation and output module is configured to obtain the final matching score of each node in the lowest hierarchical graph based on the weighted sum of the mapping weight parameter and the sum of the feature matching scores of each node in the corresponding lowest hierarchical graph, and output the probability distribution of the final matching score, i.e., the predicted optimized position of the corresponding descriptor, to output the predicted optimized position of the corresponding descriptor pair.

[0034] In some embodiments of the present application, the loss function comprises a matching loss function and a positioning loss function, as follows:

[0035] L = L match + L loc

[0036]

[0037] wherein L represents the loss function, i.e., the total loss; L match represents the matching loss function, δ represents a regularization constant for controlling the distance difference between positive and negative samples; D(·) represents a similarity calculation function; f represents the sequence number of the sample pair containing the first key point pair in each image matching pair, f = 1, 2, …, N f , N f represents the number of sample pairs. a descriptor representing a first key point contained in the fth sample of the main image A, a descriptor representing a first key point contained in a positive sample corresponding to the fth sample of the auxiliary image B, and respectively represent descriptors of negative samples corresponding to the fth sample of the main image A and the auxiliary image B; L loc represent a positioning loss function, and represent positioning losses corresponding to the main image A and the auxiliary image B respectively, and respectively represent final matching score of the i th node in the first level graph of the updated multi-level graph corresponding to the main image A and the auxiliary image B, i.e., the lowest level graph of the updated multi-level graph corresponding to the main image A and the auxiliary image B; and represent probability distribution of the final matching score of the i th node in the first level graph of the updated multi-level graph corresponding to the main image A and the auxiliary image B, i.e., the predicted optimal position of the corresponding descriptor pair; and represent the probability ground truth of the i th node in the first level graph of the updated multi-level graph corresponding to the main image A and the auxiliary image B, i.e., the actual optimal position of the corresponding descriptor pair, i = 1, 2, …, K, K represents the number of nodes.

[0038] The third aspect of the present application provides an image matching method, which comprises the following steps:

[0039] matching images in an image pair to be matched based on respective descriptors of the image pair and positions corresponding to the descriptors, the descriptors and the positions corresponding to the descriptors being obtained in advance by the image descriptor generation method according to the first aspect.

[0040] The fourth aspect of the present application provides an electronic device, which comprises a computer device comprising a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, so that the device implements the steps of the image descriptor generation method according to the first aspect, or the steps of the descriptor generation model training method according to the second aspect, or the steps of the image matching method according to the third aspect.

[0041] The fifth aspect of the present application provides a computer readable storage medium, which stores a computer program, the computer program is executed by a processor to implement the steps of the image descriptor generation method according to the first aspect, or implement the steps of the descriptor generation model training method according to the second aspect, or implement the steps of the image matching method according to the third aspect.

[0042] The sixth aspect of the present application provides a computer program product, which includes computer instructions, the computer instructions are executed by a processor to implement the steps of the image descriptor generation method according to the first aspect, or implement the steps of the descriptor generation model training method according to the second aspect, or implement the steps of the image matching method according to the third aspect.

[0043] The image descriptor generation method, the descriptor generation model training method and the image matching method of the present application, wherein the generation method guides the generation of the descriptor by extracting the key points in different scale images of the multi-level image pyramid through combining the superpixel segmentation technology and the key point fast extraction technology, and refines the generated descriptor by constructing the multi-level attention space on the multi-level image pyramid, can accurately extract the local features of the image at different scales, realizes multi-level and comprehensive feature capture, avoids the image matching limitations caused by excessive dependence on fixed regions and global features, and thus improves the accuracy of the descriptor and the matching. By fusing the multi-level features through the relationship between the features at different levels in the multi-level graph, further mining the semantic association between the features at different scales can provide more discriminative descriptors for image matching. By constraining the feature association range through the multi-level pyramid, the association region is adaptively adjusted in the feature matching process, flexibly responds to the transformation and inconsistency between images, and ensures the accurate alignment of the matching features at different scales and angles. Therefore, the method can improve the accuracy of the generated descriptor, provide more discriminative descriptors for image matching, and thus improve the precision, computational efficiency and robustness of image matching.

[0044] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following figures and detailed description thereof or can be learned by practice of the present application. The advantages and objects of the present application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0045] Those skilled in the art will appreciate that the objects and advantages of the application can be implemented and achieved by the present application, and the above and other objects of the present application will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0047] Figure 1 A flowchart of a method for generating an image descriptor in an embodiment of the present application;

[0048] Figure 2 A network structure diagram of a descriptor generation model in an embodiment of the present application;

[0049] Figure 3 A diagram of a multi-level image pyramid and a corresponding multi-level graph in an embodiment of the present application;

[0050] Figure 4 A flowchart of a method for training a descriptor generation model in an embodiment of the present application;

[0051] Figure 5 A flowchart of a method for image matching in an embodiment of the present application. DETAILED DESCRIPTION

[0052] To make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but are not used to limit the present application.

[0053] It should be noted that, in order to avoid the present application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0054] It should be emphasized that the term "comprises / comprising" when used in this specification is taken to specify the presence of stated features, elements, steps or components but does not preclude the presence or addition of one or more other features, elements, steps, components or groups thereof.

[0055] It should be noted that, if not specifically stated, the term "connected" in this specification can not only mean direct connection, but also mean indirect connection with an intermediate object.

[0056] In the following, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0057] Some of the existing descriptor generation methods extract descriptors in a fixed region of an image, which cannot effectively process global features and large-scale change images. Some other methods pay excessive attention to distant features, thus overly relying on global features and ignoring or interfering with local features, and only mine matchable features in a global range, which results in inaccurate generated descriptors and their positioning, and the calculation complexity of the image matching stage grows in a square level and the matching accuracy cannot be effectively guaranteed. Therefore, the embodiment of the present application proposes an image descriptor generation method, a descriptor generation model training method and an image matching method, which can solve the problems of inaccurate descriptors and their positioning generated by the traditional method, large matching calculation and matching errors. Moreover, the descriptors generated by the method can more accurately capture key features in a complex and variable environment, thereby significantly improving the precision, calculation efficiency and robustness of image matching.

[0058] Figure 1 and Figure 2 are respectively the flow of the image descriptor generation method and the network structure diagram of the descriptor generation model in an embodiment of the present application. As shown in Figure 1 and Figure 2 , the method comprises the following steps:

[0059] Step S110, input a target image carrying a key point detection result of a descriptor to be generated into a pre-trained descriptor generation model, the key point detection result comprising a plurality of first key points and the position of each first key point, and the descriptor generation model comprising a multi-level image pyramid construction module, a key point supplementing module, a feature extraction module, a multi-level graph construction module and a multi-scale feature semantic matching module.

[0060] Specifically, a key point detection algorithm can be used to perform global key point detection and extraction on the target image of the descriptor to be generated, to obtain a key point detection result comprising a plurality of first key points p and their corresponding position coordinates p(c x ,c y ). The first key point can be a corner point, an edge, a texture or the like of the target image, as shown by the pentagonal point in Figure 3 .

[0061] Step S120, based on the multi-scale image slices corresponding to the first key points obtained by the multi-level image pyramid construction module from the target image, the multi-level image pyramid corresponding to the first key points is constructed.

[0062] Figure 3Figure 1 is a schematic diagram of a multi-level image pyramid and a corresponding multi-level graph for an embodiment of the present application. Specifically, for each first key point on a target image, a plurality of image patches of different sizes are finely defined with the first key point as the center on the target image, such as image patches of sizes 16x16, 32x32, 64x64, 128x128, and the like, and all the image patches are uniformly scaled to a standard scale, such as 32x32, to obtain a plurality of image patches of the standard scale, and then a multi-level image pyramid corresponding to the first key point is constructed according to the plurality of image patches of the standard scale. By extracting descriptors in regions of different sizes, the defect that the position of the descriptor and the matching accuracy cannot be improved due to the fixed region extraction in the traditional method can be overcome, especially when processing multi-modal data with inconsistent feature spaces. Each level of the multi-level image pyramid has a different resolution, as shown in the following formula:

[0063] H l = H0·s l W l = W0·s l

[0064] wherein (H l ,W l ) represents the resolution of the image of the lth level, (H0,W0) represents the resolution of the target image, s represents a scale factor, l represents a level index, l = 1, 2, …, L, and L represents the number of levels of the multi-level image pyramid. The image of the lowest level (the first level) at the bottom of the pyramid has the highest resolution, and the image of the highest level (the Lth level) at the top of the pyramid has the lowest resolution. As shown in Figure 3 , one level of the multi-level image pyramid corresponds to an image patch of one scale, and the image patch of the original scale of 128x128 is located at the lowest level, and the image patch of the original scale of 16x16 is located at the highest level. Finally, a plurality of multi-level image pyramids corresponding to a plurality of first key points are obtained.

[0065] In step S130, the key point supplementing module performs superpixel segmentation, key point fast extraction and clustering on each level image of the multi-level image pyramid to obtain a plurality of superpixel regions and a plurality of key point clustering centers, and the center of the superpixel region and the key point clustering center are taken as the second key point and the third key point, respectively.

[0066] Superpixels are a key concept in image segmentation, which can segment an image into different regions under unsupervised conditions by aggregating adjacent pixels with similar features (such as color, texture, grayscale, etc.) in the image. Representative algorithms include Simple Linear Iterative Clustering (SLIC) and Watershed algorithm. By dividing the image into superpixels, the computational load can be effectively reduced while preserving important structural and semantic information in the image. The method further delimits and obtains the multi-scale attention space range of the first key point using superpixel segmentation technology on the multi-level image pyramid. Specifically, for each scale or each level image of each multi-level image pyramid, the SLIC algorithm is used to perform superpixel segmentation on the image, dividing the image into multiple superpixel regions with fine-grained feature representation. Each superpixel (i.e., superpixel region) is considered as a basic spatial unit, and these basic spatial units cover different regions in the image. That is, each level image is divided into multiple subspaces by superpixel segmentation. And the center of each superpixel region is used as a second key point to supplement the key points, such as Figure 3 The five rows and five columns of 25 dots in the middle.

[0067] In order to fine-tune the position of the key features in the image, for each level image of each multi-level image pyramid, the method uses an algorithm with dense key point fast mining capability, such as the FAST algorithm (a fast corner detection algorithm), to extract key points from the image to obtain multiple key points. This way, the first key point can be accurately labeled in the image space of different levels. However, due to the large scale of images in the multi-level image pyramid, such as image slices with an original scale of 128x128, there are a large number of highly concentrated key points after fast key point extraction. Therefore, the method can also use clustering algorithms, such as the k-means algorithm, to cluster the dense multiple key points extracted from the image for each level image in each multi-level image pyramid. In the clustering process, the dense key points of each region with similar distances are clustered to form a key point cluster center, thereby obtaining multiple key point cluster centers, which can reduce the impact of excessive redundant key points on computation and further reduce computational complexity. For example, the number of key point cluster centers is set to 9. Each key point cluster center can be regarded as the most representative key point in the corresponding region, and is also used as a third key point to further supplement the key points, such as Figure 3 The triangular points in the middle. In addition, in order to ensure that the third key points close to the first key points are given priority in subsequent steps, the method can also sort all the third key points.

[0068] Step S140, the feature extraction module extracts the features of each level image in the multi-level image pyramid to obtain the corresponding feature map.

[0069] Specifically, the feature extraction module, i.e. Figure 2 the multi-level feature extraction backbone network shown in the above formula, can use the DINOv2 model, which can effectively extract deep-level features in the image. For each multi-level image pyramid, the features in each level image are extracted by the DINOv2 model to obtain a plurality of feature maps corresponding to the plurality of level images, as shown in the following formula:

[0070]

[0071] wherein I l represents the l-th level image; represents the feature map corresponding to the l-th level image. The feature map sequence corresponding to the multi-level image pyramid formed by the plurality of feature maps of the above multi-scale is to obtain a plurality of feature map sequences corresponding to a plurality of multi-level image pyramids.

[0072] The DINOv2 (Self-supervised Learning with Vision Transformers) model is a computer vision model based on self-supervised learning. When used as an image feature extraction model, it adopts a vision Transformer (ViT) architecture. Its essence is to use a contrastive learning and self-supervised learning strategy to train a visual transformer of the autoencoder architecture, thereby generating a strong representation feature of the image. Unlike traditional supervised learning, the DINOv2 model can adapt to image features in different datasets and scenarios. It learns image features by maximizing the similarity between different perspectives of an image, thereby avoiding dependence on labels, making it have good generalization ability and high transferability even without labeled data. The specific principle is as follows: the input image is I l ∈R H×W×3 , where H and W represent the height and width of the image, respectively, and 3 represents the color channel of the image. First, the input image is divided into a plurality of image blocks x n ∈R d×d×3 , n = 1, 2, …, N, n represents the image block number, N represents the number of image blocks, and the scale of each image block is d x d, forming an image block sequence {x1, x2, …, x N}. Each image block is converted into a first feature vector by linear mapping according to the following formula, and position encoding E pos is added to maintain spatial information:

[0073] z n = W ′ ·x n+E pos,n

[0074] wherein, represents a linear mapping matrix, D represents the dimension of the embedding space, z n represents a second feature vector. That is, a feature map is a feature vector sequence composed of the second feature vectors of all image blocks.

[0075] In step S150, the multi-level graph construction module determines a respective level attention region based on the respective level image of the multi-level image pyramid, takes the first key point as a center node, takes a third key point, or a second key point and a third key point, which are within the respective level attention region and do not belong to the same superpixel region as the first key point, as a neighbor node of the center node, obtains a feature vector of each node and a similarity between nodes based on the corresponding feature map, constructs an edge connecting the nodes based on the similarity, constructs a respective level graph based on the nodes and the edges to form a multi-level graph.

[0076] Specifically, for each level image of the multi-level image pyramid, in the process of constructing the corresponding level graph (graph neural network), in this embodiment, it is required that each layer of the graph has the same number of nodes, for example, the total number of preset nodes in each layer of the graph is 9. In other embodiments, the number of nodes of each level graph can also be different. First, a region of interest matrix is defined for each level image as the region of interest of each level to form a multi-scale attention space. The multi-scale attention space is used to mark the regions of the image that have not been considered to prevent multiple key points (which can be first, second, or third key points) belonging to the same superpixel region in the image from being repeatedly selected as nodes for constructing the graph. Second, the strategy of preferentially selecting third key points as nodes and, if the total number of selected third key points and first key points does not exceed the total number of preset nodes, then selecting second key points as nodes for supplementation is used to construct the graph. In addition, although the second key points are used as a supplement to the nodes, in order to make the nodes of the constructed graph have diversity in direction and distance, the method also randomly shuffles the order of all second key points in the neighborhood space around the first key points, which can ensure that the graph construction can capture feature information in various directions and distances. The key point order after random shuffling ensures that key points in different positions can be focused on, thereby improving the generalization ability of the finally constructed graph neural network. In the specific process, it is first determined whether the second key point or the third key point and the first key point belong to the same superpixel region. If not, the second key point or the third key point is selected as a node, and the superpixel region to which the selected key point belongs is deleted in the corresponding level region of interest, which can avoid the repeated selection of key points in the same superpixel region. Therefore, the nodes in the graph can include first key points and third key points, or first key points, second key points, and third key points. Among them, the first key point is the center node, and the second key point and the third key point are the key points in the neighborhood space of the first key point, and the second key point and the third key point are both neighbor nodes of the center node.

[0077] By combining the superpixel segmentation technology, key point extraction technology, and generating a multi-scale attention space on a multi-level image pyramid, and generating a key point guided descriptor, local features of a target image can be accurately extracted at different scales, multi-level and comprehensive feature capture is achieved, which can effectively avoid the matching limitation caused by excessive dependence on global features in the image matching process, thereby improving the accuracy of image feature representation. Smaller receptive fields are used to capture subtle texture features on high-resolution images, while larger receptive fields are used to grasp the overall outline of low-resolution images.

[0078] Next, using the following formula, average pooling is applied to aggregate the feature vectors of all pixels within the superpixel regions of each node in the graph, to obtain the feature vectors of multiple nodes:

[0079]

[0080] Wherein, F(v) i ) represents the i-th node v i eigenvectors, f m This represents the feature vector of the m-th pixel within the superpixel region to which the i-th node belongs, where m = 1, 2, ..., M i M i f represents the number of pixels within the superpixel region to which the i-th node belongs. m ∈R D D represents the dimension of the feature vector of a pixel (i.e., the dimension of the embedding space). The feature vectors of all pixels within the superpixel region to which each node belongs in each level of the graph are obtained from the feature maps of the respective level of the image. By assigning the same weight to all pixel-level features within the superpixel region through the above aggregation, the features can be smoothed and the influence of local noise can be removed. The resulting feature vector of a node reflects the node's position in the corresponding feature map. Features of [the text].

[0081] To establish relationships between nodes, the similarity between node feature vectors is calculated to obtain the node similarity. Edges between nodes are then constructed based on this similarity, resulting in multiple edges in the graph. Specifically, similarity can be calculated using cosine similarity or Euclidean distance. Then, this hierarchical graph is constructed based on all nodes and all edges, resulting in a multi-level graph. Further, multiple multi-level graphs corresponding to multiple multi-level image pyramids for multiple first keypoints are obtained. In fact, different levels of the multi-level image pyramid represent the features of the target image at different scales; therefore, each level's graph G = (V, E) may have its own unique set of nodes V and edge set E.

[0082] Step S160: The multi-scale feature semantic matching module uses an attention mechanism to update each edge based on the feature vector of each node in each level of the graph. Based on the feature vector of each node distributed in each spatial region of the multi-level graph, the feature vector of each node in its respective spatial region is updated. Based on the updated feature vector of the node and the edge, the updated level graph is obtained to form the updated multi-level graph. The updated multi-level graph is then aggregated to output the descriptor corresponding to the first key point and the initial position of the descriptor. This outputs multiple descriptors corresponding to multiple first key points and the initial positions of each descriptor. The initial position of the descriptor is the position of the corresponding first key point.

[0083] In some embodiments, step S160 involves using an attention mechanism to update each edge based on the feature vectors of each node in the graph at each level, including the following steps:

[0084] In each level of the graph, the feature vectors of each node are normalized based on the feature vectors of each node.

[0085] The residual features of each neighboring node are obtained based on the difference between the normalized feature vectors of the central node and each neighboring node, and the normalized feature vector of the central node is used as the residual feature of the central node.

[0086] The attention weights between each node and its neighbors are obtained based on the normalized feature vectors of each node and each node's neighbors.

[0087] The edges between each node and its neighbors are updated based on the attention weights and the residual characteristics of each node's neighbors.

[0088] Specifically, for each level of the graph within each multi-level hierarchy, L2 normalization is used to unify the scale of node features within each level, thereby improving model stability and performance. The feature vectors of each node are normalized according to the following formula:

[0089]

[0090] in, Represents node v i The normalized feature vectors, K represents the number of nodes in the graph.

[0091] To accelerate convergence, the correlation between node features is calculated using the residual features of the nodes. First, the features of each node are updated according to the following formula to obtain the residual features of each node:

[0092]

[0093] in, Represents node v i The residual characteristics, Indicates the central node v c The normalized feature vectors, When v i =v c hour, When v i ≠v c hour, That is, when node v i When it is a central node, the residual feature of that central node is its own normalized feature vector; when node v i It is the neighbor node v of the central node p When the residual feature of a neighboring node is the difference between the normalized feature vectors of the central node and the neighboring node, the residual feature is the difference between the central node and the normalized feature vector of the neighboring node.

[0094] Secondly, calculate v for each node according to the following formula. i and its neighboring nodes v j Attention weights W ij :

[0095]

[0096] Among them, v k Represents node v i Of all neighboring nodes except neighboring node v j The kth neighbor node outside of the above, Represents node v i Of all neighboring nodes except neighboring node v j The set of neighboring nodes other than [the other node]. and Representing neighbor node v respectively j and v k The normalized feature vector, LeakyReLU(·) represents the activation function, and conv(·) represents the convolution operation.

[0097] Then, graph convolution operations are used to update the relationship representation between nodes within a hierarchy, i.e., the edges between nodes, through the residual feature information of the node's neighboring nodes, thereby better capturing the information transmission in the graph structure. Specifically, the edges between each node and each node's neighboring nodes are updated according to the convolution operation calculation formula shown in the figure below:

[0098]

[0099] Among them, e ij Indicates the connection node v i and its neighboring node vj The updated edges between; σ(·) represents the activation function, which can be ReLU or ELU, etc.; Represents neighbor node v j The residual characteristics.

[0100] The goal of the multi-scale feature semantic matching module is to model the relationships between superpixels at different scales through a multi-level graph neural network and effectively fuse their features. However, the features of each layer may come from different scales or different levels of features, and the features of superpixels in each layer may differ significantly in order of magnitude, distribution, and dynamic range. For example, low-level features may contain more detailed local information (such as edges, textures, etc.), while high-level features may contain more semantic information (such as the overall shape and category of an object, etc.). This feature imbalance problem will affect the final fusion result. The traditional method of using attention mechanisms to analyze the correlation between layers and then normalize is computationally too expensive. In fact, the nodes of each layer in a multi-level graph are scattered in different regions. Therefore, in order to optimize the computation of inter-layer feature normalization in the graph neural network, this method utilizes the characteristic that the nodes in the graph are distributed in different regions to simplify this process.

[0101] In some embodiments, step S160 involves updating the feature vectors of each node within its respective spatial region based on the feature vectors of each node distributed across various spatial regions in the multi-level graph, including:

[0102] The multi-level map is divided into multiple spatial regions.

[0103] The feature vectors of each node in each level of the map distributed across various spatial regions are updated based on the mean and standard deviation of the feature vectors of each node.

[0104] Specifically, for each multi-level graph, since the nodes in the multi-level graph are distributed across multiple different spatial regions, the neighborhood space of the central node of each level can be uniformly divided into eight equal angular intervals of 360°, with each interval ranging from -22.5° to 45° (β-1) and from 22.5° to 45° (β-1). In other words, each neighboring node in each angular interval of each level is assigned a corresponding size... Within the grid area, the eight angular intervals of each layer of the atlas and the area where the central node is located together spatially form the nine spatial regions of this multi-level atlas. Where r represents the index of the divided spatial region, r = 1, 2, ..., 9. Other partitioning methods equivalent to the above can also be used to partition the spatial regions of each multi-level map, and the number of partitioned spatial regions is not limited to 9. Then, the feature vectors of each node within each partitioned spatial region are updated according to the following formula:

[0105]

[0106] in, and These represent the spatial regions of the multi-level maps. The mean and standard deviation of the eigenvectors of all nodes, F and F2 nr These represent the spatial regions of the multi-level maps. The feature vectors of each node and the updated feature vectors. Based on the updated feature vectors F of the nodes. nr and edge e ij The updated hierarchical maps were obtained. and Let represent the updated set of nodes and the set of edges, respectively, to obtain the updated multi-level graph, and then obtain multiple updated multi-level graphs.

[0107] Finally, multi-level feature aggregation is performed on each updated multi-level map to obtain multiple descriptors describing the features of each of the multiple first keypoints in the target image. The position of the first keypoint is the initial position of the corresponding descriptor. The representation of the descriptor is shown in the following formula:

[0108]

[0109] in, β represents the descriptor corresponding to the first keypoint. l This represents the weight in the l-th layer graph used to balance the feature contributions of different levels in the multi-level graph, satisfying the following condition: Represents the central node v in the l-th layer graph. c The updated feature vector, Represents the neighbor node v in the l-th layer graph. p The updated feature vectors, p = 1, 2, ..., N p N p Represents neighbor node v p Quantity; e cp Indicates the connection used to the central node v c and its neighboring node v p The updated edges between them.

[0110] Through the above steps, the multi-scale feature semantic matching module can effectively perform joint processing of multi-level maps. By focusing on the relationships between features at different levels within the multi-level map, it achieves feature fusion of the multi-level maps, possessing a larger receptive field and further exploring the semantic associations between features at different scales. This provides more discriminative and easier-to-match feature representations, i.e., descriptors, for image matching, thereby further improving image matching accuracy. Furthermore, by constraining the feature association range through a multi-level pyramid, it adaptively adjusts the association region during feature matching, flexibly responding to transformations and inconsistencies between images, enabling precise alignment of matching features at different scales and angles.

[0111] In some embodiments, the descriptor generation model further includes a descriptor location optimization module, and the method further includes the following steps:

[0112] In step S170, the key point supplementation module also outputs the positions of the second key point and the third key point.

[0113] Specifically, the positions of the second and third key points can be calculated using the following formula:

[0114]

[0115] Among them, C q M represents the position of the q-th second or third keypoint. q This represents the number of pixels in the superpixel region to which the q-th second or third keypoint belongs, (x m ,y m () represents the coordinates of the m-th pixel in each superpixel region. In specific implementations, the total number of superpixel segments set by the SLIC algorithm can be 30, the pixel density can be 10, and the variance can be 5.

[0116] In step S180, the descriptor position optimization module obtains the final matching score of each node based on the feature vector of each node in the lowest level graph in the updated multi-level graph, and outputs the position of the node with the highest final matching score as the optimized position of the corresponding descriptor.

[0117] To address the problem of inaccurate descriptor localization leading to image matching errors in traditional methods, this method further repositions and optimizes the descriptor's location. Specifically, it precisely adjusts the local features indicated by each first keypoint in the target image, enabling accurate image matching, especially for multimodal image registration. Specifically, for each first keypoint, the updated multi-level map is optimized using the Softmax layer in the descriptor location optimization module. This transformation applies the following formula to each node's features in the updated lowest-level map with the highest image resolution, yielding the final matching score for each node:

[0118]

[0119] Among them, P 1,i This represents the lowest level graph, i.e., the i-th node v in the first level graph. i The final matching score, where K represents the number of nodes. The node v represents the first-level graph. i The feature vector is the updated feature vector mentioned above. Based on this, the final matching score {P} of each node is calculated using the argmax operation according to the following formula. 1,1 ,P 1,2 ,…,P 1,i ,…,P 1,K Find the node with the highest final matching score and output its position:

[0120]

[0121] in, This indicates the node position with the highest final match score, which could be the first, second, or third keypoint; it is the descriptor. The optimized positions are then achieved, thereby optimizing the positions of multiple descriptors generated from the target image. By re-selecting and repositioning the descriptors used for accurate matching based on the correlation between key point positions, the spatial relationships of the descriptors can be more accurately located, enabling the generated descriptors to capture key features more precisely in changing environments.

[0122] Figure 4 This is a flowchart illustrating a sub-generative model training method in one embodiment of the present invention. Figure 4 As shown, the method includes the following steps:

[0123] Step S410: Obtain a training set. The training set includes multiple image matching pairs containing main images and auxiliary images carrying keypoint detection and matching results, as well as multiple positive samples and multiple negative samples corresponding to each image matching pair. The keypoint detection and matching results include multiple first keypoint pairs composed of first keypoints matched from the main image and auxiliary image, and the positions of each first keypoint pair. The positive samples are image samples containing the first keypoints, and the negative samples are image samples containing non-first keypoints.

[0124] Specifically, auxiliary images are generally obtained from the main images through various data augmentation methods such as rotation, scaling, and cropping. The paired first keypoints in each first keypoint pair come from both the main and auxiliary images, and the positions of the first keypoint pairs include the individual positions of the paired first keypoints. For positive samples, each main image is augmented using data augmentation methods to obtain multiple auxiliary images corresponding to each main image. These multiple auxiliary images can be used as multiple positive samples, or the regions containing each first keypoint pair representing the same feature in each auxiliary image can be individually used as positive samples, resulting in multiple positive samples corresponding to each image matching pair. Negative samples include negative samples corresponding to both the main and auxiliary images. Specifically, in the main and auxiliary images, outside the regions where each first keypoint is located, multiple regions not containing the first keypoint can be randomly selected, or multiple regions not containing the first keypoint but at a certain distance from the first keypoint in the neighborhood space of the first keypoint can be selected as negative samples, resulting in multiple negative samples corresponding to each main and auxiliary image.

[0125] Step S420: Based on the training set, a preset descriptor generation model is pre-trained by minimizing a loss function to obtain a pre-trained descriptor generation model. The descriptor generation model includes a multi-level image pyramid construction module, a keypoint supplementation module, a feature extraction module, a multi-level atlas construction module, and a multi-scale feature semantic matching module. The multi-level image pyramid construction module is used to construct a multi-level image pyramid corresponding to the first keypoint or the non-first keypoint based on multi-scale image slices obtained from the main image and auxiliary image, as well as the positive and negative samples, centered on the first keypoint or the non-first keypoint. The keypoint supplementation module is used to... The multi-level image pyramid performs superpixel segmentation, rapid keypoint extraction, and clustering on each level of the image to obtain multiple superpixel regions and multiple keypoint cluster centers. The centers of the superpixel regions and the keypoint cluster centers are respectively designated as second and third keypoints. The feature extraction module extracts features from each level of the multi-level image pyramid to obtain corresponding feature maps. The multi-level map construction module determines the regions of interest for each level based on the images of each level of the multi-level image pyramid. In each level of the image, the first keypoint or the non-first keypoint is designated as the center node. Within the respective level of the region of interest, the keypoints are located in the region of interest and are adjacent to the first keypoint or the non-first keypoint. The first keypoint does not belong to the third keypoint in the same superpixel region, or the second and third keypoints are neighboring nodes of the central node. Based on the corresponding feature map, the feature vectors of each node and the similarity between each node are obtained, and edges connecting the nodes are constructed based on the similarity. Based on the nodes and edges, a multi-level graph is constructed to form a multi-level graph. The multi-scale feature semantic matching module is used to update each edge based on the feature vector of each node in each level of the graph using an attention mechanism, and to update the feature vector of each node in its respective spatial region based on the feature vector of each node distributed in each spatial region in the multi-level graph. Based on the updated feature vector of the node and the edge, a more... The new hierarchical maps are used to form an updated multi-level map. The updated multi-level map is then aggregated to output the descriptors corresponding to the first keypoint or the non-first keypoint and the initial positions of the descriptors. This outputs multiple image matching pairs, each corresponding to multiple first keypoint pairs, and the initial positions of each descriptor pair. It also outputs multiple descriptors corresponding to multiple first keypoints for multiple positive samples, multiple descriptors corresponding to multiple non-first keypoints for multiple negative samples, and the initial positions of each descriptor pair. The initial position of the descriptor pair is the position corresponding to the first keypoint pair, and the initial position of the descriptor is the position corresponding to the first keypoint or the non-first keypoint.

[0126] In other words, the initial position of each descriptor in each descriptor pair is the position of each first keypoint in its corresponding first keypoint pair.

[0127] In this embodiment, the loss function used to train the model includes a matching loss function, L. match As shown in the formula below:

[0128]

[0129] Where δ represents the regularization constant, used to control the distance difference between positive and negative samples; D(·) represents the similarity calculation function; f represents the sample pair index containing the first keypoint pair in each image matching pair, f=1,2,…,N f N f Indicates the number of sample pairs; The descriptor representing the first keypoint contained in the f-th sample of the main image A. This represents the descriptor of the first keypoint contained in the positive sample corresponding to the f-th sample of the auxiliary image B. and These represent the descriptors for the negative samples corresponding to the f-th sample of the main image A and the auxiliary image B, respectively. Each negative sample can be composed of two parts: a negative sample formed by random sampling based on the corresponding image and a difficult negative sample formed by a difficult neighborhood term. The difficult negative sample formed by the difficult neighborhood term can be the negative sample selected from the neighborhood space of the first key point whose distance to the first key point is greater than a preset threshold and whose features are most similar. The preset threshold can be 32, etc.

[0130] By optimizing the contrastive loss function used for image matching, the trained descriptor generation model can achieve more effective differentiation in the feature space. This not only strengthens the clustering of positive samples but also improves its learning ability for negative samples, including difficult negative samples. Ultimately, this matching loss function helps the trained descriptor generation model better identify and match descriptors used to describe image features, thereby improving the accuracy of feature matching and target recognition / matching.

[0131] In some embodiments, the key point supplementation module is further configured to output the positions of the second key point and the third key point;

[0132] The descriptor generation model further includes a descriptor location optimization module, which comprises a feature aggregation module, a feature matching module, a spatial mapping module, and a node score calculation and output module.

[0133] The feature aggregation module is used to update the feature vector of the central node by aggregating the feature vectors of each neighboring node in each level of the multi-level map after the corresponding update of the main image and the auxiliary image to obtain global features.

[0134] The feature matching module is used to match the feature vectors of each node in the updated hierarchical maps corresponding to the main image and the auxiliary image with the global features of the updated hierarchical maps corresponding to the auxiliary image and the main image in turn to obtain multiple feature matching degree scores of each node in the updated hierarchical maps corresponding to the main image and the auxiliary image.

[0135] The spatial mapping module is used to map each node of the lowest level map in the updated map of the main image and the auxiliary image to each higher level map in the updated map of the auxiliary image and the main image, respectively, based on a preset spatial mapping relationship, thereby obtaining each mapped node, and determining the mapping weight parameters between nodes based on whether the mapped node falls into the superpixel region of the node in the corresponding higher level map.

[0136] The node score calculation and output module is used to obtain the final matching score of each node in the lowest level graph by weighted sum of the feature matching scores of each node based on the mapping weight parameters and the corresponding lowest level graph, and outputs the probability distribution of the final matching score, i.e. the predicted optimization position of the corresponding descriptor, so as to output the predicted optimization position of the corresponding descriptor pair.

[0137] Because keypoint representations differ across image modalities, a keypoint present in one image may not exist in another. Therefore, to address issues such as inconsistent feature space representations and mismatched salient feature locations during multimodal image registration, and to effectively perform image matching, especially multimodal image matching, this method addresses the problem of inaccurate descriptor localization by repositioning and optimizing the generated descriptor locations. Specifically, firstly, for each level of the updated multi-level atlas corresponding to the main and auxiliary images, the feature vector of the center node, i.e., the aforementioned updated feature vector F, is processed according to the following feature aggregation formula. nr (v c Perform another update to obtain the global features:

[0138]

[0139] Among them, F g (v c ) represents the central node v c The updated feature vector, i.e., the global feature; F nr (vp ) represents the neighbor node v of the central node. p The feature vector (i.e., the updated feature vector mentioned above), Let w represent the set of neighboring nodes of the central node, and b represent the learned weights and biases, respectively. || represents the concatenation operation. Through the above feature aggregation process, the feature vector of the central node is updated again based on the feature information of its neighborhood, thus enabling the global features of the generated hierarchical maps to capture richer local semantic information. In this process, the update of the central node features at each level reflects the high-level semantic information of the first keypoint at different scale resolutions. In fact, if the local features of a certain subspace (superpixel region) in image A are most similar to the global features of a certain image slice in image B, then it can be said that the subspace in image A better represents the features of that image slice in image B, i.e., it is more suitable for characterizing the specific spatial location of the corresponding descriptor in image B.

[0140] Many images, such as remote sensing images, often exhibit multi-scale characteristics. This method matches and compares the local features (i.e., the updated feature vectors of each node) at each level of the main image (auxiliary image) with the global features at all levels of the auxiliary image (main image), and performs mutual matching between the main image and the auxiliary image according to the following formula:

[0141]

[0142] in, This represents the feature matching score between the i-th local feature of the l-th layer in the updated multi-level map corresponding to the main image A and the global feature of the h-th layer in the updated multi-level map corresponding to the auxiliary image B. This represents the feature matching score between the i-th local feature of the l-th layer in the updated multi-level map corresponding to the auxiliary image B and the global feature of the h-th layer in the updated multi-level map corresponding to the main image A. and Let A and B represent the feature vectors of the i-th node in the l-th layer of the updated multi-level graph in the main image A and the auxiliary image B, respectively. and represents the global features of the h-th layer in the updated multi-level map in the main image A and the auxiliary image B, respectively, and Matching(·) represents the matching function.

[0143] Secondly, for each updated multi-level map corresponding to the main image and the auxiliary image, a weighted decision algorithm is used to sum the feature matching scores of each node in the first-level (lowest-level) map of each of the multiple updated multi-level maps corresponding to the main image and the auxiliary image, so as to obtain the final matching score of each node in the first-level map of each of the multiple updated multi-level maps corresponding to the main image and the auxiliary image. In the weighting process, this method emphasizes that only features at the same position are included in the calculation between different levels of the multi-level map. Therefore, in the process of matching the main image and the auxiliary image, a mapping relationship from high-resolution space to low-resolution space is set, as shown in the following formula:

[0144]

[0145] in, This represents the i-th node in the first-level map with the highest resolution. The (main image or auxiliary image) is mapped to a position in the lower-resolution h-th level map (auxiliary image or main image), i.e., the mapped position; scale 1→h This represents the spatial mapping scale or spatial mapping relationship from the first layer to the h-th layer, where h = 2, 3, ..., L; Represents a node The position in the first-level map, that is, the position before mapping.

[0146] Based on this, the following indicator function is defined to represent the t-th node in the h-th level graph. and nodes in the first-level graph Mapping weight parameters between:

[0147]

[0148] in, The node corresponding to the main image A Nodes corresponding to auxiliary image B Mapping weight parameters between Represents the node corresponding to the auxiliary image B The node corresponding to the main image A The mapping weight parameters between them. According to the above formula, if the node... After being mapped and falling into the node Within the range of the superpixel region, the node and The mapping weight parameter between them is 1 if one is true and 0 otherwise. Therefore, the final matching score of each node in the first level of the updated multi-level atlas corresponding to the main image and the auxiliary image is calculated according to the following weighted sum formula:

[0149]

[0150] in, and These represent the i-th nodes in the lowest level of the updated multi-level graph corresponding to the main image A and the auxiliary image B, respectively. The final match score, This represents all nodes V of the updated multilevel graph corresponding to the main image A. A In addition to all nodes of the first-level graph All nodes in the other hierarchical graphs, This represents all nodes V of the updated multilevel map corresponding to the auxiliary image B. B In addition to all nodes of the first-level graph All nodes in the other hierarchical graphs, or

[0151] The essence of descriptor location optimization is to find the best matching point between the main image and the auxiliary image. Therefore, during model training, the location optimization problem can be transformed into a classification problem. For each multi-level map corresponding to the updated main image A and auxiliary image B, the probability distribution of the final matching degree score of each node in the lowest level map can be obtained through the Softmax layer according to the following formula:

[0152]

[0153] in, and These represent the i-th nodes in the lowest level of the updated multi-level graph corresponding to the main image A and the auxiliary image B, respectively. The probability distribution of the final matching score corresponds to the predicted optimized position of the descriptor pair corresponding to the first keypoint pair. Furthermore, the center nodes in the lowest level map of the updated multi-level map corresponding to the main image A (auxiliary image B) are... After mapping to the lowest level of the updated multi-level map corresponding to the auxiliary image B (main image A), based on the central node... Does the mapped position fall within a node of the lowest-level map of another image? The judgment result within the superpixel region is obtained by using the following formula to obtain the results for each node in the ground truth one-hot encoding matrix. The corresponding probability truth value and That is, the actual optimized positions of the descriptor pairs corresponding to the first keypoint pairs in the main image A and the auxiliary image B:

[0154]

[0155] Then, the cross-entropy loss function is used to calculate the difference between the one-hot encoding of the predicted optimized position and the actual optimized position of the descriptor according to the following formula, thereby obtaining the localization loss corresponding to the main image A and the auxiliary image B respectively. and And by and Construct the localization loss function together:

[0156]

[0157] Among them, L loc This represents the localization loss function. In this embodiment, the loss function used to train the model includes the aforementioned matching loss function and localization loss function, which is obtained according to the following formula:

[0158] L = L match +L loc

[0159] Where L represents the loss function, i.e., the total loss.

[0160] Figure 5 This is a schematic flowchart of an image matching method according to an embodiment of the present invention. Figure 5 As shown, the method includes the following steps:

[0161] Step S510: Match the images in the image pair based on the descriptors of each image pair to be matched and the positions corresponding to the descriptors. The descriptors and the positions corresponding to the descriptors are obtained in advance by the image descriptor generation method as described in any of the foregoing embodiments.

[0162] The position corresponding to the descriptor can be the initial position of the first keypoint, or the optimized position obtained through the descriptor position optimization module. Image matching based on the optimized position avoids matching errors caused by inaccurate descriptor positions in traditional methods, thereby improving the accuracy of image matching.

[0163] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the aforementioned method.

[0164] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned method. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0165] This invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the aforementioned method.

[0166] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0167] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0168] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0169] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating image descriptors, characterized in that, The method includes: The target image carrying the key point detection results to be generated is input into the pre-trained descriptor generation model. The key point detection results include multiple first key points and the position of each first key point. The descriptor generation model includes a multi-level image pyramid construction module, a key point supplementation module, a feature extraction module, a multi-level map construction module, and a multi-scale feature semantic matching module. The multi-level image pyramid construction module constructs a multi-level image pyramid corresponding to the first key point based on the multi-scale image slices obtained from the target image with the first key point as the center. The key point supplementation module performs superpixel segmentation and key point extraction and clustering on each level of the multi-level image pyramid to obtain multiple superpixel regions and multiple key point cluster centers. The center of the superpixel region and the key point cluster center are respectively used as the second key point and the third key point. The feature extraction module extracts features from each level of the multi-level image pyramid to obtain the corresponding feature map. The multi-level graph construction module determines the regions of interest at each level based on the images at each level of the multi-level image pyramid. In each image, the first key point is taken as the center node, and the third key point, or the second key point and the third key point, which are not in the same superpixel region as the first key point within the respective region of interest are taken as the neighbor nodes of the center node. Based on the corresponding feature map, the feature vectors of each node and the similarity between each node are obtained, and the edges connecting the nodes are constructed based on the similarity. The graphs at each level are constructed based on the nodes and the edges to form a multi-level graph. The multi-scale feature semantic matching module uses an attention mechanism to update each edge based on the feature vector of each node in each level of the graph. Based on the feature vector of each node distributed in each spatial region of the multi-level graph, the feature vector of each node in its respective spatial region is updated. Based on the updated node feature vectors and edges, the updated level graphs are obtained to form the updated multi-level graph. The updated multi-level graphs are then aggregated to output the descriptor corresponding to the first key point and the initial position of the descriptor. This outputs multiple descriptors corresponding to multiple first key points and the initial positions of each descriptor, where the initial position of the descriptor is the position of the corresponding first key point.

2. The method according to claim 1, characterized in that, The descriptor generation model further includes a descriptor position optimization module, and the method further includes: The key point supplementation module also outputs the positions of the second key point and the third key point; The descriptor position optimization module obtains the final matching score of each node based on the feature vector of each node in the lowest level graph in the updated multi-level graph, and outputs the position of the node with the highest final matching score as the optimized position of the corresponding descriptor.

3. The method according to claim 1, characterized in that, An attention mechanism is used to update each edge based on the feature vectors of each node in the graph at each level, including: In each level of the graph, the feature vectors of each node are normalized based on the feature vectors of each node. The residual features of each neighboring node are obtained based on the difference between the normalized feature vectors of the central node and each neighboring node, and the normalized feature vector of the central node is used as the residual feature of the central node. The attention weights between each node and its neighbors are obtained based on the normalized feature vectors of each node and each node's neighbors. The edges between each node and its neighbors are updated based on the attention weights and the residual characteristics of each node's neighbors.

4. The method according to claim 1, characterized in that, The feature vectors of each node within each spatial region are updated based on the feature vectors of each node distributed in each spatial region of the multi-level graph, including: The multi-level map is divided into multiple spatial regions. The feature vectors of each node in each level of the map distributed across various spatial regions are updated based on the mean and standard deviation of the feature vectors of each node.

5. A method for training a descriptor generation model, characterized in that, The method includes: Obtain a training set, which includes multiple image matching pairs containing main images and auxiliary images carrying keypoint detection and matching results, as well as multiple positive samples and multiple negative samples corresponding to each image matching pair. The keypoint detection and matching results include multiple first keypoint pairs composed of first keypoints matched from the main image and auxiliary image, and the positions of each first keypoint pair. The positive samples are image samples containing the first keypoints, and the negative samples are image samples containing non-first keypoints. Based on the training set, a pre-trained descriptor generation model is obtained by minimizing the loss function. The pre-trained descriptor generation model includes a multi-level image pyramid construction module, a keypoint supplementation module, a feature extraction module, a multi-level atlas construction module, and a multi-scale feature semantic matching module. The multi-level image pyramid construction module is used to construct a multi-level image pyramid corresponding to the first keypoint or the non-first keypoint based on multi-scale image slices obtained from the main image and auxiliary image, as well as the positive and negative samples, centered on the first keypoint or the non-first keypoint. The keypoint supplementation module is used to perform multi-level image pyramid matching on the multi-level image pyramid. The image pyramid performs superpixel segmentation and rapid keypoint extraction and clustering on each level of the image, resulting in multiple superpixel regions and multiple keypoint cluster centers. The centers of the superpixel regions and the keypoint cluster centers are designated as the second and third keypoints, respectively. The feature extraction module extracts features from each level of the multi-level image pyramid to obtain corresponding feature maps. The multi-level map construction module determines the regions of interest at each level based on the images at each level of the multi-level image pyramid. In each level of the image, the first keypoint or the non-first keypoint is designated as the center node. Within the respective level of the region of interest, the keypoints are located at the first keypoint or the non-first keypoint. The third keypoint, which does not belong to the same superpixel region, or the second and third keypoints, are considered as neighboring nodes of the central node. Based on the corresponding feature maps, the feature vectors of each node and the similarity between nodes are obtained, and edges connecting the nodes are constructed based on the similarity. Based on the nodes and edges, a multi-level graph is constructed. The multi-scale feature semantic matching module uses an attention mechanism to update each edge based on the feature vectors of each node in each level of the graph. It also updates the feature vectors of each node within its respective spatial region based on the feature vectors of each node distributed within each spatial region in the multi-level graph. Based on the updated node feature vectors and edges, an updated graph is obtained. The various levels of the atlas are used to form an updated multi-level atlas. The updated multi-level atlas is then aggregated to output the descriptors corresponding to the first keypoint or the non-first keypoint and the initial positions of the descriptors. This outputs multiple image matching pairs, each corresponding to multiple first keypoint pairs, and the initial positions of each descriptor pair. It also outputs multiple descriptors corresponding to multiple first keypoints for multiple positive samples, multiple descriptors corresponding to multiple non-first keypoints for multiple negative samples, and the initial positions of each descriptor pair. The initial position of the descriptor pair is the position corresponding to the first keypoint pair, and the initial position of the descriptor is the position corresponding to the first keypoint or the non-first keypoint.

6. The method according to claim 5, characterized in that, The key point supplementation module is also used to output the positions of the second key point and the third key point; The descriptor generation model further includes a descriptor location optimization module, which comprises a feature aggregation module, a feature matching module, a spatial mapping module, and a node score calculation and output module. The feature aggregation module is used to update the feature vector of the central node by aggregating the feature vectors of each neighboring node in each level of the multi-level map after the corresponding update of the main image and the auxiliary image to obtain global features. The feature matching module is used to match the feature vectors of each node in the updated hierarchical maps corresponding to the main image and the auxiliary image with the global features of the updated hierarchical maps corresponding to the auxiliary image and the main image in turn to obtain multiple feature matching degree scores of each node in the updated hierarchical maps corresponding to the main image and the auxiliary image. The spatial mapping module is used to map each node of the lowest level map in the updated map of the main image and the auxiliary image to each higher level map in the updated map of the auxiliary image and the main image, respectively, based on a preset spatial mapping relationship, thereby obtaining each mapped node, and determining the mapping weight parameters between nodes based on whether the mapped node falls into the superpixel region of the node in the corresponding higher level map. The node score calculation and output module is used to obtain the final matching score of each node in the lowest level graph by weighted sum of the feature matching scores of each node based on the mapping weight parameters and the corresponding lowest level graph, and outputs the probability distribution of the final matching score, i.e. the predicted optimization position of the corresponding descriptor, so as to output the predicted optimization position of the corresponding descriptor pair.

7. The method according to claim 6, characterized in that, The loss function includes a matching loss function and a localization loss function, as shown in the following formula: L=L match +L loc Where L represents the loss function, i.e., the total loss; L match The matching loss function is represented by δ, which is a regularization constant used to control the distance difference between positive and negative samples; D(·) represents the similarity calculation function; f represents the sample pair index containing the first keypoint pair in each image matching pair, f = 1, 2, ..., N. f N f Indicates the number of sample pairs; The descriptor representing the first keypoint contained in the f-th sample of the main image A. This represents the descriptor of the first keypoint contained in the positive sample corresponding to the f-th sample of the auxiliary image B. and L represents the descriptors of the negative samples corresponding to the f-th sample of the main image A and the auxiliary image B, respectively; loc S represents the localization loss function. 1,i P represents the lowest level graph of the updated multi-level graph, i.e., the final matching score of the i-th node in the first-level graph; 1,i y represents the probability distribution of the final matching score of the i-th node in the first-level graph of the updated multi-level graph, i.e., the prediction optimization position of the corresponding descriptor; i This represents the true probability of the i-th node in the first level of the updated multi-level graph, which is the actual optimized position of the corresponding descriptor, i = 1, 2, ..., K, where K represents the number of nodes.

8. An image matching method, characterized in that, The method includes: The images in the image pair are matched based on their respective descriptors and the positions corresponding to those descriptors, wherein the descriptors and the positions corresponding to those descriptors are obtained in advance by the method described in any one of claims 1 to 4.

9. An electronic device comprising a processor, a memory, and computer instructions stored in the memory, characterized in that, The processor is configured to execute the computer instructions, and when the computer instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method as described in any one of claims 1 to 8.

Images