Image Descriptor Update Model and Method Using Graph Structure, and Image Matching Device and Program

Abstract

To provide an image descriptor update model, a method, and a program that enable machine-learning-based image matching processing with suppressed calculation cost.SOLUTION: An image descriptor update model 1, which is a model for updating descriptors used in matching of two images, causes a computer to function as: a graph generation unit 12 which receives a key point group and a descriptor group for each of the images, and generates a graph including the received key points as nodes and information related to a pair of the node key point and a neighboring key point in the same image and / or information related to a pair of the node key point and a key point in the other image, corresponding to the node key point within a predetermined error or with a predetermined similarity or higher, as edges; and a descriptor update unit 13 which receives the descriptor group for each of the images, applies graph attention processing using the generated graph to the received descriptors, and generates updated descriptors.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to image matching technology. [Background technology]

[0002] Image matching has become an important technique for selecting and generating visual artifacts from images, such as image retrieval, visual exploration, and three-dimensional (3D) image construction.

[0003] In many cases, image matching is performed based on the visual similarity between images and the degree of agreement of the positions of keypoints as image features using image features (keypoints and their descriptors) such as SIFT (Scale-Invariant Feature Transform). Specifically, visual similarity is determined based on the distance between the descriptor of each keypoint in one image and the descriptor of the nearest keypoint in the other image, and the degree of agreement of the positions of the matched keypoints is verified.

[0004] Conventionally, in such image matching, it has been common to manually intervene in verifying the degree of agreement of keypoint positions, as in the methods disclosed in Patent Document 1, Patent Document 2, and Non-Patent Document 1. In contrast, recently, methods have been proposed that perform image matching based on machine learning in order to improve the descriptiveness (the degree to which image features are accurately described without missing anything) of local image features and to make more accurate judgments of visual similarity, as in the methods disclosed in Non-Patent Document 2 and Non-Patent Document 3.

[0005] Here, the image matching method disclosed in Non-Patent Document 2 employs a Transformer, which is a deep learning model that uses an attention mechanism that weights and considers the influence of different input image parts. This Transformer is trained to perform image matching based on both the descriptor and the position of the keypoint, and includes (a) a keypoint encoder that grasps the position of the keypoint, (b) a self / cross attention layer (9 layers) related to the features in each image or between images in the keypoint and the descriptor, and (c) a matching layer including a Sinkhorn algorithm for matching two embedded representation features.

[0006] In addition, the image matching method disclosed in Non-Patent Document 3 also has a transform configuration, similar to the method in Non-Patent Document 2. However, this image matching method does not match keypoints, but performs matching for each pixel along the height and width in the two images. Specifically, therefore, a ResNet backbone that extracts local features from images and two transforms that match low-resolution and high-resolution pixels, respectively, are adopted.

[0007] Here, the low-resolution transformer is equipped with (a) a position encoder that grasps the position of each pixel, (b) a multi-head self / cross attention layer (four-layer configuration), and (c) a nearest neighbor layer that performs low-resolution pixel matching. On the other hand, the high-resolution transformer is composed only of a self / cross attention layer (four-layer configuration), from which the high-resolution matching results are derived. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Chinese Patent No. 102194133 [Patent Document 2] U.S. Patent No. 8,805,067 [Non-patent literature]

[0009] [Non-Patent Document 1] Luca Cavalli et al., “Revisiting Handcrafted Outlier Detection”, Conference: European Conference on Computer Vision,<https: / / arxiv.org / pdf / 2006.04250.pdf> , 2020 [Non-Patent Document 2] Paul-Edouard Sarlin et al., “SuperGlue: Learning Feature Matching with Graph Neural Networks”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,<https: / / arxiv.org / pdf / 1911.11763.pdf> , 2020 [Non-Patent Document 3] Jiaming Sun et al., “LoFTR: Detector-Free Local Feature Matching with Transformers”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,<https: / / arxiv.org / pdf / 2104.00680.pdf> , 2021 Summary of the Invention [Problem to be solved by the invention]

[0010] As explained above, in image matching methods that require human intervention, the processing accuracy is limited by the artificial initial design contents, whereas in machine learning-based image matching methods, end-to-end machine learning eliminates artificial design as much as possible and makes it possible to consistently perform feature extraction / updating, feature matching processing, and keypoint position verification with a high degree of accuracy.

[0011] However, conventional end-to-end machine learning based image matching methods such as those disclosed in Non-Patent Document 2 and Non-Patent Document 3 have the problem of increased computational costs, and in particular the use of transforms requires huge amounts of memory and computation time.

[0012] For example, in the specific example described in Non-Patent Document 2, when the batch size is 16, about 56 gigabytes of GPU (Graphics Processing Unit) memory is required, and, although it is speculated, in this case, it is thought that the calculation time for matching 2048 keypoints, for example, is about 120 milliseconds. Furthermore, in the method of Non-Patent Document 3, even a small-sized image of, for example, 640 pixels x 480 pixels, requires a calculation time of about 116 milliseconds.

[0013] Therefore, the present invention aims to provide an image descriptor update model and an image descriptor update method that enable a descriptor update process that realizes machine learning-based image matching processing with reduced computational costs, as well as an image matching device and an image matching program that enable such image matching processing. [Means for solving the problem]

[0014] According to the present invention, the descriptors used to match two images are updated. like 1. An image descriptor update model for operating a computer, comprising: a graph generating means for receiving a set of keypoints and a set of descriptors for each of the two images, and generating a graph with the received keypoints as nodes and information relating to pairs of the keypoints as nodes with nearby keypoints in the same image and / or information relating to pairs of the keypoints as nodes with keypoints in the other image that correspond to the keypoints within a predetermined error or with a similarity equal to or greater than a predetermined value as edges; a descriptor update means for receiving a set of descriptors for each of the two images, and performing graph attention processing using the graph on the received descriptors to generate updated descriptors; An image descriptor updating model is provided that operates a computer in accordance with the

[0015] As one embodiment of the image descriptor updating model according to the present invention, it is also preferable that the image descriptor updating model further causes the computer to function as keypoint matching means which receives a set of keypoints and a set of descriptors for each of the two images, performs a matching process for the two images, selects matching keypoints from the received keypoints, and causes the graph generation means to receive the selected keypoints and their corresponding descriptors.

[0016] In another embodiment of the image descriptor update model according to the present invention, it is also preferable that the descriptor update means performs the graph attention process on the received descriptor multiple times, and in each time from the second time onwards, generates an updated descriptor for that time using the updated descriptor generated in the previous time, the updated descriptor generated in the first time or a preset stabilization bias term, and the descriptive information generated by the graph attention process.

[0017] Furthermore, as yet another preferred embodiment of the image descriptor updating model according to the present invention, it is also preferable that the image descriptor updating model performs at least one process pair of a process of receiving the updated descriptor by the graph generation means and updating a graph having the edge by the graph generation means, and a process of further generating an updated descriptor using the updated graph by the descriptor updating means.

[0018] In addition, as yet another preferred embodiment of the image descriptor update model according to the present invention, The graph generating means determines information relating to a pair of the received keypoint and a nearby keypoint in the same image as an edge for self-attention, and also determines information relating to a pair of the received keypoint and a keypoint in another image that corresponds to the received keypoint within a predetermined error or with a similarity equal to or greater than a predetermined error as an edge for cross-attention, The descriptor update means includes: a self-attention processing means for performing a self-attention process, which is a graph attention process using the self-attention edges, on the received descriptor for each of the two images to generate a self-attention processed descriptor; a cross-attention processing means for performing a cross-attention process, which is a graph attention process using the cross-attention edges, on the self-attention processed descriptor for each of the two images to generate an updated cross-attention processed descriptor; and performing the graph attention processing as a processing pair of the self-attention processing and the cross-attention processing once, or having a self-attention processing means receive an updated descriptor that has been subjected to the cross-attention processing, and performing the graph attention processing as the processing pair a plurality of times. It is also preferable.

[0019] According to the present invention, there is also provided an image feature extraction means for extracting a group of key points and a group of descriptors from each of the two images; an image descriptor updating means for updating the extracted descriptor group using the image descriptor updating model described above; image matching means for matching the two images using the extracted keypoints and the updated descriptors; An image matching apparatus is provided having:

[0020] According to the present invention, there is further provided an image feature extraction means for extracting a group of key points and a group of descriptors from each of the two images; an image descriptor updating means for updating the extracted descriptor group using the image descriptor updating model described above; image matching means for matching the two images using the extracted keypoints and the updated descriptors; An image matching program is provided that causes a computer to function as:

[0021] According to the present invention, 、2 1. A computer-implemented method for updating image descriptors used to match two images, comprising the steps of: receiving keypoints and descriptors for each of the two images, and generating a graph with the received keypoints as nodes and information relating to pairs of the keypoints as nodes with nearby keypoints in the same image and / or information relating to pairs of the keypoints as nodes with keypoints in the other image that correspond to the keypoints within a predetermined error or with a similarity equal to or greater than a predetermined value as edges; receiving a set of descriptors for each of the two images, and performing graph attention processing using the graph on the received descriptors to generate updated descriptors; An image descriptor updating method is provided having: Effect of the Invention

[0022] According to the image descriptor update model and the image descriptor update method of the present invention, it is possible to perform a descriptor update process that realizes a machine learning-based image matching process with reduced computational costs. Also, according to the image matching device and the image matching program of the present invention, it is possible to perform a machine learning-based image matching process with reduced computational costs. [Brief description of the drawings]

[0023] [Figure 1] FIG. 2 is a schematic diagram illustrating one embodiment of an image descriptor updating model in accordance with the present invention; [Diagram 2] 1 is a schematic diagram showing a specific example for explaining matched keypoints between an original image A and a target image B and their neighboring areas. FIG. [Diagram 3] 2 is a flow chart outlining one embodiment of a method for updating image descriptors in accordance with the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0025] [Image Descriptor Update Model] FIG. 1 is a schematic diagram illustrating one embodiment of an image descriptor updating model according to the present invention.

[0026] The image descriptor update model 1 of this embodiment shown in FIG. (a) A set of pre-extracted keypoints from each of the source image A and target image B that are the subject of image matching. A K and Descriptor Sets A D and keypoints B K and Descriptor Sets B Take D as input, (b) An updated set of descriptors used to match source image A and target image B, which enables image matching processing with higher accuracy. A D * and Descriptor Group B D* Output It is a machine learning model.

[0027] Here, a keypoint is a part of an image that is judged to be characteristic (a part that is important for image classification) when the target image is viewed from a set, predetermined viewpoint. A descriptor is a feature extracted from the image area around a keypoint, and is a high-dimensional vector that describes the semantic information of the keypoint.

[0028] In order to realize the above-mentioned processing, the image descriptor update model 1 specifically: (A) For each of the source image A and the target image B, a set of key points ( A K, B K) and the descriptor group ( A D, B D) and use the received key points as nodes. (a1) Information on pairs of keypoints that are nodes and nearby keypoints in the same image, and (a2) Information on pairs of keypoints that have been set as nodes and keypoints in another image that correspond to the keypoints within a predetermined error or with a similarity level equal to or greater than a predetermined level a graph generation unit 12 that generates a "graph" with either one or both of the above as edges; (B) For each of the source image A and the target image B, the descriptor set ( A D, B D), and performs graph attention (GAT) processing on the received descriptor using the generated “graph” to generate an updated descriptor. This makes the computer function.

[0029] In this way, image descriptor update model 1 realizes updating of descriptors using GAT processing, which requires relatively small computational costs, without relying on a transformer, which has been a problem in the past due to its huge computational costs. To achieve this, a "graph" for GAT processing is generated separately in advance based on the degree of matching of keypoint positions.

[0030] For example, when 1024 keypoints were handled using image descriptor update model 1 and the descriptor update process was performed using GAT processing (a cross-attention process, which will be described later), the calculation time was 18 milliseconds. In contrast, when the existing transformer was used to update the descriptor under the same conditions, it was confirmed that the calculation time was approximately 80 milliseconds. Also, with this image descriptor update model 1, the amount of memory required when the batch size is 1 is kept to less than 2 gigabytes, whereas the existing transformer is known to require approximately 3.7 gigabytes of memory under the same conditions.

[0031] Furthermore, the updated descriptor also contains information about the degree of match between the positional relationship of key points (nodes) as edge information of the graph and their surroundings as a result of the GAT process, which makes it possible to perform image matching with higher accuracy by using the updated descriptor. Also, this final image matching process can be performed using a lightweight and high-speed method such as the conventional nearest neighbor matching.

[0032] As described above, it is understood that the image descriptor update model 1 makes it possible to perform a descriptor update process that realizes machine learning-based image matching processing with reduced computational costs. This makes it possible to provide an image matching application that performs image search, visual search, 3D image construction, etc. in real time.

[0033] In the embodiment shown below as the GAT process (B) above, self-attention processing and cross-attention processing are performed, but it is also possible to perform only one of the processes. For example, a processing pair of graph generation processing and cross-attention processing may be performed once or multiple times. However, by adopting both self-attention processing and cross-attention processing, an updated descriptor with higher descriptiveness (the degree to which the image features are accurately described without missing any) can be obtained.

[0034] [Model configuration] The configuration of the image descriptor update model 1 of this embodiment will be described in more detail below. Referring to FIG. 1, the image descriptor update model 1 of this embodiment is as follows: (a) Keypoint matching unit 11, (a) a graph generating unit 12, and (c) A descriptor update unit 13 having a self-attention processing unit 131 and a cross-attention processing unit 132 The above are functional components (embodied by a program installed in the computer). In other words, it is a model that makes the computer function as these functional components. Each of the above-mentioned functional components will be specifically described below.

[0035] <Keypoint matching method> Also in FIG. 1, the keypoint matching unit 11 determines a set of keypoints ( A K, B K) and the descriptor group ( A D, B D), and performs a matching process for both images using these, selects matching keypoints from the received keypoints, and causes the graph generation unit 12 to receive the selected keypoints and their corresponding descriptors.

[0036] Here, it is also preferable that the matching process in the keypoint matching unit 11 is performed by a known lightweight and high-speed method, for example, so-called nearest neighbor matching based on the degree of coincidence of keypoint positions. In any case, this matching process corresponds to pre-processing for the subsequent graph generation.

[0037] Incidentally, the keypoint matching unit 11 can be omitted, but by selecting (matched) keypoints to be used in the subsequent graph generation unit 12 in this keypoint matching unit 11 and limiting the number of keypoints, it is possible to further reduce the calculation cost for training the entire model 1. For example, the number of dimensions of the hidden features in the image descriptor update model 1 can actually be suppressed to, for example, 64 or 128, but in the transformer, it is actually necessary to make it larger, for example, 256.

[0038] In this embodiment, the keypoint matching unit 11 receives a set of keypoints ( A K, B K) and the descriptor group ( A D, B D) can be extracted from the source image A and the target image B by an external image feature extraction means. For example, it may be the well-known SIFT (Scale-Invariant Feature Transform) or SuperPoint. Here, SuperPoint is described in the non-patent document: D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description”, In IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops,<https: / / arxiv.org / pdf / 1712.07629.pdf> This is an image feature disclosed in 2018.

[0039] As a further modification, these key points ( A K, B K) and the descriptor group ( A D, B D) can also be generated by an end-to-end trained deep neural network (DNN).

[0040] <Graph generation method> Also in FIG. 1, in this embodiment, the graph generation unit 12 receives the matched keypoints and descriptors of the source image A and the target image B from the keypoint matching unit 11, and for each image, (a1) Information related to pairs of keypoints that are nodes and their neighboring keypoints in the same image is called "self-attention edges" ( A ε self , B ε self ) and generate a self-attention graph. (a2) Information relating to a pair of a keypoint set as a node and a keypoint in another image that corresponds to this keypoint within a predetermined error or with a similarity degree equal to or higher than a predetermined degree is called a "cross-attention edge" ( A ε cros , B ε cros ) to generate a cross-attention graph.

[0041] Hereinafter, the graph generation process in the graph generation unit 12 will be described with reference to FIG. 2 to explain the matched keypoints between the source image A and the target image B and their neighboring regions.

[0042] First, the self-attention edge in (a1) above A ε self In this embodiment, the matched key points received from the key point matching unit 11 are treated as a seed point group, and the seed point i is treated as a set of pairs (i, j) of a seed point j located within a preset neighborhood area of ​​the seed point i. That is, (1) A ε self (l) =∪ i∈AS ∪ j∈AN_i (i, j) Self-attention for edges A ε self (l) is calculated.

[0043] As will be described in detail later, in this embodiment, a set of the graph generation process and the descriptor update process, which will be described later, is executed multiple times. A ε self (l) The superscript (l) in indicates that this edge is generated in the l-th graph generation process and is used in the l-th descriptor update process.

[0044] In addition, AS in the above formula (1) is a set of (index i of) seed points of the original image A, and AN_i is a set of (indexes of) seed points j located in the neighborhood of the seed point i. i∈AS ∪ j∈ANi indicates a union operator that forms a set including all pairs (i, j) (of indices) of all seed points i and all seed points j located in the neighborhood of the seed point i. Note that the self-attention edge of the target image B is B ε self (l) is calculated in the same manner as above.

[0045] In this embodiment, the above-mentioned neighborhood regions of the seed points are circular regions such as regions A.1 and A.2 (source image A) and regions B.1 and B.2 (target image B) shown in Fig. 2. Here, in Fig. 2, the seed points related to the circular regions A.1 and A.2 match with the seed points related to the circular regions B.1 and B.2, respectively. Of course, it is also possible to adopt a region other than a circular region as the neighborhood region, for example, an elliptical region or a square region.

[0046] Next, the cross-attention edge of the original image A in (a2) above A ε cros (Target image B B ε cros In this embodiment, the matched keypoints received from the keypoint matching unit 11 are treated as a seed point group, and the set of pairs (i, j) of a seed point i and a seed point j in the other image (here, target image B) that corresponds to the seed point i within a certain error is treated as a set of pairs (i, j).

[0047] In this embodiment, a predetermined key point matching process is performed to generate the cross-attention edge, and the correspondence between the key points of each image seen from different viewpoints, specifically, the affine transformation Φ that transforms the seed point of the source image A to the seed point of the target image B, is derived. A ε cros (l) Using this affine transformation Φ, (2) A ε cros (l) =∪ (i, j):|ΦAP_i-BP_j|≦ε (i, j) It is calculated as follows.

[0048] In the above formula (2), AP_i and BP_j are the position (coordinate value) of the seed point i in the source image A and the position (coordinate value) of the seed point j in the target image B, respectively. (i, j):|ΦAP_i-BP_j|≦ε denotes a union operator that forms a set containing all (index) pairs (i, j) between a seed point i in source image A and a seed point j in target image B that is mapped to this seed point i by an affine transformation Φ within a given error ε.

[0049] As specific examples of such pairs ((i, j): |ΦAP_i-BP_j|≦ε), FIG. 2 shows two seed points (circles) at both ends of line C, two seed points (diamonds) at both ends of line D, and two seed points (squares) at both ends of line E. Incidentally, in the specific example of FIG. 2, AdaLAM disclosed in Non-Patent Document 1 is used as a predetermined keypoint matching process (deriving process of affine transformation Φ) for generating edges for cross attention.

[0050] Furthermore, as another embodiment of the above (a2) of generating edges for cross attention, A ε cros (Target image B B ε cros It is also possible to use the matched keypoint received from the keypoint matching unit 11 as a seed point i, and to use the matched keypoint received from the keypoint matching unit 11 as a set of pairs (i, j) of the seed point i and a seed point j in the other image (target image B in this case) that corresponds to the seed point i with a similarity equal to or higher than a predetermined level. That is, in this embodiment, the cross-attention edges are generated based on visual similarity, and the calculation cost is smaller than that of the method of the above formula (2), and the cross-attention edges can be generated in a shorter calculation time.

[0051] Specifically, S(i, j) is the similarity score (similarity) between the descriptor of seed point i in source image A and the descriptor of seed point j in target image B, and MNN(AS) is an operator that receives a set of seed points i in source image A (AS) and derives a pair of seed points (i, j) that are in a nearest neighbor relationship, as follows: (3) A ε cros (l) ={(i, j)|(i, j)∈MNN(AS), S(i, j)>θ} Cross attention edge A ε cros (l) is calculated, where θ is a predefined similarity score threshold.

[0052] Above are the cross-attention edges for original image A. A ε cros We will explain the generation process of the cross-attention edge of the target image B. B ε cros It has been mentioned that the same can be generated, but as a modification, A ε cros (or B ε cros ) may be used as a common cross-attention edge in the source image A and the target image B. In this case, A ε cros and B ε cros Since it is not necessary to calculate both, it is possible to reduce the calculation cost of the graph generation process.

[0053] <Descriptor update means> As also shown in FIG. 1, in this embodiment, the descriptor update unit 13 includes a self-attention processing unit 131 and a cross-attention processing unit 132. Among them, the self-attention processing unit 131 initially receives a set of descriptors ( A D, B D), and for the received descriptor, the self-attention edge ( A ε self , B ε self ) to generate an updated self-attention processed descriptor.

[0054] Details of attention processing, including the self-attention processing performed here, can be found, for example, in the non-patent literature: Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN, Kaiser, L. ukasz, & Polosukhin, I., “Attention is All you Need”, In Advances in Neural Information Processing Systems (Vol. 30). Curran Associates, Inc.<https: / / proceedings.neurips.cc / paper / 2017 / file / 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf> , disclosed in 2017.

[0055] In this embodiment, first, as a preprocessing step of the self-attention processing, the Conv1D unit 131a of the self-attention processing unit 131 converts the received descriptor x_i of the original image A into a query Q i (=x_i), value V i (=x_i), and key K i (=x_i), and then perform convolution processing on these to obtain the feature quantities Q_i, V_i, and K_i, that is, (4) Q_i = CONV1D(Q i ), V_i=CONV1D(V i ), K_i=CONV1D(K i ) Generate.

[0056] Next, the Attention unit 131b of the self-attention processing unit 131 applies the self-attention edge input from the graph generating unit 12 to the received descriptor of the original image A. A ε self (l) We perform self-attention processing using

[0057] Specifically, for a descriptor x_i, A εself (l) The influence of the surrounding descriptor x_j defined by x_i is summed up with its weight (determined by the relationship of x_j to x_i) to create a message, m ε→i , as follows: (5) m ε→i =Σ j:(i,j)∈Aεself(l) V_j×Softmax j (Q_i, K_j) Here, Σ j:(i,j)∈Aεself(l) is, (i, j) is A ε self (l) The sum over j such that the elements of j is the softmax function with respect to j.

[0058] Next, the concatenation unit 131c, the MLP unit 131d, and the addition unit 131f of the self-attention processing unit 131 calculate the descriptor x_i (l) , as follows: (6) x_i (l) =x_i (l-1) +MLP(Concat(Q_i, m ε→i )) Here, as will be described in detail later, in this embodiment, the process set of the graph generation process and the descriptor update process is executed multiple times. The l in the superscript (l) in the above formula (6) indicates that the quantity to be subscripted is the quantity generated in the l-th descriptor update process (self-attention process).

[0059] In addition, in the above equation (6), x_i (l-1) is the descriptor updated by the (l-1)th process immediately before the current descriptor update process (self-attention process), which is the l (≧2)th process. (0) is set to the descriptor initially received (as external input). Also, Concat(Q_i, m ε→i ) is Q_i and m by the concatenation unit 131c.ε→i and MLP(Concat(Q_i, m ε→i )) represents the result of processing this concatenated result by a multilayer perceptron (MLP) in the MLP unit 131d.

[0060] Also, the updated descriptor x_i (l) As another preferred embodiment for deriving the l (≧1)-th self-attention process, the descriptor x_i (l) , as follows: (7) x_i (l) = α (l) ×x_i bias +β (l) ×x_i (l-1) +MLP(Concat(Q_i, m ε→i )) It is also preferable to calculate using the following formula:

[0061] where x_i bias is a stabilization bias term devised by the inventors of the present application, and is the updated descriptor x_i (l) In this embodiment, the calculation result of x_i bias is set to the descriptor received as external input, i.e., x_i (0) (x_i bias =x_i (0) ). This allows the descriptor x_i to be updated in the l(≧2)th self-attention process. (l) Specifically, the descriptor received as external input (i.e., x_i bias ) and the previous (l-1th) updated descriptor x_i stored in the memory section 131e. (1-1) and the output of MLP section 131d are passed to adder 131f for processing to generate the signal.

[0062] In addition, in the above formula (7), α (l) and β (l) are the stabilizing bias terms x_i biasand the weights of the previous updated descriptors, which are the parameters learned by training the entire model 1. Here, as a modification, MLP(Concat(Q_i, m ε→i )) term also has a weight γ (l) or weight β (l) Let {x_i (l-1) +MLP(Concat(Q_i, m ε→i It is also possible to multiply it by the {)} term.

[0063] The above describes the Conv1D unit 131a, Attention unit 131b, Concatenation unit 131c, MLP unit 131d, Memory unit 131e, and adder 131f of the self-attention processing unit 131, which sequentially processes the received descriptor of the original image A to generate an updated descriptor. Meanwhile, the Conv1D unit 131g, Attention unit 131h, Concatenation unit 131i, MLP unit 131j, Memory unit 131k, and adder 131l of the self-attention processing unit 131 also shown in Fig. 1 sequentially perform processes similar to those of the functional configuration units of the same names described above on the received descriptor of the target image B to generate an updated descriptor.

[0064] However, unlike the attention unit 131b, the attention unit 131h does not select the self-attention edge input from the graph generation unit 12. B ε self (l) Specifically, for the descriptor x_i of the target image B, B ε self (l) The influence of the surrounding descriptor x_j defined by x_i is summed up with its weight (determined by the relationship of x_j to x_i) to create a message, m ε→i , as follows: (8) m ε→i =Σ j:(i,j)∈Bεself(l) V_j×Softmax j (Q_i, K_j) Here, it is calculated by Σj:(i,j)∈Bεself(l) is, (i, j) is B ε self (l) The sum is the sum over j such that

[0065] Next, the cross-attention processing unit 132 of the descriptor updating unit 13 will be described.

[0066] Also in FIG. 1, in this embodiment, the cross-attention processing unit 132 receives the updated descriptor groups of each of the source image A and the target image B from the self-attention processing unit 131, and applies the cross-attention edges ( A ε cros , B ε cros ) to generate a further updated descriptor that has been cross-attention processed.

[0067] Specifically, the Conv1D unit 132a, Attention unit 132b, Concatenation unit 132c, MLP unit 132d, Memory unit 132e and adder 132f of the cross-attention processing unit 132 sequentially perform processing on the updated descriptor of the original image A received from the self-attention processing unit 131 in the same manner as the functional components of the same names (131a, 131b, 131c, 131d, 131e, 131f) described above in the self-attention processing unit 131, thereby generating a further updated descriptor.

[0068] However, unlike the Conv1D unit 131a, the Conv1D unit 132a converts the updated descriptor x_i of the received original image A into a query Q as preprocessing of the cross-attention processing. i (=x_i), and furthermore, the updated descriptor x_j of the target image B received from the self-attention processing unit 131 is set as a value V j (=x_j) and key K j(=x_j), and then perform convolution processing on these to obtain the features Q_i, V_j, and K_j, i.e. (9) Q_i = CONV1D(Q i ), V_j=CONV1D(V j ), K_j=CONV1D(K j ) Generate.

[0069] Next, the attention unit 132b applies the cross-attention edge input from the graph generation unit 12 to the updated descriptor of the original image A. A ε cros (l) Cross attention processing is performed using the following equation: (10) m ε→i =Σ j:(i,j)∈Aεcros(l) V_j×Softmax j (Q_i, K_j) By message (description information) m ε→i Calculate Σ j:(i,j)∈Aεcros(l) is, (i, j) is A ε cros (l) is the sum over j such that

[0070] Meanwhile, the Conv1D unit 132g, Attention unit 132h, Concatenation unit 132i, MLP unit 132j, Memory unit 132k, and adder 132l of the cross-attention processing unit 132 also shown in Figure 1 sequentially perform processing on the updated descriptor of the received target image B in the same manner as the functional components of the same names (132a, 132b, 132c, 132d, 132e, 132f) in the cross-attention processing unit 132 described above, to generate a further updated descriptor.

[0071] However, unlike the Conv1D unit 132a, the Conv1D unit 132g converts the updated descriptor x_i of the received target image B into a query Q as preprocessing of the cross-attention processing. i(=x_i), and further, the updated descriptor x_j of the original image A received from the self-attention processing unit 131 is set as a value V j (=x_j) and key K j (=x_j), and then perform convolution processing on these to obtain the features Q_i, V_j, and K_j, i.e. (11) Q_i = CONV1D(Q i ), V_j=CONV1D(V j ), K_j=CONV1D(K j ) Generate.

[0072] Next, the attention unit 132h applies the cross-attention edge input from the graph generation unit 12 to the updated descriptor of the target image B. B ε cros (l) Cross attention processing is performed using the following equation: (12) m ε→i =Σ j:(i,j)∈Bεcros(l) V_j×Softmax j (Q_i, K_j) By message (description information) m ε→i Here, we calculate Σ j:(i,j)∈Bεcros(l) is, (i, j) is B ε cros (l) is the sum over j such that

[0073] The above provides a detailed explanation of the self-attention processing unit 131 and the cross-attention processing unit 132 of the descriptor update unit 13. Here, as the descriptor update process, it is also possible to execute the "GAT process" as a process pair of the self-attention process by the self-attention processing unit 131 and the cross-attention process by the cross-attention processing unit 132 only once.

[0074] However, in order to generate a more descriptive descriptor, in this embodiment, the self-attention processing unit 131 receives the updated descriptor that has undergone the cross-attention processing, and performs the "GAT processing" for the processing pair L (≧2) times. A ε self (l) or x_i (l) The l in the superscript (l) of the above is an integer greater than or equal to 1 and less than or equal to L (l = 1, 2, ..., L).

[0075] Specifically, after the first processing (l=1) is completed, (a) The output from the adder 132f of the cross-attention processing unit 132 (the descriptor of the original image A that has been subjected to the cross-attention processing) is input to the Conv1D unit 131a of the self-attention processing unit 131, and the output from the adder 132l of the cross-attention processing unit 132 (the descriptor of the target image B that has been subjected to the cross-attention processing) is input to the Conv1D unit 131g of the self-attention processing unit 131 to perform the self-attention processing again; (b) After that, the output from the adder 131f of the self-attention processing unit 131 (the descriptor of the original image A that has been subjected to the self-attention processing) and the output from the adder 131l (the descriptor of the target image B that has been subjected to the self-attention processing) are input to the Conv1D units 132a and 132g of the cross-attention processing unit 132 to perform the cross-attention processing again. These processes (a) and (b) are repeated at least once ((L-1) times from the second time onwards).

[0076] Here, in the l-th self-attention process in (a) above, the self-attention edge ( A ε self (l) ,B ε self (l) ) is used.

[0077] In the l-th cross-attention process in (b) above, the cross-attention edge ( A ε cros (l) , B ε cros (l) That is, in this embodiment, a pair of processes is additionally performed at least once (L-1 times), the pair being a process of receiving an updated descriptor in the graph generating unit 12, updating a graph (edge) by the graph generating unit 12, and a process of further generating an updated descriptor by using the updated graph (edge) by the descriptor updating unit 13 (the self-attention processing unit 131 and the cross-attention processing unit 132).

[0078] This makes it possible to generate a more descriptive descriptor in which the information related to the degree of match between the positional relationships of repeatedly updated keypoints (nodes) and their surrounding conditions is successively embedded as updated edge information, and information that may cause mismatches is appropriately removed. Furthermore, by performing image matching processing using such a descriptor, it becomes possible to obtain image matching results that are more accurate and include a larger number of truly matched keypoints.

[0079] In addition, when the self-attention processing unit 131 and the cross-attention processing unit 132 perform the descriptor update process using the stabilization bias term (shown in the above formula (7)), in each iteration from the second iteration (l=2) onwards, the updated descriptor x_i (l-1) and the updated descriptor x_i generated in the first step (1) The stabilizing bias term x_i is (or is preset) bias and the generated message (description information) m ε→i Using the above, the updated descriptor x_i (l)As a result, during the dynamic update process, the updated descriptor x_i (l) This makes the derivation of the final output (x_i (L) ) can also be obtained.

[0080] In any case, the descriptor update unit 13 (image descriptor update model 1) finally updates the set of descriptors ( A D * , B D * These are more descriptive descriptors that can realize suitable image matching between source image A and target image B.

[0081] As described above, the descriptor update unit 13 (image descriptor update model 1) realizes updating of the descriptor using the GAT process, which requires relatively small calculation costs, without relying on a transformer, which has been a problem in the past due to its huge calculation costs. For this purpose, a graph for the GAT process is generated separately in advance by the graph generation unit 12. This makes it possible to perform the descriptor update process with reduced calculation costs.

[0082] Incidentally, the image descriptor update model 1, especially the descriptor update unit 13, is trained using a loss function of the output (updated) descriptor according to a method similar to normal backpropagation, for example, α (l) and β (l) This can be achieved by learning various parameters including the number of iterations L and the number of repetitions L. Note that the number of iterations L may be a preset value rather than being a learning target.

[0083] [How to update image descriptors] 3 is a flowchart outlining an image descriptor updating method according to an embodiment of the present invention. The process flow in steps S101 to S106 of this method will be described below.

[0084] (S101) Key points of source image A and target image B (A K, B K) and the descriptor group ( A D, B D) is obtained from an external source. (S102) A keypoint matching process is performed using the acquired image feature amounts.

[0085] Next, the graph generation and descriptor update process is performed by repeating steps S103 to S105 a predetermined number of times (L times). It has been confirmed that good results can be obtained by setting the predetermined number of times to, for example, 4 to 8 times (4≦L≦8). (S103) The first time, the keypoint matching result of step S102 is used, and from the second time onwards, the processing result (updated descriptor) of step S105 described below is used to generate a self-attention graph including self-attention edges and a cross-attention graph including cross-attention edges for each of the original image A and the target image B.

[0086] (S104) Using the generated self-attention graph, self-attention processing is performed on the descriptors of each of the source image A and the target image B. (S105) Cross-attention processing is performed on the descriptors of the source image A and the target image B that have been subjected to the self-attention processing, and updated descriptors are generated.

[0087] (S106) After the predetermined number of times (L times), the updated descriptors generated in step S105 are added to the updated descriptor set ( A D * , B D * )

[0088] [Image matching device / program] Returning to FIG. 1, the image matching device 9 of this embodiment shown in the lower part of the figure is as follows: (a) an input unit 91 for inputting an original image A and a target image B which are to be subjected to image matching; (b) From each of the input source image A and the target image B, a set of key points ( A K, B K) and the descriptor group ( A D, B D), and an image feature extraction unit 92 for extracting the image feature (c) Using the image descriptor update model 1 described in detail above, the extracted descriptor set ( A D, B D) and update the updated descriptors ( A D * , B D * ), and an image descriptor update unit 93 that outputs (d) Extracted keypoints ( A K, B K) and updated descriptors ( A D * , B D * ) to perform image matching processing between the source image A and the target image B; (e) an output unit 95 that provides the image matching result (a set of matching key points) of (d) to an external device; It has.

[0089] Here, the image feature extraction unit 92, the image descriptor update unit 93, and the image matching unit 94 are main functional components for implementing an embodiment of the image matching method according to the present invention, and further, are functions of a processor / memory that stores an embodiment of the image matching program according to the present invention. Therefore, the image matching device 9 may be a dedicated device for image matching processing, but it can also be, for example, a cloud server, a non-cloud server device, a personal computer (PC), a notebook or tablet computer, or a smartphone, etc., equipped with the image matching program according to the present invention.

[0090] Furthermore, the input unit 91 may be, for example, one that has a communication function and receives the original image A and the target image B from an external information processing device, for example, a camera image management server, or it may be an interface such as a USB (Universal Serial Bus) that allows the user to directly input the original image A and the target image B. The output unit 95 may also be, for example, one that has a communication function and transmits the image matching results to an external information processing device, for example, a camera image management server that is the image provider, or it may be a display that can display the image matching results.

[0091] The image feature extraction unit 92 can extract known image features such as SIFT or SuperPoint from the source image A and the target image B, or can extract a set of key points ( A K, B K) and the descriptor group ( A D, B D).

[0092] Furthermore, in this embodiment, the image matching unit 94 performs image matching on the extracted key points ( A K, B K) and updated descriptors ( A D * , B D * ) is used to implement a lightweight and high-speed keypoint matching method, such as nearest neighbor matching, and output the image matching result (a set of matching keypoints) between the source image A and the target image B. In this way, even with a lightweight and high-speed keypoint matching method, the descriptors used are those updated by the image descriptor update model 1 and have improved descriptive properties, making it possible to implement image matching with higher accuracy.

[0093] For example, as a typical example, even if the original image A and the target image B are camera images of a single building taken from different viewpoints and the lighting (sunlight) conditions, such as the amount of shading, are different, the image matching device 9 of this embodiment can successfully perform matching processing between both such images.

[0094] As described above in detail, the present invention makes it possible to update a descriptor using a GAT process that requires a relatively small computational cost, without relying on a transformer, which has been a problem in the past due to its huge computational cost. In other words, the present invention makes it possible to perform a descriptor update process that realizes a machine learning-based image matching process with a lower computational cost. As a result, such an image matching process can also be performed. Furthermore, this makes it possible to provide an image matching application that performs, for example, image search, visual search, and 3D image construction in real time.

[0095] Furthermore, the highly accurate image matching process of the present invention can be utilized in the analysis of a huge amount of camera images from numerous security cameras installed in a city, which can facilitate prediction and detection of trouble and crime occurrence, and even arrest of perpetrators, etc. In other words, the present invention can contribute to Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations.

[0096] Furthermore, the highly accurate image matching process of the present invention can be used to analyze satellite and aerial photographs of target areas, particularly agricultural areas, forest areas, and arid areas, as well as satellite and aerial photographs of target sea areas, to investigate various conditions in such areas and sea areas, such as the growth status of crops, the current state of ecosystems, and the impact of climate change. In other words, the present invention can contribute to Goal 13 "Take urgent action to combat climate change and its impacts," Goal 14 "Conserve and sustainably use the oceans and marine resources," and Goal 15 "Sustainably manage forests, combat desertification, halt and reverse land degradation, and halt biodiversity loss" in the Sustainable Development Goals (SDGs) led by the United Nations.

[0097] Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents. [Explanation of symbols]

[0098] 1 Image Descriptor Update Model 11 Keypoint Matching Section 12 Graph Generation Unit 13 Descriptor Update Section 131 Self-attention processing unit 131a, 131g, 132a, 132g Conv1D section 131b, 131h, 132b, 132h Attention part 131c, 131i, 132c, 132i Concatenation part 131d, 131j, 132d, 132j ​​MLP Department 131e, 131k, 132e, 132k Memory section 131f, 131l, 132f, 132l adders 132 Cross-attention processing unit 9. Image Matching Device 91 Input section 92 Image feature extraction unit 93 Image Descriptor Update Unit 94 Image Matching Section 95 Output section

Claims

1. 1. An image descriptor updating model for causing a computer to update a descriptor used to match two images, comprising: a graph generating means for receiving a set of keypoints and a set of descriptors for each of the two images, and generating a graph with the received keypoints as nodes and information relating to pairs of the keypoints selected as nodes with nearby keypoints in the same image and / or information relating to pairs of the keypoints selected as nodes with keypoints in the other image that correspond to the keypoints selected as nodes within a predetermined error or with a similarity equal to or greater than a predetermined value as edges; a descriptor update means for receiving a set of descriptors for each of the two images, and performing graph attention processing using the graph on the received descriptors to generate updated descriptors; The image descriptor updating model is characterized in that the image descriptor updating model is

2. 2. The image descriptor updating model according to claim 1, further comprising a computer that functions as a keypoint matching means for receiving a set of keypoints and a set of descriptors for each of the two images, performing a matching process for the two images, selecting matching keypoints from the received keypoints, and causing the graph generating means to receive the selected keypoints and their corresponding descriptors.

3. The image descriptor update model described in claim 1 or 2, characterized in that the descriptor update means performs the graph attention processing on the received descriptor multiple times, and in each processing from the second processing onwards, generates an updated descriptor for the current processing using the updated descriptor generated in the previous processing, the updated descriptor generated in the first processing or a stabilization bias term that is set in advance, and the description information generated by the graph attention processing.

4. The image descriptor update model according to any one of claims 1 to 3, characterized in that the image descriptor update model performs at least one process pair of a process of having the graph generation means receive the updated descriptor and updating a graph having the edge by the graph generation means, and a process of further generating an updated descriptor using the updated graph by the descriptor update means.

5. the graph generating means defines information relating to a pair of the received keypoint and a nearby keypoint in the same image as an edge for self-attention, and defines information relating to a pair of the received keypoint and a keypoint in another image that corresponds to the received keypoint within a predetermined error or with a similarity equal to or greater than a predetermined degree as an edge for cross-attention; The descriptor update means a self-attention processing means for performing a self-attention process, which is a graph attention process using the self-attention edges, on the received descriptor for each of the two images to generate a self-attention processed descriptor; a cross-attention processing means for performing a cross-attention process, which is a graph attention process using the cross-attention edges, on the self-attention processed descriptor for each of the two images to generate an updated cross-attention processed descriptor; and performing the graph attention processing as a processing pair of the self-attention processing and the cross-attention processing once, or performing the graph attention processing as the processing pair a plurality of times by having the self-attention processing means receive an updated descriptor that has been subjected to the cross-attention processing. An image descriptor updating model according to any one of claims 1 to 4.

6. an image feature extraction means for extracting a set of key points and a set of descriptors from each of the two images; an image descriptor updating means for updating the extracted descriptors using the image descriptor updating model according to any one of claims 1 to 5; an image matching means for matching the two images using the extracted keypoints and the updated descriptors; 1. An image matching device comprising:

7. an image feature extraction means for extracting a set of key points and a set of descriptors from each of the two images; an image descriptor updating means for updating the extracted descriptors using the image descriptor updating model according to any one of claims 1 to 5; an image matching means for matching the two images using the extracted keypoints and the updated descriptors; The image matching program causes a computer to function as follows.

8. A computer-implemented image descriptor updating method for updating a descriptor used to match two images, comprising: receiving a set of keypoints and a set of descriptors for each of the two images, and generating a graph in which the received keypoints are treated as nodes and information relating to pairs of the keypoints treated as nodes with nearby keypoints in the same image and / or information relating to pairs of the keypoints treated as nodes with keypoints in the other image that correspond to the keypoints treated as nodes within a predetermined error or with a similarity equal to or greater than a predetermined degree are treated as edges; receiving a set of descriptors for each of the two images, and performing a graph attention process using the graph on the received descriptors to generate updated descriptors; 13. A method for updating an image descriptor, comprising:

Images