Coarse-to-fine image dense matching method, system, device and storage medium

Abstract

The present invention discloses a method, system, device and storage medium for dense image matching from coarse to fine. During coarse matching, RANSAC is used to fit candidate sparse pairs between source and target images to obtain a homography estimation matrix, and global registration is performed to complete the rough positioning of the object, overcoming the problem of inaccurate matching caused by geometric transformation differences such as scale, rotation and perspective; for the image after coarse matching, an optical flow field prediction of the matching area is generated through an optical flow estimation network to complete the matching of object details, overcoming the influence of different colors, appearances and textures. The above-mentioned dense image matching scheme from coarse to fine has good generalization ability, and can not only solve the dense matching problem of weak texture, no texture, repeated texture, large parallax, and large foreground interference image pairs, but also can achieve pixel-level alignment on image pairs to be registered with large intra-class differences and different modalities, which fully proves that the method of the present invention has good application prospects and application value.

Description

Technical Field

[0001] The present invention relates to the technical field of dense image matching, and in particular to a method, system, device and storage medium for dense image matching from coarse to fine. Background Art

[0002] Image matching technology is one of the important directions of computer vision. Its purpose is to quickly and accurately establish the correspondence between similar objects or scenes, and transfer the information of known images to new images through estimated correspondence. This can not only directly obtain various information of new images, but also assist the execution of other visual tasks.

[0003] According to the matching method, image matching technology can be roughly divided into two categories: image feature point matching and image dense matching. Image feature point matching first extracts feature points in the image, and then seeks matching corresponding points in another image to establish the correspondence between the two images. Since image feature point matching can only estimate the overall similarity between images, but cannot determine the local correspondence, it is difficult to transfer the information of a known image to a new image. In order to obtain a more accurate correspondence, image dense matching came into being. The main purpose of image dense matching is to establish a pixel-level matching relationship between images, seek the corresponding pixel of each pixel in the source image in the target image, and thus transfer the attributes of the known image to the new image.

[0004] Among the existing dense image matching methods, the patent "A Dense Matching Method Suitable for Deformed Images" (publication number CN106023230A) uses polynomial correction and coordinate correspondence preservation mechanisms for dense matching of deformed images based on the manual extraction of matching feature point pairs. This method has high labor costs and poor robustness, which limits the application of image matching technology; the patent "A Dense Matching Method and System Based on Sparse Matching and Image Edges" (publication number CN106548482A) combines edge images obtained by edge detection and feature point disparity values ​​obtained by sparse matching to perform image matching. It cannot accurately find pixel matching points in weak texture areas, and the detail matching effect is poor; the patent "An Image Optical Flow Estimation Method and System Based on Non-rigid Dense Matching" (publication number CN108986150A) performs dense block matching on two frames of images, and predicts the motion information of the object by calculating the non-rigid dense motion field and variational optical flow between images. This method consumes a lot of time and space, and has poor generalization ability for large disparity and large displacement image pairs in real scenes.

[0005] There are many urgent problems to be solved in the above-mentioned existing dense image matching methods: 1) How to accurately calculate the dense matching relationship between images under large-scale spatial transformation; 2) How to effectively perceive the structural information of the object to improve robustness; 3) How to get better performance on image details; 4) How to overcome the semantic gap and achieve pixel-level matching on different domain images with large differences in content, grayscale, texture, etc. Therefore, designing a set of technical solutions to improve the accuracy and robustness of image dense matching has important practical value and realistic significance. Summary of the invention

[0006] The purpose of the present invention is to provide a method, system, device and storage medium for dense image matching from coarse to fine, which can realize dense image matching at different scales, different angles and different scenes, correctly estimate the correspondence of each pixel in the image, and improve the robustness and generalization ability of dense image matching.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A method for dense image matching from coarse to fine, characterized by comprising:

[0009] Global registration stage: Processing multi-scale feature matching point pairs of the source image and the target image by a random sampling consensus algorithm, transforming the source image by using the homography estimation matrix obtained by the processing, and obtaining a transformed source image;

[0010] Pixel matching stage: The optical flow field of the transformed source image and the target image is predicted through the optical flow estimation network, and the optical flow field is used to perform optical flow mapping on the transformed source image to achieve dense image matching.

[0011] A coarse-to-fine image dense matching system is implemented based on the above method, and the system includes:

[0012] A coarse matching module for global registration is applied to the global registration stage, processes multi-scale feature matching point pairs of the source image and the target image through a random sampling consensus algorithm, and transforms the source image using the homography estimation matrix obtained through the processing to obtain a transformed source image;

[0013] The pixel matching fine alignment module is applied to the pixel matching stage, predicts the optical flow field of the transformed source image and the target image through the optical flow estimation network, and uses the optical flow field to perform optical flow mapping on the transformed source image to achieve dense image matching.

[0014] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0016] A readable storage medium stores a computer program, characterized in that the aforementioned method is implemented when the computer program is executed by a processor.

[0017] It can be seen from the technical solution provided by the present invention that the hierarchical matching strategy from coarse matching to fine alignment is used to estimate the optical flow field, which overcomes the limitation that geometric transformation cannot perform pixel alignment and optical flow registration cannot solve large displacement, making dense matching more accurate and robust; specifically, during coarse matching, RANSAC (random sampling consensus algorithm) is used to fit the candidate sparse pairs between the source image and the target image to obtain the homography estimation matrix, and global registration is performed to complete the rough positioning of the object, overcoming the inaccurate matching problem caused by geometric transformation differences such as scale, rotation and perspective; given two roughly aligned images, the optical flow field prediction of the matching area is generated through the optical flow estimation network, the matching of the object details is completed, and the influence of different colors, appearances and textures is overcome. The above-mentioned coarse-to-fine image dense matching scheme has good generalization ability, not only can it solve the dense matching problem of weak texture, no texture, repeated texture, large parallax, and large foreground interference image pairs, but also can achieve pixel-level alignment on the image pairs to be registered with large intra-class differences and different modalities, which fully proves that the method of the present invention has good application prospects and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 A processing flow chart of a method for dense image matching from coarse to fine provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of an overall process of dense image matching from coarse to fine provided by an embodiment of the present invention;

[0021] Figure 3 A diagram of a coarse-to-fine image dense matching training process provided by an embodiment of the present invention;

[0022] Figure 4 An illustration of a coarse-to-fine image dense matching training data set provided by an embodiment of the present invention;

[0023] Figure 5A schematic diagram of a coarse-to-fine image dense matching result provided by an embodiment of the present invention;

[0024] Figure 6 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0026] First, the terms that may be used in this article are explained as follows:

[0027] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0028] The following is a detailed description of a coarse-to-fine image dense matching scheme provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer are followed. If the manufacturer of the reagents or instruments used in the embodiments of the present invention is not specified, they are all conventional products that can be purchased commercially.

[0029] Embodiment 1

[0030] The embodiment of the present invention provides a coarse-to-fine image dense matching method, which solves the problem of dense matching of images at different scales, different angles, and different scenes, correctly estimates the correspondence between each pixel in the image, and improves the robustness and generalization ability of the model. The method mainly includes two stages:

[0031] 1) Global registration stage: The multi-scale feature matching point pairs of the source image and the target image are processed by the random sampling consensus algorithm (RANSAC), and the source image is transformed using the homography estimation matrix obtained by the processing to obtain the transformed source image.

[0032] 2) Pixel matching stage: The optical flow field of the transformed source image and the target image (i.e., the relative displacement of each pixel between the two images) is predicted through the optical flow estimation network, and the optical flow field is used to perform optical flow mapping on the transformed source image to achieve dense image matching.

[0033] In the embodiment of the present invention, in order to ensure the dense matching effect of the image, the optical flow estimation network is trained in an end-to-end manner. During the training, the structural similarity measurement and image consistency between the distorted source image pixels and the target image are optimized through self-supervised training, and the alignment can be refined to the pixel level without relying on the constant brightness assumption; at the same time, in order to accelerate the convergence of the model and measure the alignment effect of the image at the pixel level, the matching mask is used as the pixel-level weight of the reconstruction loss and the consistency loss. The matching mask mainly measures the alignment effect of the image at the pixel level and serves as the weight of the unaligned pixels. After the training is completed, in the pixel matching stage, the pixel-level alignment transformation optical flow field is directly used to perform optical flow mapping (such as a bilinear interpolation algorithm) on the transformed source image, and a pixel-level correspondence between the source image and the target image is established, thereby completing the dense matching of the image.

[0034] In the embodiment of the present invention, the optical flow field is a matrix data, and its size is the same as that of the source image and the target image. The value of each position element in the optical flow field represents the relative displacement of the pixels at the corresponding position of the transformed source image and the target image.

[0035] like Figure 1 As shown, a processing flow of a coarse-to-fine image dense matching method is demonstrated.

[0036] For ease of understanding, the preferred implementation methods of the above two stages and the model training process are described in detail below.

[0037] 1. Global registration.

[0038] In the embodiment of the present invention, a convolutional neural network is used to construct a source image I s and the target image I t In order to better integrate local information and global information, the source image I is fixed. s and the target image I t The aspect ratio is scaled at multiple scales, features are extracted at multiple scales and dimensionally transformed; the multiple scale features after dimension transformation are concatenated to obtain a feature matrix, and the dot product of the feature matrix is ​​performed to obtain the index value with the largest row and column values, that is, the feature matching point pair.

[0039] After that, the source image is geometrically transformed by using the random sampling consensus algorithm (RANSAC) to obtain the homography estimation transformation matrix. s ′ and target image It Input together into the optical flow network for fine alignment.

[0040] 2. Pixel matching.

[0041] The overall process can be described as: using the optical flow estimation network to predict the optical flow field of the transformed source image and the target image, using the optical flow field to perform optical flow mapping on the transformed source image, establishing a pixel-level correspondence between the source image and the target image, and thus completing image dense matching.

[0042] Specifically, the pixel matching stage includes two parts: optical flow estimation and optical flow mapping. Optical flow estimation calculates the relative displacement of each pixel between two images with relative displacement, while optical flow mapping uses optical flow to map the source image in the image to be registered to the target image to complete fine alignment.

[0043] 1) Optical flow estimation.

[0044] In order to better integrate the feature map information between image pairs in optical flow estimation, a similarity tensor is calculated to quantify the correlation between two sets of features: the transformed source image I s ′ and target image I t After a full convolution layer, feature extraction is performed to obtain the corresponding feature map. The transformed source image I s The feature map of ′ is recorded as f s , target image I t The feature map is denoted as f t ; For the feature map f s With f t Normalize and get the normalized feature map f s ′ and f t ′; The normalized feature map f s The vector f at each position in ′ s ′(x′, y′) and the normalized feature map f t The vector f in the domain D = 2k+1 in ′ t ′(x, y) is dot-producted to get a similar tensor of size w×h×(2k+1) 2 ; where k is a coefficient used to ensure that the domain D (i.e., D*D range) is an odd number, w and h are the width and height of the feature map, respectively, and (x, y) is the target image I t The position coordinates of the pixel in, (x′, y′) are the transformed source image I s The position coordinates of the corresponding pixels in ′; the similarity tensor is passed through the two branches of the full convolution prediction network to obtain the optical flow field and the matching mask. The training process is the same as the use process, except that the matching mask is no longer required when using it.

[0045] 2) Optical flow mapping.

[0046] The optical flow mapping method during training and use is the same, both of which use the optical flow field to perform optical flow mapping on the transformed source image obtained by global registration to establish a pixel-level correspondence between the source image and the target image.

[0047] From the introduction of the optical flow estimation section above, we can know that the same pixel in the transformed source image I s ′ and target image I t The position coordinates of are different, which are (x′, y′) and (x, y) respectively. Therefore, by using the optical flow field to transform the transformed source image I s The pixel position coordinates in ′ are mapped from (x′, y′) to (x, y). At the same time, the size of the original input source image is the same as that of the target image. Therefore, the value range of (x′, y′) and (x, y) is exactly the same. The optical flow mapping is expressed as:

[0048] (x, y) = F s→t (x′, y′)

[0049] Through the above formula, the transformed source image I s ′, find the image that matches the target image I t The pixel corresponding to the pixel in (position coordinates are (x′, y′)) is mapped to the pixel corresponding to the target image I t Aligned position (x, y); in the above formula, F s→t The optical flow mapping function uses the optical flow field F to transform the transformed source image I s ′Move pixel by pixel to the target image I t The position of the corresponding pixel, the source image of the same pixel after optical flow mapping With the target image I t The position coordinates are the same, so they can be uniformly expressed as (x, y).

[0050] like Figure 2 As shown in Figure 1, the overall process of dense image matching from coarse to fine is shown. Figure 2 In the above figure, CNN is a deep neural network, H represents the homography estimation matrix, the coarse matching module of global registration executes the process of the global registration stage, and the fine alignment module of pixel matching executes the process of the pixel matching stage.

[0051] 3. Model training.

[0052] In the embodiment of the present invention, the model mainly refers to a network model for realizing dense image matching, which is called an image dense matching model, and mainly includes a convolutional neural network used in the global registration stage and an optical flow estimation network used in the pixel matching stage, both of which are deep neural networks.

[0053] In the embodiment of the present invention, the convolutional neural network can be implemented using the ResNet-50 network pre-trained on the ImageNet data, and the corresponding features can be directly obtained through the conv4 layer of the pre-trained ResNet-50 network, and there is no need to train and update the model parameters of the network. That is, the model training is mainly performed on the optical flow estimation network.

[0054] In the embodiment of the present invention, the loss function combines the reconstruction loss and the image consistency loss as an unsupervised loss function to train the optical flow estimation network. Figure 3 The main training process is shown. The optical flow estimation network is trained in an end-to-end manner. The input is two images that have been globally registered, that is, the transformed source image and the target image. The output is the optical flow field of the two images. The registration result is obtained by aligning the source image to the target image through optical flow mapping. In order to make the network model converge faster and better, a matching mask M is introduced to measure the pixel-level alignment effect of the image. t (x, y) = M t→s (x,y)M s→t (x′, y′) is used as the weight value of the loss function. The matching mask is a matrix with the same pixel structure as the image. It is obtained by calculating the similarity between the transformed source image and the target image. s→t (x′, y′) represents the transformed source image I s ′ to the target image I t The matching degree, M t→s (x, y) represents the target image I t To the transformed source image I s The closer the matching degree is to 1, the better the pixel alignment effect is.

[0055] In the embodiment of the present invention, the loss function calculates the reconstruction loss and the image consistency loss after the pixels of two images are matched.

[0056] 1) The reconstruction loss minimizes the brightness difference between the two images and penalizes the sudden change of optical flow, which is expressed as:

[0057]

[0058] The above formula minimizes the brightness difference between the two images after registration and penalizes the sudden change of optical flow to ensure the consistency of the image pair. Represents the source image after optical flow mapping With the target image I t The structural similarity of (x, y) represents the position coordinates of the pixel, SSIM is a structural similarity index ranging from -1 to 1. When the two images are equal, the SSIM value is 1.

[0059] 2) Image consistency loss is used to smooth local edges while keeping the image structure aligned, expressed as:

[0060]

[0061] Among them, ||.||2 represents the L2 norm.

[0062] 3) The loss function is a combination of the above two losses, expressed as:

[0063]

[0064] Among them, α1 is the image consistency loss The weight of .

[0065] Using the above loss function The parameters of the optical flow estimation network are optimized. The optimization process can be implemented by referring to conventional techniques, which will not be described in detail in the present invention.

[0066] The above scheme of the embodiment of the present invention adopts a hierarchical matching strategy from coarse matching to fine alignment to estimate the optical flow field, overcomes the limitation that geometric transformation cannot perform pixel alignment and optical flow registration cannot solve large displacement, and makes dense matching more accurate and robust; specifically, during coarse matching, RANSAC (random sampling consensus algorithm) is used to fit the candidate sparse pairs between the source image and the target image to obtain the homography estimation matrix, and global registration is performed to complete the rough positioning of the object, overcoming the inaccurate matching problem caused by geometric transformation differences such as scale, rotation and perspective; given two roughly aligned images, the optical flow field prediction of the matching area is generated through the optical flow estimation network, the matching of the object details is completed, and the influence of different colors, appearances and textures is overcome. The above-mentioned coarse-to-fine image dense matching scheme has good generalization ability, not only can solve the dense matching problem of weak texture, no texture, repeated texture, large parallax, and large foreground interference image pairs, but also can achieve pixel-level alignment on the image pairs to be registered with large intra-class differences and different modalities, which fully proves that the method of the present invention has good application prospects and application value.

[0067] Based on the above introduction, the following is a comprehensive description, which includes four parts: dataset construction, image dense matching method, model training, and model use. It is worth noting that the image content, size, various network parameters, and various parameters set during training involved in the following comprehensive description are all examples.

[0068] 1. Dataset construction.

[0069] Since the entire image dense matching model only needs to train the optical flow estimation network, and the optical flow estimation method needs to meet the small displacement assumption, that is, the local optical flow value should not be too large, the rough positioning of the image pair to be registered in the global registration stage is used. By setting a threshold for the number of matching feature points, if the matching feature points of the image pair to be registered after feature extraction and feature matching in the global registration stage are higher than the threshold, it will be geometrically transformed and used as the training set and verification set of the optical flow estimation network, otherwise it will be discarded.

[0070] As an example, the image pairs to be registered come from an image stitching dataset UDIS-D obtained from variable motion videos. By extracting video frames at different intervals, image pairs with different overlap rates (displacement sizes) are obtained, such as Figure 4 Part (b) of the dataset shows examples of image pairs with different overlap rates. Moreover, these videos are not shot by a camera rotating around the optical center, and the scenes shot are far from being planar structures, which means that the dataset contains different degrees of parallax, such as Figure 4 Part (c) of the figure shows examples of image pairs with different parallaxes. In addition, the image stitching dataset includes variable scenes such as indoor, outdoor, night, dark, snowy, and zoomed scenes. Figure 4 As shown in part (a), it is an example of an image pair of a variable scene. The threshold can be set to 100, and a total of 10440 pairs of training sets and 1106 pairs of validation sets are obtained. The convolutional neural networks involved in the present invention are all based on three-dimensional data, so there is no need to change the original image in dimension. Because the grayscale distribution between images is uneven, the present invention normalizes the image, performs a linear transformation on the grayscale value in the image, and maps the image grayscale value to the interval [0, 1].

[0071] 2. Dense image matching method.

[0072] The overall process can also be seen Figure 1 Image preprocessing mainly involves scaling the source image and the target image to a certain minimum size and then sending them to the convolutional neural network for sampling at different scales to obtain a feature matrix. The large-scale features contain the global information description of the image, while the small-scale features contain the description of the image detail features. Combining these features can better match the feature points between image pairs. Then, the classic random sampling consensus algorithm (RANSAC) is used to analyze a set of corresponding feature points, and the geometric transformation matrix between the image pairs to be registered is fitted. Finally, the source image is mapped to the target image through the interpolation algorithm to complete the global registration and obtain a rough matching image pair.

[0073] As an example, the conv4 layer of the ResNet-50 network pre-trained on ImageNet data is used as the downsampling network, the minimum size is 480, the scale factors of different scales are [2.0, 1.66, 1.33, 1.0, 0.83, 0.66, 0.5], the downsampling multiple is 16, the geometric transformation matrix is ​​the homography estimation matrix, and the interpolation algorithm is bilinear interpolation.

[0074] The optical flow estimation network receives two roughly matched image pairs with relative displacement as input. It first extracts features through four convolution blocks (full convolution layers). Each convolution block contains a convolution layer, a batch normalization layer, and a Relu activation function. The similarity tensor obtained by performing a dot product operation on the normalized feature vector is sent to the full convolution prediction network to obtain a dense optical flow field with a certain downsampling multiple.

[0075] As an example, the convolution block for feature extraction is similar to the conv3 feature extractor of ResNet-18. The difference is that the first 7×7 convolution kernel of the network is replaced by a 3×3 convolution kernel without stride. In order to minimize the loss of spatial resolution in the network, the output feature map is 1 / 8 of the input image resolution. The fully convolutional network consists of three convolution blocks and one convolution layer.

[0076] 3. Dense matching model training.

[0077] like Figure 3 As shown in the figure, after the source image and the target image are densely matched, the transformed optical flow field of the two images is obtained (that is, the relative displacement of each pixel between the images to be registered), and then the two registered images are obtained through optical flow mapping. and I t The unsupervised loss function does not use any true value labels, and directly calculates the reconstruction loss and image consistency loss between the two registered images. Finally, the loss function is derived layer by layer through the chain rule to iteratively update the parameters of the network layer in the dense matching model. Among them, the image reconstruction loss minimizes the brightness difference between the two images after registration and penalizes the sudden change of optical flow to ensure the structural similarity of the image pair, and the consistency loss makes the local smoothness keep the edge of the image structure aligned.

[0078] By using the constructed data set and the deep learning framework, the corresponding hyperparameters are set, and the loss function value is reduced and the model converges through the back-propagation algorithm and gradient descent strategy, and finally a trained image dense matching model is obtained.

[0079] As an example, the loss function of the network model is divided into two parts, namely the image reconstruction loss and image consistency loss The total loss function is

[0080]

[0081] Among them, α1 is the weight corresponding to the image consistency loss, the parameter is set to 1, bilinear interpolation is used for upsampling in optical flow mapping, the network optimization method selects stochastic gradient descent, the learning rate is 1e-4, the optimizer is Adam, the weight decay coefficient is set to 1e-5, and a total of 200 epochs are trained.

[0082] 4. Use of dense image matching model.

[0083] The constructed source image and target image are sent to the trained image dense matching model to obtain the homography estimation matrix and dense transformation optical flow field of the image pair to be registered for global registration and pixel matching. This process does not require the use of the matching mask M in the fine alignment module of pixel matching. t (x, y).

[0084] In order to better verify the generalization ability and application prospects of the method of the present invention, multiple image matching datasets of real complex scenes are fine-tuned based on the UDIS-D dataset training model, and the visualization results of the corresponding scene test sets are obtained on the fine-tuned model.

[0085] As an example, we performed model fine-tuning and forward inference on a cross-domain image dataset with large differences in content, grayscale, texture, etc. to obtain experimental results of dense image matching, such as Figure 5 As shown, in order to well demonstrate the overall process and each module effect of the inventive method, Figure 5 From left to right, each column corresponds to the source image, the target image, the coarse matching image pair fusion of global registration, the fine alignment image pair fusion of pixel matching, and the visualization result of the unmatched image pair fusion of the original input, where the image pair fusion method is to weight the pixels of the two images by 0.5.

[0086] Embodiment 2

[0087] The present invention also provides a coarse-to-fine image dense matching system, which is mainly implemented based on the method provided in the above embodiment 1, see Figure 1 and Figure 2 The system mainly includes:

[0088] A coarse matching module for global registration is applied to the global registration stage, processes multi-scale feature matching point pairs of the source image and the target image through a random sampling consensus algorithm, and transforms the source image using the homography estimation matrix obtained through the processing to obtain a transformed source image;

[0089] The pixel matching fine alignment module is applied to the pixel matching stage, predicts the optical flow field of the transformed source image and the target image through the optical flow estimation network, and uses the optical flow field to perform optical flow mapping on the transformed source image to achieve dense image matching.

[0090] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0091] In addition, the relevant technical details involved in the above two modules in the system have been introduced in detail in the above-mentioned embodiment 1, so they are not repeated here.

[0092] Embodiment 3

[0093] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided by the aforementioned embodiments.

[0094] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0095] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0096] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0097] The output device may be a display terminal;

[0098] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0099] Embodiment 4

[0100] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0101] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0102] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A coarse-to-fine image dense matching method, characterized in that: include: Global registration stage: Processing multi-scale feature matching point pairs of the source image and the target image by a random sampling consensus algorithm, transforming the source image by using the homography estimation matrix obtained by the processing, and obtaining a transformed source image; Pixel matching stage: The optical flow field of the transformed source image and the target image is predicted by the optical flow estimation network, and the optical flow field is used to perform optical flow mapping on the transformed source image to achieve dense image matching; The pixel matching stage includes: using the optical flow estimation network to predict the optical flow field and matching mask of the transformed source image and the target image, and using the optical flow field to perform optical flow mapping on the transformed source image; the matching mask is obtained by calculating the similarity between the transformed source image and the target image, and is used to measure the alignment effect of the image at the pixel level, and is used as the weight of the misaligned pixel points to participate in the calculation of the loss function; The step of using the optical flow estimation network to predict the optical flow field and the matching mask of the transformed source image and the target image comprises: The transformed source image I s ′ and target image I t After a full convolution layer, feature extraction is performed to obtain the corresponding feature map. The transformed source image I s The feature map of ′ is recorded as f s , target image I t The feature map is denoted as f t ; For the feature map f s With f t Normalize and get the normalized feature map f s ′ and f t ′; The normalized feature map f s The vector f at each position in ′ s ′(x′, y′) and the normalized feature map f t The vector f in the domain D = 2k+1 in ′ t ′(x, y) is dot-producted to get the similarity tensor; k is a coefficient to ensure that the domain D is an odd number; (x, y) is the target image I t The position coordinates of the pixel in the image, (x′, y′) are the transformed source image I s ′The position coordinates of the corresponding pixels in ; The similarity tensor is passed through two branches of the fully convolutional prediction network to obtain the optical flow field and the matching mask.

2. The method for dense image matching from coarse to fine according to claim 1, characterized in that: Before processing the multi-scale feature matching point pairs of the source image and the target image by the random sampling consensus algorithm, the steps of multi-scale feature extraction and feature matching are also included. The related steps include: Constructing source image I using convolutional neural network s and the target image I t The feature matrix of the fixed source image I s and the target image I t The aspect ratio is scaled at multiple scales, features are extracted at multiple scales, and dimension transformation is performed; The multiple scale features after dimension transformation are concatenated to obtain a feature matrix. After performing dot product on the feature matrix, the index value with the largest row and column values ​​is obtained, which is the feature matching point pair.

3. The method for dense image matching from coarse to fine according to claim 1, characterized in that: The optical flow mapping of the transformed source image using the optical flow field is expressed as: (x,y)=F s→t (x′,y′) The same pixel in the transformed source image I s ′ and target image I t The position coordinates of are different, and are recorded as (x′, y′) and (x, y) respectively. Through the above formula, in the transformed source image I s ′ to find the image I t The pixels in the image are mapped to the pixels corresponding to the pixels in the target image I t The position coordinates of the corresponding pixel (x, y); in the above formula, F s→t The optical flow mapping function uses the optical flow field F to transform the transformed source image I s ′Move pixel by pixel to the target image I t The position of the corresponding pixel.

4. The method for dense image matching from coarse to fine according to claim 1, characterized in that: The optical flow estimation network is trained in an end-to-end manner, with the input being two images that have been globally registered, i.e., the transformed source image and the target image, and the output being the optical flow field and the matching mask of the two images; The loss function calculates the reconstruction loss and image consistency loss after pixel matching of two images; the reconstruction loss minimizes the brightness difference between the two images and penalizes the sudden change of optical flow, and the image consistency is used to keep the edges of image structure aligned; the loss function is used as an unsupervised loss function to train the optical flow estimation network.

5. The method for dense image matching from coarse to fine according to claim 4, characterized in that: The loss function is expressed as: in, To rebuild the losses, is the image consistency loss, α1 is the image consistency loss The weight of Reconstruction losses It is expressed as: Among them, M t (x, y) represents the matching mask, which has the same structure as the image pixels. Represents the source image after optical flow mapping With the target image I t The structural similarity of the same pixel in the source image after optical flow mapping With the target image I t The position coordinates are the same, expressed as (x, y); Image consistency loss It is expressed as: Among them, ||.||2 represents the L2 norm, F s→t The optical flow mapping function uses the optical flow field F to transform the transformed source image I s ′Move pixel by pixel to the target image I t The position of the corresponding pixel, the same pixel in the transformed source image I s ′ and target image I t The position coordinates are different, namely (x′, y′) and (x, y).

6. A coarse-to-fine image dense matching system, characterized in that: The method according to any one of claims 1 to 5 is implemented, and the system comprises: A coarse matching module for global registration is applied to the global registration stage, processes multi-scale feature matching point pairs of the source image and the target image through a random sampling consensus algorithm, and transforms the source image using the homography estimation matrix obtained through the processing to obtain a transformed source image; The pixel matching fine alignment module is applied to the pixel matching stage, predicts the optical flow field of the transformed source image and the target image through the optical flow estimation network, and uses the optical flow field to perform optical flow mapping on the transformed source image to achieve dense image matching.

7. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images