Image processing method, device, apparatus and storage medium

Abstract

Embodiments of the present application provide an image processing method, device and equipment and storage medium, relating to the technical field of image processing. In the embodiments of the present application, the spatial transformation relationship between two images can be obtained by firstly estimating the coarse geometric transformation parameters between the two images and then estimating the fine geometric transformation parameters between the two images. Through experiments, it is found that when the method is configured as an algorithm, the algorithm has good generalization ability. Furthermore, the coarse geometric transformation parameters and the fine geometric transformation parameters obtained based on the method are used for image registration. The estimation of the spatial transformation between two images with large appearance difference and noise is more accurate, the accuracy of image registration is improved, and the effect of image registration is better.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, an image processing apparatus, an electronic device, and a storage medium. Background Technology

[0002] Image registration refers to the process of spatially aligning corresponding features of one image with those of the other, given two images, through spatial transformation. Image registration has wide applications in fields such as aerial image processing, computer image processing, facial recognition technology, autonomous driving, and medicine.

[0003] However, in practical applications, the estimation accuracy of the spatial transformation between two images with significant differences in appearance due to clutter is not high, resulting in poor image registration. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide an image processing method that overcomes or at least partially solves the above problems, in order to solve the problem of poor image registration effect for two images with large appearance differences in the presence of clutter.

[0005] Accordingly, embodiments of the present invention also provide an image processing apparatus, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0006] To address the above problems, embodiments of the present invention disclose an image processing method, comprising:

[0007] Acquire a first input image and a second input image, and perform coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature;

[0008] Feature matching is performed on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image;

[0009] Based on the coarse feature point matching pairs and their feature values, estimate the coarse geometric transformation parameters from the first input image to the second input image;

[0010] The first input image is geometrically transformed according to the coarse geometric transformation parameters to obtain the third input image;

[0011] Fine-grained features are extracted from the third input image and the second input image respectively to obtain the first fine-grained feature and the second fine-grained feature;

[0012] Perform feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image;

[0013] Based on the fine feature point matching pairs and their feature values, the fine geometric transformation parameters from the third input image to the second input image are estimated.

[0014] Optionally, the step of performing coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature includes:

[0015] The first input image and the second input image are respectively input into the first convolutional neural network for coarse feature extraction, and the first initial feature image and the second initial feature image are output.

[0016] Each subject in the first initial feature image and the second initial feature image is cropped to obtain a first cropped feature image and a second cropped feature image;

[0017] The first cropped feature image and the second cropped feature image are respectively subjected to L2 norm normalization to obtain the first coarse feature and the second coarse feature.

[0018] Optionally, the step of performing feature matching on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image includes:

[0019] The first coarse feature is represented as a first coarse feature tensor, the second coarse feature is represented as a second coarse feature tensor, and the first coarse feature tensor and the second coarse feature tensor are input into the first feature matching network;

[0020] Calculate the dot product between the first coarse feature tensor and the second coarse feature tensor to obtain the first similarity tensor between the first coarse feature and the second coarse feature;

[0021] Based on the first similarity tensor, a coarse feature point matching pair between the first input image and the second input image is determined;

[0022] The coarse features of each coarse feature point matching pair are regularized.

[0023] The coarse features after regularization are normalized using the L2 norm to obtain the feature values ​​of the coarse feature point matching pairs, which are then output through the first feature matching network.

[0024] Optionally, estimating the coarse geometric transformation parameters from the first input image to the second input image based on the coarse feature point matching pairs and their feature values ​​includes:

[0025] The feature values ​​of the coarse feature point matching pairs are input into a first regression network to estimate the coarse geometric transformation parameters from the first input image to the second input image through the first regression network, and the coarse geometric transformation parameters are output; the coarse geometric transformation parameters include affine transformation parameters.

[0026] Optionally, the first convolutional neural network, the first feature matching network, and the first regression network are trained using a first training image set that has undergone affine transformation.

[0027] Optionally, the step of extracting fine features from the third input image and the second input image respectively to obtain first fine features and second fine features includes:

[0028] The third input image and the second input image are respectively input into the second convolutional neural network for fine feature extraction, and the third initial feature image and the fourth initial feature image are output.

[0029] Each subject in the third initial feature image and the fourth initial feature image is cropped to obtain the third cropped feature image and the fourth cropped feature image;

[0030] The third and fourth cropped feature images are subjected to L2 norm normalization to obtain the first and second fine features.

[0031] Optionally, the step of performing feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image includes:

[0032] The first fine feature is represented as a first fine feature tensor, the second fine feature is represented as a second fine feature tensor, and the first fine feature tensor and the second fine feature tensor are input into the second feature matching network;

[0033] Calculate the dot product of the first fine feature tensor and the second fine feature tensor to obtain the second similarity tensor;

[0034] Based on the second similarity tensor, a fine feature point matching pair between the third input image and the second input image is determined;

[0035] Regularization is performed on the fine features of each fine feature point matching pair;

[0036] The regularized fine features are normalized using the L2 norm to obtain the feature values ​​of the fine feature point matching pairs, which are then output through the second feature matching network.

[0037] Optionally, the step of performing feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image includes:

[0038] The feature values ​​of the fine feature point matching pairs are input into the second regression network to estimate the fine geometric transformation parameters from the third input image to the second input image, and the fine geometric transformation parameters are output; the fine geometric transformation parameters include thin plate spline interpolation transformation parameters.

[0039] Optionally, the second convolutional neural network, the second feature matching network, and the second regression network are trained using a second training image set that has undergone thin-plate spline interpolation transformation.

[0040] Optionally, the first convolutional neural network and the second convolutional neural network include a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a first pooling layer, a third convolutional layer, a third activation function layer, a fourth convolutional layer, a fourth activation function layer, a second pooling layer, a fifth convolutional layer, a fifth activation function layer, a sixth convolutional layer, a sixth activation function layer, a seventh convolutional layer, a seventh activation function layer, a third pooling layer, an eighth convolutional layer, an eighth activation function layer, a ninth convolutional layer, a ninth activation function layer, a tenth convolutional layer, a tenth activation function layer, and a fourth pooling layer of a VGG-16 convolutional neural network.

[0041] Optionally, the first regression network and the second regression network include an eleventh convolutional layer, a first batch normalization layer, an eleventh activation function layer, a twelfth convolutional layer, a second batch normalization layer, a twelfth activation function layer, and a fully connected layer;

[0042] The eleventh convolutional layer has added local geometric constraints.

[0043] Optionally, after estimating the fine geometric transformation parameters from the third input image to the second input image based on the fine feature point matching pairs, the method further includes:

[0044] Based on the coarse geometric transformation parameters and the fine geometric transformation parameters, the first input image and the second input image are image registered.

[0045] This invention also discloses an image processing apparatus, comprising:

[0046] A coarse feature extraction module is used to acquire a first input image and a second input image, and to perform coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature;

[0047] The coarse feature matching module is used to perform feature matching on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image;

[0048] The first estimation module is used to estimate the coarse geometric transformation parameters from the first input image to the second input image based on the coarse feature point matching pairs and their feature values.

[0049] A transformation module is used to perform a geometric transformation on the first input image according to the coarse geometric transformation parameters to obtain a third input image;

[0050] A fine feature extraction module is used to perform fine feature extraction on the third input image and the second input image respectively to obtain the first fine feature and the second fine feature;

[0051] The fine feature matching module is used to perform feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image.

[0052] The second estimation module is used to estimate the fine geometric transformation parameters from the third input image to the second input image based on the fine feature point matching pairs and their feature values.

[0053] Optionally, the coarse feature extraction module is specifically used for:

[0054] The first input image and the second input image are respectively input into the first convolutional neural network for coarse feature extraction, and the first initial feature image and the second initial feature image are output.

[0055] Each subject in the first initial feature image and the second initial feature image is cropped to obtain a first cropped feature image and a second cropped feature image;

[0056] The first cropped feature image and the second cropped feature image are respectively subjected to L2 norm normalization to obtain the first coarse feature and the second coarse feature.

[0057] Optionally, the coarse feature matching module is specifically used for:

[0058] The first coarse feature is represented as a first coarse feature tensor, the second coarse feature is represented as a second coarse feature tensor, and the first coarse feature tensor and the second coarse feature tensor are input into the first feature matching network;

[0059] Calculate the dot product between the first coarse feature tensor and the second coarse feature tensor to obtain the first similarity tensor between the first coarse feature and the second coarse feature;

[0060] Based on the first similarity tensor, a coarse feature point matching pair between the first input image and the second input image is determined;

[0061] The coarse features of each coarse feature point matching pair are regularized.

[0062] The coarse features after regularization are normalized using the L2 norm to obtain the feature values ​​of the coarse feature point matching pairs, which are then output through the first feature matching network.

[0063] Optionally, the first estimation module is specifically used for:

[0064] The feature values ​​of the coarse feature point matching pairs are input into a first regression network to estimate the coarse geometric transformation parameters from the first input image to the second input image through the first regression network, and the coarse geometric transformation parameters are output; the coarse geometric transformation parameters include affine transformation parameters.

[0065] Optionally, the first convolutional neural network, the first feature matching network, and the first regression network are trained using a first training image set that has undergone affine transformation.

[0066] Optionally, the fine feature extraction module is specifically used for:

[0067] The third input image and the second input image are respectively input into the second convolutional neural network for fine feature extraction, and the third initial feature image and the fourth initial feature image are output.

[0068] Each subject in the third initial feature image and the fourth initial feature image is cropped to obtain the third cropped feature image and the fourth cropped feature image;

[0069] The third and fourth cropped feature images are subjected to L2 norm normalization to obtain the first and second fine features.

[0070] Optionally, the fine feature matching module is specifically used for:

[0071] The first fine feature is represented as a first fine feature tensor, the second fine feature is represented as a second fine feature tensor, and the first fine feature tensor and the second fine feature tensor are input into the second feature matching network;

[0072] Calculate the dot product of the first fine feature tensor and the second fine feature tensor to obtain the second similarity tensor;

[0073] Based on the second similarity tensor, a fine feature point matching pair between the third input image and the second input image is determined;

[0074] Regularization is performed on the fine features of each fine feature point matching pair;

[0075] The regularized fine features are normalized using the L2 norm to obtain the feature values ​​of the fine feature point matching pairs, which are then output through the second feature matching network.

[0076] Optionally, the second estimation module is specifically used for:

[0077] The feature values ​​of the fine feature point matching pairs are input into the second regression network to estimate the fine geometric transformation parameters from the third input image to the second input image, and the fine geometric transformation parameters are output; the fine geometric transformation parameters include thin plate spline interpolation transformation parameters.

[0078] Optionally, the second convolutional neural network, the second feature matching network, and the second regression network are trained using a second training image set that has undergone thin-plate spline interpolation transformation.

[0079] Optionally, the first convolutional neural network and the second convolutional neural network include a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a first pooling layer, a third convolutional layer, a third activation function layer, a fourth convolutional layer, a fourth activation function layer, a second pooling layer, a fifth convolutional layer, a fifth activation function layer, a sixth convolutional layer, a sixth activation function layer, a seventh convolutional layer, a seventh activation function layer, a third pooling layer, an eighth convolutional layer, an eighth activation function layer, a ninth convolutional layer, a ninth activation function layer, a tenth convolutional layer, a tenth activation function layer, and a fourth pooling layer of a VGG-16 convolutional neural network.

[0080] Optionally, the first regression network and the second regression network include an eleventh convolutional layer, a first batch normalization layer, an eleventh activation function layer, a twelfth convolutional layer, a second batch normalization layer, a twelfth activation function layer, and a fully connected layer;

[0081] The eleventh convolutional layer has added local geometric constraints.

[0082] Optionally, the device further includes:

[0083] The registration module is used to perform image registration between the first input image and the second input image based on the coarse geometric transformation parameters and the fine geometric transformation parameters.

[0084] This invention also discloses an electronic device, including a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the programs containing instructions for performing the following operations:

[0085] Acquire a first input image and a second input image, and perform coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature;

[0086] Feature matching is performed on the first coarse feature and the second coarse feature to obtain a coarse feature point matching pair between the first input image and the second input image;

[0087] Based on the coarse feature point matching pairs, estimate the coarse geometric transformation parameters from the first input image to the second input image;

[0088] The first input image is geometrically transformed according to the coarse geometric transformation parameters to obtain the third input image;

[0089] Fine-grained features are extracted from the third input image and the second input image respectively to obtain the first fine-grained feature and the second fine-grained feature;

[0090] Perform feature matching on the first fine feature and the second fine feature to obtain a fine feature point matching pair between the third input image and the second input image;

[0091] Based on the fine feature point matching pairs, the fine geometric transformation parameters from the third input image to the second input image are estimated.

[0092] This invention also discloses a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the image processing method described above.

[0093] The embodiments of the present invention have the following advantages:

[0094] In this embodiment of the invention, by first estimating the coarse geometric transformation parameters between two images, and then estimating the fine geometric transformation parameters, the spatial transformation relationship between the two images can be obtained. Experiments show that when this method is configured as an algorithm, it enables the algorithm to have good generalization ability. Furthermore, image registration based on the coarse and fine geometric transformation parameters obtained by this method is more accurate in estimating the spatial transformation between two images with significant appearance differences in the presence of clutter, thus improving the accuracy of image registration and resulting in better image registration performance. Attached Figure Description

[0095] Figure 1 This is a flowchart of the steps of an image processing method according to the present invention;

[0096] Figure 2 This is a flowchart of another image processing method according to the present invention;

[0097] Figure 3 This is a flowchart of another image processing method according to the present invention;

[0098] Figure 4 This is a network architecture diagram of a first / second convolutional neural network according to the present invention;

[0099] Figure 5 This is a schematic diagram illustrating the calculation of a similarity tensor according to the present invention;

[0100] Figure 6 This is a network architecture diagram of a first / second regression network according to the present invention;

[0101] Figure 7 This is a structural block diagram of an image processing device according to the present invention;

[0102] Figure 8 This is a structural block diagram of an electronic device for processing images, according to an exemplary embodiment. Detailed Implementation

[0103] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0104] Reference Figure 1 The diagram illustrates a flowchart of an image processing method according to the present invention, which may specifically include the following steps:

[0105] Step 101: Obtain the first input image and the second input image, and perform coarse feature extraction on the first input image and the second input image respectively to obtain the first coarse feature and the second coarse feature.

[0106] In this embodiment of the invention, the first input image can be the image to be registered, and the second input image can be the target image. That is, the spatial transformation relationship between the first input image and the second input image can be estimated first, and then the first input image can be registered with the second input image after spatial transformation.

[0107] Spatial transformations from one image to another can be categorized into two types: coarse geometric transformations and fine geometric transformations. Coarse geometric transformations, also known as simple geometric transformations, primarily affect larger and more prominent features in an image, such as affine transformations (Aff). Fine geometric transformations, on the other hand, are complex geometric transformations that primarily affect smaller and more detailed features in an image, such as thin-plate-spline interpolation (TPS).

[0108] Since it is difficult to directly estimate a complex and comprehensive geometric transformation when there is clutter in the image, and the algorithm is also difficult to implement, the embodiments of the present invention start by estimating a simple geometric transformation. The image processing algorithm is faster and more stable, and it also provides a foundation for estimating more complex geometric transformations, which can make the estimation of complex geometric transformations more accurate.

[0109] Accordingly, in this embodiment of the invention, after obtaining the first input image to be registered and the second input image as the registration target image, coarse feature extraction can be performed on the first input image to obtain the first coarse feature, and coarse feature extraction can be performed on the second input image to obtain the second coarse feature. The first coarse feature and the second coarse feature can be used to estimate what kind of simple geometric transformation has been performed from the first input image to the second input image.

[0110] Step 102: Perform feature matching on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image.

[0111] Then, feature matching can be performed on the first coarse features extracted from the first input image and the second coarse features extracted from the second input image.

[0112] In the feature matching process, the features in an image are first described using descriptors, such as SIFT (Scale-invariant feature transform) descriptors. Then, the similarity (or correlation) of the descriptors is used to calculate and find coarse feature points that match between the two images, forming coarse feature point matching pairs.

[0113] In this embodiment of the invention, similar portions of the descriptors of two images can be retained, while descriptors of dissimilar portions will be discarded, because all useful information for geometric estimation is already contained in the similar descriptors. However, in practical applications, the determination of coarse feature point matching pairs may also be inaccurate. Therefore, alternatively, some correction mechanisms can be employed. For example, the image can be thresholded to distinguish the image background from image noise (i.e., image clutter), thereby excluding coarse feature points that were successfully matched due to image noise in the coarse feature point matching pairs; or, for another example, matching pairs in the neighborhood of coarse feature points can be retained to modify the matching pairs.

[0114] In addition, geometric estimation only needs to pay attention to the spatial location and similarity of descriptors, and does not need to focus on the descriptors themselves.

[0115] Step 103: Estimate the coarse geometric transformation parameters from the first input image to the second input image based on the coarse feature point matching pairs and their feature values.

[0116] In this step, based on the feature data of coarse feature points that have a one-to-one correspondence in the two images, the coarse geometric transformation parameters between the two images can be estimated by regression, so as to know what kind of simple geometric transformation has been performed between the two images.

[0117] Step 104: Perform a geometric transformation on the first input image according to the coarse geometric transformation parameters to obtain the third input image.

[0118] In this step, a geometric transformation can be performed on the first input image based on the estimated coarse geometric transformation parameters, thereby filtering out the simple geometric transformation between the first and second input images. For the third input image obtained after filtering out the simple geometric transformation, there is only a complex geometric transformation between the third and second input images, rather than a superposition of simple and complex geometric transformations. Therefore, the type of complex geometric transformation performed between the third and second input images can be estimated subsequently.

[0119] Step 105: Perform fine feature extraction on the third input image and the second input image respectively to obtain the first fine feature and the second fine feature.

[0120] Similar to the coarse feature extraction method in step 101, this step focuses on extracting fine features from the two images to estimate what kind of complex geometric transformation has been performed from the first input image to the second input image.

[0121] Step 106: Perform feature matching on the first and second fine features to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image.

[0122] Referring to the feature matching method in step 102, in this step, feature matching can be performed on the first fine feature extracted from the third input image and the second fine feature extracted from the second input image.

[0123] Step 107: Estimate the fine geometric transformation parameters from the third input image to the second input image based on the fine feature point matching pairs and their feature values.

[0124] Referring to the estimation method of the coarse geometric transformation parameters in step 103, in this step, the fine geometric transformation parameters between the two images can be estimated by regression based on the feature data of the fine feature points that have a one-to-one correspondence in the two images, so as to know what kind of complex geometric transformation has been performed between the two images.

[0125] Steps 101-107 above can be simplified to: Figure 2 The image processing flow is shown.

[0126] In this embodiment of the invention, by first estimating the coarse geometric transformation parameters between two images, and then estimating the fine geometric transformation parameters, the spatial transformation relationship between the two images can be obtained. Experiments show that when this method is configured as an algorithm, it enables the algorithm to have good generalization ability. Furthermore, image registration based on the coarse and fine geometric transformation parameters obtained by this method is more accurate in estimating the spatial transformation between two images with significant appearance differences in the presence of clutter, thus improving the accuracy of image registration and resulting in better image registration performance.

[0127] Before introducing another embodiment of the image processing method of the present invention, the convolutional neural network involved in this embodiment will be introduced first.

[0128] Convolutional Neural Networks (CNNs) are deep learning networks designed to mimic the sensory cortex of organisms. They extract features from various dimensions through convolutional and pooling layers, then synthesize these features into a higher-level set using fully connected layers. Furthermore, activation functions are used to mitigate the non-linearity of CNNs and enhance their expressive power. CNNs update their weights through forward and backward propagation, and regularization is used to suppress overfitting. While CNNs are just one type of deep neural network, they offer advantages such as weight sharing and local connections compared to other deep neural networks. Therefore, they require fewer parameters, and their outstanding performance in computer vision, natural language processing, and other areas makes them a popular choice for deep learning research.

[0129] Convolutional layers are the core layers for feature extraction in convolutional neural networks. When an image is used as input to a convolutional neural network, a color image is first separated into three color channels: R (red), G (green), and B (blue), forming three grayscale layers: R, G, and B. Then, each pixel is converted into a numerical representation based on its brightness. Since an image has many pixels, the convolutional kernels in the convolutional layers can extract features from each pixel in the input image, ultimately extracting useful features.

[0130] For the input image, the convolutional layer moves the convolutional kernel across the pixel value map according to a set stride, while simultaneously calculating feature values. The stride refers to the unit of movement of the convolutional kernel across the input data, i.e., the distance the kernel moves each time. The feature value calculation method involves mapping each pixel value covered by the convolutional kernel to the value on the kernel, multiplying them, and then summing the results. The sum is then filled into a pre-defined feature map. Then, according to the set stride, the convolutional kernel moves to the next region, covering the pixel values ​​of other regions and calculating the next feature value.

[0131] In practical applications, pixel values ​​in the image can be padded before convolution operations on feature values. This involves filling around the input data with fixed values, such as 0. Padding in convolutional layers serves two purposes: first, it ensures the output feature values ​​maintain the same size as the input data, preventing the information obtained after convolution from becoming increasingly smaller; second, it preserves edge information. Without padding during convolution, edge data would be ignored in each operation, masking its importance and reducing its reference value.

[0132] Pooling layers in convolutional neural networks can reduce dimensionality, remove redundant information, and compress features, thereby simplifying network complexity and reducing computational cost. Commonly used pooling layers include max pooling and average pooling.

[0133] Among the many convolutional neural networks (CNNs), the VGG network model is a classic example. Its greatest advantage lies in simplifying the structure of CNNs, effectively reducing network complexity. A typical representative of the VGG network model is the VGG-16 CNN, which has 16 weighted layers, hence the name VGG-16. The VGG-16 CNN reduces the number of weights, speeding up computation, and also prevents overfitting.

[0134] Figure 3 A flowchart of the steps of an image processing method according to the present invention is shown, with reference to... Figure 3 The method includes the following steps:

[0135] Step 201: Obtain the first input image and the second input image; input the first input image and the second input image into the first convolutional neural network respectively, perform coarse feature extraction, and output the first initial feature image and the second initial feature image; crop each subject in the first initial feature image and the second initial feature image respectively to obtain the first cropped feature image and the second cropped feature image; perform L2 norm normalization processing on the first cropped feature image and the second cropped feature image respectively to obtain the first coarse feature and the second coarse feature.

[0136] Based on the above-described content on deep networks, embodiments of the present invention can select a deep learning network for image feature extraction. Specifically, the deep learning network can be a convolutional neural network.

[0137] In an optional implementation, a variant of the VGG-16 convolutional neural network can be used as the first convolutional neural network to extract features from the input image. Specifically, refer to... Figure 4 The first convolutional neural network may include the following layers of a VGG-16 convolutional neural network: a first convolutional layer (Conv1), a first activation function layer (ReLU1), a second convolutional layer (Conv2), a second activation function layer (ReLU2), a first pooling layer (Pool1), a third convolutional layer (Conv3), a third activation function layer (ReLU3), a fourth convolutional layer (Conv4), a fourth activation function layer (ReLU4), a second pooling layer (Pool2), a fifth convolutional layer (Conv5), a fifth activation function layer (ReLU5), a sixth convolutional layer (Conv6), a sixth activation function layer (ReLU6), a seventh convolutional layer (Conv7), a seventh activation function layer (ReLU7), a third pooling layer (Pool3), an eighth convolutional layer (Conv8), an eighth activation function layer (ReLU8), a ninth convolutional layer (Conv9), a ninth activation function layer (ReLU9), a tenth convolutional layer (Conv10), a tenth activation function layer (ReLU10), and a fourth pooling layer (Pool4). The activation function layer may use the ReLU activation function.

[0138] The first convolutional neural network is used to extract coarse features from two input images. These coarse features represent a coarse geometric transformation between the two input images, such as an affine transformation. Therefore, the first convolutional neural network can be trained using a training image set that has undergone a coarse geometric transformation. In an optional implementation, the first convolutional neural network can be trained using a first training image set that has undergone an affine transformation.

[0139] After coarse feature extraction by the first convolutional neural network, a first initial feature image corresponding to the first input image and a second initial feature image corresponding to the second input image are obtained. Then, each subject (such as a person, object, etc.) in the first and second initial feature images can be cropped. Each cropped image (i.e., the cropped feature image) is a regular rectangular image, and each cropped image contains only a single subject. Each subject occupies a large area in its corresponding cropped feature image. Therefore, compared with the no-cropping scheme, subject cropping can highlight the influence of subject features, reduce the influence of noise on the training of the first convolutional neural network, enable the first convolutional neural network to produce better learning results, and increase the stability of the first convolutional neural network.

[0140] Then, the cropped feature image can be normalized using the L2 norm to obtain the first coarse feature and the second coarse feature. By setting the L2 norm normalization, gradient explosion or vanishing can be prevented during the training of the convolutional neural network, and the adverse effects caused by singular sample data can also be eliminated during training.

[0141] Step 202: Represent the first coarse feature as a first coarse feature tensor, and the second coarse feature as a second coarse feature tensor. Input the first and second coarse feature tensors into the first feature matching network. Calculate the dot product of the first and second coarse feature tensors to obtain the first similarity tensor between the first and second coarse features. Determine the coarse feature point matching pairs between the first and second input images based on the first similarity tensor. Perform regularization processing on the coarse features of each coarse feature point matching pair. Perform L2 norm normalization processing on the regularized coarse features to obtain the feature values ​​of the coarse feature point matching pairs and output them through the first feature matching network.

[0142] After obtaining the first coarse features of the first input image and the second coarse features of the second input image through feature extraction, the extracted coarse features can be combined into a tensor and used as the input of the first feature matching network.

[0143] In the first feature matching network, a rough feature point matching pair between the first input image and the second input image can be determined by dot product similarity.

[0144] In one alternative implementation, the first similarity tensor can be obtained by directly calculating the dot product (also called inner product, scalar product, or dot product) between each feature value in the first coarse feature tensor and each feature value in the second coarse feature tensor.

[0145] In another alternative implementation, refer to Figure 5 First, the first coarse feature tensor f can be...A Divided into (w×h) first feature blocks (i k ,j k ), and the second coarse feature tensor f B It is also divided into (w×h) second feature blocks (i,j), where each first feature block may include (d×d) first coarse features, and each second feature block may include (d×d) second coarse features. Optionally, d can be 2. Then, for each first feature block (i... k ,j k Each feature value in the first feature block (i) is convolved to obtain each first feature block (i) k ,j k The convolution value of each feature in the first feature block (i,j) is calculated by performing convolution on each feature value in the second feature block (i,j) separately. Then, the convolution value of each feature in the first feature block (i,j) is calculated. k ,j k The first similarity tensor is obtained by taking the dot product (also called inner product, scalar product, or dot product) between the convolution value of the first feature block (i,j) and the convolution value of each second feature block (i,j). The calculation formula is as follows:

[0146] C AB (i,j,k)=f B (i,j) T f A (i k ,j k )

[0147] Among them, C AB (i,j,k) represents the first similarity tensor, f A (i k ,j k ) represents the first coarse feature tensor f A The first feature block (i) in k ,j k The convolution value of f B (i,j) represents the second coarse feature tensor f. B The convolution value of the second feature block (i,j) in the matrix, where k is an auxiliary variable, k = h(j k -1)+i k .

[0148] In the first implementation described above, the dot product similarity between tensors is calculated on a per-feature-value basis. Therefore, the dot product similarity requires calculating w×h×d×(w×h×d) values, resulting in a computational complexity of w×h×d×(w×h×d).

[0149] In the second implementation described above, due to the first coarse feature tensor f ASecond coarse feature tensor f B Each feature is divided into (w×h) feature blocks, and the dot product similarity between tensors is calculated on a per-feature-block basis. Therefore, the dot product similarity requires calculating w×h×(w×h) values, resulting in a computational complexity of w×h×(w×h). Thus, the second implementation method can reduce the complexity of similarity calculation.

[0150] The first similarity tensor reflects the correlation between feature points in the first and second input images from the perspective of coarse features. Feature points with high correlation actually represent the same target, but they have different representations in the first and second input images only because of spatial transformation.

[0151] Then, the first feature matching network can determine the coarse feature point matching pairs between the first input image and the second input image based on the first similarity tensor. The coarse feature points with high similarity between the first input image and the second input image have high correlation and can be considered as feature points expressing the same target. Therefore, the coarse feature points with high correlation in the two images can be matched one-to-one, thereby obtaining multiple coarse feature point matching pairs between the first input image and the second input image.

[0152] Subsequently, the first feature matching network can perform regularization on the coarse features of each coarse feature point matching pair, where the coarse features of each pair come from the first and second coarse features. This regularization reduces the model complexity of the first feature matching network, thereby improving the model's generalization ability.

[0153] Furthermore, the first feature matching network can perform L2 norm normalization on the coarse features after regularization and output the processed feature values. By setting L2 norm normalization, gradient explosion or vanishing can be prevented during the training of the feature matching network, and the adverse effects caused by singular sample data can also be eliminated during training.

[0154] In practice, two scenarios may occur: First, for a coarse feature point in one input image, there may be only one corresponding coarse feature point in another input image. In this case, L2 norm normalization will increase the matching weights. Second, when there is clutter in the input image, i.e., when there are many noise points, for a coarse feature point in one input image, there may be multiple corresponding coarse feature points in another input image. In this case, L2 norm normalization will decrease the matching weights. However, since similarity calculation and normalization calculation are differentiable relative to the descriptor (forward propagation is calculating partial derivatives, and backpropagation is calculating differentiation), the weights can still be updated through backpropagation, thus preserving the advantages of normalization while ensuring the performance of the feature matching network.

[0155] Step 203: Input the feature values ​​of the coarse feature point matching pairs into the first regression network to estimate the coarse geometric transformation parameters from the first input image to the second input image through the first regression network, and output the coarse geometric transformation parameters; the coarse geometric transformation parameters include the affine transformation parameter θ_Aff.

[0156] In this embodiment of the invention, a regression method can be used to estimate the coarse geometric transformation parameters from the first input image to the second input image, thereby determining which simple geometric transformation the first input image to the second input image has undergone.

[0157] In an alternative embodiment, refer to Figure 6 The first regression network includes an eleventh convolutional layer Conv11, a first batch normalization layer BN1, an eleventh activation function layer ReLU11, a twelfth convolutional layer Conv12, a second batch normalization layer BN2, a twelfth activation function layer ReLU12, and a fully connected layer FC; wherein, the eleventh convolutional layer Conv11 has local geometric constraints added.

[0158] The first regression network uses convolutional layers to vote on geometric changes, and then processes and aggregates these votes through subsequent layers. The first convolutional layer in the first regression network can incorporate local geometric constraints. These constraints preserve the matching relationships within the spatial neighborhood of feature points, allowing the first convolutional layer to output a result to the next layer only when the nearest neighbor descriptors in the first input image match those in the second input image. Compared to global geometric constraints, local geometric constraints, by achieving neighborhood consistency, can better fit local geometric deformations and have better noise resistance.

[0159] Batch normalization layers can scale data to a suitable range, thereby accelerating the training speed of the first regression network and avoiding gradient vanishing during the training process, making the training of the first regression network easier and more stable.

[0160] The activation function layer is used to increase the nonlinearity of the first regression network. Optionally, the activation function layer can use the ReLU activation function.

[0161] Fully connected layers can act as classifiers, mapping the distributed feature representations learned by the first regression network to the label space.

[0162] When the coarse geometric transformation parameters include the affine transformation parameter θ_Aff, the first convolutional neural network, the first feature matching network, and the first regression network are all trained using the first training image set that has undergone the affine transformation. In this way, the type of affine transformation between two images can be estimated through the trained first convolutional neural network, the first feature matching network, and the first regression network.

[0163] Step 204: Perform a geometric transformation on the first input image according to the coarse geometric transformation parameters to obtain the third input image.

[0164] In this step, after estimating the coarse geometric transformation parameters from the first input image to the second input image, a geometric transformation can be performed on the first input image using these parameters, resulting in the third input image. This coarse geometric transformation eliminates the influence of the initial coarse geometric transformation between the first and second input images, ensuring that only a fine geometric transformation relationship exists between the transformed third input image and the second input image. This allows for the estimation of the fine geometric transformation relationship between the third and second input images in subsequent steps.

[0165] Step 205: Input the third input image and the second input image into the second convolutional neural network respectively, perform fine feature extraction, and output the third initial feature image and the fourth initial feature image; crop each subject in the third initial feature image and the fourth initial feature image respectively to obtain the third cropped feature image and the fourth cropped feature image; perform L2 norm normalization processing on the third cropped feature image and the fourth cropped feature image respectively to obtain the first fine feature and the second fine feature.

[0166] Step 206: Represent the first fine feature as a first fine feature tensor, and the second fine feature as a second fine feature tensor. Input the first and second fine feature tensors into the second feature matching network. Calculate the dot product of the first and second fine feature tensors to obtain the second similarity tensor. Based on the second similarity tensor, determine the fine feature point matching pairs between the third and second input images. Perform regularization processing on the fine features of each fine feature point matching pair. Perform L2 norm normalization processing on the regularized fine features to obtain the feature values ​​of the fine feature point matching pairs and output them through the second feature matching network.

[0167] Step 207: Input the feature values ​​of the fine feature point matching pairs into the second regression network to estimate the fine geometric transformation parameters from the third input image to the second input image through the second regression network, and output the fine geometric transformation parameters; the fine geometric transformation parameters include the thin plate spline interpolation transformation parameter θ_tps.

[0168] The implementation principle of steps 205-207 can be referred to steps 201-203. The second convolutional neural network has the same network architecture as the first convolutional neural network, the second feature matching network has the same network architecture as the first feature matching network, and the second regression network has the same network architecture as the first regression network.

[0169] The difference is that when the fine geometric transformation parameters include the thin-plate spline interpolation transformation parameters θ_tps, the second convolutional neural network, the second feature matching network, and the second regression network are trained using the second training image set that has undergone the thin-plate spline interpolation transformation. In this way, the type of thin-plate spline interpolation transformation between the two images can be estimated through the trained second convolutional neural network, the second feature matching network, and the second regression network.

[0170] Different network weights can be trained using different training data, enabling the first convolutional neural network, the first feature matching network, and the first regression network to estimate simple geometric transformation relationships between two images, and enabling the second convolutional neural network, the second feature matching network, and the second regression network to estimate complex geometric transformation relationships between two images.

[0171] Step 208: Based on the coarse geometric transformation parameters and the fine geometric transformation parameters, perform image registration between the first input image and the second input image.

[0172] After estimating the coarse and fine geometric transformation parameters from the first input image to the second input image, the two images can be registered based on these parameters.

[0173] Image registration maps one image to another by finding a spatial transformation (obtained through steps 201-207), ensuring a one-to-one correspondence between identical spatial points in the two images, thus achieving information fusion. Through image registration, two images can be represented in the same coordinate system. By using identical objects in the two images for image stitching and fusion, different content in the two images can be integrated. The fused image expresses the information from both images within the same system, thereby unifying the information in the two images under a single standard.

[0174] In one specific embodiment, the MatConvNet library can be used to train the entire image processing network using stochastic gradient descent. For example, the learning rate can be set to 0.001, the momentum to 0.9, with no weight decay, and the training iterations can be 10. The input dimension of the image is 227*227, and after feature extraction, it enters the feature matching network with a dimension of 15*15.

[0175] The training process requires fully supervised training data consisting of image pairs and known geometric relationships between them, such as affine transformations and thin-plate spline interpolation transformations. Optionally, 20,000 image pairs can be artificially synthesized, with 10,000 image pairs used as the training set and the other 10,000 image pairs used as the validation set.

[0176] In practical applications, a grid can be constructed in the training images. The geometric transformation between image pairs can be represented by the geometric transformation between the grid points. Since the geometric transformations (ground truth values) between pairs of grid points in an image pair are known, a geometric transformation (the estimated value output by the network) can also be estimated using an image processing network. Using the ground truth and estimated geometric transformation values ​​of the pairs of grid points, a loss function required for training the network can be constructed, and an effective network can be obtained by minimizing this loss function. Optionally, the loss function can be a mean squared error loss function.

[0177] In this embodiment of the invention, a convolutional neural network is first used to extract coarse features from two images. Then, a feature matching network is used to perform feature matching on the extracted coarse features. Next, a regression network is used to estimate the coarse geometric transformation parameters between the two images. Similarly, a convolutional neural network, feature matching network, and regression network are used to estimate the fine geometric transformation parameters between the two images, thereby deriving the spatial transformation relationship between the two images. Finally, the two images are registered based on the coarse and fine geometric transformation parameters. Experiments show that when configured as an algorithm, this method exhibits good generalization ability. Furthermore, image registration based on the coarse and fine geometric transformation parameters obtained by this method is more accurate in estimating the spatial transformation between two images with significant appearance differences in the presence of clutter, thus improving the accuracy of image registration and resulting in better image registration performance.

[0178] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0179] Reference Figure 7 The diagram shows a structural block diagram of an image processing apparatus according to the present invention. The apparatus 300 may specifically include the following modules:

[0180] The coarse feature extraction module 301 is used to acquire a first input image and a second input image, and to perform coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature;

[0181] The coarse feature matching module 302 is used to perform feature matching on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image;

[0182] The first estimation module 303 is used to estimate the coarse geometric transformation parameters from the first input image to the second input image based on the coarse feature point matching pairs and their feature values.

[0183] Transformation module 304 is used to perform geometric transformation on the first input image according to the coarse geometric transformation parameters to obtain a third input image;

[0184] The fine feature extraction module 305 is used to perform fine feature extraction on the third input image and the second input image respectively to obtain the first fine feature and the second fine feature;

[0185] The fine feature matching module 306 is used to perform feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image.

[0186] The second estimation module 307 is used to estimate the fine geometric transformation parameters from the third input image to the second input image based on the fine feature point matching pairs and their feature values.

[0187] Optionally, the coarse feature extraction module is specifically used for:

[0188] The first input image and the second input image are respectively input into the first convolutional neural network for coarse feature extraction, and the first initial feature image and the second initial feature image are output.

[0189] Each subject in the first initial feature image and the second initial feature image is cropped to obtain a first cropped feature image and a second cropped feature image;

[0190] The first cropped feature image and the second cropped feature image are respectively subjected to L2 norm normalization to obtain the first coarse feature and the second coarse feature.

[0191] Optionally, the coarse feature matching module is specifically used for:

[0192] The first coarse feature is represented as a first coarse feature tensor, the second coarse feature is represented as a second coarse feature tensor, and the first coarse feature tensor and the second coarse feature tensor are input into the first feature matching network;

[0193] Calculate the dot product between the first coarse feature tensor and the second coarse feature tensor to obtain the first similarity tensor between the first coarse feature and the second coarse feature;

[0194] Based on the first similarity tensor, a coarse feature point matching pair between the first input image and the second input image is determined;

[0195] The coarse features of each coarse feature point matching pair are regularized.

[0196] The coarse features after regularization are normalized using the L2 norm to obtain the feature values ​​of the coarse feature point matching pairs, which are then output through the first feature matching network.

[0197] Optionally, the first estimation module is specifically used for:

[0198] The feature values ​​of the coarse feature point matching pairs are input into a first regression network to estimate the coarse geometric transformation parameters from the first input image to the second input image through the first regression network, and the coarse geometric transformation parameters are output; the coarse geometric transformation parameters include affine transformation parameters.

[0199] Optionally, the first convolutional neural network, the first feature matching network, and the first regression network are trained using a first training image set that has undergone affine transformation.

[0200] Optionally, the fine feature extraction module is specifically used for:

[0201] The third input image and the second input image are respectively input into the second convolutional neural network for fine feature extraction, and the third initial feature image and the fourth initial feature image are output.

[0202] Each subject in the third initial feature image and the fourth initial feature image is cropped to obtain the third cropped feature image and the fourth cropped feature image;

[0203] The third and fourth cropped feature images are subjected to L2 norm normalization to obtain the first and second fine features.

[0204] Optionally, the fine feature matching module is specifically used for:

[0205] The first fine feature is represented as a first fine feature tensor, the second fine feature is represented as a second fine feature tensor, and the first fine feature tensor and the second fine feature tensor are input into the second feature matching network;

[0206] Calculate the dot product of the first fine feature tensor and the second fine feature tensor to obtain the second similarity tensor;

[0207] Based on the second similarity tensor, a fine feature point matching pair between the third input image and the second input image is determined;

[0208] Regularization is performed on the fine features of each fine feature point matching pair;

[0209] The regularized fine features are normalized using the L2 norm to obtain the feature values ​​of the fine feature point matching pairs, which are then output through the second feature matching network.

[0210] Optionally, the second estimation module is specifically used for:

[0211] The feature values ​​of the fine feature point matching pairs are input into the second regression network to estimate the fine geometric transformation parameters from the third input image to the second input image, and the fine geometric transformation parameters are output; the fine geometric transformation parameters include thin plate spline interpolation transformation parameters.

[0212] Optionally, the second convolutional neural network, the second feature matching network, and the second regression network are trained using a second training image set that has undergone thin-plate spline interpolation transformation.

[0213] Optionally, the first convolutional neural network and the second convolutional neural network include a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a first pooling layer, a third convolutional layer, a third activation function layer, a fourth convolutional layer, a fourth activation function layer, a second pooling layer, a fifth convolutional layer, a fifth activation function layer, a sixth convolutional layer, a sixth activation function layer, a seventh convolutional layer, a seventh activation function layer, a third pooling layer, an eighth convolutional layer, an eighth activation function layer, a ninth convolutional layer, a ninth activation function layer, a tenth convolutional layer, a tenth activation function layer, and a fourth pooling layer of a VGG-16 convolutional neural network.

[0214] Optionally, the first regression network and the second regression network include an eleventh convolutional layer, a first batch normalization layer, an eleventh activation function layer, a twelfth convolutional layer, a second batch normalization layer, a twelfth activation function layer, and a fully connected layer;

[0215] The eleventh convolutional layer has added local geometric constraints.

[0216] Optionally, the device further includes:

[0217] The registration module is used to perform image registration between the first input image and the second input image based on the coarse geometric transformation parameters and the fine geometric transformation parameters.

[0218] In this embodiment of the invention, by first estimating the coarse geometric transformation parameters between two images, and then estimating the fine geometric transformation parameters, the spatial transformation relationship between the two images can be obtained. Experiments show that when this method is configured as an algorithm, it enables the algorithm to have good generalization ability. Furthermore, image registration based on the coarse and fine geometric transformation parameters obtained by this method is more accurate in estimating the spatial transformation between two images with significant appearance differences in the presence of clutter, thus improving the accuracy of image registration and resulting in better image registration performance.

[0219] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0220] Figure 8 This is a structural block diagram illustrating an electronic device 400 for processing images, according to an exemplary embodiment. For example, the electronic device 400 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0221] Reference Figure 8 The electronic device 400 may include one or more of the following components: processing component 402, memory 404, power supply component 406, multimedia component 408, audio component 410, input / output (I / O) interface 412, sensor component 414, and communication component 416.

[0222] Processing component 402 typically controls the overall operation of electronic device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0223] Memory 404 is configured to store various types of data to support the operation of device 400. Examples of this data include instructions for any application or method operating on electronic device 400, contact data, phonebook data, messages, pictures, videos, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0224] Power supply component 406 provides power to various components of electronic device 400. Power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 400.

[0225] Multimedia component 408 includes a screen that provides an output interface between the electronic device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera and / or a rear-facing camera. When the electronic device 400 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0226] Audio component 410 is configured to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) configured to receive external audio signals when electronic device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0227] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0228] Sensor assembly 414 includes one or more sensors for providing state assessments of various aspects of electronic device 400. For example, sensor assembly 414 may detect the on / off state of device 400, the relative positioning of components such as the display and keypad of electronic device 400, changes in position of electronic device 400 or a component of electronic device 400, the presence or absence of user contact with electronic device 400, orientation or acceleration / deceleration of electronic device 400, and temperature changes of electronic device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0229] Communication component 416 is configured to facilitate wired or wireless communication between electronic device 400 and other devices. Electronic device 400 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0230] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0231] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, which can be executed by a processor 420 of an electronic device 400 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0232] A readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform an image processing method, the method comprising:

[0233] Acquire a first input image and a second input image, and perform coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature;

[0234] Feature matching is performed on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image;

[0235] Based on the coarse feature point matching pairs and their feature values, estimate the coarse geometric transformation parameters from the first input image to the second input image;

[0236] The first input image is geometrically transformed according to the coarse geometric transformation parameters to obtain the third input image;

[0237] Fine-grained features are extracted from the third input image and the second input image respectively to obtain the first fine-grained feature and the second fine-grained feature;

[0238] Perform feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image;

[0239] Based on the fine feature point matching pairs and their feature values, the fine geometric transformation parameters from the third input image to the second input image are estimated.

[0240] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0241] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0242] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0243] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0244] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0245] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0246] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0247] The foregoing has provided a detailed description of an image processing method and apparatus, an electronic device, and a storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image processing method, characterized in that, include: Acquire a first input image and a second input image, and perform coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature; Feature matching is performed on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image; Based on the coarse feature point matching pairs and their feature values, estimate the coarse geometric transformation parameters from the first input image to the second input image; The first input image is geometrically transformed according to the coarse geometric transformation parameters to obtain the third input image; Fine-grained features are extracted from the third input image and the second input image respectively to obtain the first fine-grained feature and the second fine-grained feature; Perform feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image; Based on the fine feature point matching pairs and their feature values, estimate the fine geometric transformation parameters from the third input image to the second input image; Based on the coarse geometric transformation parameters and the fine geometric transformation parameters, the first input image and the second input image are image registered.

2. The method according to claim 1, characterized in that, The step of performing coarse feature extraction on the first input image and the second input image respectively to obtain first coarse features and second coarse features includes: The first input image and the second input image are respectively input into the first convolutional neural network for coarse feature extraction, and the first initial feature image and the second initial feature image are output. Each subject in the first initial feature image and the second initial feature image is cropped to obtain a first cropped feature image and a second cropped feature image; The first cropped feature image and the second cropped feature image are respectively subjected to L2 norm normalization to obtain the first coarse feature and the second coarse feature.

3. The method according to claim 2, characterized in that, The step of performing feature matching on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image includes: The first coarse feature is represented as a first coarse feature tensor, the second coarse feature is represented as a second coarse feature tensor, and the first coarse feature tensor and the second coarse feature tensor are input into the first feature matching network; Calculate the dot product between the first coarse feature tensor and the second coarse feature tensor to obtain the first similarity tensor between the first coarse feature and the second coarse feature; Based on the first similarity tensor, a coarse feature point matching pair between the first input image and the second input image is determined; The coarse features of each coarse feature point matching pair are regularized. The coarse features after regularization are normalized using the L2 norm to obtain the feature values ​​of the coarse feature point matching pairs, which are then output through the first feature matching network.

4. The method according to claim 3, characterized in that, The step of estimating the coarse geometric transformation parameters from the first input image to the second input image based on the coarse feature point matching pairs and their feature values ​​includes: The feature values ​​of the coarse feature point matching pairs are input into a first regression network to estimate the coarse geometric transformation parameters from the first input image to the second input image through the first regression network, and the coarse geometric transformation parameters are output; the coarse geometric transformation parameters include affine transformation parameters.

5. The method according to claim 4, characterized in that, The first convolutional neural network, the first feature matching network, and the first regression network are trained using a first training image set that has undergone affine transformation.

6. The method according to claim 4, characterized in that, The step of extracting fine features from the third input image and the second input image respectively to obtain first fine features and second fine features includes: The third input image and the second input image are respectively input into the second convolutional neural network for fine feature extraction, and the third initial feature image and the fourth initial feature image are output. Each subject in the third initial feature image and the fourth initial feature image is cropped to obtain the third cropped feature image and the fourth cropped feature image; The third and fourth cropped feature images are subjected to L2 norm normalization to obtain the first and second fine features.

7. The method according to claim 6, characterized in that, The step of performing feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image includes: The first fine feature is represented as a first fine feature tensor, the second fine feature is represented as a second fine feature tensor, and the first fine feature tensor and the second fine feature tensor are input into the second feature matching network; Calculate the dot product of the first fine feature tensor and the second fine feature tensor to obtain the second similarity tensor; Based on the second similarity tensor, a fine feature point matching pair between the third input image and the second input image is determined; Regularization is performed on the fine features of each fine feature point matching pair; The regularized fine features are normalized using the L2 norm to obtain the feature values ​​of the fine feature point matching pairs, which are then output through the second feature matching network.

8. The method according to claim 7, characterized in that, The step of performing feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image includes: The feature values ​​of the fine feature point matching pairs are input into the second regression network to estimate the fine geometric transformation parameters from the third input image to the second input image, and the fine geometric transformation parameters are output; the fine geometric transformation parameters include thin plate spline interpolation transformation parameters.

9. The method according to claim 8, characterized in that, The second convolutional neural network, the second feature matching network, and the second regression network are trained using a second training image set that has undergone thin-plate spline interpolation transformation.

10. The method according to claim 6, characterized in that, The first convolutional neural network and the second convolutional neural network include a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a first pooling layer, a third convolutional layer, a third activation function layer, a fourth convolutional layer, a fourth activation function layer, a second pooling layer, a fifth convolutional layer, a fifth activation function layer, a sixth convolutional layer, a sixth activation function layer, a seventh convolutional layer, a seventh activation function layer, a third pooling layer, an eighth convolutional layer, an eighth activation function layer, a ninth convolutional layer, a ninth activation function layer, a tenth convolutional layer, a tenth activation function layer, and a fourth pooling layer of a VGG-16 convolutional neural network.

11. The method according to claim 8, characterized in that, The first regression network and the second regression network include an eleventh convolutional layer, a first batch normalization layer, an eleventh activation function layer, a twelfth convolutional layer, a second batch normalization layer, a twelfth activation function layer, and a fully connected layer; The eleventh convolutional layer has added local geometric constraints.

12. An image processing apparatus, characterized in that, include: A coarse feature extraction module is used to acquire a first input image and a second input image, and to perform coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature; The coarse feature matching module is used to perform feature matching on the first coarse feature and the second coarse feature to obtain coarse feature point matching pairs and their feature values ​​between the first input image and the second input image; The first estimation module is used to estimate the coarse geometric transformation parameters from the first input image to the second input image based on the coarse feature point matching pairs and their feature values. A transformation module is used to perform a geometric transformation on the first input image according to the coarse geometric transformation parameters to obtain a third input image; A fine feature extraction module is used to perform fine feature extraction on the third input image and the second input image respectively to obtain the first fine feature and the second fine feature; The fine feature matching module is used to perform feature matching on the first fine feature and the second fine feature to obtain fine feature point matching pairs and their feature values ​​between the third input image and the second input image. The second estimation module is used to estimate the fine geometric transformation parameters from the third input image to the second input image based on the fine feature point matching pairs and their feature values. The registration module is used to perform image registration between the first input image and the second input image based on the coarse geometric transformation parameters and the fine geometric transformation parameters.

13. An electronic device, characterized in that, It includes memory and one or more programs, wherein one or more programs are stored in memory and configured to be executed by one or more processors, the programs containing instructions for performing the following operations: Acquire a first input image and a second input image, and perform coarse feature extraction on the first input image and the second input image respectively to obtain a first coarse feature and a second coarse feature; Feature matching is performed on the first coarse feature and the second coarse feature to obtain a coarse feature point matching pair between the first input image and the second input image; Based on the coarse feature point matching pairs, estimate the coarse geometric transformation parameters from the first input image to the second input image; The first input image is geometrically transformed according to the coarse geometric transformation parameters to obtain the third input image; Fine-grained features are extracted from the third input image and the second input image respectively to obtain the first fine-grained feature and the second fine-grained feature; Perform feature matching on the first fine feature and the second fine feature to obtain a fine feature point matching pair between the third input image and the second input image; Based on the fine feature point matching pairs, estimate the fine geometric transformation parameters from the third input image to the second input image; Based on the coarse geometric transformation parameters and the fine geometric transformation parameters, the first input image and the second input image are image registered.

14. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image processing method as described in any one of claims 1-11.

Images