Image registration method, device, equipment and computer readable storage medium

Abstract

The present disclosure relates to an image registration method, device, equipment and computer-readable storage medium. After obtaining a group of images to be registered, the present disclosure obtains the feature images that are helpful for registration and the feature images that are not helpful for registration included in each image to be registered in the group of images to be registered. When calculating the spatial position relationship between each image to be registered, only the feature images that are helpful for registration are referred to, and the feature images that are not helpful for registration are ignored. Therefore, the spatial position relationship between each image to be registered is calculated only based on the feature images that are helpful for registration, which not only improves the calculation efficiency, but also improves the accuracy of the spatial position relationship. Then, each image to be registered is registered according to the calculated spatial position relationship, which improves the accuracy of image registration.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning technology, and in particular to an image registration method, device, equipment and computer-readable storage medium. Background Art

[0002] Image registration is the process of matching and superimposing two or more images acquired at different times, using different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.).

[0003] In related technologies, image registration mostly relies on optimizing a similarity index, such as correlation ratio and structural feature descriptor, to achieve registration by maximizing the similarity between images. However, these methods often require a lot of calculations, and the registration based on a single similarity index has a low accuracy rate.

[0004] On the other hand, in related technologies, deep learning methods are also used for image registration. However, the deep learning method itself is difficult to train, and noise is introduced during the training process, resulting in low accuracy of image registration. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides an image registration method, device, equipment and computer-readable storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides an image registration method, comprising:

[0007] Obtaining a group of images to be registered;

[0008] Acquire a feature image that is helpful for registration and a feature image that is not helpful for registration, included in each image to be registered in the group of images to be registered;

[0009] Calculating the spatial position relationship between the images to be registered based on the feature images that are helpful for registration and included in the images to be registered;

[0010] Based on the spatial position relationship, each image to be registered is registered.

[0011] In a second aspect, an embodiment of the present disclosure provides an image registration device, comprising:

[0012] A first acquisition module, used for acquiring a group of images to be registered;

[0013] A second acquisition module is used to acquire the feature images that are helpful for registration and the feature images that are not helpful for registration included in each image to be registered in the image group to be registered;

[0014] A calculation module, used for calculating the spatial position relationship between the images to be registered based on the feature images that are helpful for registration and included in the images to be registered;

[0015] The registration module is used to register the images to be registered based on the spatial position relationship.

[0016] In a third aspect, an embodiment of the present disclosure provides an image registration device, including:

[0017] Memory;

[0018] Processor; and

[0019] Computer programs;

[0020] The computer program is stored in the memory and is configured to be executed by the processor to implement the image registration method of the first aspect.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the image registration method of the first aspect.

[0022] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and the computer program or instructions implement the above image registration method when executed by a processor.

[0023] The present disclosure relates to an image registration method, device, equipment and computer-readable storage medium. After obtaining a group of images to be registered, the present disclosure obtains the feature images that are helpful for registration and the feature images that are not helpful for registration included in each image to be registered in the group of images to be registered. When calculating the spatial position relationship between each image to be registered, only the feature images that are helpful for registration are referred to, and the feature images that are not helpful for registration are ignored. Therefore, the spatial position relationship between each image to be registered is calculated only based on the feature images that are helpful for registration, which not only improves the calculation efficiency, but also improves the accuracy of the spatial position relationship. Then, each image to be registered is registered according to the calculated spatial position relationship, which improves the accuracy of image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1A flowchart of an image registration method provided by an embodiment of the present disclosure;

[0027] Figure 2 A schematic diagram of a flow chart of another image registration method provided by an embodiment of the present disclosure;

[0028] Figure 3 A schematic diagram of another image registration method provided in an embodiment of the present disclosure;

[0029] Figure 4 A flowchart of a decoupled convolutional sparse coding model training process provided by an embodiment of the present disclosure;

[0030] Figure 5 A schematic diagram of the structure of an image registration device provided in an embodiment of the present disclosure;

[0031] Figure 6 A schematic diagram of the structure of an image registration device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0034] In the prior art, image registration is mainly performed by traditional image registration methods and deep learning-based image registration methods.

[0035] Traditional image registration methods usually use traditional feature point extraction algorithms first, then eliminate outliers through random sampling consistency algorithms, iterative reweighted least squares algorithms, etc., and estimate the homography matrix through the coordinates of four pairs of matching points. However, this method is heavily dependent on the quality of the extracted image. When the number of matching points is insufficient or the quality is too poor, the registration will fail. At the same time, such methods are often time-consuming and require extremely high computational complexity, which not only leads to low registration efficiency, but also low image registration accuracy.

[0036] One of the image registration methods based on deep learning is to extract the feature maps of multimodal images through deep neural networks and achieve registration by optimizing certain similarity metrics, but this method is highly dependent on similarity metrics and often only performs well on a certain metric. Another method is to generate adversarial networks, but this method is difficult to train and will also introduce noise, resulting in low image registration accuracy.

[0037] In view of the above problems, the embodiments of the present disclosure provide an image registration method, apparatus, device and computer-readable storage medium. The image registration method is first introduced below in conjunction with specific embodiments.

[0038] In the disclosed embodiment, the image registration method can be performed by an image registration device, wherein the image registration device can be an electronic device or a server. The electronic device can include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. The server can be an independent server or a cluster of multiple servers, including servers built locally and servers built in the cloud.

[0039] Figure 1 A flowchart of an image registration method provided in an embodiment of the present disclosure.

[0040] like Figure 1 As shown, the image registration method mainly includes the following steps:

[0041] S110: Obtain a group of images to be registered.

[0042] In the embodiment of the present disclosure, the image group to be registered is a set of at least two images having partially or completely the same features. Each image included in the above-mentioned image group to be registered is an image to be registered, and each image to be registered can be registered.

[0043] Taking the example that the image group to be registered includes two images to be registered, the registration of the two images to be registered can specifically be to find a spatial transformation to map one image to another image so that the points of the two images to be registered corresponding to the same position in space correspond one to one to achieve the purpose of registration.

[0044] Among them, the image can be a depth image taken by a lidar, a multispectral image taken by a multispectral camera, an RGB image taken by a camera, etc., which is not limited in the embodiments of the present disclosure.

[0045] Meanwhile, the images in the image group to be registered may be single-modality images or multi-modality images, which is not limited in the embodiments of the present disclosure.

[0046] S120: Acquire feature images that are helpful for registration and feature images that are not helpful for registration, included in each image to be registered in the group of images to be registered.

[0047] Generally speaking, the information between images acquired by different imaging devices at different time points and in different environments varies greatly, so not all features are helpful for registration, and some features are even harmful to registration.

[0048] Exemplarily, the depth image includes depth information but does not include color information, texture information, etc. Therefore, when the depth image is aligned with the RGB image, the color information and texture information included in the RGB image are useless features for the alignment of the depth image with the RGB image.

[0049] Therefore, in the embodiments of the present disclosure, it is necessary to distinguish between features that are helpful for registration and features that are not helpful for registration.

[0050] Furthermore, after distinguishing the features that are helpful for registration and the features that are not helpful for registration, the features that are helpful for registration are combined into a feature image that is helpful for registration, and the features that are not helpful for registration are combined into a feature image that is not helpful for registration.

[0051] S130 , calculating the spatial position relationship between the images to be registered based on the feature images that are helpful for registration and are included in the images to be registered.

[0052] In the embodiment of the present disclosure, S130 may specifically include: based on the feature images that are helpful for registration and included in each image to be registered, calculating the rigid deformation field and non-rigid deformation field of the feature images that are helpful for registration and included in each image to be registered; determining the spatial position relationship of the images to be registered according to the rigid deformation field and the non-rigid deformation field.

[0053] Generally speaking, image registration mainly includes rigid registration and non-rigid registration.

[0054] Therefore, in the embodiment of the present disclosure, the image registration device performs rigid registration on the feature images that are helpful for registration and included in each image to be registered, and obtains a rigid deformation field. The rigid registration often includes translation, rotation, scaling and shearing operations.

[0055] At the same time, the image registration device performs non-rigid registration on the feature images that are helpful for registration and included in each image to be registered, and obtains a non-rigid deformation field. The non-rigid registration often includes a coordinate value offset.

[0056] The rigid deformation field and non-rigid deformation field calculated above are the spatial position relationship of the characteristic images that are helpful for registration and included in each image to be registered, and the spatial position relationship of the characteristic images that are helpful for registration and included in the images to be registered are used as the spatial position relationship of the images to be registered.

[0057] S140: Register the images to be registered based on the spatial position relationship.

[0058] In the embodiment of the present disclosure, the image registration device performs spatial changes on each image to be registered according to the rigid deformation field to achieve rigid registration of each image to be registered.

[0059] And according to the non-rigid deformation field, each image to be registered is spatially changed to achieve non-rigid registration of each image to be registered.

[0060] After completing the above rigid registration and non-rigid registration, the registration of each image to be registered can be achieved.

[0061] It can be seen that after obtaining the group of images to be registered, the embodiment of the present disclosure obtains the feature images that are helpful for registration and the feature images that are not helpful for registration included in each image to be registered in the group of images to be registered. When calculating the spatial position relationship between the images to be registered, only the feature images that are helpful for registration are referred to, and the feature images that are not helpful for registration are ignored. Therefore, the spatial position relationship between the images to be registered is calculated only based on the feature images that are helpful for registration, which not only improves the calculation efficiency but also improves the accuracy of the spatial position relationship. Furthermore, the images to be registered are registered based on the calculated spatial position relationship, thereby improving the accuracy of image registration.

[0062] Figure 2 A flowchart of another image registration method provided by an embodiment of the present disclosure is shown in FIG. Figure 3 Another image registration method provided by the present disclosure is shown below. Figure 3 right Figure 2 Another image registration method is introduced as shown in Figure 2 As shown, the image registration method includes the following steps:

[0063] S210: Obtain a group of images to be registered.

[0064] In the embodiment of the present disclosure, this step is the same as step S110 and will not be described again here.

[0065] S220. Based on a pre-established decoupled convolutional sparse coding model, decouple each image to be registered in the image group to be registered, and output feature images that are helpful for registration and feature images that are not helpful for registration included in each image to be registered; wherein the decoupled convolutional sparse coding model is a model established based on a convolutional sparse representation algorithm.

[0066] Specifically, in the disclosed embodiment, a decoupled convolutional sparse coding model is established based on the convolutional sparse representation algorithm, and the decoupled convolutional sparse coding model is trained. After the decoupled convolutional sparse coding model is trained, each image to be registered in the image group to be registered is input into the decoupled convolutional sparse coding model, and the model can output feature images that are helpful for registration and feature images that are not helpful for registration, which are included in each image to be registered.

[0067] Taking the case where the image group to be registered includes two images to be registered as an example, the following introduces the establishment of a decoupled convolutional sparse coding model based on the convolutional sparse representation algorithm and the training process of the decoupled convolutional sparse coding model.

[0068] Among them, establishing a decoupled convolutional sparse coding model based on the convolutional sparse representation algorithm can be: based on the convolutional sparse representation algorithm, obtaining the convolutional dictionary atomic items of the feature images that are helpful for registration and included in each image to be registered, and configuring the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are helpful for registration and included in each image to be registered; based on the convolutional sparse representation algorithm, obtaining the convolutional dictionary atomic items of the feature images that are not helpful for registration and included in each image to be registered, and configuring the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are not helpful for registration and included in each image to be registered; establishing a decoupled convolutional sparse coding model based on the convolutional dictionary atomic items of the feature images that are helpful for registration, the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are helpful for registration, the convolutional dictionary atomic items of the feature images that are not helpful for registration, and the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are not helpful for registration.

[0069] Because, usually, when registering the images to be registered, the feature images that are helpful for the registration contain the same features, and the same features can link the images to be registered.

[0070] Therefore, in the embodiment of the present disclosure, when a decoupled convolutional sparse coding model is established based on the convolutional sparse representation algorithm, the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are helpful for registration are configured to be the same; and the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are not helpful for registration are configured to be different.

[0071] Therefore, the decoupled convolutional sparse coding model established in the embodiment of the present disclosure can be specifically represented by formula (1):

[0072]

[0073] Among them, I x is the image to be registered x, b p is the convolution dictionary atomic item of the pth feature image that is not helpful for registration, included in the image to be registered x, p∈{1,2,3...P}, u p is the convolution sparse response coefficient corresponding to the convolution dictionary atomic item of the pth feature image that is not helpful for registration included in the image to be registered x, d k is the convolution dictionary atomic item of the kth feature image that helps to register the image x, k∈{1,2,3...K}, r kis the convolution sparse response coefficient corresponding to the convolution dictionary atomic item of the kth feature image that helps to register included in the image to be registered x, I y is the image to be registered y, e q is the convolution dictionary atomic item of the qth non-contributory feature image included in the image to be registered y, q∈{1,2,3...Q}, v q is the convolution sparse response coefficient corresponding to the convolution dictionary atomic item of the qth feature image that is not helpful for registration included in the image to be registered y, h k is the convolution dictionary atomic item of the kth feature image that helps to register the image y to be registered, k∈{1,2,3...K}, r k is the convolution sparse response coefficient corresponding to the convolution dictionary atomic item of the kth feature image that helps to register included in the image to be registered y, * is the convolution symbol, The image to be registered y is mapped to the image to be registered x, so as to realize the registration of the image to be registered x and the image to be registered y.

[0074] In some embodiments, in order to simplify the subsequent calculation of related content and the training of the decoupled convolutional sparse coding model, the formula (1) is replaced by Move to the right side of the equation, therefore, the above formula (1) can be further transformed into the following formula (2):

[0075]

[0076] in, Map the image x to be registered to the image y to be registered.

[0077] It can be seen that in the embodiment of the present disclosure, when a decoupled convolutional sparse coding model is established based on the convolutional sparse representation algorithm, the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that contribute to the registration and are included in each image to be registered are set to be the same, so that the features included in each image to be registered are coupled and the images to be registered are associated.

[0078] Furthermore, after the decoupled convolutional sparse coding model is established, the decoupled convolutional sparse coding model needs to be trained.

[0079] Figure 4 A flowchart of the decoupled convolutional sparse coding model training process provided by the embodiment of the present disclosure is shown in FIG. Figure 4 As shown, training the decoupled convolutional sparse coding model specifically includes the following steps.

[0080] In the embodiment of the present disclosure, before training the decoupled convolutional sparse coding model, the training target of the decoupled convolutional sparse coding model is first determined from a mathematical perspective based on the basic principles of the convolutional sparse representation algorithm.

[0081] Specifically, in the embodiment of the present disclosure, the training target of the decoupled convolutional sparse coding model can be determined from a mathematical perspective as formula (3):

[0082]

[0083] Among them, R s is a spatial transformation module, which can be a custom fixed function, Ω is the spatial position relationship when the image to be registered x is mapped to the image to be registered y, ψ(r k ) is the constrained convolution sparse response coefficient r k Function of sparsity, λ r is the convolution sparse response coefficient r k The regularization coefficient, ψ(u p ) is the constrained convolution sparse response coefficient u p Function of sparsity, λ u is the convolution sparse response coefficient λ u The regularization coefficient, ψ(v q ) is the constrained convolution sparse response coefficient v q Function of sparsity, λ v is the convolution sparse response coefficient λ v The regularization coefficient of .

[0084] In some embodiments, ψ(r k ),ψ(u p ) and ψ(v q ) can be constrained by the L1 norm.

[0085] Since the convolution dictionary atomic items included in each image to be registered are usually known, when the convolution dictionary atomic items included in the image to be registered are known, it can be calculated by u p 、v q 、r k and Ω, so that the decoupled convolutional sparse coding model can achieve the training goal from a mathematical perspective.

[0086] S410 , based on a convolutional sparse representation algorithm, calculating convolutional sparse response coefficients corresponding to convolutional dictionary atomic items of feature images that are not helpful for registration and are included in each image to be registered.

[0087] In the embodiment of the present disclosure, based on the above training objectives, an iterative update calculation method is adopted to first calculate the convolution sparse response coefficient corresponding to the convolution dictionary atomic item of the feature image that is not helpful for registration, that is, to calculate u in formula (3). p 、v q The iterative update calculation method may specifically be: when calculating a certain variable, other variables are fixed.

[0088] Below is up 、v q The calculation process is introduced.

[0089] A.u p The calculation process.

[0090] Specifically, calculate u p When v q 、r k and Ω is fixed, that is, calculate u p When setting q 、r k and Ω are fixed values.

[0091] E q 、r k After and Ω are set to fixed values, the above formula (3) can be transformed into formula (4):

[0092]

[0093] Further, The above formula (4) can be converted into formula (5):

[0094]

[0095] in, It is the feature image that is not helpful for registration and is included in the image to be registered x.

[0096] Furthermore, using the convolutional sparse representation algorithm, u in the above formula p Calculate and solve for u p ,Right now Figure 3 The u shown p , and the above calculation u p The process is converted to I x The non-contributory feature image extraction module corresponding to the non-contributory feature image is included, that is, Figure 3 I x Included are the nRAFPMs corresponding to the feature images that do not contribute to the registration.

[0097] B.v q The calculation process.

[0098] Specifically, calculate v q When u p 、r k and Ω is fixed, that is, calculate v q When setting u p 、r k and Ω are fixed values.

[0099] In some embodiments, u pThe fixed value of can be the value obtained by the above calculation.

[0100] Therefore, the above formula (3) can be transformed into formula (6):

[0101]

[0102] Afterwards, The above formula (6) can be converted into formula (7):

[0103]

[0104] in, It is the feature image that is not helpful for registration and is included in the image y to be registered.

[0105] Furthermore, using the convolutional sparse representation algorithm, v in formula (7) q Calculate and solve for v q ,Right now Figure 3 The v shown q , and convert the above calculation v q The process is converted to I y The non-contributory feature image extraction module corresponding to the non-contributory feature image is included, that is, Figure 3 I y Included are the nRAFPMs corresponding to the feature images that do not contribute to the registration.

[0106] S420, based on the convolution sparse representation algorithm, calculate the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration and included in each image to be registered.

[0107] In the embodiment of the present disclosure, after the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are not helpful for registration are calculated in step S210, the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration included in each image to be registered are calculated by an iterative update calculation method, that is, r in formula (3) k . The following is r k The calculation process is introduced.

[0108] C.r k The calculation process.

[0109] Specifically, calculate r k When u p 、v q and Ω is fixed, that is, calculate r k When setting u p 、v q and Ω are fixed values.

[0110] In some embodiments, calculating r k When p 、v q The fixed value of can be u calculated in step S210 p 、v q The value of .

[0111] Since the calculation r k During the process, I x with I y r k , a new intermediate variable s needs to be introduced k To distinguish k .

[0112] In some embodiments, I x Calculate r k , that is, calculate r k When x r k No change, I y r k Set to variable s k .

[0113] In other embodiments, it is also possible to y Calculate r k , that is, calculate r k When y r k No change, I x Medium k The setting is variable s k .

[0114] Through I x Calculate r k For example, therefore, formula (3) can be transformed into formula (8)

[0115]

[0116] It should be noted that when using I x Calculate r k When you need to establish k =r k The constraint relationship is to ensure I x with I y Includes correlation of features that aid in registration.

[0117] Further, The above formula (8) can be further transformed into formula (9).

[0118]

[0119] in, is the feature image that helps with registration included in the image to be registered x, that is, Figure 3 shown

[0120] Then use the convolutional sparse representation algorithm to k Calculate and solve for r k ,Right now Figure 3 The r shown k , and the above calculation r k The process is converted to I x The feature image extraction module that helps with registration corresponds to the feature image that helps with registration, that is, Figure 3 RA FPM shown.

[0121] S430: Acquire a group of registered images as training samples.

[0122] In the embodiment of the present disclosure, in order to ensure that the above calculated r k More accurately, the r calculated in step S420 can be calculated using the registered image group. k Supervision is performed to further verify the r calculated in step S420. k .

[0123] Therefore, in the embodiment of the present disclosure, it is necessary to first obtain a group of registered images as training samples.

[0124] S440, obtaining, for each registered image in the registered image group, a feature image that is helpful for registration as a first label, a convolution dictionary atomic item of the feature image that is helpful for registration, and a feature image that is not helpful for registration as a second label, a convolution dictionary atomic item of the feature image that is not helpful for registration.

[0125] S450, inputting the convolution dictionary atomic items of the feature images that contribute to the registration and the convolution dictionary atomic items of the feature images that do not contribute to the registration included in each registered image in the registered image group into the decoupled convolutional sparse coding model, and training the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that contribute to the registration in the decoupled convolutional sparse coding model until the feature images that contribute to the registration output by the decoupled convolutional sparse coding model are the first label or converge to the first label.

[0126] Specifically, the calculated r k Substitute it into formula (3) and judge its second term, that is,

[0127]

[0128] Whether the minimum value is reached. When it cannot be reached, the loss function of the feature image extraction module that helps with registration is used. Back propagation further updates I x and I y r k , until formula (3) obtains the minimum value, the calculation of r is completed. k Therefore, the training of the decoupled convolutional sparse coding model is completed.

[0129] Among them, the loss function of the feature image extraction module that helps with registration is Specifically, it can be:

[0130]

[0131] in, is the feature image included in the registered image x that helps with registration, It is the feature image included in the registered image y that helps with the registration.

[0132] Further, get is the feature image that helps with registration included in the image to be registered, that is, Figure 3 shown

[0133] S230: Calculate the spatial position relationship between the images to be registered based on the feature images that are helpful for registration and are included in the images to be registered.

[0134] As in the above step S130 , rigid registration often includes translation, rotation, scaling and shearing operations, and non-rigid registration often includes coordinate value offset.

[0135] Therefore, in the embodiment of the present disclosure, the main task of rigid registration is to calculate the homography matrix between the feature images that are helpful for registration and included in each image to be registered.

[0136] The main task of non-rigid registration is to calculate the non-rigid deformation field between the feature images that are helpful for registration and included in each image to be registered. The calculation of the non-rigid deformation field can specifically be to calculate the coordinate value offset between the feature images that are helpful for registration and included in each image to be registered. The calculation process of the homography matrix and the non-rigid deformation field is introduced below.

[0137] D. Homography matrix (H matrix) calculation process.

[0138] Although the feature images that are helpful for registration are obtained in the above, there are still some feature points in the feature images that are helpful for registration that are included in each image to be registered that are more important during registration. Therefore, in the embodiment of the present disclosure, a Convolutional Block Attention Module (CBAM) is introduced. This module can assign different weights to each feature point in the feature images that are helpful for registration, and the weight can be used to indicate the importance of the feature point.

[0139] In the disclosed embodiment, the input of the attention mechanism module is the feature image that is helpful for registration and included in each image to be registered output by the above-mentioned decoupled convolutional sparse coding model, and the output is a saliency map. The saliency map is an image after weights are assigned to each feature point included in the feature image that is helpful for registration and included in each image to be registered.

[0140] Furthermore, the neural network model VGG16-Net is used to make predictions based on the input saliency map, select four feature points, and output the offset values ​​of the vertex coordinates of the four feature points, and then calculate the homography matrix (H matrix) based on the Tensor Direct Linear Transform (Tensor DLT) algorithm.

[0141] In some embodiments, the above-mentioned CBAM module and VGG16-Net module can be combined into a rigid deformation field model, the input of which is a feature image that helps with registration and included in each image to be registered, and the output is a homography matrix of the feature image that helps with registration and included in each image to be registered.

[0142] Rigid deformation field model The input is the feature image x included in the image to be registered that helps with registration And the feature image to be registered y includes the feature image that helps with registration For example, it can be expressed as follows:

[0143]

[0144] Among them, H is the homography matrix, that is, Figure 3 H shown. is the rigid deformation field module, that is Figure 3 shown

[0145] E. Calculation process of non-rigid deformation field.

[0146] Specifically, in the embodiment of the present disclosure, the same method as that for calculating the homography matrix is ​​firstly adopted, using the CBAM module to output the saliency map.

[0147] Furthermore, a semantic segmentation network module U-Net network module is used to make predictions based on the input saliency map and output coordinate value offsets.

[0148] In some embodiments, the above-mentioned CBAM module and U-Net can be combined into a non-rigid deformation field model, the input of which is a feature image that helps with registration and is included in each image to be registered, and the output is the coordinate value offset of the feature image that helps with registration and is included in each image to be registered.

[0149] Taking the non-rigid deformation field model as The input is the feature image x included in the image to be registered that helps with registration And the feature image to be registered y includes the feature image that helps with registration For example, it can be expressed as follows:

[0150]

[0151] in, is a non-rigid deformation field, that is Figure 3 shown is a non-rigid deformation field module, that is Figure 3 shown

[0152] In some embodiments, in order to avoid excessive offset of coordinate values ​​resulting in distortion of feature images included in each image to be registered that are helpful for registration, a deformation field smoothing loss function is used. Limit the non-rigid deformation field gradient.

[0153] Specifically, the deformation field smoothing loss function for:

[0154]

[0155] in, is a non-rigid deformation field The gradient in the x direction at point p is, is a non-rigid deformation field The gradient in the y direction at point p, P is the non-rigid deformation field A feature point of The number of all feature points in .

[0156] Optionally, in some embodiments, the I obtained above can be x The non-contributory feature image extraction module corresponding to the non-contributory feature image of the registration is included, yThe image is reconstructed by including a feature image extraction module corresponding to the feature image that is not helpful for registration, a feature image extraction module corresponding to the feature image that is helpful for registration, the spatial position relationship and the convolution dictionary to further verify the accuracy of the feature images extracted by the above-mentioned feature extraction modules and the obtained spatial position relationship.

[0157] Specifically, if the reconstructed images have a high degree of similarity with the images to be registered, it is considered that the feature images extracted by the above feature extraction modules and the obtained spatial position relationships are highly accurate; otherwise, it is considered that the feature images extracted by the above feature extraction modules and the obtained spatial position relationships are not accurate enough.

[0158] Specifically, the reconstruction can be performed in the following ways:

[0159]

[0160]

[0161] Among them, f nRA is a feature image extraction module that is not helpful for registration, f RA represents the feature image extraction module that helps with registration, R s is the space transformation module, that is Figure 3 R s , B p For I x Convolutional dictionary of feature images that do not contribute to registration, D k for Convolutional dictionary of feature images included to aid in registration, E q For I y Convolutional dictionary of feature images that do not contribute to registration, H k for Convolutional dictionaries of feature images included to aid in registration, is the image obtained after reconstruction, that is Figure 3 shown

[0162] In some embodiments, to reconstruct the image and Consistent with the original input image, define the reconstruction loss function Specifically, reconstruct the loss function for:

[0163]

[0164] S240: Register the images to be registered based on the spatial position relationship.

[0165] In the embodiment of the present disclosure, the final registered image may be:

[0166]

[0167] Right now Figure 3 The registration process shown in Figure 2. is the image x after registration.

[0168] In some embodiments, after the images to be registered are registered based on the spatial position relationship in step S240, in order to further improve the accuracy of the registration, the registered images and the real images can be supervised, specifically by adding a registration loss function. Registration loss function as follows:

[0169]

[0170] in, is the image x after registration, I gt For real images.

[0171] In some embodiments, when registering each image to be registered, an overall loss function Loss may be set. Specifically, the overall loss function Loss is:

[0172]

[0173] Among them, λ R ,λ C ,λ G ,λ s represents the weight of each loss, is the registration loss function, To reconstruct the loss function, is the loss function of the feature image extraction module that helps with registration, is the deformation field smoothing loss function.

[0174] It should be noted that when rigid registration is performed on each image to be matched, supervision of smoothing loss is not required. Therefore, when rigid registration is performed on each image to be matched, let λ s =0.

[0175] It can be seen that in the embodiment of the present disclosure, the training target of the decoupled convolutional sparse coding model is first determined from a mathematical perspective, and then the convolutional sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are not helpful for registration and the convolutional sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration are calculated in turn through the convolutional sparse representation algorithm, so that the calculation process is converted into a neural network. The calculation process explains the training process of the neural network and opens the "black box" of the neural network.

[0176] Furthermore, in the embodiments of the present disclosure, the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature image that assists in the registration are verified by using the registered image, thereby further improving the accuracy of the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature image that assists in the registration. Thus, the registration is performed based on the feature image that assists in the registration, and the registration result obtained is more accurate.

[0177] Furthermore, in the embodiment of the present disclosure, a CBAM module is introduced when calculating the homography matrix and the coordinate value offset. By configuring weights for each feature point included in the feature image that helps with the registration, feature points that are more helpful for the registration are further determined in the feature image that helps with the registration, thereby further improving the accuracy of the registration of the image to be registered.

[0178] Figure 5 The image registration device provided in the embodiment of the present disclosure is a schematic diagram of the structure of the image registration device. The image registration device may be the image registration device in the above embodiment, or the image registration device may be a component or assembly in the image registration device. The image registration device provided in the embodiment of the present disclosure may execute the processing flow provided in the image registration method embodiment, such as Figure 5 As shown, the image registration device 500 includes:

[0179] A first acquisition module 510 is used to acquire a group of images to be registered;

[0180] The second acquisition module 520 is used to acquire the feature images that are helpful for registration and the feature images that are not helpful for registration included in each image to be registered in the image group to be registered;

[0181] A calculation module 530, configured to calculate the spatial position relationship between the images to be registered based on the feature images that are helpful for registration and are included in the images to be registered;

[0182] The registration module 540 is used to register the images to be registered based on the spatial position relationship.

[0183] Optionally, the second acquisition module 520 includes a decoupling unit and an output unit. The decoupling unit is used to perform decoupling processing on each image to be registered in the image group to be registered based on a pre-established decoupled convolutional sparse coding model. The output unit is used to output the feature images that are helpful for registration and the feature images that are not helpful for registration included in each image to be registered; wherein the decoupled convolutional sparse coding model is a model established based on a convolutional sparse representation algorithm.

[0184] Optionally, the second acquisition module 520 includes an establishment unit, which is used to obtain, based on a convolutional sparse representation algorithm, convolutional dictionary atomic items of feature images that are helpful for registration and included in each image to be registered, and configure convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are helpful for registration and included in each image to be registered; based on a convolutional sparse representation algorithm, obtain convolutional dictionary atomic items of feature images that are not helpful for registration and included in each image to be registered, and configure convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are not helpful for registration and included in each image to be registered; establish a decoupled convolutional sparse coding model based on the convolutional dictionary atomic items of the feature images that are helpful for registration, the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are helpful for registration, the convolutional dictionary atomic items of the feature images that are not helpful for registration, and the convolutional sparse response coefficients corresponding to the convolutional dictionary atomic items of the feature images that are not helpful for registration.

[0185] Optionally, when the establishing unit is used to configure the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration included in each image to be registered, the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration included in each image to be registered are specifically configured to be the same.

[0186] Optionally, when the establishing unit is used to configure the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are not helpful for registration included in each image to be registered, it is specifically used to: configure the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are not helpful for registration included in each image to be registered to be different.

[0187] Optionally, the image registration device 500 includes a training module for calculating, based on a convolutional sparse representation algorithm, convolutional sparse response coefficients corresponding to convolutional dictionary atomic items of feature images that are not helpful for registration and included in each image to be registered; calculating, based on a convolutional sparse representation algorithm, convolutional sparse response coefficients corresponding to convolutional dictionary atomic items of feature images that are helpful for registration and included in each image to be registered; obtaining a group of registered images as training samples; obtaining a feature image that is helpful for registration and included in each registered image in the group of registered images as a first label and a convolutional dictionary atomic item of the feature image that is helpful for registration; Item, a feature image that is not helpful for registration as the second label, and a convolution dictionary atomic item of the feature image that is not helpful for registration; the convolution dictionary atomic items of the feature image that is helpful for registration and the convolution dictionary atomic items of the feature image that are not helpful for registration included in each registered image in the registered image group are input into the decoupled convolutional sparse coding model, and the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature image that is helpful for registration in the decoupled convolutional sparse coding model are trained until the feature image that is helpful for registration output by the decoupled convolutional sparse coding model is the first label or converges to the first label.

[0188] Optionally, the calculation module 530 is used to calculate the spatial position relationship between each image to be registered based on the feature image that helps with registration included in each image to be registered. Specifically, it is used to calculate the rigid deformation field and non-rigid deformation field of the feature image that helps with registration included in each image to be registered based on the feature image that helps with registration included in each image to be registered; determine the spatial position relationship of the images to be registered based on the rigid deformation field and the non-rigid deformation field.

[0189] Figure 5 The image registration device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar and will not be described in detail here.

[0190] Figure 6 The image registration device provided in the embodiment of the present disclosure can execute the processing flow provided in the image registration method embodiment, such as Figure 6 As shown, the image registration device 60 includes: a memory 61, a processor 62, a computer program and a communication interface 63; wherein the computer program is stored in the memory 61 and is configured so that the processor 62 executes the above image registration method.

[0191] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the image registration method of the above embodiment.

[0192] In addition, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the image registration method of the above embodiment is implemented.

[0193] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0194] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image registration method, characterized in that: The method comprises: Obtaining a group of images to be registered; Acquire a feature image that is helpful for registration and a feature image that is not helpful for registration, included in each image to be registered in the group of images to be registered; Calculating the spatial position relationship between the images to be registered based on the feature images that are helpful for registration and are included in the images to be registered; Based on the spatial position relationship, registering the images to be registered; The step of acquiring the feature images that are helpful for registration and the feature images that are not helpful for registration included in each image to be registered in the group of images to be registered includes: Based on a pre-established decoupled convolutional sparse coding model, each image to be registered in the group of images to be registered is decoupled, and feature images that are helpful for registration and feature images that are not helpful for registration included in each image to be registered are output; wherein the decoupled convolutional sparse coding model is a model established based on a convolutional sparse representation algorithm.

2. The method according to claim 1, characterized in that The decoupled convolutional sparse coding model is a model established based on a convolutional sparse representation algorithm, including: Based on the convolution sparse representation algorithm, the convolution dictionary atomic items of the feature images that are helpful for registration included in each of the images to be registered are obtained, and the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration included in each of the images to be registered are configured; Based on the convolution sparse representation algorithm, the convolution dictionary atomic items of the feature images that are not helpful for registration and included in each of the images to be registered are obtained, and the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are not helpful for registration and included in each of the images to be registered are configured; The decoupled convolutional sparse coding model is established based on the convolution dictionary atomic items of the feature image that helps with registration, the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature image that helps with registration, the convolution dictionary atomic items of the feature image that does not help with registration, and the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature image that does not help with registration.

3. The method according to claim 2, characterized in that The configuration of the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration and that are included in each image to be registered includes: The convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration and included in the images to be registered are configured to be the same.

4. The method according to claim 2, characterized in that: The convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are not helpful for registration and are included in each image to be registered include: The convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are not helpful for registration and are included in each image to be registered are different.

5. The method according to claim 2, characterized in that: The decoupled convolutional sparse coding model is trained by the following steps: Based on the convolution sparse representation algorithm, calculate the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are not helpful for registration and are included in each of the images to be registered; Based on the convolution sparse representation algorithm, calculating the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that are helpful for registration and included in each of the images to be registered; Obtain a group of registered images as training samples; Acquire, as a first label, a convolution dictionary atomic item of the feature image that is helpful for registration, and, as a second label, a convolution dictionary atomic item of the feature image that is not helpful for registration, each registered image in the registered image group includes a feature image that is helpful for registration; The convolution dictionary atomic items of the feature images that assist in registration and the convolution dictionary atomic items of the feature images that do not assist in registration included in each registered image in the registered image group are input into a decoupled convolutional sparse coding model, and the convolution sparse response coefficients corresponding to the convolution dictionary atomic items of the feature images that assist in registration in the decoupled convolutional sparse coding model are trained until the feature images that assist in registration output by the decoupled convolutional sparse coding model are the first label or converge to the first label.

6. The method according to claim 1, characterized in that The calculating the spatial position relationship between the images to be registered based on the feature images that are helpful for registration and are included in the images to be registered comprises: Based on the feature images that are helpful for registration and included in each of the images to be registered, calculating the rigid deformation field and the non-rigid deformation field of the feature images that are helpful for registration and included in each of the images to be registered; The spatial position relationship of the images to be registered is determined according to the rigid deformation field and the non-rigid deformation field.

7. An image processing device, characterized in that: The device comprises: A first acquisition module, used for acquiring a group of images to be registered; A second acquisition module is used to acquire the feature images that are helpful for registration and the feature images that are not helpful for registration included in each image to be registered in the group of images to be registered; A calculation module, used for calculating the spatial position relationship between the images to be registered based on the feature images that are helpful for registration and are included in the images to be registered; A registration module, used for registering the images to be registered based on the spatial position relationship; The second acquisition module includes a decoupling unit and an output unit; the decoupling unit is used to perform decoupling processing on each image to be registered in the group of images to be registered based on a pre-established decoupled convolutional sparse coding model; the output unit is used to output feature images that are helpful for registration and feature images that are not helpful for registration included in each image to be registered; wherein the decoupled convolutional sparse coding model is a model established based on a convolutional sparse representation algorithm.

8. An image registration device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the image registration method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the image registration method according to any one of claims 1 to 6.

Images