Multimodal image registration method based on attention mechanism and contrastive learning

Abstract

The present invention relates to the field of image registration technology, and discloses a multimodal image registration method based on attention mechanism and contrastive learning, comprising inputting an initial modal image A and an initial modal image B into a registration network, generating a deformation field, and optimizing the generation of the deformation field; inputting the initial modal image A into a translation network and obtaining a translated modal image B. t ; Translate modal image B t The image deformed by the deformation field is subjected to pixel block level contrast loss with the initial modal image A; the translated modal image B t The image deformed by the deformation field is subjected to a pixel-by-pixel L1 loss compared to the initial modal image B, prompting the registration network to generate a smooth deformation field. The translation network and registration network are trained and optimized to minimize the sum of the pixel-level contrast losses. This method achieves high registration accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and in particular to a multimodal image registration method based on attention mechanism and contrast learning. Background Art

[0002] From classic disciplines like medical imaging and remote sensing to emerging fields like autonomous driving, using different sensors to collect scene data is a common data collection method in various fields. Multimodal sensors are one of the most commonly used sensor types. Multimodal sensors allow for the collection of a wide range of physical properties, resulting in richer scene representations.

[0003] For multimodal sensors, accurate alignment of multimodal images is crucial to accurately utilize different aspects of sensor information. However, this remains a challenging task due to the complex and unknown spatial correspondences between different modalities.

[0004] To address this, there are some existing registration methods for multimodal images, including registration methods based on image translation. By converting images of one modality into images of another modality, the more complex and difficult-to-evaluate multimodal registration problem is converted into a simpler and easier-to-evaluate single-modal registration problem, thereby achieving effective registration of multimodal images. However, most registration methods based on image translation use GAN (Generative Adversarial Network) for data processing. Since the training process of GAN is an adversarial training process, in which the generator and discriminator compete with each other, there are problems of difficulty in convergence and mode collapse. The adversarial process also introduces inconsistencies and artifacts. The discriminator is prone to mistakenly encode deformations in a specific field as indispensable appearance features and encourages the generator to reproduce deformations. This often brings unnecessary difficulties to the registration task and greatly affects the registration accuracy. Summary of the Invention

[0005] The present invention aims to provide a multimodal image registration method based on attention mechanism and contrastive learning, which can achieve high registration accuracy and efficiency.

[0006] The basic solution provided by the present invention is: a multimodal image registration method based on attention mechanism and contrastive learning, comprising the following steps:

[0007] S1, input the initial modality image A and the initial modality image B into the registration network to generate a deformation field, and optimize the generation of the deformation field according to the optimization function;

[0008] S2, input the initial modal image A into the translation network to obtain the translated modal image B t ; Translate modal image B tA pixel-level contrast loss is applied to the initial modality image A to encourage the translation network to preserve object shapes;

[0009] S3, translates the modal image B t The image deformed by the deformation field is subjected to pixel-by-pixel L1 loss with the initial modal image B to maximize the appearance similarity;

[0010] S4, performing pixel block level contrast loss on the image after deformation of the initial modality image A and the initial modality image B, so as to enable the registration network to align at the pixel block level;

[0011] S5, calculating the pixel-by-pixel L1 loss of the transformed image of the initial modal image A and the translated image of the initial modal image B;

[0012] S6, uses deformation field regularization terms to encourage the registration network to generate a smooth deformation field;

[0013] S7, with the goal of minimizing the sum of the contrast loss at the pixel block level, trains and optimizes the translation network and the registration network.

[0014] The working principle and advantages of the present invention are:

[0015] In this solution, the translation network and registration network are trained and optimized by performing multiple pixel block-level contrast loss processing on different image combinations, and then optimizing the pixel block-level contrast loss sum. The overall process is simple and can effectively improve the registration performance.

[0016] The key point is this: Compared to existing registration methods, while this solution also utilizes a translation network, it does not use a discriminator. By removing the discriminator and employing contrastive learning for image translation, this solution effectively addresses the difficulties in network convergence and pattern collapse during adversarial training caused by GAN networks in existing registration methods, achieving higher registration accuracy. Furthermore, this solution trains with the goal of minimizing the sum of the aforementioned pixel-level contrast losses, resulting in a more stable and shorter training process. Furthermore, the attention mechanism employed in the pixel-level contrast loss (i.e., the attention-based contrastive learning loss) effectively improves the performance of contrastive learning for image translation, thereby further enhancing registration performance. Furthermore, the application of the attention mechanism does not introduce additional parameters, improving training effectiveness without excessively increasing training costs.

[0017] In particular, this scheme overcomes a difficulty in improving existing registration methods. Conventional registration methods that utilize translation networks often retain the discriminator to ensure the performance of the translation network and ensure that the image translation quality meets registration requirements. However, the presence of the discriminator can easily mistakenly encode domain-specific deformations as essential appearance features, resulting in registration errors. Furthermore, removing the discriminator directly impacts the performance of the translation network, making translation quality difficult to guarantee, and the impact of registration errors even greater. Therefore, most existing registration schemes retain the discriminator, resulting in a relatively low upper limit on registration accuracy. This scheme, however, removes the discriminator and, through multiple contrastive learning corruptions and a pixel-by-pixel L1 loss, effectively addresses the difficulty in ensuring translation quality caused by the removal of the discriminator. Through loss optimization, this improves translation quality and fundamentally eliminates registration errors that may be caused by the discriminator, resulting in higher overall registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the overall method flow of an embodiment of a multimodal image registration method based on attention mechanism and contrastive learning of the present invention;

[0019] Figure 2 Schematic diagram of the pixel block level contrast loss steps of an embodiment of the multimodal image registration method based on attention mechanism and contrast learning of the present invention. DETAILED DESCRIPTION

[0020] The following is a further detailed description through specific implementation methods:

[0021] The embodiment is basically as shown in the attached Figure 1 As shown: A multimodal image registration method based on attention mechanism and contrastive learning includes the following steps:

[0022] S1: Input the initial modality image A and the initial modality image B into the registration network to generate a deformation field, and optimize the deformation field according to the optimization function.

[0023] Specifically, the registration network is a registration network based on a U-net structure.

[0024] The deformation field is a two-dimensional deformation field that represents the offset of the horizontal and vertical coordinates of each pixel in the corresponding initial modal image. Specifically, in this embodiment, a series of optimization functions are provided to optimize the generation of the deformation field. These optimization functions are a combination of the loss functions involved in the following steps. By optimizing the loss, a more accurate deformation field can be generated, which helps improve image registration accuracy.

[0025] S2, input the initial modal image A into the translation network to obtain the translated modal image B t ; Translate modal image Bt A pixel-level contrast loss is applied to the initial modality image A to encourage the translation network to preserve object shapes.

[0026] The translation network is an encoder-decoder. After the translation network, only the modality of the image changes, while the content of the image remains unchanged. The translated image can retain the overall shape of the image, which is conducive to extracting global information.

[0027] The contrast loss is an attention-based contrast learning loss. Figure 2 As shown, the pixel block level contrast loss includes the following steps:

[0028] T1: Encode the initial modal image A through the encoder to obtain a three-dimensional feature map; compress the dimension of the three-dimensional feature map to obtain the features of each pixel block in the three-dimensional feature map, and obtain the correlation matrix of each pixel block through transposition and matrix multiplication;

[0029] T2, select key pixels from the 3D feature map through entropy sorting to obtain the correlation matrix of key pixel blocks;

[0030] T3, multiply the correlation matrix of the key pixel block with the feature matrix of each pixel block to obtain the features of the key pixel block;

[0031] T4, perform the same operations as T1 to T3 on the initial modality image B;

[0032] T5: Use contrastive learning loss on the features of the key pixel blocks of the initial modality image A and the initial modality image B. This loss corresponds to a contrastive loss at the pixel block level.

[0033] S3, translates the modal image B t The image deformed by the deformation field is subjected to a pixel-by-pixel L1 loss with the initial modal image B to maximize the appearance similarity. The L1 loss refers to the absolute value of the loss.

[0034] S4, performs pixel block level contrast loss on the image after deformation of the initial modal image A through the deformation field and the initial modal image B, so that the registration network can align at the pixel block level.

[0035] S5 , calculating the pixel-by-pixel L1 loss of the image obtained by deforming the translated initial modal image A and the image obtained by transforming the translated initial modal image B.

[0036] S6, using a deformation field regularization term to force the registration network to generate a smooth deformation field. The deformation field regularization term is an L2 norm.

[0037] In this embodiment, the L2 norm is the Euclidean norm, which refers to the square root of the sum of the squares of the elements in the vector. By setting a smooth deformation field, excessive deformation of the image can be effectively avoided.

[0038] The smooth deformation field is the final registration result output by the registration network; after the initial image is deformed by the smooth deformation field, the registration of the image is completed.

[0039] S7, with the goal of minimizing the sum of the contrast loss at the pixel block level, trains and optimizes the translation network and the registration network.

[0040] This embodiment provides a multimodal image registration method based on an attention mechanism and contrastive learning. This method removes the discriminator from the image translation process, avoiding the instability issues such as convergence difficulties and mode collapse associated with previous adversarial training. Furthermore, the use of contrastive learning effectively improves image translation performance. Furthermore, the use of an attention mechanism to select key points for contrastive learning further enhances the effectiveness of contrastive learning and contributes to achieving superior registration performance.

[0041] The above is only an embodiment of the present invention. Common knowledge such as the specific structure and characteristics of the scheme is not described in detail here. Ordinary technicians in the relevant field are aware of all common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the guidance of this application. Some typical well-known structures or well-known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several variations and improvements can be made, which should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A multimodal image registration method based on attention mechanism and contrastive learning, characterized by: The following steps are involved: S1, input the initial modality image A and the initial modality image B into the registration network to generate a deformation field, and optimize the generation of the deformation field according to the optimization function; S2, input the initial modal image A into the translation network to obtain the translated modal image ; will translate the modal image Perform pixel-level contrast loss with the initial modality image A to encourage the translation network to preserve object shape; S3, will translate the modal image The image deformed by the deformation field is subjected to pixel-by-pixel L1 loss with the initial modal image B to maximize the appearance similarity; S4, performing pixel-level contrast loss on the image after deformation of the initial modality image A and the initial modality image B, so that the registration network can align at the pixel level; S5, calculating the pixel-by-pixel L1 loss of the transformed image of the initial modal image A and the translated image of the initial modal image B; S6, uses deformation field regularization terms to encourage the registration network to generate a smooth deformation field; S7: Training and optimizing the translation network and the registration network with the goal of minimizing the sum of the above-mentioned pixel-level contrast loss, where the contrast loss is an attention-based contrast learning loss.

2. The multimodal image registration method based on attention mechanism and contrastive learning according to claim 1, characterized in that: In S2, the pixel-level contrast loss includes the following steps: T1: Encode the initial modal image A through the encoder to obtain a three-dimensional feature map; compress the dimension of the three-dimensional feature map to obtain the features of each pixel in the three-dimensional feature map, and obtain the correlation matrix of each pixel through transposition and matrix multiplication; T2, select key pixels from the three-dimensional feature map through entropy sorting and obtain the correlation matrix of key pixels; T3, multiply the correlation matrix of the key pixel points by the feature matrix of each pixel point to obtain the features of the key pixel points; T4, perform the same operations as T1 to T3 on the initial modality image B; T5, contrastive learning loss is used for the features of key pixels of the initial modality image A and the initial modality image B.

3. The multimodal image registration method based on attention mechanism and contrastive learning according to claim 1, characterized in that: The registration network is a registration network based on the U-net structure.

4. The multimodal image registration method based on attention mechanism and contrastive learning according to claim 1, characterized in that: The deformation field is a two-dimensional deformation field, and the deformation field is used to display the offset of the horizontal and vertical coordinates of each pixel point in the corresponding initial modality image.

5. The multimodal image registration method based on attention mechanism and contrastive learning according to claim 1, characterized in that: The translation network is an encoder-decoder.

6. The multimodal image registration method based on attention mechanism and contrastive learning according to claim 1, characterized in that: The deformation field regularization term is the L2 norm.

Images