Multi-frame agent-based multi-modal medical image flexible registration method

Abstract

The present application relates to a kind of multi-modal medical image flexible registration algorithm based on multi-frame intelligent agent, based on reinforcement learning design a new end-to-end multi-modal image registration method, by soft actor-critic algorithm SAC driven training, can imitate the step-by-step registration process of human expert, improve the accuracy of high-dimensional registration action.In view of the extremely complex multi-modal environment, there are severe challenges in pixel-level control in three-dimensional space, the present application combines reinforcement learning with planner network, encourages artificial agent to learn more accurate registration action explicitly from the state frame that has been generated, overcomes the challenge from multi-modal and high-dimensional continuous action space with the advantage of space-time dimension, avoids introducing deep neural network with huge amount of parameters, and has strong robustness and generalization ability, can drive model to twist and move image in the correct direction.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a flexible registration method for multimodal medical images based on multi-frame intelligent agents. Background Technology

[0002] In the field of medical imaging, images of different modalities contain different anatomical information. For example, T1-weighted magnetic resonance (MR) imaging can reflect anatomical images that conform to human intuition, while T2-weighted MR imaging is affected by the water content of lesion tissue and presents localized highlights. This makes it easier to effectively observe the differences between normal and diseased tissues. Stable multimodal image registration can fuse information from different modalities to achieve complementary advantages, thereby helping doctors to accurately diagnose diseases.

[0003] Medical image registration is the process of mapping image pairs to the same spatial coordinate system by finding spatial correspondences between them. Existing methods for medical image registration include traditional feature-based methods and learning-based methods. Traditional image registration methods rely on manually extracting features to calculate the similarity measure of image pairs, but they struggle to extract effective features when dealing with complex multimodal images. Learning-based methods can automatically capture high-dimensional abstract features, enhancing the robustness of the registration algorithm. This method typically completes image registration using a one-time alignment, but it is difficult to handle images with large deformations or displacements.

[0004] Traditional flexible image registration is formulated as an optimization process aimed at facilitating image alignment by maximizing the similarity metric between moving and stationary images, while penalizing non-smooth distortions in the deformation field. However, this approach is typically computationally intensive and time-consuming. Consequently, traditional registration methods often struggle to obtain good image features for accurate global alignment.

[0005] In recent years, the continuous development of deep learning technology has promoted progress in the field of image registration. Deep learning (DL) methods typically use convolutional neural networks to automatically capture high-dimensional abstract features from input image pairs, overcoming the obstacle of manual feature extraction and further improving registration accuracy. However, to further enhance the ability to capture abstract features, deep network structures are often used, which undoubtedly increases computational overhead. This method generally completes image registration in a one-time alignment manner, making it difficult to handle images with large deformations or displacements, and the computation is extremely complex. Deep reinforcement learning (DRL) methods, on the other hand, treat the registration process as a Markov decision process, allowing the DRL agent to freely explore a predefined action space, accumulating experience through trial and error. Finally, it can quickly make corresponding decisions based on specific environments to achieve high-precision registration. This makes each step of image registration visible, overcoming the opacity inherent in deep learning methods. However, due to the difficulty in training DRL and the complexity of multimodal processing, and because flexible image registration has a huge number of parameters, complex spatial mapping relationships, and a high-dimensional and continuous action space, its application to flexible image registration remains very challenging.

[0006] The shortcomings of existing technical solutions (especially those relevant to this invention) are as follows: Current deep learning methods typically use a one-time alignment approach for image registration, which is difficult to handle images with large deformations or displacements. The results also lack good interpretability, and the extraction of high-dimensional abstract features usually requires deeper and more complex networks, increasing training instability and placing higher demands on computational performance. Due to the difficulty in training reinforcement learning and the complexity of multimodal registration environments, these methods can generally only handle rigid multimodal transformations, and reinforcement learning framework designers often neglect the extraction of temporal features. Deep reinforcement learning methods mostly require large amounts of memory to store historical experience; therefore, simplifying the registration model and improving registration efficiency are urgent problems to be solved in medical image registration. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a flexible registration method for multimodal medical images based on a multi-frame intelligent agent. The method is characterized by a multi-frame intelligent agent framework for 3D multimodal flexible image registration based on a soft actor-critic algorithm. This framework introduces the concept of a planner into the traditional actor-critic framework. The planner observes multiple consecutive states and generates a low-dimensional plan, which serves as a template for generating high-dimensional actions of the actor and participates in the critic's evaluation. This allows for the prediction of high-dimensional actions from multi-frame state fusion in a completely unsupervised manner, guiding the model to complete image generation and policy control. Specifically, this includes:

[0008] Step 1: Prepare the image dataset to be registered, including structural images T1w and T2w, where T1w is a fixed image and T2w is a moving image;

[0009] Step 2: Resample all images to be registered to a size of 128*128*128 using max-min scaling normalization, and set T1w to a fixed image I. f T2w is the moving image I m Initial state S t=0 For image pair {I m ,I f};

[0010] Step 3: Create a state queue Q of size 3 to store the state of the most recent 3 frames [S] t-2 ,S t-1 ,S t The queue consists of 3 S t=0 Initially, three networks are constructed: a planner network, an actor network, and a critic network, specifically:

[0011] The planner network consists of 5 downsampling modules, each containing two convolutional layers and one residual module. The residual module also consists of two convolutional layers, and the LeakyReLU activation function is used on the output of each convolutional layer.

[0012] The actor network contains five upsampling modules, and its structure is consistent with that of the planner network. The actor network also has an additional output module, which takes the output of the fifth upsampling module as its input and generates the deformation field. The input of each upsampling module is formed by concatenating the output of the previous module and the feature map provided by the skip connection in a channel manner.

[0013] The critic network consists of 5 downsampling modules, in which spectral normalization is used to smooth the network gradient. After the 5th downsampling module, an additional output module is introduced, which consists of convolutional layers, LeakyReLU activation layers and linear layers. The tensors output by the convolutional layers are flattened and fed into the linear layers.

[0014] Step 4: Input the state queue Q created in Step 3 into the planner network. The planner simultaneously observes the three most recent states [S]. t-2 ,S t-1 ,S t ], and downsample it to the corresponding low-dimensional representation [Z t-2 Z t-1 Z t Z is represented in low dimension. t The offset approximates the spatiotemporal information, i.e., [Z t -Z t-1 Z t-1 -Z t-2 The mean and variance are calculated from the low-dimensional representation obtained by downsampling and the offset, and then fitted to generate a Gaussian distribution. The potential plan P... t It is obtained by random sampling from a Gaussian distribution; therefore, the potential plan P t The spatiotemporal characteristics of the registration process were explicitly considered;

[0015] Step 5: Develop potential plans P t The input is processed into the actor network, and the feature maps output by each downsampling module in the planner network are concatenated into the actor network according to the number of channels via skip connections. Simultaneously, a detailed high-dimensional deformation field is reconstructed using the detailed features and low-dimensional latent plans from the planner network. This high-dimensional deformation field is then treated as an action and applied to the moving image I through a spatial transformation network. m This causes it to undergo flexible deformation to obtain the predicted image, and the predicted image and the fixed image I are then compared. f The mutual information loss and the spatial smoothing term of the deformation field are used as unsupervised registration losses. Image pairs {I} are computed using an unsupervised modality-independent neighborhood description operator in a flexible registration environment. m ,I f} and the deformed image pair {I m ,I f The similarity metric is used as an unsupervised feedback reward.

[0016] Step 6: Set the current state S t and potential plans P t Input the critic network, and the critic network captures the current state S. t Abstract the characteristics and analyze them, then compare these characteristics with the potential plan P. t The features are concatenated along the channel dimension to form a fusion feature. Finally, the output module of the critic network analyzes the fusion feature again and outputs the maximum expected reward for executing the decision action in the current state as the loss for reinforcement learning. The deformed image is then used to evaluate {I}. m ,I f Update the state and state queue Q in the registration environment;

[0017] Step 7: Repeat the above steps until the deformed moving image I is obtained. m and fixed image I f When the modality-independent neighborhood descriptor value is less than the set value of 0.006, the precise registration state is reached, and the current registration round ends;

[0018] Step 8: Finally, a perfectly aligned and registered image is obtained.

[0019] The beneficial effects of this invention are as follows:

[0020] 1. This invention utilizes the differences between low-dimensional representations of multi-frame states to represent spatiotemporal information, thereby encouraging agents to generate more accurate high-dimensional registration actions. The framework is based on an unsupervised loss function, and the reward function of this invention is designed to provide real-time decision feedback to the agent, with training performed in an end-to-end manner. Experimental results on a 3D multimodal dataset demonstrate that this invention outperforms existing classical methods and deep learning methods.

[0021] 2. The multi-frame intelligent agent framework for 3D multimodal flexible image registration driven by the soft actor-critic algorithm proposed in this invention introduces a planner network into the traditional reinforcement learning framework, mapping multiple high-dimensional state frames into a low-dimensional latent space. The actor network includes an upsampling structure that re-decodes the low-dimensional plan into a high-dimensional deformation field. In this framework, the critic no longer directly evaluates high-dimensional actions but instead optimizes low-dimensional latent actions. The critic can promote model convergence by evaluating low-dimensional actions.

[0022] 3. In this invention, spectral normalization is used in the downsampling module of the critic network to smooth the network gradient, which promotes the training effect of the registration framework. The introduction of spectral normalization makes it possible to use deeper, larger neural networks in future work. Attached Figure Description

[0023] Figure 1 This is the flexible registration framework of the method of the present invention;

[0024] Figure 2 This is a schematic diagram of the flexible registration network of the present invention;

[0025] Figure 3 yes Figure 2 A schematic diagram of the specific structure of the middle module;

[0026] Figure 4 A schematic diagram of the multi-frame plan generated by this invention;

[0027] Figure 5 This is a comparison chart of the experimental results of various methods on the HCP dataset;

[0028] Figure 6 It is a step-by-step visual comparison chart of different reinforcement learning frameworks. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0030] The following is a detailed explanation with reference to the accompanying drawings.

[0031] To address the issue that deep learning methods typically employ a one-time alignment approach for registration, this invention utilizes a reinforcement learning framework to implement a stepwise image registration process, making the process visualized. Due to the excellent performance of the soft actor-critic algorithm in high-dimensional motion space continuous control tasks, this invention combines this algorithm with a planner, enabling the planner to guide actors in generating high-dimensional continuous movements. To improve registration accuracy in complex multimodal environments, this invention uses multi-frame planning to leverage redundant states generated during the stepwise registration process, extracting spatiotemporal features to achieve pixel-level control in complex multimodal environments, realizing image registration from coarse to fine.

[0032] To address the shortcomings of existing technologies, this invention proposes a multimodal medical image flexible registration method based on a multi-frame intelligent agent. This registration method proposes a multi-frame intelligent agent framework for 3D multimodal flexible image registration based on a soft actor-critic algorithm. This framework introduces the concept of a planner into the traditional actor-critic framework. The planner observes multiple consecutive states at once and generates a low-dimensional plan, which serves as a template for generating high-dimensional actions of the actor and participates in the critic's evaluation. It predicts high-dimensional actions from multi-frame state fusion in a completely unsupervised manner, guiding the model to complete image generation and policy control. Figure 1 This is the flexible registration framework of the method of the present invention. Figure 2 This is a schematic diagram of the flexible registration network of the present invention. Figure 1 and Figure 2 As shown, the registration method of the present invention specifically includes:

[0033] Step 1: Prepare the image dataset to be registered, including structural images T1w and T2w, where T1w is a fixed image and T2w is a moving image;

[0034] Step 2: Resample all images to be registered to a size of 128*128*128 using max-min scaling normalization, and set T1w to a fixed image I. f T2w is the moving image I mInitial state S t=0 For image pair {I m ,I f};

[0035] Step 3: Create a state queue Q of size 3 to store the state of the most recent 3 frames [S] t-2 ,S t-1 ,S t The queue consists of 3 S t=0 Initially, three networks are constructed: a planner network, an actor network, and a critic network, specifically:

[0036] The planner network consists of 5 downsampling modules, each containing two convolutional layers and one residual module. The residual module also consists of two convolutional layers, and the LeakyReLU activation function is used on the output of each convolutional layer.

[0037] The actor network contains five upsampling modules, and its structure is consistent with that of the planner network. The actor network also has an additional output module, which takes the output of the fifth upsampling module as its input and generates the deformation field. The input of each upsampling module is formed by concatenating the output of the previous module and the feature map provided by the skip connection in a channel manner.

[0038] The critic network consists of 5 downsampling modules, in which spectral normalization is used to smooth the network gradient. After the 5th downsampling module, an additional output module is introduced, which consists of convolutional layers, LeakyReLU activation layers, and linear layers. The tensors output by the convolutional layers are flattened and fed into the linear layers.

[0039] Figure 3 yes Figure 2 The diagram illustrates the specific structure of the middle section modules. Specifically, it includes sampling modules for planners and actor networks, a critic sampling module, actor output modules, and critic output modules, as detailed below. Figure 3 As shown.

[0040] Figure 4 This invention provides a schematic diagram of generating a multi-frame plan. The planner network simultaneously observes the current state and the states of the two past time points, generating corresponding low-dimensional representations for each. Then, it uses their offsets to learn the differences between these state frames, ultimately generating a latent plan P. t The code encodes spatiotemporal information, and the latent plan can be used to guide the generation of more refined deformation fields.

[0041] Step 4: Input the state queue Q created in Step 3 into the planner network. The planner simultaneously observes the three most recent states [S]. t-2 ,S t-1 ,S t ], and downsample it to the corresponding low-dimensional representation [Zt-2 Z t-1 Z t Z is represented in low dimension. t The offset approximates the spatiotemporal information, i.e., [Z t -Z t-1 Z t-1 -Z t-2 The mean and variance are calculated from the low-dimensional representation obtained by downsampling and the offset, and a Gaussian distribution is generated by fitting the result. The potential program P... t It is obtained by random sampling from a Gaussian distribution; therefore, the potential plan P t The spatiotemporal characteristics of the registration process were explicitly considered;

[0042] Step 5: Develop potential plans P t The input is fed into the actor network, and the feature maps output by each downsampling module in the planner network are concatenated into the actor network by channel number via skip connections. Simultaneously, detailed features and low-dimensional latent plans from the planner network are used to reconstruct a refined high-dimensional deformation field. This high-dimensional deformation field is then treated as an action and applied to the moving image I through a Spatial Transformer Network (STN). m This causes it to undergo flexible deformation to obtain the predicted image, and the predicted image and the fixed image I are then compared. f The mutual information loss and the spatial smoothing term for the deformation field are used as unsupervised registration losses. Image pairs {I} are computed using the unsupervised Modality Independent Neighbourhood Descriptor (MIND) in a flexible registration environment. m ,I f} and the deformed image pair {I m ,I f The similarity metric is used as an unsupervised feedback reward.

[0043] Step 6: Set the current state S t and potential plans P t Input the critic network, and the critic network captures the current state S. t Abstract the characteristics and analyze them, then compare these characteristics with the potential plan P. t The features are concatenated along the channel dimension to form a fusion feature. Finally, the output module of the critic network analyzes the fusion feature again and outputs the maximum expected reward for executing the decision action in the current state as the loss for reinforcement learning. The deformed image is then used to evaluate {I}. m ,I f Update the state in the environment and the state queue Q;

[0044] Step 7: Repeat the above actions until the deformed I mand I f When the modality-independent neighborhood descriptor value is less than the set value of 0.006, the precise registration state is reached, and the current registration round ends;

[0045] Step 8: Finally, a perfectly aligned and registered image is obtained.

[0046] This invention focuses on flexible registration of three-dimensional multimodal images, using Dice score and average surface distance (ASD) as evaluation metrics. The Dice score measures similarity by calculating the degree of overlap between the labels of two segmentation maps; its value is between [0,1], with higher values ​​indicating greater similarity between images. If multiple anatomical structures of interest exist, the Dice score for each structure is calculated separately, and the average is taken. The average surface distance (ASD) is used to compare the average distance between the anatomical markers in the deformed segmentation map and the corresponding segmentation map of the fixed image; the lower the ASD value, the higher the similarity between the two images.

[0047] Table 1 Comparison of objective evaluation metrics for different input frame numbers

[0048]

[0049] The intelligent agent in this invention fully utilizes the redundant state information generated during the registration process. To further evaluate the effectiveness of the multi-frame planning module, it was compared with registration methods that only consider the most recent 1 frame, 2 frames, 3 frames, and 5 frames. The experimental results are shown in Table 1, comparing the objective evaluation metrics of this method under different numbers of input frames on the HCP dataset, which is a public dataset from the Human Connectome Project. This indicates that in reinforcement-based registration tasks, the addition of historical state information significantly improves the performance of the intelligent agent. Moreover, the results show that using the most recent 3 frames already achieves good results; further increasing the number of input state frames does not yield significant performance improvements but instead increases the network burden.

[0050] Table 2 Comparison of Objective Evaluation Criteria for Each Comparison Method

[0051]

[0052] Table 2 summarizes the performance of the proposed method and other advanced methods on the HCP human brain dataset. The proposed method outperforms traditional methods, deep learning-based methods, and other reinforcement learning-based methods on this dataset, achieving state-of-the-art performance. Flexible registration of multimodal medical images is far more challenging than rigid registration. Symmetric regularization registration and elastic registration methods in advanced normalization tools, as two traditional methods, lack understanding of high-level semantics and are therefore ill-suited for this task. Voxel deformation registration is a classic flexible registration method with a significant advantage in testing speed; however, this one-time registration method is ineffective when processing moving images with large deformations. Reinforcement learning-based multimodal image registration methods ignore spatiotemporal information during the registration process, and the one-dimensional vectors used in the intermediate layers further lose detail spatial features, making it difficult to decode accurate high-dimensional deformation fields. Therefore, reinforcement learning-based multimodal image registration methods fail in more complex multimodal registration environments, resulting in the loss of most image details in the final generated images.

[0053] Experiments on the HCP dataset demonstrate that the method of this invention has good robustness when dealing with complex multimodal data. Figure 5 This is a comparison chart of the experimental results of various methods on the HCP dataset. The light gray image represents the difference map between the registration result and the fixed image; the lighter the color, the better the registration effect. Figure 5 (a) is a fixed image. Figure 5 (b) is a moving image. Figure 5 (c) is the registration result obtained by the conversion normalization registration method in the advanced normalization tool. Figure 5 (d) shows the registration result of the flexible registration method. Figure 5 (e) shows the registration result of the voxel degeneration registration method. Figure 5 (f) shows the registration results of a multimodal image registration method based on reinforcement learning. Figure 5 (g) is the registration result of the method of the present invention. From Figure 5 It can be seen that the difference map produced by the method of the present invention has the lightest color and the best registration effect.

[0054] To further discuss the effectiveness of the framework of this invention from the perspective of reinforcement learning, this section will compare it with two other methods. The first method removes the reinforcement learning control part from the framework of this invention, retaining only the part used for image generation. Specifically, it eliminates the use of critics and abandons the optimization path of reinforcement learning. Furthermore, all training methods remain consistent with the framework of this invention, still employing a stepwise registration strategy. The second method rewrites the reinforcement learning control part of this invention using the Proximal Policy Optimization (PPO) method. The PPO method is similar to the Advantage Actor-Critic (A2C) method, but it introduces an importance sampling mechanism, making it a more stable algorithm. Other settings in the experiment remain consistent with the original method. The objective evaluation metrics of the experiments are shown in Table 3.

[0055] Table 3 Comparison of objective evaluation metrics for different reinforcement learning algorithms

[0056]

[0057] As shown in Table 3, the model without reinforcement learning only achieved slightly better performance than the classic deep learning registration framework, voxel deformation registration. This improvement is due to the use of residual modules to deepen the network structure, but compared to the complete framework that incorporates reinforcement learning, the performance loss is significant. Proximal policy optimization methods can also play a positive role in the registration process.

[0058] Figure 6 These are step-by-step visualizations comparing different reinforcement learning frameworks, illustrating the registration process in detail. Figure 6 The first and second lines represent the method of this invention, wherein the light gray image represents the difference map between the registration result and the fixed image; the lighter the color, the better the registration effect. Figure 6 The third and fourth rows show the registration results and corresponding difference plots for the proximal policy optimization method. Figure 6 The fifth and sixth rows show the registration results and corresponding difference maps without using reinforcement learning. The method proposed in this invention can align a moving image to a fixed image more quickly and stably.

[0059] Table 4 Comparison of registration results with and without spectral normalization layer.

[0060]

[0061] Training reinforcement learning frameworks is often difficult, and the performance of large networks is evident as the exploration of network architectures deepens. However, the gradients of networks oscillate wildly with increasing depth, leading to increased difficulty in model convergence. Applying spectral normalization to the network allows its parameters to satisfy Lipshitz constraints, thereby facilitating training. To evaluate the effectiveness of the spectral normalization operation in this invention's framework, it was compared with an intelligent agent without a spectral normalization layer. As shown in Table 4, spectral normalization has a positive effect on the registration framework of this invention. Note that in this invention's framework, the reviewer only used a shallow network, while the introduction of spectral normalization makes it possible to use deeper, larger neural networks in future work.

[0062] The above process was used to perform flexible registration of multimodal medical images using a multi-frame intelligent agent. The experimental results show that the multi-frame intelligent agent framework for 3D multimodal flexible image registration driven by the soft actor-critic algorithm proposed in this invention achieves state-of-the-art performance. The experimental results demonstrate that the algorithm proposed in this invention is highly effective and can accurately register multimodal medical images.

[0063] This invention introduces the concept of a planner on top of the soft actor-critic framework. The planner maps the high-dimensional input state to a low-dimensional plan. The planner network guides the actor to generate high-dimensional continuous actions; in other words, the planner network is essentially a template for generating deformation fields, a coarse high-level action. Therefore, this invention differs from the classic actor-critic framework in that the actor's input is not a high-dimensional state, but a low-dimensional plan. The critic can promote model convergence by evaluating the low-dimensional plan. To further handle complex multimodal registration, this invention proposes explicitly utilizing redundant states in the registration process to learn more accurate registration actions from the generated state frames, thereby enabling precise pixel-level control in multimodal environments.

[0064] It should be noted that the specific embodiments described above are exemplary, and those skilled in the art can devise various solutions inspired by the disclosure of this invention. These solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents.

Claims

1. A flexible registration method for multimodal medical images based on multi-frame intelligent agents, characterized in that, The proposed registration method presents a multi-frame intelligent agent framework for 3D multimodal flexible image registration based on a soft actor-critic algorithm. This framework introduces the concept of a planner into the traditional actor-critic framework. The planner observes multiple consecutive states and generates a low-dimensional plan, which serves as a template for generating high-dimensional actor actions and participates in the critic's evaluation. It predicts high-dimensional actions from multi-frame state fusion in a completely unsupervised manner, guiding the model to complete image generation and policy control, specifically including: Step 1: Prepare the image dataset to be registered, including structural images T1w and T2w, where T1w is a fixed image and T2w is a moving image; Step 2: Resample all images to be registered to a size of 128*128*128 using max-min scaling normalization, and set T1w to a fixed image I. f T2w is the moving image I m Initial state S t=0 For image pair {I m ,I f }; Step 3: Create a state queue Q of size 3 to store the state of the most recent 3 frames [S] t-2 ,S t-1 ,S t The queue consists of 3 S t=0 Initially, three networks are constructed: a planner network, an actor network, and a critic network, specifically: The planner network consists of 5 downsampling modules, each containing two convolutional layers and one residual module. The residual module also consists of two convolutional layers, and the LeakyReLU activation function is used on the output of each convolutional layer. The actor network contains five upsampling modules, and its structure is consistent with that of the planner network. The actor network also has an additional output module, which takes the output of the fifth upsampling module as its input and generates the deformation field. The input of each upsampling module is formed by concatenating the output of the previous module and the feature map provided by the skip connection in a channel manner. The critic network consists of 5 downsampling modules, in which spectral normalization is used to smooth the network gradient. After the 5th downsampling module, an additional output module is introduced, which consists of convolutional layers, LeakyReLU activation layers and linear layers. The tensors output by the convolutional layers are flattened and fed into the linear layers. Step 4: Input the state queue Q created in Step 3 into the planner network. The planner simultaneously observes the three most recent states [S]. t-2 ,S t-1 ,S t ], and downsample it to the corresponding low-dimensional representation [Z t-2 Z t-1 Z t Z is represented in low dimension. t The offset approximates the spatiotemporal information, i.e., [Z t -Z t-1 Z t-1 -Z t-2 The mean and variance are calculated from the low-dimensional representation obtained by downsampling and the offset, and then fitted to generate a Gaussian distribution. The potential plan P... t It is obtained by random sampling from a Gaussian distribution; therefore, the potential plan P t The spatiotemporal characteristics of the registration process were explicitly considered; Step 5: Develop potential plans P t The input is processed into the actor network, and the feature maps output by each downsampling module in the planner network are concatenated into the actor network according to the number of channels via skip connections. Simultaneously, a detailed high-dimensional deformation field is reconstructed using the detailed features and low-dimensional latent plans from the planner network. This high-dimensional deformation field is then treated as an action and applied to the moving image I through a spatial transformation network. m This causes it to undergo flexible deformation to obtain the predicted image, and the predicted image and the fixed image I are then compared. f The mutual information loss and the spatial smoothing term of the deformation field are used as unsupervised registration losses. Image pairs {I} are computed using an unsupervised modality-independent neighborhood description operator in a flexible registration environment. m ,I f } and the deformed image pair {I m ,I f The similarity metric is used as an unsupervised feedback reward. Step 6: Set the current state S t and potential plans P t Input the critic network, and the critic network captures the current state S. t Abstract the characteristics and analyze them, then compare these characteristics with the potential plan P. t The features are concatenated along the channel dimension to form a fusion feature. Finally, the output module of the critic network analyzes the fusion feature again and outputs the maximum expected reward for executing the decision action in the current state as the loss for reinforcement learning. The deformed image is then used to evaluate {I}. m ,I f Update the state and state queue Q in the registration environment; Step 7: Repeat the above steps until the deformed moving image I is obtained. m and fixed image I f When the modality-independent neighborhood descriptor value is less than the set value of 0.006, the precise registration state is reached, and the current registration round ends; Step 8: Finally, a perfectly aligned and registered image is obtained.

Images