Lung 4D CT medical image registration method and system based on deep reinforcement learning

Abstract

The application belongs to the technical field of medical image registration, and provides a lung 4DCT medical image registration method and system based on deep reinforcement learning, acquires a source image and a target image, performs multi-scale feature coding on the source image and the target image, obtains features of different scales, calculates residuals of the features of different scales to obtain a residual tensor; based on the deformed source image and the target image, uncertainty estimation is performed to obtain an uncertainty map; according to a current deformation, a Jacobian risk is calculated, the residual tensor, the uncertainty map and the Jacobian risk are spliced to obtain a current state; based on the current state, deep reinforcement learning registration under a constrained continuous control Markov decision process is performed, and in the process of the deep reinforcement learning registration, alignment similarity improvement, deformation regularization and topological constraint, and cross-phase time sequence consistency are taken as rewards to obtain a medical registration result, and the registration effect is improved.

Description

Technical Field

[0001] This invention belongs to the field of medical image registration technology, specifically relating to a method and system for registering 4DCT medical images of the lungs based on deep reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Four-dimensional computed tomography (4DCT) of the lungs is of significant value in radiotherapy, respiratory motion analysis, functional imaging assessment, and interventional navigation. Compared to static 3DCT, 4DCT explicitly records morphological changes during the respiratory cycle, enabling clinicians to more accurately characterize the temporal trajectory of tumors and normal organs, thereby reducing radiation margin, minimizing dose to normal tissues, and improving the reliability of target coverage. However, 4DCT acquisition methods typically rely on respiratory phase separation and reconstruction, making them susceptible to respiratory irregularities, phase separation errors, motion artifacts, and reconstruction noise. Simultaneously, lung tissue exhibits low contrast, fine texture, and significant non-rigid deformation characteristics (e.g., overall displacement driven by the diaphragm, local lobar expansion and contraction, and complex deformation of blood vessels / bronchioles). These factors collectively contribute to the significant difficulty and challenge of registration. For subsequent tasks, the registration quality directly determines the reliability of motion field estimation and further affects the stability of dose accumulation, anatomical structure propagation, lesion tracking and quantitative analysis. Therefore, it is of continuous and urgent research significance to construct a robust, high-precision and physically reasonable lung 4DCT registration method.

[0004] Traditional lung registration methods are mostly based on explicit optimization frameworks, solving for the optimal deformation field to align two or more images by defining image similarity terms and deformation regularization terms. Methods represented by free deformation (FFD) and B-spline parameterization can express smooth and continuous non-rigid deformations in the form of control points and gradually approximate them through multi-resolution strategies; Demons-like methods and their variants start from optical flow and mechanical analogies, iteratively advancing the deformation field through local update rules; variational frameworks express registration as an energy functional minimization problem, which is theoretically easy to introduce different regularization terms, boundary constraints, and prior knowledge. The above methods are widely used in different scenarios, but in the typical large deformation, strong temporal sequence and high noise task of lung 4DCT, there are still several common bottlenecks: On the one hand, the energy function is often non-convex, the optimization process is sensitive to the initial value, step size and regularization weight, and unstable local minima are easy to appear in low contrast or artifact areas; on the other hand, the real lung motion has the characteristics of local compressibility, boundary slip and tissue heterogeneity, and simple smoothing regularization is difficult to simultaneously take into account fine structure alignment and deformation physical rationality; in addition, 4D data inherently carries the requirements of cross-phase trajectory continuity and period consistency, but traditional two-frame registration usually only solves independently between phase pairs, and cross-phase consistency relies more on additional post-processing or weak constraints, making it difficult to explicitly integrate "full-cycle motion rationality" as the core optimization objective into the solution process.

[0005] In recent years, deep learning has driven a paradigm shift in registration research. End-to-end learning-based registration typically utilizes convolutional neural networks to extract features and directly regress dense deformation fields, combined with differentiable space transformations to achieve image resampling, thus gaining a significant speed advantage in the inference stage. Unsupervised or weakly supervised training can achieve learning through similarity loss and deformation regularization terms, avoiding dependence on dense ground truth deformation. Simultaneously, researchers have proposed strategies such as multi-scale pyramids, inverse consistency constraints, cyclic consistency, and auxiliary supervision based on anatomical structures or segmentation priors to improve stability and generalization ability under complex deformations. Nevertheless, these methods still face a key contradiction: lung 4DCT deformations often span multiple scales, and the deformation amplitude and credible cues differ significantly across regions. A single forward regression needs to simultaneously handle both "large-scale alignment for coarse registration" and "local refinement for fine registration" within the same network, easily leading to underfitting or oversmoothing. Even with iterative refinement, the update rule is often fixed as gradient descent or a pre-defined iterator, lacking a mechanism to adaptively adjust the strategy based on the current alignment state. In other words, the registration process naturally has the decision-making attribute of "gradual approximation": in some stages, large-scale displacement should be corrected first; in some areas, fine-tuning should be done carefully to avoid folding or discontinuity; and in areas with severe artifacts or poor texture, stronger regularization and uncertainty suppression should be relied upon. These decision-making logics of "when to update, where to update, and how much to update" are not easily expressed completely by a fixed optimizer or a single loss weight. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a lung 4DCT medical image registration method and system based on deep reinforcement learning. This invention iteratively predicts deformation increments through a multi-scale, staged strategy, introduces velocity field integration to obtain approximately differential homeomorphic topology-preserving deformation, and simultaneously constructs cross-phase... Figure 1 Consistent rewards, by explicitly constraining the rationality of whole-cycle motion, and combined with uncertainty-driven state representation and constrained reinforcement learning, achieve a stable and controllable learning process, which can improve the effectiveness of lung respiratory motion registration tasks.

[0007] According to some embodiments, the present invention adopts the following technical solution: A lung 4DCT medical image registration method based on deep reinforcement learning includes the following steps: Acquire source and target images, resample the source image under the current deformation to obtain deformed source image, perform multi-scale feature encoding on deformed source and target images to obtain features at different scales, calculate the residuals of features at different scales, use the feature residuals to characterize the alignment error, and obtain the residual tensor; Based on the deformation source image and target image under the current deformation, uncertainty estimation is performed to obtain the uncertainty map; The Jacobian risk is calculated based on the current deformation. The residual tensor, uncertainty diagram and Jacobian risk are then concatenated to obtain the current state. Based on the current state, deep reinforcement learning registration is performed under a constrained continuous control Markov decision process. During the deep reinforcement learning registration process, alignment similarity enhancement, deformation regularization and topological constraints, as well as cross-phase temporal consistency are used as rewards to obtain medical registration results.

[0008] As an alternative implementation, multi-scale feature encoding is performed on the source image and the target image to obtain features at different scales. The process of calculating the residuals of the features at different scales and obtaining the residual tensor includes: in the current deformation... The source image will be displayed below. Resampling and with the target image Extracting pyramid features from a multi-scale encoding network At each scale l, the feature space residual is used as the core expression of the alignment error.

[0009] As an alternative implementation, the process of performing uncertainty estimation based on the deformation source image and the target image to obtain an uncertainty map includes: uncertainty... Where norm(·) represents linear normalization to [0,1] after 1%–99% quantile truncation in the voxel dimension, and the uncertainty U(x) is downsampled to 1 / 8 resolution using average pooling as the uncertainty space channel to obtain the uncertainty map, where, For the current deformation, For the source image, The target image.

[0010] As an alternative implementation, the process of calculating the Jacobian risk based on the current deformation includes: calculating the Jacobian determinant of the current overall deformation. Perform truncation and scaling, then downsample to 1 / 8 resolution as a single spatial risk channel, while simultaneously calculating the global scalar minimum. negative proportion With 5% quantile The Jacobian risk is obtained, which consists of one Jacobian space channel and three broadcast global scalar channels.

[0011] As an alternative implementation, the process of deep reinforcement learning registration under a constrained continuous control Markov decision process includes: formulating lung 4DCT registration as a constrained continuous control problem, and characterizing the iterative solution process of deformation using a Markov decision process. Specifically, in the k-th iteration, the system is in the current alignment state. The intelligent agent outputs continuous actions As a deformation update control variable, the environment updates the deformation and resamples to obtain a new alignment state. At the same time, return instant rewards Through sequential decision modeling, it learns to adopt different update modes at different stages: in the coarse alignment stage, it prioritizes correcting large-scale displacements, in the refinement stage, it performs smaller-amplitude local corrections, and in areas with poor texture or severe artifacts, it adopts more conservative updates to avoid unstable deformation.

[0012] As an alternative implementation, during the deep reinforcement learning registration process under constrained continuous control Markov decision process, the optimal update strategy is learned by maximizing the cumulative reward, and is subject to topological and geometric constraints such as avoiding folding, limiting excessive compression / expansion, and maintaining smooth deformation.

[0013] As an alternative implementation, in the deep reinforcement learning registration process under constrained continuous control Markov decision process, a hierarchical continuous action is adopted, combined with velocity field integration to construct a more topologically stable deformation update mechanism. The high-level policy is responsible for determining the overall strength and spatial allocation of this update, the low-level policy outputs the specific deformation increment, and the high-level output includes the gating graph. With step size Low-level output velocity field increment This is used to express the local update direction and detailed structure, and to obtain the modulated velocity field; The modulated velocity field is parameterized using SVF, and the velocity field is integrated into deformation increments through exponential mapping, achieving a result greater than a set value during the coarse alignment stage. and Achieve rapid global alignment by reducing [the size of the target area] during the refinement phase. And By focusing on areas with significant residuals to achieve local refinement, a natural iterative behavior is formed.

[0014] As an alternative implementation, the process of rewarding alignment similarity enhancement, deformation regularization, and topological constraints includes using local normalized cross-correlation as the primary similarity measure and using similarity enhancement as an immediate reward. A velocity field smoothing regularization is added as a reward and penalty term to constrain the updated spatial roughness, and a soft constraint penalty is introduced on the Jacobian determinant to encourage it to remain positive and stay away from excessively small regions.

[0015] As an alternative implementation, the process of rewarding cross-phase temporal consistency includes: explicitly introducing cross-phase consistency into the reward, characterizing the forward and reverse combinations as close to an identity mapping based on the deformation between adjacent phases using a cyclic consistency term, and introducing a time smoothing term for adjacent velocity fields.

[0016] A lung 4DCT medical image registration system based on deep reinforcement learning, comprising: The residual tensor calculation module is configured to acquire source image and target image, resample source image under current deformation to obtain deformed source image, perform multi-scale feature encoding on deformed source image and target image to obtain features at different scales, calculate residuals of features at different scales, use the feature residuals to characterize alignment error, and obtain residual tensor. The uncertainty map calculation module is configured to perform uncertainty estimation based on the deformation source image and target image under the current deformation, and obtain the uncertainty map. The Jacobian risk calculation module is configured to calculate Jacobian risk based on the current deformation. The splicing module is configured to splice the residual tensor, uncertainty graph, and Jacobian risk to obtain the current state; The deep reinforcement learning registration module is configured to perform deep reinforcement learning registration under a constrained continuous control Markov decision process based on the current state. During the deep reinforcement learning registration process, alignment similarity enhancement, deformation regularization and topological constraints, as well as cross-phase temporal consistency are used as rewards to obtain medical registration results.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention iteratively predicts deformation increments through a multi-scale, staged strategy, introduces velocity field integration to obtain topologically preserving deformation with approximate differential homeomorphism, and simultaneously constructs cross-phase... Figure 1 Consistent rewards explicitly constrain the rationality of the entire cycle of motion, and combined with uncertainty-driven state representation and constraint reinforcement learning, achieve a stable and controllable learning process.

[0018] This invention introduces a hierarchical strategy and ROI gating to achieve adaptive updates from coarse to fine and from global to local; based on SVF exponential mapping integrals and safe action projection, it enhances the reversibility and topological rationality of deformation; and cross-phase... Figure 1 Consistent rewards enable registration to not only optimize the alignment of two phases but also ensure consistency of respiratory time trajectories; uncertainty-driven state encoding and Lagrange-constrained reinforcement learning enable interpretable decision-making with multi-objective trade-offs.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a deep reinforcement learning registration process under a constrained continuous control MDP, as described in one embodiment. Figure 2 An example of an uncertainty-driven state tensor Construction: splicing together multi-scale residuals, uncertainty plots, and Jacobian risk channels; Figure 3 This is one embodiment of hierarchical actions and SVF-based deformation update process: gating-step modulation, exponential mapping integration and secure projection; Figure 4 A multi-objective reward in one embodiment The components are: LNCC similarity gain, velocity field smoothing regularization, Jacobian topological constraints, and cross-phase consistency. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in these embodiments have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0026] Example 1 A lung 4DCT medical image registration method based on deep reinforcement learning includes the following steps: Acquire source and target images, resample the source image under the current deformation to obtain deformed source image, perform multi-scale feature encoding on deformed source and target images to obtain features at different scales, calculate the residuals of features at different scales, use the feature residuals to characterize the alignment error, and obtain the residual tensor; Based on the deformation source image and target image under the current deformation, uncertainty estimation is performed to obtain the uncertainty map; The Jacobian risk is calculated based on the current deformation. The residual tensor, uncertainty diagram and Jacobian risk are then concatenated to obtain the current state. Based on the current state, deep reinforcement learning registration is performed under a constrained continuous control Markov decision process. During the deep reinforcement learning registration process, alignment similarity enhancement, deformation regularization and topological constraints, as well as cross-phase temporal consistency are used as rewards to obtain medical registration results.

[0027] The following is a detailed introduction.

[0028] First, the goal of 4DCT registration of the lungs is to establish a dense spatial correspondence between different respiratory phases, thereby obtaining a three-dimensional deformation field that can describe respiratory deformation. Let the three-dimensional volumetric image be defined in the spatial domain. Above, the source phase image is denoted as The target phase image is denoted as This embodiment aims to estimate a deformation mapping from the target space to the source space. This ensures that the deformed source image is anatomically aligned with the target image. Using backward warping, the deformed source image can be represented as: ; In the specific implementation, Usually determined by displacement field Parameterization, that is: ; In a 4DCT scenario, given a phase sequence within a respiratory cycle This embodiment focuses on both registration between adjacent phases and motion consistency modeling across phases. For any phase pair (p, q), the deformation mapping is denoted as... It maps position x in the target phase q to the corresponding position in the source phase p, thus defining the deformation relationship: ; Furthermore, lung deformation should meet certain geometric rationality requirements, especially avoiding local folding. Therefore, this embodiment focuses on the Jacobian determinant of the deformation: ; The method employs a deformation generation mechanism in conjunction with reward / constraint design to suppress [the spread of the virus]. The value should be too small or negative to ensure the availability and stability of the deformation field in the topology.

[0029] This invention formulates lung 4DCT registration as a constrained continuous control problem and uses a Markov decision process (MDP) to characterize the iterative solution process of deformation. Traditional registration typically minimizes the energy function directly in the parameter space, while this embodiment emphasizes the decision property of "gradually updating deformation": at the k-th iteration, the system is in the current alignment state. The intelligent agent outputs continuous actions As a deformation update control variable, the environment updates the deformation and resamples to obtain a new alignment state. At the same time, return instant rewards Through this sequential decision modeling, the policy can learn to adopt different update patterns at different stages: prioritizing the correction of large-scale displacements in the coarse alignment stage, performing smaller-amplitude local corrections in the refinement stage, and adopting more conservative updates in areas with poor texture or severe artifacts to avoid unstable deformation. The overall state-action-environment update-reward / cost feedback loop is as follows: Figure 1 As shown.

[0030] Regarding the reinforcement learning objective, this embodiment learns the optimal update strategy by maximizing cumulative reward: ; in Let θ be the policy network with parameter θ, and γ∈(0,1] be the discount factor. Unlike two-frame registration, 4DCT is essentially temporal data, and the benefit of single-step alignment cannot fully guarantee the rationality of the entire cycle motion. Therefore, this embodiment incorporates cross-phase consistency into the reward structure, so that the policy can pursue the improvement of local similarity while taking into account the overall coordination on the phase chain. On the other hand, lung registration also needs to meet topological and geometric constraints, such as avoiding folding, limiting excessive compression / expansion, and maintaining smooth deformation. To this end, this embodiment adopts the idea of ​​"constrained reinforcement learning", which achieves these requirements through rewards and penalties and constraints, so that the goal of policy learning is not only to align better, but also to align better in a stable, usable and physiologically intuitive way.

[0031] State Representation: Uncertainty-Driven Registration Situation Encoding To enable the agent to perceive "where there is misalignment, where clues are unreliable, and where deformation poses a risk," this embodiment constructs a state representation for the registration task and designs it as a multi-scale, structured tensor input containing uncertainty and geometric risk information. First, in the current deformation... The source image will be resampled to obtain and with the target image Extracting pyramid features from a multi-scale encoding network At each scale l, this embodiment uses the feature space residual as the core expression for the alignment error: ; Compared to pure intensity differences, these feature residuals are more resistant to noise, artifacts, and insufficient local contrast common in 4DCT, and can provide more stable error signals for structures such as blood vessels, bronchi, and lung textures. To further enhance correspondence perception, the state can also implicitly include local correlation responses (response maps formed by the attention / correlation modules within the encoding network), enabling the policy to distinguish between "structural misalignment" and "spurious differences caused by noise," thereby outputting more reliable updates.

[0032] Introducing uncertainty into the state is a key design feature of this embodiment. The encoding network adds an uncertainty prediction branch to output a voxel-level uncertainty graph. This is used to estimate the reliability of the current error signal. Regions with high uncertainty often correspond to breathing phase errors, motion artifacts, or texture-poor areas. Blindly following error updates in these regions may lead to deformation oscillations, local folding, or "misalignment" of the structure. By explicitly inputting uncertainty into the policy network, the policy can learn an adaptive mechanism: more aggressive updates for low-uncertainty regions to improve alignment efficiency, and a stronger smoothing tendency and smaller update amplitude for high-uncertainty regions to improve stability.

[0033] Furthermore, lung registration not only concerns image alignment but must also ensure that deformations are available. To this end, the status also includes deformation geometry risk indicators, such as Jacobian determinants. Statistical information or its low-resolution mapping can be used to help policies perceive topology security margins.

[0034] when When the value is too small or even negative, it indicates a risk of local folding or topological disruption. Therefore, using it as a risk channel input can prompt the strategy to automatically reduce updates and change local update allocation when approaching fold boundaries, thereby forming an endogenous adjustment capability for deformation safety. In summary, the state representation in this embodiment achieves a unified encoding of "alignment error - reliability - geometric safety margin," providing a sufficient, interpretable, and highly task-consistent information foundation for continuous control strategies.

[0035] like Figure 2 As shown, the state input consists of multi-scale feature residuals. Uncertainty diagram and Jacobian risk channel Composed of components, and obtained by channel-dimensional stitching at a uniform resolution. .

[0036] To ensure the state input is realizable and reproducible, we implement the state representation as a combination of "a multi-channel 3D tensor with uniform resolution + a small number of global scalars". First, we extract 3D feature maps at three scales (resolutions of 1 / 4, 1 / 8, and 1 / 16 of the original volume data, respectively) from the encoder of the registration network / feature extractor, denoted as F(1 / 4), F(1 / 8), and F(1 / 16). We then calculate the source image after the current deformation... The feature residuals of the deformed image and the target image are obtained, and the residuals at each scale are compressed to 8 channels through 1×1×1 convolution, then upsampled to 1 / 8 resolution, and then concatenated in the channel dimension to obtain the multi-scale residual tensor: ; Secondly, the uncertainty diagram is constructed using the "amplitude of the strength residual after current deformation": Let Where norm(·) represents linear normalization to [0,1] after 1%–99% quantile truncation in the voxel dimension. U(x) is downsampled to 1 / 8 resolution using average pooling and used as a single-channel input, resulting in: ; Wherein, D, H, and W represent the number of voxels in the depth (z), height (y), and width (x) directions of the input three-dimensional medical image (volume data), respectively; and D / 8, H / 8, and W / 8 represent the dimensions of the volume data after downsampling to 1 / 8 resolution in the three directions.

[0037] Finally, the Jacobian risk channel uses the current overall deformed Jacobian determinant. This embodiment is for Perform truncation and scaling (cropping to [0,2] and linearly scaling to [0,1]), then downsample to 1 / 8 resolution as a single spatial risk channel; simultaneously compute three global scalars: minimum value negative proportion With 5% quantile The global scalar is expanded into a constant 3D feature map via broadcasting and concatenated with the spatial risk channel, enabling the policy to simultaneously obtain both "local risk distribution" and "global safety margin." In summary, the final state tensor is: ; in It includes one Jacobian space channel and three broadcast global scalar channels.

[0038] Action Space and Deformation Update: Hierarchical Strategy and Near-Homeomorphic Velocity Field Integral Lung 4DCT registration involves both large-scale respiratory motion and local fine structural deformation. Directly performing continuous control on a high-dimensional displacement field significantly increases the learning difficulty and instability. Therefore, this embodiment employs hierarchical continuous motion combined with velocity field integration to construct a more topologically stable deformation update mechanism. Overall, the higher-level strategy determines the overall intensity and spatial allocation of the update, while the lower-level strategy outputs the specific deformation increments. The higher-level output includes a gating graph. With step size This is used to express "where updates should be more proactive and how large the overall update should be"; low-level output velocity field increment. This is used to express the local update direction and detailed structure. This results in the modulated velocity field: ; To structurally enhance the reversibility and topology preservation of the deformation, this embodiment employs stationary velocity field (SVF) parameterization and integrates the velocity field into deformation increments via an exponential mapping. The deformation update is written as: ; The exp(·) function achieves differentiable numerical integration through scaling and squaring. The advantage of SVF lies in its construction of deformation as an exponential mapping of the velocity field, which is closer to the differential homeomorphic deformation generation mechanism, thus maintaining good topological stability even under large deformation scenarios. The combination of hierarchical actions and SVF integration allows the strategy to achieve significant improvements in the coarse alignment stage. With a wider range Achieve rapid global alignment by reducing [the size of the target area] during the refinement phase. And By focusing on regions of significant residuals to achieve local refinement, a natural "coarse-to-fine" iterative behavior is formed. The above describes the action... to deformation increment Then to combination update The generation link is as follows Figure 3 As shown.

[0039] In actual training and inference, numerical errors and extreme local differences may still induce folding trends. Therefore, this embodiment adds a safety constraint projection to the action-to-deformation mapping link to form a differentiable safety barrier. Rapid estimation is performed after each update. The lower bound of the threshold, when it is locally below the threshold, for The update is projected back into the safe region by scaling the magnitude of the changes or locally suppressing the gating graph. This mechanism can significantly reduce unusable deformations caused by policy exploration in the early stages of training, making the learning process more stable and enabling the final policy to maintain good deformation availability while pursuing high alignment gains.

[0040] The reward function determines the direction and convergence behavior of policy learning. In this embodiment, the reward consists of three parts: alignment similarity enhancement, deformation regularization and topological constraints, and cross-phase temporal consistency. It is organized around the core idea of ​​"the gain from each update step," thus better aligning with the process of iterative registration. The meaning of each reward item and its summarization method are as follows: Figure 4 As shown.

[0041] In the similarity section, this embodiment uses Local Normalized Cross-Correlation (LNCC) as the primary similarity measure and uses similarity enhancement as an immediate reward: ; This approach directly measures the marginal contribution of actions to registration quality, shifting the policy learning objective from "predicting results" to "learning to effectively update action sequences" and gradually improving alignment through multiple iterations.

[0042] Secondly, to avoid instability caused by excessively drastic local deformation, this embodiment incorporates a velocity field smoothing regularization as a reward / penalty term to constrain the updated spatial roughness: ; Meanwhile, to explicitly suppress the folding tendency and ensure topological rationality, this embodiment introduces a soft constraint penalty on the Jacobian determinant, encouraging it to remain positive and stay away from excessively small regions: ; This ensures that the strategy is always constrained by deformability availability when pursuing higher similarity, thereby reducing the situation where "alignment improves but deformability becomes unusable".

[0043] Finally, regarding the temporal properties of 4DCT, this embodiment explicitly introduces cross-phase consistency in the reward, ensuring that the deformations between different phases are not only well aligned individually but also maintain overall coordination along the phase chain. Let the deformation between adjacent phases be... Then, the cycle-consistent term characterizing forward and reverse combinations should approximate the identity mapping: ; Meanwhile, to ensure the continuity of motion over time, a time smoothing term for adjacent velocity fields is introduced: ; Where Ω represents the image domain (voxel space). This represents the summation over the voxel space; For the target image, Source image; For the current deformation in the k-th iteration, For the updated deformation, Indicates deformation composite; LNCC ( , This is a locally normalized cross-correlation similarity measure; Let the velocity field be the velocity field in the k-th iteration. Indicates its spatial gradient; Represents the L2 norm, Denotes the norm 1; τ is the Jacobian determinant of the deformation at position x; τ>0 is the Jacobian security threshold; p is the 4DCT phase index. This represents the deformation mapping from phase p to phase p+1. Id represents its reverse mapping, and Id represents the identity mapping; >0 represents the weighting coefficient for each reward item.

[0044] The comprehensive reward is written as: ; Through the aforementioned multi-objective rewards, the strategy is simultaneously driven by "alignment improvement", "deformation smoothing and topology safety" and "cross-phase consistency" in each update step, making the final learned update strategy more in line with the physiological continuity and clinical usability requirements of 4D lung breathing motion.

[0045] Training Algorithm and Implementation Details This embodiment employs Actor-Critic-based Proximal Policy Optimization (PPO) to train the policy network, in order to handle stable updates in a continuous action space. Output the Gaussian distribution parameters (mean and log-variance) for action a, value network The predicted state value is determined by the shared encoder backbone, with a separate policy head and value head at a higher level. The advantage function employs Generalized Advantage Estimation (GAE), and the policy update magnitude is constrained by the PPO pruning objective to improve training stability. Specifically, the hyperparameters are fixed as follows: discount factor γ = 0.99, GAE parameter λ = 0.95, and PPO pruning coefficient. =0.2, learning rate 3×10 4 Each round samples 2048 environment steps, the mini-batch size is 64, each round optimizes for 10 epochs, the entropy regularization coefficient is 0.01, the value loss coefficient is 0.5, and the gradient clipping threshold is 0.5.

[0046] To achieve deformation safety control in "constrained reinforcement learning", we employ the Lagrange multiplier method to integrate topological safety constraints into the optimization process: Let the constraint cost be c(s,a): c(s,a) characterizes the folding risk (e.g., voxel ratio or The spatial mean is used, and the expected cost is specified to not exceed a threshold d. During training, the augmented target is maximized. ; Here, β≥0 represents a learnable Lagrange multiplier used for adaptive trade-offs between registration gains and deformation safety. The multiplier update employs a simple dual gradient ascent: ,in The average cost of the current batch, ηβ=10 3. Thus, when the risk of folding exceeds the threshold, β increases and strengthens the penalty, thereby prompting the policy to actively reduce high-risk updates; when the risk is under control, β tends to stabilize, allowing the policy to focus more on improving registration accuracy. To further avoid numerical instability in the early stages of training, all rewards and costs are standardized within each batch (subtract the mean, divide by the standard deviation), and extreme values ​​are truncated.

[0047] experiment This embodiment was experimentally validated on two publicly available datasets, DIR-Lab and POPI. The DIR-Lab4DCT dataset contains 4DCT volumetric data of the lungs of multiple subjects and provides sparse corresponding anatomical markers for evaluation, enabling objective calculation of target registration error in the absence of a dense ground truth deformation field. The commonly used subset of DIR-Lab (Lung 4DCT) contains 10 4DCT scans, each containing multi-phase volumetric data within one respiratory cycle; it provides 300 pairs of manually labeled corresponding anatomical points between the maximal inspiration and maximal expiration phases, and additionally provides a set of markers that can be used for intermediate phase evaluation.

[0048] The POPI dataset provides images and point annotations related to chest respiratory motion, which can be used to supplement and validate the applicability of the method under different acquisition conditions and motion modes. POPI (POPI-model / POPI lung 4DCT) provides multiple chest 4DCT sequences (one respiratory cycle typically consists of 10 three-dimensional phase volumes), and provides manually annotated corresponding anatomical points in key inspiratory / expiratory phases (commonly set to 100 markers per case); some cases also provide markers for intermediate phases to facilitate cross-phase registration consistency assessment.

[0049] The combination of the two can evaluate the robustness and generalization ability of the model in the lung respiratory movement registration task from different perspectives.

[0050] To ensure experimental comparability, this embodiment performed uniform preprocessing on all volume data: images were resampled to a uniform isotropic voxel spacing, and intensity cropping and normalization were performed within the lung region. To reduce interference from the background and chest wall on registration similarity calculations and enhance the anatomical rationality of deformation, this embodiment performed coarse segmentation of the lung field, which was used for mask constraints in similarity measurement and the definition of deformation constraint regions. Lightweight data augmentation, including random intensity perturbations and small-amplitude deformation perturbations, was introduced during the training phase to improve the model's robustness to noise and intensity fluctuations; a multi-scale iterative update strategy was adopted during the inference phase to gradually converge the registration process from coarse to fine.

[0051] This embodiment uses target registration error (TRE) as the main evaluation index. Assume there are N pairs of corresponding marker points between the source phase and the target phase. ,in For the source phase point, For the target phase corresponding point, the registered TRE is defined as follows: ; The experiment will report a comparison of TRE before and after registration, and further analyze the TRE changes under different phase spacings (e.g., adjacent phases and spanning multiple phases) to demonstrate the overall performance of the method in small and large deformation scenarios.

[0052] To verify the effectiveness of the proposed deep reinforcement learning registration framework, this embodiment sets classical non-rigid registration methods and representative deep learning registration frameworks as comparative baselines to provide a reference at the TRE level. Simultaneously, this embodiment conducts ablation experiments to analyze the contributions of key designs, including removing cross-phase consistency rewards, removing uncertainty-driven state encoding, removing Jacobian constraints, or replacing deformation parameterization methods, and discusses the changes in TRE and deformation field geometry for each variant. Through the combination of comparison and ablation, the experiments systematically illustrate the mechanism by which the proposed method improves alignment accuracy, deformation stability, and cross-phase consistency.

[0053] Example 2 A lung 4DCT medical image registration system based on deep reinforcement learning, comprising: The residual tensor calculation module is configured to acquire source image and target image, resample source image under current deformation to obtain deformed source image, perform multi-scale feature encoding on deformed source image and target image to obtain features at different scales, calculate residuals of features at different scales, use the feature residuals to characterize alignment error, and obtain residual tensor. The uncertainty map calculation module is configured to perform uncertainty estimation based on the deformation source image and target image under the current deformation, and obtain the uncertainty map. The Jacobian risk calculation module is configured to calculate Jacobian risk based on the current deformation. The splicing module is configured to splice the residual tensor, uncertainty graph, and Jacobian risk to obtain the current state; The deep reinforcement learning registration module is configured to perform deep reinforcement learning registration under a constrained continuous control Markov decision process based on the current state. During the deep reinforcement learning registration process, alignment similarity enhancement, deformation regularization and topological constraints, as well as cross-phase temporal consistency are used as rewards to obtain medical registration results.

[0054] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can be implemented in one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0055] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0056] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lung 4DCT medical image registration method based on deep reinforcement learning, characterized in that, Includes the following steps: Acquire source and target images, resample the source image under the current deformation to obtain deformed source image, perform multi-scale feature encoding on deformed source and target images to obtain features at different scales, calculate the residuals of features at different scales, use the feature residuals to characterize the alignment error, and obtain the residual tensor; Based on the deformation source image and target image under the current deformation, uncertainty estimation is performed to obtain the uncertainty map; The Jacobian risk is calculated based on the current deformation. The residual tensor, uncertainty diagram and Jacobian risk are then concatenated to obtain the current state. Based on the current state, deep reinforcement learning registration is performed under a constrained continuous control Markov decision process. During the deep reinforcement learning registration process, alignment similarity enhancement, deformation regularization and topological constraints, as well as cross-phase temporal consistency are used as rewards to obtain medical registration results.

2. The lung 4DCT medical image registration method based on deep reinforcement learning as described in claim 1, characterized in that, The process of performing multi-scale feature encoding on the source and target images to obtain features at different scales, calculating the residuals of features at different scales, and obtaining the residual tensor includes: [The text abruptly ends here, so the translation stops as well.] The source image will be displayed below. Resampling and with the target image Extracting pyramid features from a multi-scale encoding network At each scale l, the feature space residual is used as the core expression of the alignment error.

3. The lung 4DCT medical image registration method based on deep reinforcement learning as described in claim 1, characterized in that, The process of performing uncertainty estimation and obtaining an uncertainty map based on the deformation source image and target image includes: uncertainty estimation. Where norm(·) represents linear normalization to [0,1] after 1%–99% quantile truncation in the voxel dimension. The uncertainty U(x) is downsampled to 1 / 8 resolution using average pooling and used as a single-channel input to obtain the uncertainty map, where, For the current deformation, For the source image, The target image.

4. The lung 4DCT medical image registration method based on deep reinforcement learning as described in claim 1, characterized in that, The process of calculating the Jacobian risk based on the current deformation includes: the Jacobian determinant of the current overall deformation. Perform truncation and scaling, then downsample to 1 / 8 resolution as a single spatial risk channel, while simultaneously calculating the global scalar minimum. negative proportion With 5% quantile The Jacobian risk is obtained, which consists of one Jacobian space channel and three broadcast global scalar channels.

5. The lung 4DCT medical image registration method based on deep reinforcement learning as described in claim 1, characterized in that, The process of deep reinforcement learning registration under constrained continuous control Markov decision process includes: formulating lung 4DCT registration as a constrained continuous control problem, and using Markov decision process to characterize the iterative solution process of deformation. Specifically, in the k-th iteration, the system is in the current alignment state. The intelligent agent outputs continuous actions As a deformation update control variable, the environment updates the deformation and resamples to obtain a new alignment state. At the same time, return instant rewards Through sequential decision modeling, it learns to adopt different update modes at different stages: in the coarse alignment stage, it prioritizes correcting large-scale displacements, in the refinement stage, it performs smaller-amplitude local corrections, and in areas with poor texture or severe artifacts, it adopts more conservative updates to avoid unstable deformation.

6. The lung 4DCT medical image registration method based on deep reinforcement learning as described in claim 1, characterized in that, In the process of deep reinforcement learning registration under constrained continuous control Markov decision process, the optimal update strategy is learned by maximizing the cumulative reward, and is subject to topological and geometric constraints such as avoiding folding, limiting excessive compression / expansion, and maintaining smooth deformation.

7. The lung 4DCT medical image registration method based on deep reinforcement learning as described in claim 1, characterized in that, In the deep reinforcement learning registration process under constrained continuous control Markov decision process, a hierarchical continuous action is adopted, combined with velocity field integral to construct a more topologically stable deformation update mechanism. The high-level policy is responsible for determining the overall strength and spatial allocation of the update, the low-level policy outputs the specific deformation increment, and the high-level output includes the gating graph. With step size Low-level output velocity field increment This is used to express the local update direction and detailed structure, and to obtain the modulated velocity field; The modulated velocity field is parameterized using SVF, and the velocity field is integrated into deformation increments through exponential mapping, achieving a result greater than a set value during the coarse alignment stage. and Achieve rapid global alignment by reducing [the size of the target area] during the refinement phase. And By focusing on areas with significant residuals to achieve local refinement, a natural iterative behavior is formed.

8. The lung 4DCT medical image registration method based on deep reinforcement learning as described in claim 1, characterized in that, The process of using alignment similarity enhancement, deformation regularization, and topological constraints as rewards involves using local normalized cross-correlation as the primary similarity measure and using similarity enhancement as an immediate reward. A velocity field smoothing regularization is added as a reward and penalty term to constrain the updated spatial roughness, and a soft constraint penalty is introduced on the Jacobian determinant to encourage it to remain positive and stay away from excessively small regions.

9. The lung 4DCT medical image registration method based on deep reinforcement learning as described in claim 1, characterized in that, The process of rewarding cross-phase temporal consistency includes: explicitly introducing cross-phase consistency into the reward; using a cyclic consistency term to characterize the near identity mapping of forward and reverse combinations based on the deformation between adjacent phases; and introducing a time smoothing term for adjacent velocity fields.

10. A lung 4DCT medical image registration system based on deep reinforcement learning, characterized in that, include: The residual tensor calculation module is configured to acquire source image and target image, resample source image under current deformation to obtain deformed source image, perform multi-scale feature encoding on deformed source image and target image to obtain features at different scales, calculate residuals of features at different scales, use the feature residuals to characterize alignment error, and obtain residual tensor. The uncertainty map calculation module is configured to perform uncertainty estimation based on the deformation source image and target image under the current deformation, and obtain the uncertainty map. The Jacobian risk calculation module is configured to calculate Jacobian risk based on the current deformation. The splicing module is configured to splice the residual tensor, uncertainty graph, and Jacobian risk to obtain the current state; The deep reinforcement learning registration module is configured to perform deep reinforcement learning registration under a constrained continuous control Markov decision process based on the current state. During the deep reinforcement learning registration process, alignment similarity enhancement, deformation regularization and topological constraints, as well as cross-phase temporal consistency are used as rewards to obtain medical registration results.

Images