A lung multi-phase CT registration method and apparatus

Abstract

The present application relates to the technical field of medical image processing, and discloses a lung multi-time phase CT registration method and device, comprising: acquiring a to-be-registered image pair for lung medical image registration, the to-be-registered image pair comprising a floating image and a fixed image, the floating image and the fixed image being lung medical images of a target object at different time phases; splicing and downsampling the floating image and the fixed image along a channel dimension to obtain a first scale to-be-registered image; performing multi-scale deformation field prediction on the first scale to-be-registered image by using a pre-trained multi-scale registration network to obtain target deformation field data, the multi-scale registration network being pre-trained based on lung medical training images at different time phases by a first scale registration network, a second scale registration network and a third scale registration network; and performing spatial transformation on the floating image based on the target deformation field data to obtain a target registration image. The present application improves the registration accuracy of cross-time phase lung CT images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a lung multi-phase CT registration method and device. BACKGROUND

[0002] In the field of medical image analysis, lung multi-phase image registration is a key task with both challenges and importance. Lung multi-phase images contain anatomical structure and physiological function information of the lungs at different time points, which plays an irreplaceable role in disease diagnosis, treatment plan development, and efficacy evaluation. However, the physiological activities such as breathing and heartbeat of the human body at different time phases will cause complex non-rigid deformation of the lung tissue, which makes the registration work between lung multi-phase images very difficult.

[0003] In recent years, convolutional neural networks have made significant progress in image registration, but there are still deficiencies in multi-phase lung image processing. Currently, the main lung multi-phase image registration methods include single-scale deep learning registration method and multi-scale deep learning registration method. The single-scale deep learning registration method often directly predicts the deformation field, which has limited ability to capture complex deformation, and the feature extraction and expression are difficult to obtain multi-scale information and context-dependent relationships, resulting in low registration accuracy. Although the multi-scale deep learning registration method can gradually capture deformation information, it has the problem of insufficient information fusion between scales, resulting in information transmission loss or inaccuracy. Therefore, the existing technology has poor registration accuracy in multi-phase image registration. SUMMARY

[0004] Therefore, the present application provides a lung multi-phase CT registration method and device to solve the problem of poor registration accuracy in multi-phase image registration.

[0005] In a first aspect, the present application provides a lung multi-phase CT registration method, comprising: obtaining a pair of to-be-registered images for lung medical image registration, the pair of to-be-registered images comprising a floating image and a fixed image, the floating image and the fixed image being lung medical images of a target object at different time phases; concatenating and downsampling the floating image and the fixed image along the channel dimension to obtain a first scale to-be-registered image; performing multi-scale deformation field prediction on the first scale to-be-registered image using a pre-trained multi-scale registration network to obtain target deformation field data, the multi-scale registration network being pre-trained based on lung medical training images at different time phases by a first scale registration network, a second scale registration network, and a third scale registration network; and performing spatial transformation on the floating image based on the target deformation field data to obtain a target registration image.

[0006] In an optional implementation, the first-scale to-be-registered image is subjected to multi-scale deformation field prediction by using a pre-trained multi-scale registration network to obtain target deformation field data, including: performing deformation field prediction on the first-scale to-be-registered image by using a first-scale registration network to obtain initial deformation field data; performing up-sampling on the initial deformation field data to obtain first-scale deformation field data; performing spatial transformation on the floating image based on the first-scale deformation field data to obtain a first-scale deformed image; concatenating and down-sampling the fixed image and the first-scale deformed image along the channel dimension to obtain a second-scale to-be-registered image; performing residual deformation field prediction on the second-scale to-be-registered image by using a second-scale registration network to obtain first residual deformation field data; performing up-sampling on the first residual deformation field data to obtain second-scale deformation field data; performing spatial transformation on the floating image based on the second-scale deformation field data to obtain a second-scale deformed image; concatenating the fixed image and the second-scale deformed image along the channel dimension to obtain a third-scale to-be-registered image; performing residual deformation field prediction on the third-scale to-be-registered image by using a third-scale registration network to obtain target deformation field data.

[0007] In an optional implementation, the up-sampling of the first residual deformation field data to obtain second-scale deformation field data includes: performing up-sampling on the first residual deformation field data to obtain first-scale residual deformation field data; and superimposing the first-scale deformation field data and the first-scale residual deformation field data to obtain second-scale deformation field data.

[0008] In an optional implementation, the residual deformation field prediction on the third-scale to-be-registered image by using a third-scale registration network to obtain target deformation field data includes: performing residual deformation field prediction on the third-scale to-be-registered image by using a third-scale registration network to obtain second residual deformation field data; and superimposing the second-scale deformation field data and the second residual deformation field data to obtain target deformation field data.

[0009] In an optional implementation, the first-scale registration network includes an encoder, a decoder, and an attention skip connection layer; and the deformation field prediction on the first-scale to-be-registered image by using the first-scale registration network to obtain initial deformation field data includes: performing stepwise down-sampling on the first-scale to-be-registered image by using the encoder to obtain multi-scale semantic features; performing weighted fusion on the multi-scale semantic features by using the attention skip connection layer to obtain weighted low-level features; and performing stepwise up-sampling on the weighted low-level features by using the decoder to restore the resolution and perform deformation field prediction to obtain initial deformation field data.

[0010] In an optional implementation, the multi-scale semantic features are weighted and fused by using an attention skip connection layer to obtain weighted low-level features, including: adding low-level features in the multi-scale semantic features and high-level features output by the decoder by using the attention skip connection layer to obtain fused features; sequentially processing the fused features by using an activation function to generate attention weights; and multiplying the attention weights and the low-level features element by element to obtain the weighted low-level features.

[0011] The lung multi-time phase CT registration method provided by the application further includes: obtaining a multi-time phase floating training image and a fixed training image of the same object; performing multi-scale deformation field prediction on the floating training image and the fixed training image by using an initial multi-scale registration network to obtain multi-scale deformation field data; performing spatial transformation on the floating training image based on the multi-scale deformation field data to obtain a multi-scale deformation image; constructing a first loss based on the multi-scale deformation image, the fixed training image, the floating training image and the multi-scale deformation field data; constructing a second loss based on the multi-scale deformation image and the fixed training image; adding the first loss and the second loss to obtain a total loss; and based on the total loss, iteratively optimizing network parameters of the initial multi-scale registration network by back propagation to complete training of the initial multi-scale registration network and obtain a multi-scale registration network.

[0012] In an optional implementation, the first loss is constructed based on the multi-scale deformation image, the fixed training image, the floating training image and the multi-scale deformation field data, including: calculating a non-similarity measure based on the multi-scale deformation image and the fixed training image; performing smoothing regularization processing on the multi-scale deformation field data to obtain regular deformation field data; calculating a difference value based on the fixed training image and the floating training image; calculating an adaptive weight coefficient based on the difference value; and performing loss function calculation based on the non-similarity measure, the regular deformation field data and the adaptive weight coefficient to obtain the first loss.

[0013] In an optional implementation, the second loss is constructed based on the multi-scale deformation image and the fixed training image, including: performing blood vessel segmentation and normalization processing on the multi-scale deformation image by using a threshold method to obtain a multi-scale deformation blood vessel image; performing blood vessel segmentation and normalization processing on the fixed training image by using a threshold method to obtain a fixed blood vessel training image; and performing loss function calculation based on the multi-scale deformation blood vessel image and the fixed blood vessel training image to obtain the second loss.

[0014] In a second aspect, the application provides a lung multi-time phase CT registration device, including: a memory and a processor, the memory and the processor are connected in communication with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the lung multi-time phase CT registration method of the first aspect or any of the corresponding embodiments.

[0015] The application provides a lung multi-time CT registration method, which comprises the following steps: first, obtaining a to-be-registered image pair for lung medical image registration, wherein the to-be-registered image pair comprises lung CT images of a same target object at different time, namely a floating image and a fixed image, and the to-be-registered image pair can be preprocessed and intensity value normalized, which not only clearly defines the registration object and can reflect the lung changes over time, but also can highlight the key registration regions such as the lung and the liver, thereby providing accurate image data for subsequent registration; then, the floating image and the fixed image are spliced and down-sampled along the channel dimension to obtain a first scale to-be-registered image, the splicing can integrate the information of the two images, and the down-sampling can reduce the subsequent calculation amount and help the model extract global features; then, a multi-scale registration network composed of a first scale registration network, a second scale registration network and a third scale registration network which are pre-trained based on different time lung medical training images is used to perform multi-scale deformation field prediction on the first scale to-be-registered image, so as to obtain target deformation field data, the three scale networks cooperate with each other and progress layer by layer, and the image information under different scales is fully utilized, so that the deformation relationship between the floating image and the fixed image can be more comprehensively and accurately described; finally, the floating image is subjected to spatial transformation based on the target deformation field data to obtain a target registration image. This method effectively overcomes the local detail mismatching problem of the traditional single scale registration method in complex organ deformation modeling and the low precision and slow speed problem of the existing multi-scale registration method, and significantly improves the registration precision of the cross-time lung CT image.

[0016] The application processes different time lung medical training images by using the initial multi-scale registration network. A progressive registration strategy from coarse to fine is realized by three deep models with different scales. The low resolution / same resolution deformation field output by the upper network is superimposed with the deformation field of the lower network, and the final deformation field is output. This method fully utilizes the image information under different scales, and effectively improves the anatomical structure alignment accuracy of the cross-time lung CT image by combining the comprehensive optimization of the first loss and the second loss. At the same time, the training images of different time and the vascular prior knowledge are introduced, so that the network can better adapt to the morphological changes of the lung during the breathing process, and the registration ability of the fine organ is improved, so as to complete the training of the initial multi-scale registration network. The multi-scale registration network obtained effectively improves the registration accuracy and the capture ability of the fine structure, so that the registration result is more accurate and reliable, and provides more accurate image registration support for the diagnosis and treatment of lung diseases. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the drawings needed in the following specific embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flowchart of a lung multi-phase CT registration method according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a processing procedure of a floating image and a fixed image according to an embodiment of the present application;

[0020] Figure 3 is a structural diagram of a first scale registration network according to an embodiment of the present application;

[0021] Figure 4 is a structural diagram of an attention skip connection layer according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0023] According to an embodiment of the present application, a lung multi-phase CT registration method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0024] In the present embodiment, a lung multi-phase CT registration method is provided, which can be used in the mobile terminal described above, such as a mobile phone, a tablet computer, etc. (the execution subject is described in combination with the actual situation), Figure 1 is a flowchart of a lung multi-phase CT registration method according to an embodiment of the present application, as shown in Figure 1 the flowchart includes the following steps:

[0025] In step S101, a to-be-registered image pair for lung medical image registration is acquired, the to-be-registered image pair includes a floating image and a fixed image, and the floating image and the fixed image are lung medical images of a target object at different time phases.

[0026] The fixed image in the present embodiment is a reference standard in the registration task, and the floating image is an object to be adjusted in the registration task, which may have differences in spatial position, direction, scale, etc. from the target state, and needs to be aligned with the fixed image through certain transformation operations (such as translation, rotation, scaling, etc.). With fixed image It involves taking CT (Computed Tomography) images of the same target object at different time points. The registration task selects any two images from different time phases as floating images. With fixed image This allows them to combine and form the image pair to be registered.

[0027] This embodiment can also include floating images in the image pair to be registered. and fixed image Preprocessing is performed by setting reasonable maximum and minimum intensity thresholds (within these threshold ranges, regions that highlight registration relationships, such as the lungs and liver). The floating image is then processed according to these maximum and minimum intensity thresholds. With fixed image Intensity values ​​are normalized, and the resulting normalized floating and normalized fixed images can be used for subsequent registration processing.

[0028] Step S102: The floating image and the fixed image are stitched together and downsampled along the channel dimension to obtain the first scale image to be registered.

[0029] like Figure 2 As shown, the floating image and fixed image The concatenation operation along the channel dimension integrates information from two images. In deep learning, image data often exists in tensor form, whose dimensions are generally represented as (batch size, number of channels, height, width). The channel dimension represents the depth of the image / feature map, with each channel corresponding to a feature map that captures different feature information from the input data. The process of concatenating along the channel dimension involves connecting images along the channel number dimension. Specifically, it involves concatenating floating images... and fixed image Connect the floating images end-to-end along the channel dimension. After all channels are arranged sequentially, a fixed image is then arranged. All channels of the image are used to integrate the information from the two images. After stitching, the number of channels in the new image is equal to the sum of the number of channels in the floating image and the number of channels in the fixed image. Then, the stitched image is downsampled (e.g., 4x downsampling), which reduces the height and width of the image to 1 / 4 of their original size. This reduces the amount of computation required and helps the model extract global features, ultimately resulting in the first-scale image to be registered, which can be used as input to the subsequent registration model.

[0030] Step S103, using the pre-trained multi-scale registration network to perform multi-scale deformation field prediction on the first scale to-be-registered image, to obtain target deformation field data, and the multi-scale registration network is pre-trained based on lung medical training images at different time phases by the first scale registration network, the second scale registration network and the third scale registration network.

[0031] In the embodiment, the multi-scale registration network is composed of the first scale registration network, the second scale registration network and the third scale registration network. As shown in the figure, Figure 2 by means of three processing stages and in cooperation with the three scale registration networks, the floating image and the fixed image can be progressively completed in the order from coarse to fine for lung image registration. The specific structures of the first scale registration network, the second scale registration network and the third scale registration network are consistent, and since the resolution of the image obtained at each processing stage changes from low to high, a distinct multi-scale feature is presented, which is the key to efficient registration, so that the three scale registration networks can process the lung image at different levels, thereby efficiently realizing the lung image registration from coarse to fine.

[0032] The multi-scale registration network is pre-trained by using lung medical training images (such as lung CT) at different time phases, and in the training process, the network parameters are constantly adjusted to minimize the difference between the predicted deformation field and the real deformation field, so that the network learns the common deformation patterns and features of the lung image at different time phases. After training, the first scale to-be-registered image is input into the pre-trained multi-scale registration network, and the network will perform multi-scale deformation field prediction in sequence by using the three sub-networks (i.e. the first scale registration network, the second scale registration network and the third scale registration network), and finally obtain the target deformation field data. The three scale networks cooperate with each other and progress layer by layer to form a complete multi-scale registration system, fully utilizing the image information at different scales, so as to more comprehensively and accurately describe the deformation relationship between the floating image and the fixed image. The deformation field data can describe how the floating image needs to be deformed to accurately align with the fixed image to realize image registration.

[0033] Step S104, performing spatial transformation on the floating image based on the target deformation field data to obtain a target registration image.

[0034] The target deformation field data obtained in step S103 is used to perform spatial transformation on the floating image. The target deformation field specifies how the pixels move to align with the fixed image, based on which the pixels of the floating image are repositioned and adjusted, and through coordinate transformation and interpolation calculation, the floating image is spatially matched with the fixed image, and finally a target registration image is obtained, providing a reliable basis for subsequent medical applications.

[0035] The embodiment provides a lung multi-time CT registration method, which comprises the following steps: first, obtaining a to-be-registered image pair for lung medical image registration, wherein the to-be-registered image pair comprises lung CT images of a same target object at different time, namely a floating image and a fixed image, and the to-be-registered image pair can be preprocessed and intensity value normalized, which not only clearly defines a registration object and can reflect lung changes over time, but also highlights key registration regions such as lungs and livers, thereby providing accurate image data for subsequent registration; then, the floating image and the fixed image are spliced and down-sampled along a channel dimension to obtain a first scale to-be-registered image, the splicing can integrate information of the two images, and the down-sampling can reduce subsequent calculation amount and help a model extract global features; then, a multi-scale registration network composed of a first scale registration network, a second scale registration network and a third scale registration network which are pre-trained based on different time lung medical training images is used to perform multi-scale deformation field prediction on the first scale to-be-registered image, so as to obtain target deformation field data, the three scale networks cooperate with each other and progress layer by layer, and image information at different scales is fully utilized, so that the deformation relationship between the floating image and the fixed image can be more comprehensively and accurately described; finally, the floating image is subjected to spatial transformation based on the target deformation field data to obtain a target registration image. The method effectively overcomes the local detail mismatching problem of traditional single-scale registration methods in complex organ deformation modeling and the low precision and slow speed problem of existing multi-scale registration methods, and significantly improves the registration precision of cross-time lung CT images.

[0036] In some optional embodiments, the process of using the pre-trained multi-scale registration network to perform multi-scale deformation field prediction on the first scale to-be-registered image to obtain target deformation field data in the step S103 mainly comprises the following steps:

[0037] S1031, performing deformation field prediction on the first scale to-be-registered image by using the first scale registration network to obtain initial deformation field data.

[0038] As shown in Figure 2 , the first scale to-be-registered image is input into the first scale registration network, the first scale registration network performs deformation field prediction on the first scale to-be-registered image, and outputs the initial deformation field data.

[0039] S1032, performing up-sampling on the initial deformation field data to obtain first scale deformation field data.

[0040] As shown in Figure 2 , the initial deformation field data is up-sampled (for example, 4x up-sampling) to obtain the first scale deformation field data .

[0041] S1033, performing spatial transformation on the floating image based on the first scale deformation field data to obtain a first scale deformation image.

[0042] AsFigure 2 As shown, the first-scale deformation field data Apply to floating images The floating image's pixels are adjusted according to the first-scale deformation field data. After repositioning and adjustment, the first-scale deformation image was finally obtained. .

[0043] S1034, the fixed image and the first-scale deformed image are stitched together and downsampled along the channel dimension to obtain the second-scale image to be registered.

[0044] like Figure 2 As shown, first fix the image and first-scale deformation image The images are stitched together along the channel dimension, merging information from both images in that direction. This results in a new image containing more features, aiding in subsequent analysis of the differences and relationships between the two images. Next, the stitched image is downsampled (e.g., 2x downsampling) to reduce its size in both length and width. This reduces subsequent computational load and complexity, facilitating faster processing while extracting global features and abstract information. Finally, the stitching and downsampling operations yield a second-scale image for registration, used in a subsequent second-scale registration network to more accurately predict the deformation field and achieve finer image registration.

[0045] S1035, the second-scale registration network is used to predict the residual deformation field of the second-scale image to be registered, and the first residual deformation field data is obtained.

[0046] like Figure 2 As shown, the image to be registered at the second scale is input into the second-scale registration network. The second-scale registration network predicts the residual deformation field for the subtle local differences remaining after the first-scale registration. After calculation and analysis, the first residual deformation field data is output, providing a key basis for subsequent registration optimization.

[0047] S1036, Upsample the first residual deformation field data to obtain the second-scale deformation field data.

[0048] like Figure 2 As shown, the first residual deformation field data is upsampled (e.g., 2x upsampling) to obtain the second-scale deformation field data. .

[0049] S1037, Spatial transformation is performed on the floating image based on the second-scale deformation field data to obtain the second-scale deformation image.

[0050] like Figure 2 As shown, the second-scale deformation field data Apply to floating images This allows each pixel of the floating image to be based on the second-scale deformation field data. After repositioning and adjustment, the second-scale deformation image was finally obtained. .

[0051] S1038, the fixed image and the second-scale deformed image are stitched together along the channel dimension to obtain the third-scale image to be registered.

[0052] like Figure 2 As shown, the image will be fixed. Second-scale deformation image By stitching along the channel dimension and merging the information of the two images in the channel direction, the new image contains more features, which helps to analyze the differences and correlations between the two images in the subsequent analysis. This results in a second-scale image to be registered, which is then used in the subsequent third-scale registration network to further accurately predict the deformation field and complete a more refined image registration.

[0053] S1039, using a third-scale registration network to predict the residual deformation field of the third-scale image to be registered, and obtaining the target deformation field data.

[0054] like Figure 2 As shown, the image to be registered at the third scale is input into the third-scale registration network. The third-scale registration network predicts the residual deformation field to account for the subtle local differences remaining after the second-scale registration. After calculation and analysis, the target deformation field data is output. The target deformation field data Apply to floating images The final target registration image can then be obtained. .

[0055] This embodiment utilizes a pre-trained multi-scale registration network. First, a first-scale registration network predicts the deformation field of the first-scale image to be registered, obtaining an initial deformation field. Then, after upsampling and spatial transformation, a first-scale deformed image is obtained. This first-scale image is then stitched with a fixed image to generate a second-scale image to be registered. The second-scale registration network predicts the residual deformation field, and subsequent operations yield a second-scale deformed image. This process continues until a third-scale registration network predicts the residual deformation field of the third-scale image to obtain the target deformation field data. This method cascades three registration networks of different scales, implementing a progressive registration strategy from coarse to fine. Through iterative optimization of the deformation field, it gradually corrects the remaining subtle local differences after each registration step, effectively overcoming the local detail mismatch problem inherent in traditional single-scale registration methods in modeling complex organ deformation. The pre-trained multi-scale registration network efficiently learns the features and deformation patterns of images at different scales. The registration process employs upsampling and downsampling operations to balance computational load and feature information utilization. This approach fully leverages features while reducing redundant computation, balancing accuracy and registration speed. It overcomes the problems of low accuracy and slow speed in existing multi-scale registration methods, significantly improving the alignment accuracy of anatomical structures in cross-temporal lung CT images.

[0056] In some optional implementations, the process of upsampling the first residual deformation field data to obtain the second-scale deformation field data in step S1036 mainly includes:

[0057] S10361, Upsample the first residual deformation field data to obtain the first-scale residual deformation field data.

[0058] like Figure 2 As shown, the first residual deformation field data is upsampled (e.g., 2x upsampling) to obtain the first-scale residual deformation field data. .

[0059] S10362, the first-scale deformation field data and the first-scale residual deformation field data are superimposed to obtain the second-scale deformation field data.

[0060] like Figure 2 As shown, the first-scale deformation field data Compared with the first-scale residual deformation field data By superimposing the data, the deformation field data at the second scale is obtained. .

[0061] In this embodiment, during the process of obtaining the second-scale deformation field data, the first residual deformation field data is upsampled to obtain the first-scale residual deformation field data, which is then superimposed on the first-scale deformation field data. This method is similar to how the low-resolution deformation field output by the upper-level network is upsampled and passed to the current scale, and superimposed on the deformation field of the current scale to output the final deformation field. This method can integrate deformation information from different stages, gradually correct and improve the deformation field, so that the final deformation field more accurately reflects the true deformation relationship between images, thereby further improving the accuracy of multi-temporal CT image registration for the lungs.

[0062] In some optional implementations, the process of using a third-scale registration network to predict the residual deformation field of the third-scale image to be registered, and obtaining the target deformation field data in step S1039 above, mainly includes:

[0063] S10391, the residual deformation field of the image to be registered at the third scale is predicted using the third-scale registration network to obtain the second residual deformation field data.

[0064] like Figure 2 As shown, the image to be registered at the third scale is input into the third-scale registration network. The third-scale registration network performs residual deformation field prediction for the remaining subtle local differences after the second-scale registration, thus obtaining the second residual deformation field data. .

[0065] S10391, the second-scale deformation field data and the second residual deformation field data are superimposed to obtain the target deformation field data.

[0066] like Figure 3 As shown, the second-scale deformation field data Second residual deformation field data Superimposed to obtain the target deformation field data .

[0067] This embodiment first uses a third-scale registration network to predict the residual deformation field of the third-scale image to be registered, obtaining second residual deformation field data. This data accurately captures details that are not yet aligned between images, addressing the subtle local differences remaining after the second-scale registration. Then, the second-scale deformation field data is superimposed with the second residual deformation field data to obtain the target deformation field data. This method of combining residual prediction and deformation field superposition can integrate deformation information at different scales, gradually correcting and improving the deformation field, effectively compensating for deficiencies in the preceding registration steps, further improving the accuracy of multi-temporal CT image registration of the lungs, and achieving more precise alignment of anatomical structures.

[0068] like Figure 3 As shown, the first-scale registration network includes an encoder, a decoder, and an attention-skipping connection layer.

[0069] Furthermore, the process of using a first-scale registration network to predict the deformation field of the first-scale image to be registered, and obtaining initial deformation field data in step S1031 above, includes:

[0070] S10311, the encoder is used to progressively downsample the first-scale image to be registered to obtain multi-scale semantic features.

[0071] like Figure 3 As shown, the first-scale image to be registered is input into the encoder of the first-scale registration network. The encoder performs four downsampling operations, gradually compressing the image resolution from the original 1 to the lowest resolution 1 / 16. Low-level features (such as edge, texture and other details) and deep low-resolution semantic features (such as the overall shape of organs, global positional relationships, etc.) are extracted. These features together constitute multi-scale semantic features, and the features of each resolution scale are input into the attention jump connection layer of the corresponding layer.

[0072] S10312 utilizes an attention-skipping connection layer to weightedly fuse multi-scale semantic features, resulting in weighted low-level features.

[0073] like Figure 3 As shown, the attention skip connection layer connects the encoder and decoder at the same resolution. When processing multi-scale semantic features, the attention skip connection layer assigns specific weight values ​​to the semantic features fed by the encoder of the corresponding layer and the decoder of the previous layer, and performs weighted fusion. In this way, more important features can be highlighted, while flexibly controlling the importance of features at different spatial locations. After this processing, weighted low-level features are output, which are then fed into the decoder of the corresponding layer to aid in subsequent analysis and processing.

[0074] S10313 uses a decoder to progressively upsample the weighted low-level features to restore the resolution and perform deformation field prediction to obtain the initial deformation field data.

[0075] like Figure 4 As shown, the decoder gradually restores the weighted low-level features delivered by the attention jump connection layer from the lowest resolution 1 / 16 to the original resolution 1 through four upsampling operations. It receives the weighted fusion features (including details and semantics) output by the attention jump connection layer, generates the initial deformation field data through layer-by-layer feature refinement, and passes it to the end spatial transformer to guide image registration.

[0076] The second-scale registration network and the third-scale registration network have the same network structure as the first-scale registration network, and will not be described again here.

[0077] The first scale registration network in the embodiment extracts multi-scale semantic features through encoder downsampling, giving consideration to both details and global information; the attention skip connection layer weights and fuses the features of the encoder, highlighting key information; the decoder upsamples to restore resolution and generate initial deformation field data, and image registration is guided by the spatial transformer, thereby improving the image registration accuracy and the ability of the first scale registration network to handle geometric transformations.

[0078] In some optional embodiments, the process of weighting and fusing the multi-scale semantic features by the attention skip connection layer in step S10312 to obtain the weighted low-level features includes:

[0079] S103121, adding the low-level features in the multi-scale semantic features and the high-level features output by the decoder by using the attention skip connection layer to obtain fused features.

[0080] In the first scale registration network, the encoder obtains multi-scale semantic features through downsampling, which contains low-level features (such as edge, texture, and other detail information) and high-level features (such as the overall shape of an organ, global positional relationship, and other semantic information) obtained by the decoder of the previous layer through upsampling. The attention skip connection layer adds these two features of different levels to preliminarily integrate the feature information of different scales and prepare for generating attention weights. This step is to simply fuse features of different scales so that the importance of each feature can be determined based on the fused features.

[0081] S103122, sequentially processing the fused features by an activation function to generate attention weights.

[0082] For example, the attention weights are calculated in the following manner:

[0083] ,

[0084] ,

[0085] wherein, represents the size of the voxel space, represents the fused features, the activation function introduces nonlinearity, represents the high-level features of the i-th layer, represents a specific position, represents a convolution kernel with a size of 1×1×1 for linear transformation of the high-level features, represents a transpose operator of a matrix, represents the low-level features of the i-th layer, ​​This represents the convolution parameters with a kernel size of 1×1×1, used for linear transformation of low-level features. Indicates an upsampling operation. This represents the bias term in the network. Show the first Attention weights of layers Activation function This represents the convolution parameters with a kernel size of 1×1×1, used to map the fused features. This represents the bias term in the network.

[0086] S103123, multiply the attention weights element-wise with the low-level features to obtain the weighted low-level features.

[0087] For example, weighted low-level features are obtained using the following method:

[0088] ,

[0089] in, Indicates the first Layer position First Weighted low-level features of each channel, Indicates channel size. Indicates the first Layer position First Low-level features of the eigenvalues ​​of each channel.

[0090] Attention weight It is for location It is calculated jointly by all channels at a given location, and it represents the importance of that location's features. and By multiplying element-wise, the eigenvalues ​​of each channel can be adjusted according to their importance, resulting in... That is, the first Layer position First Weighted low-level features of each channel.

[0091] The above steps can be used as a reference. ​ Low-level and high-level features are input into the attention jump connection layer. The low-level features are linearly transformed by a convolutional layer with a kernel size of 1×1×1. The high-level features are linearly transformed by a convolutional layer with a kernel size of 1×1×1 after 2x upsampling. Then, the two convolutional features are added together, and the added features are then processed... The activation function introduces non-linearity to generate fused features. These fused features are then linearly transformed through a convolutional layer with a kernel size of 1×1×1. The activation function processing generates attention weights, and finally, the attention weights are element-wise multiplied with the low-level features to obtain weighted low-level features corresponding to the attention skip connection layer.

[0092] In this embodiment, the attention skip connection layer is used to weight and fuse the multi-scale semantic features. First, the low-level and high-level features are added to preliminarily integrate the information, which prepares for generating the attention weights. Then, the activation function processing is used to generate the attention weights from the fused features, so as to realize adaptive feature selection. Finally, the attention weights are multiplied with the low-level features to obtain the weighted low-level features. The attention mechanism module is used to strengthen the accuracy and semantic correlation of feature extraction, which provides reliable support for image registration and the like, and improves the accuracy of subsequent registration.

[0093] In some optional embodiments, the multi-scale registration network pre-trained in step S103 is pre-trained based on the lung medical training images of different time phases by the first scale registration network, the second scale registration network and the third scale registration network. The specific training process includes:

[0094] Step one, obtaining multi-time floating training images and fixed training images of the same object.

[0095] The lung CT images of any target object taken at different time points are obtained as floating training images and fixed training images. Taking 10 time phase images as an example, these time phases represent 10 time points or states that are evenly divided in a complete respiratory cycle (usually from the end of one expiration to the end of the next expiration). These 10 time phases are not simply "1st second, 2nd second", but 3D lung CT images representing different anatomical positions in the respiratory cycle, which are synchronously collected and reconstructed according to the patient's breathing signal, including T0, T1, T2, T3, T4, T5, T6, T7, T8 and T9. Among them, T0 represents stability, and T4 and T5 represent extreme states in the breathing process. These images of different time phases contain morphological change information of the lung during the breathing process, which provides a rich data basis for subsequent network training.

[0096] Step two, using the initial multi-scale registration network to predict the multi-scale deformation field of the floating training image and the fixed training image to obtain multi-scale deformation field data.

[0097] The initial multi-scale registration network (including the first scale registration network, the second scale registration network and the third scale registration network) is used to predict the multi-scale deformation field of the floating training image and the fixed training image to obtain multi-scale deformation field data.

[0098] Step three, performing spatial transformation on the floating training image based on the multi-scale deformation field data to obtain multi-scale deformation images.

[0099] The multi-scale deformation field data obtained by the initial multi-scale registration network is used to act on the floating training image, so as to move the pixel points in the floating training image to the corresponding positions, thereby obtaining the multi-scale deformation image after deformation. These deformation images are the results of adjusting the floating training image under different scales, and are closer to the morphology of the fixed training image.

[0100] Step four, constructing a first loss based on the multi-scale deformation image, the fixed training image, the floating training image and the multi-scale deformation field data.

[0101] The first loss is constructed to measure the differences between the multi-scale deformation image and the fixed training image and the floating training image. By calculating these differences, the matching degree between the deformed image and the target image (the fixed training image) and whether the deformation process is reasonable can be understood. This loss value will be used for subsequent network parameter optimization to enable the initial multi-scale registration network to learn more accurate deformation field prediction capability.

[0102] Step five, constructing a second loss based on the multi-scale deformation image and the fixed training image.

[0103] In the training process, in order to better guide the network to learn the fine organ registration and the overall fitting direction, the embodiment introduces the segmentation result with blood vessel prior knowledge as auxiliary information. The second loss is constructed based on the multi-scale deformation image and the fixed training image, which considers the registration of fine structures such as blood vessels, further constrains the training of the initial multi-scale registration network, and enables the network to more accurately register the fine organs of the lung.

[0104] Step six, adding the first loss and the second loss to obtain a total loss.

[0105] The first loss and the second loss are added to obtain the total loss, so that the matching degree of the whole image and the registration condition of the fine organs can be considered comprehensively. The total loss will be used as an overall indicator to measure the performance of the network under the current parameters, and to provide a basis for subsequent back propagation and parameter update.

[0106] Step seven, based on the total loss, the network parameters of the initial multi-scale registration network are iteratively optimized through back propagation to complete the training of the initial multi-scale registration network, and a multi-scale registration network is obtained.

[0107] The embodiment uses the total loss to iteratively optimize the parameters of the initial multi-scale registration network through the back propagation algorithm. The back propagation algorithm calculates the gradient of each network parameter according to the total loss, and then adjusts the parameters according to the direction of the gradient, so that the total loss gradually decreases. After multiple iterations, the parameters of the network are continuously optimized, and finally the training is completed to obtain a multi-scale registration network with better performance.

[0108] The embodiment processes the lung medical training images of different time phases by using an initial multi-scale registration network. A progressive registration strategy from coarse to fine is realized by three deep models of different scales. The low resolution / same resolution deformation field output by the upper network is superimposed with the deformation field of the lower network, and the final deformation field is output. This way makes full use of the image information at different scales, and effectively improves the alignment accuracy of the anatomical structure of the cross-time lung CT image through the comprehensive optimization of the first loss and the second loss. At the same time, the multi-time training image and the blood vessel prior knowledge are introduced, so that the network can better adapt to the morphological changes of the lung during the breathing process, and the registration ability of the fine organs is improved. To complete the training of the initial multi-scale registration network, the multi-scale registration network obtained effectively improves the registration accuracy and the capture ability of the fine structure, so that the registration result is more accurate and reliable, and provides more accurate image registration support for the diagnosis and treatment of lung diseases.

[0109] In some optional embodiments, the process of constructing the first loss based on the multi-scale deformation image, the fixed training image, the floating training image and the multi-scale deformation field data mainly includes:

[0110] Step one, calculate the non-similarity measure based on the multi-scale deformation image and the fixed training image.

[0111] The three scale deformation images obtained by the first scale registration network, the second scale registration network and the third scale registration network are respectively calculated with the fixed training image to measure the difference between the registration results at different scales and the target fixed training image.

[0112] Step two, perform smoothing regularization processing on the multi-scale deformation field data to obtain regular deformation field data.

[0113] The three scale deformation field data predicted by the first scale registration network, the second scale registration network and the third scale registration network are processed to control the smoothness of the deformation field.

[0114] Step three, calculate the difference value based on the fixed training image and the floating training image.

[0115] Exemplarily, the difference value is calculated in the following way:

[0116] ,

[0117] Wherein, represents the difference value of the fixed training image and the floating training image, represents the absolute value operation, represents the minimum value operation, represents the time phase of the current floating training image, indicates the phase of the current fixed training image, indicates the total number of phases of the current CT training image.

[0118] Step four, calculate adaptive weight coefficient based on difference value.

[0119] According to the calculation of step three, for example, for 10 phase training images, there are five cases of difference values 0, 1, 2, 3, and 4 in total. Using the above formula, it is easy to obtain that the difference value The case of 4 will only appear 4 times in pairwise matching, that is, (T0, T4), (T0, T5), (T9, T4), (T9, T5), that is, the difference value of phase 0 and phase 4 or 5, and the difference value of phase 9 and phase 4 or 5 can be quantified as 4 units. Similarly, the difference value There are 8 groups of combinations of (T0, T3), (T9, T6), (T1, T4), (T8, T5), (T1, T5), (T8, T4), (T9, T3), (T0, T6); the difference value There are 12 groups of combinations of (T0, T2), (T9, T7), (T1, T3), (T8, T6), (T2, T4), (T7, T5), (T0, T7), (T9, T2), (T1, T6), (T8, T3), (T2, T5), (T7, T4); the difference value There are 16 groups of combinations of (T0, T1), (T9, T8), (T1, T2), (T8, T7), (T2, T3), (T7, T6), (T3, T4), (T6, T5), (T0, T8), (T9, T1), (T1, T7), (T8, T2), (T2, T6), (T7, T3), (T3, T5), (T6, T4); the difference value There are 5 groups of combinations of (T0, T9), (T1, T8), (T2, T7), (T3, T6), (T4, T5). According to the difference value between different phases, the adaptive weight Then, the adaptive weight coefficient corresponding to different difference values is:

[0120] .

[0121] Step five, calculate the loss function based on the non-similarity measure, the regular deformation field data and the adaptive weight coefficient to obtain the first loss.

[0122] Exemplarily, the first loss is calculated in the following way:

[0123]

[0124]

[0125]

[0126] ,

[0127] wherein, represents the first loss, represents the floating training image, represents the fixed training image, represents the multi-scale deformation image, represents the deformation field data predicted by the first scale registration network, represents the deformation field data predicted by the second scale registration network, represents the deformation field data predicted by the third scale registration network, represents the non-similarity measure between the multi-scale deformation image and the fixed training image , is a regularization term, which controls the smoothness of the deformation field, , , and are compromise parameters, controls the weight of the regularization term, which can be determined according to specific tasks, and generally, the greater the value of , and can be determined according to specific tasks.

[0128] The embodiment can accurately measure the difference between the registration result at different scales and the target fixed training image by calculating the non-similarity measure between the multi-scale deformation image and the fixed training image. The smoothness of the deformation field can be effectively controlled by performing smooth regularization processing on the multi-scale deformation field data, so that the registration result is more consistent with the actual physiological characteristics. Then, by calculating the difference value between the fixed training image and the floating training image at different time phases, and obtaining the adaptive weight coefficient accordingly, the loss weight of image registration between different time phases can be flexibly adjusted according to the fluctuation degree of the lung of the target object in the training data. Finally, the first loss is calculated by comprehensively considering the non-similarity measure, the regular deformation field data and the adaptive weight coefficient, which effectively improves the accuracy of large displacement registration, and enables the initial multi-scale registration network to better learn the accurate deformation field between different time phase images during the training process, thereby providing a strong guarantee for the accurate registration of lung medical images.

[0129] In some optional embodiments, the process of constructing the second loss based on the multi-scale deformation image and the fixed training image mainly includes:

[0130] Step one, the multi-scale deformation image is segmented and normalized by threshold method to obtain a multi-scale deformation blood vessel image.

[0131] Exemplarily, the multi-scale deformation image is segmented and normalized by the following method:

[0132] ,

[0133] wherein, represents the multi-scale deformation image the multi-scale deformation blood vessel image after threshold blood vessel segmentation and normalization, is a normalization method, is a specific threshold method, is a specific threshold.

[0134] Step two, the fixed training image is segmented and normalized by threshold method to obtain a fixed blood vessel training image.

[0135] Exemplarily, the multi-scale deformation image is segmented and normalized by the following method:

[0136] ,

[0137] wherein, represents the fixed training image the fixed blood vessel training image after threshold blood vessel segmentation and normalization.

[0138] Step three, the loss function is calculated based on the multi-scale deformation blood vessel image and the fixed blood vessel training image to obtain a second loss.

[0139] Exemplarily, the second loss is calculated by the following method:

[0140] ,

[0141] wherein, represents the second loss with blood vessel prior knowledge, is a loss function.

[0142] Therefore, the total loss can be calculated according to the first loss and the second loss :

[0143] .

[0144] ​The embodiment is based on a multi-scale deformation image and a fixed training image to construct a second loss, which has important beneficial effects. First, the multi-scale deformation image and the fixed training image are respectively subjected to blood vessel segmentation and normalization processing by using a threshold method to obtain a multi-scale deformation blood vessel image and a fixed blood vessel training image, and this process integrates blood vessel prior knowledge into the initial multi-scale registration network training. As an important component of fine organs, the segmentation result of the blood vessel can provide key auxiliary information for registration. Then, the second loss, i.e., the loss with blood vessel prior knowledge, is calculated based on the two types of images, and the loss reflects the matching degree of the blood vessel part after registration. Finally, the first loss and the second loss are combined to obtain a total loss, so that the initial multi-scale registration network not only pays attention to the overall registration effect (reflected by the first loss) during the training process, but also fully utilizes the blood vessel prior knowledge to guide the registration of fine organs and the learning of the overall fitting direction, so as to obtain the final multi-scale registration network, so that the multi-scale registration network after training effectively improves the registration accuracy and the capture ability of fine structures, makes the registration result more accurate and reliable, and provides stronger support for the application of medical image registration in fine organ analysis and the like.

[0145] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in a computer readable medium includes but is not limited to source files, executable files, installation package files, etc. Correspondingly, the way of executing computer program instructions by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0146] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method of lung multi-phase CT registration, the method comprising: The method comprises: acquiring a to-be-registered image pair for lung medical image registration, the to-be-registered image pair comprising a floating image and a fixed image, the floating image and the fixed image being lung medical images of a target object at different time phases; stitching and down-sampling the floating image and the fixed image along a channel dimension to obtain a first-scale to-be-registered image; performing multi-scale deformation field prediction on the first-scale to-be-registered image by using a pre-trained multi-scale registration network to obtain target deformation field data, the multi-scale registration network being pre-trained based on lung medical training images at different time phases by a first-scale registration network, a second-scale registration network and a third-scale registration network; performing spatial transformation on the floating image based on the target deformation field data to obtain a target registered image; wherein the multi-scale deformation field prediction on the first-scale to-be-registered image by using the pre-trained multi-scale registration network to obtain the target deformation field data comprises: performing deformation field prediction on the first-scale to-be-registered image by using the first-scale registration network to obtain initial deformation field data; up-sampling the initial deformation field data to obtain first-scale deformation field data; performing spatial transformation on the floating image based on the first-scale deformation field data to obtain a first-scale deformed image; stitching and down-sampling the fixed image and the first-scale deformed image along a channel dimension to obtain a second-scale to-be-registered image; performing residual deformation field prediction on the second-scale to-be-registered image by using the second-scale registration network to obtain first residual deformation field data; up-sampling the first residual deformation field data to obtain second-scale deformation field data; performing spatial transformation on the floating image based on the second-scale deformation field data to obtain a second-scale deformed image; stitching the fixed image and the second-scale deformed image along a channel dimension to obtain a third-scale to-be-registered image; performing residual deformation field prediction on the third-scale to-be-registered image by using the third-scale registration network to obtain the target deformation field data.

2. The method of claim 1, wherein, The up-sampling of the first residual deformation field data to obtain second-scale deformation field data comprises: up-sampling the first residual deformation field data to obtain first-scale residual deformation field data; superimposing the first-scale deformation field data and the first-scale residual deformation field data to obtain the second-scale deformation field data.

3. The method of claim 1, wherein, The residual deformation field prediction on the third-scale to-be-registered image by using the third-scale registration network to obtain the target deformation field data comprises: performing residual deformation field prediction on the third-scale to-be-registered image by using the third-scale registration network to obtain second residual deformation field data; superimposing the second-scale deformation field data and the second residual deformation field data to obtain the target deformation field data.

4. The method of claim 1, wherein, The first-scale registration network comprises an encoder, a decoder and an attention skip connection layer; The deformation field prediction on the first-scale to-be-registered image by using the first-scale registration network to obtain initial deformation field data comprises: The first scale image to be registered is stepwise down-sampled by using the encoder to obtain multi-scale semantic features; The multi-scale semantic features are weighted and fused by using the attention skip connection layer to obtain weighted low-level features; The weighted low-level features are stepwise up-sampled by using the decoder to restore the resolution and perform deformation field prediction to obtain the initial deformation field data.

5. The method of claim 4, wherein, The multi-scale semantic features are weighted and fused by using the attention skip connection layer to obtain weighted low-level features, including: The low-level features in the multi-scale semantic features and the high-level features output by the decoder are added by using the attention skip connection layer to obtain fused features; The fused features are sequentially processed by an activation function to generate attention weights; The attention weights are multiplied with the low-level features element by element to obtain weighted low-level features.

6. The method according to any one of claims 1-5, characterized in that, Further comprising: Obtaining multi-temporal floating training images and fixed training images of the same object; Multi-scale deformation field prediction is performed on the floating training images and the fixed training images by using an initial multi-scale registration network to obtain multi-scale deformation field data; Spatial transformation is performed on the floating training images based on the multi-scale deformation field data to obtain multi-scale deformation images; A first loss is constructed based on the multi-scale deformation images, the fixed training images, the floating training images and the multi-scale deformation field data; A second loss is constructed based on the multi-scale deformation images and the fixed training images; The first loss and the second loss are added to obtain a total loss; Based on the total loss, the network parameters of the initial multi-scale registration network are iteratively optimized by back propagation to complete the training of the initial multi-scale registration network, thereby obtaining a multi-scale registration network.

7. The method of claim 6, wherein, The first loss is constructed based on the multi-scale deformation images, the fixed training images, the floating training images and the multi-scale deformation field data, including: Non-similarity metrics are calculated based on the multi-scale deformation images and the fixed training images; Regularization processing is performed on the multi-scale deformation field data to obtain regular deformation field data; Difference values are calculated based on the fixed training images and the floating training images; Adaptive weight coefficients are calculated based on the difference values; Loss function calculation is performed based on the non-similarity metrics, the regular deformation field data and the adaptive weight coefficients to obtain the first loss.

8. The method of claim 6, wherein, The second loss is constructed based on the multi-scale deformation images and the fixed training images, including: The multi-scale deformation images are segmented and normalized by using a threshold method to obtain multi-scale deformation vessel images; The fixed training images are segmented and normalized by using a threshold method to obtain fixed vessel training images; Loss function calculation is performed based on the multi-scale deformation vessel images and the fixed vessel training images to obtain the second loss.

9. A lung multi-phase CT registration apparatus, characterized by, Comprising: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to cause the processor to perform a lung multi-phase CT registration method according to any one of claims 1-8.

Images