Image non-rigid registration method and system based on deep learning

Abstract

The invention provides an image non-rigid registration method and system based on deep learning. The method comprises the steps of obtaining a first image of a user and a second image obtained through rigid registration; establishing a non-rigid registration model, and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model; a to-be-registered image of the same type as the second image is input into the target model, a registration result is output, and specifically, the non-rigid registration model comprises a first encoder, a second encoder and a decoder; the first encoder and the second encoder are respectively composed of a plurality of sliding window transformation modules and are used for respectively calculating characteristic vector sequences of different scales of the first image and the second image; and the decoder is used for determining the position information of the voxels on the second image matched with each position on the first image according to the feature vector sequence, so that the problem of poor registration effect caused by inconsistent body coverage ranges of the first image and the second image can be effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of non-rigid image registration, and in particular relates to a method and system for non-rigid image registration based on deep learning. Background Art

[0002] Radiotherapy is one of the important means of treating cancer. It requires accurate positioning of each patient's tumor and surrounding organs at risk, and reduces damage to organs at risk while irradiating the lesion area with high doses, thereby improving the personalized treatment effect. During different fractionated treatments, factors such as positioning errors, organ movement and deformation may cause the target area to be missed or nearby organs at risk to receive higher doses of radiation, thus affecting the prognosis of radiotherapy patients. Therefore, it is necessary to transform the doses between different fractions into the same reference space through image deformation registration, perform dose superposition, and monitor and evaluate the actual total dose of the patient during the entire radiotherapy period. Therefore, image deformation registration has important research significance in the field of radiotherapy.

[0003] Image registration is the process of converting medical images acquired at different times and with different devices into a unified spatial coordinate system, so that the image information at the same spatial position corresponds to the same anatomical structure. Due to the differences in imaging methods and principles of different modalities, there are inherent differences in the appearance of images of different modalities, such as the inconsistency between the intensity of CBCT images and the intensity (CT value) of traditional CT images, and the inconsistency between the structural representation of MR images and CT images. Therefore, finding an effective and accurate image similarity metric is a challenging problem. Traditional image registration assumes that all parts of the image conform to the same deformation model, and cannot adaptively select a deformation model that conforms to the actual situation according to the deformation differences of each organ at risk. For example, the free deformation model assumes that the deformation field at each position in space can be controlled by the deformation field of certain control points, and the deformation field at all positions is estimated by interpolation using the sparse control points. Elastic deformation regards all organs as elastic entities and satisfies the Navier-Cauchy partial differential equation. The viscous fluid flow model assumes that the deformation field conforms to the characteristics of viscous fluid and is constrained by the Navier-Stokes equation. The optical flow model assumes that the deformation field of each pixel is the temporal accumulation of its motion speed, which meets the requirements of the Lagrange transport equation. At present, the free deformation model is widely used, such as the model of deformable image registration of the commercial software RayStation and MIMMaestro software. This model has the advantage of short processing time, but the disadvantage is that it cannot well preserve the topological structure and it is difficult to add regularization constraints to the deformation field. For example, Marchant’s research found that CT images based on free deformation NiftyReg registration cannot be accurately deformed and registered in cases where there are large differences in bladder filling, movement of small intestinal loops, or the presence of air pockets in the rectum in patients with cervical cancer. When there is a large range of motion in the target area, most deformation registration algorithms based on intensity methods cannot reproduce the CT values ​​in the image well. Summary of the invention

[0004] Based on this, an embodiment of the present invention provides a deep learning-based image non-rigid registration method and system, which aims to solve the problem of poor registration effect caused by the inconsistency between the reference image and the moving image in covering the body range.

[0005] A first aspect of an embodiment of the present invention provides a method for non-rigid image registration based on deep learning, the method comprising: Acquire a first image of the user and a second image obtained by rigid body registration of the first image to generate a training set; Establishing a non-rigid registration model, and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model; Inputting an image to be registered of the same type as the second image into the target model, and outputting a registration result; The non-rigid registration model includes a decoder, and a first encoder and a second encoder that are independent of each other; The first encoder and the second encoder are both composed of a plurality of sliding window transformation modules, and are used to respectively calculate feature vector sequences of different scales of the first image and the second image; The decoder is used to determine the position information of voxels on the second image that match each position on the first image based on the feature vector sequence, wherein the feature vector sequence with the smallest size is subjected to global attention processing, and the feature vector sequences of other sizes are subjected to in-window attention processing.

[0006] Furthermore, after being input into the non-rigid registration model, the first image and the second image are firstly subjected to linear embedding processing and rearrangement processing to convert the images into feature vector sequences.

[0007] Furthermore, by combining the sliding window transformation modules, a sliding window transformation module group is obtained, and the sliding window transformation module groups in the first encoder and the second encoder are matched, wherein one end of the output of the sliding window transformation module group is connected to the next level sliding window transformation module group through a sub-block merging layer, and the other end is connected to the corresponding interactive attention module, and when the sliding window transformation module group is the last level sliding window transformation module group, the output is only connected to the first cross attention module, and the first cross attention module is used to perform global attention processing on the feature vector sequence with the smallest size.

[0008] Furthermore, the second cross-attention module is used to perform in-window attention processing on feature vector sequences of other sizes, and the first cross-attention module and the second cross-attention module are connected through an upsampling module.

[0009] Furthermore, the inputs of the first cross-attention module and the second cross-attention module are a feature vector sequence output by a sliding window transformation module group of a corresponding level of the first image, a feature vector sequence output by a sliding window transformation module group of a corresponding level of the second image, and position labels corresponding to the feature vector sequence output by a sliding window transformation module group of a corresponding level of the second image.

[0010] Furthermore, in the step of establishing a non-rigid registration model and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model, the parameters of the non-rigid registration model are randomly initialized and optimized under the guidance of a loss function, wherein the loss function is composed of an image similarity measure, a labeling constraint, and a matching position continuity constraint, and the expression is:

[0011] Among them, Loss is expressed as the total loss, α, β, and γ are non-negative real numbers used to control the contribution of different losses to the overall loss. Expressed as image similarity measure loss, Denoted as the label constraint loss of the same organ in the first image and the second image, It is expressed as the matching position continuity constraint loss.

[0012] A second aspect of an embodiment of the present invention provides a deep learning-based image non-rigid registration system, which is used to implement the deep learning-based image non-rigid registration method provided in the first aspect, and the system includes: An acquisition module, used to acquire a first image of a user and a second image obtained by rigid body registration of the first image, and generate a training set; A training module, used for establishing a non-rigid registration model, and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model; An input module, used for inputting an image to be registered of the same type as the second image into the target model, and outputting a registration result; The non-rigid registration model includes a decoder, and a first encoder and a second encoder that are independent of each other; The first encoder and the second encoder are both composed of a plurality of sliding window transformation modules, and are used to respectively calculate feature vector sequences of different scales of the first image and the second image; The decoder is used to determine the position information of voxels on the second image that match each position on the first image based on the feature vector sequence, wherein the feature vector sequence with the smallest size is subjected to global attention processing, and the feature vector sequences of other sizes are subjected to in-window attention processing.

[0013] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep learning-based image non-rigid registration method provided in the first aspect.

[0014] The fourth aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the deep learning-based image non-rigid registration method provided in the first aspect is implemented.

[0015] A method and system for non-rigid image registration based on deep learning is provided in an embodiment of the present invention. The method generates a training set by acquiring a first image of a user and a second image obtained by rigid registration of the first image; establishing a non-rigid registration model, and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model; inputting an image to be registered of the same type as the second image into the target model, and outputting a registration result. Specifically, the non-rigid registration model includes a decoder, and a first encoder and a second encoder that are independent of each other; the first encoder and the second encoder are both composed of a plurality of sliding window transformation modules, which are used to respectively calculate feature vector sequences of different scales of the first image and the second image; the decoder is used to calculate feature vector sequences of different scales of the first image and the second image according to the feature vector sequences. , determine the position information of the voxels on the second image that match each position on the first image, wherein the feature vector sequence with the smallest size is processed with global attention, and the feature vector sequences of other sizes are processed with in-window attention. It should be noted that an independent encoder design is adopted, so the registration model can not only be used for images of the same modality collected from the same patient at different times, but also for registration of images of different modalities collected from the same patient. In addition, considering the computational efficiency and video memory occupancy in the decoding network, global attention processing and in-window attention processing are used to process feature vector sequences of different sizes respectively, so as to finally improve the problem of poor registration effect caused by inconsistent coverage of the body range between the first image and the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of an implementation method of a deep learning-based non-rigid image registration method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the network structure of the non-rigid registration model; Figure 3 This is the comparison result of the annotation on CBCT and the annotation on CT after registration; Figure 4 A structural block diagram of a deep learning-based image non-rigid registration system provided in Embodiment 3 of the present invention; Figure 5 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0017] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0018] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0020] Embodiment 1 See also Figure 1 , Figure 1 A flowchart of an implementation of a deep learning-based non-rigid image registration method provided in Embodiment 1 of the present invention is shown. The deep learning-based non-rigid image registration method specifically includes steps S01 to S03.

[0021] Step S01, acquiring a first image of a user and a second image obtained by rigid body registration of the first image to generate a training set.

[0022] Specifically, the first image and the second image are images of the same patient acquired at different times, images of the same modality or images of different modalities, and the two images have been processed by rigid body registration, have the same size and resolution, and the body parts covered by the first image and the second image have no missing image information, that is, all points in the body area of ​​the first image of interest can find matching positions in the second image. In this embodiment, the first image refers to a reference image, and the second image refers to a moving image, that is, the reference image and the moving image of the same size and resolution are used as training sets for the subsequent non-rigid body registration model.

[0023] Step S02: establishing a non-rigid registration model, and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model.

[0024] See also Figure 2, is a schematic diagram of the network structure of the non-rigid registration model, wherein the deformation field vector is the matching position information of each pixel of the first image on the second image, and is a three-dimensional vector. Since the first image is a 3D image, the deformation field size is a matrix of D×H×W×3. It should be noted that the non-rigid registration model includes a first encoder, a second encoder and a decoder; the first encoder and the second encoder are both composed of a plurality of sliding window transformation modules, which are used to respectively calculate the feature vector sequences of different scales of the first image and the second image. Specifically, by combining the sliding window transformation modules, a sliding window transformation module group is obtained, and the sliding window transformation module groups in the first encoder and the second encoder are matched, wherein one end of the output of the sliding window transformation module group is connected to the next level sliding window transformation module group through a sub-block merging layer, and the other end is connected to the corresponding interactive attention module. When the sliding window transformation module group is the last level sliding window transformation module group, the output is only connected to the first cross attention module, and the first cross attention module is used to perform global attention processing on the feature vector sequence with the smallest size. In addition, the second cross-attention module is used to perform in-window attention processing on feature vector sequences of other sizes. The first cross-attention module and the second cross-attention module are connected via an upsampling module. It should be noted that both the first cross-attention module and the second cross-attention module are cross-transformation modules without multi-layer perceptrons and residual connections. Furthermore, the inputs of the first cross-attention module and the second cross-attention module are the feature vector sequence output by the sliding window transformation module group of the corresponding level of the first image, the feature vector sequence output by the sliding window transformation module group of the corresponding level of the second image, and the position labels corresponding to the feature vector sequence output by the sliding window transformation module group of the corresponding level of the second image.

[0025] It can be understood that the image encoders of the reference image and the moving image are independent of each other, and thus support image registration between different modalities, for example, the reference image is a CBCT image and the moving image is a CT image, or the reference image is an MR image and the moving image is a CT image.

[0026] The decoder is used to determine the position information of the voxels on the second image that match each position on the first image according to the feature vector sequence, wherein the feature vector sequence with the smallest size is processed with global attention, and the feature vector sequences of other sizes are processed with in-window attention. It should be noted that if the feature vector sequences of each level of the reference image and the moving image are processed with global attention, it will take a lot of computing time and occupy a lot of storage space. Therefore, only the feature vector sequence of the highest level (smallest size) is processed with global attention, that is, the matching position is searched in the global range. In other level processing, in-window attention is used to control the model calculation amount and occupied space. When the high-level matching position information is obtained, it is used as the center of the initial larger search window and the in-window attention is processed. In this way, the matching position information at each scale is restored step by step, and the matching position estimation is finally completed.

[0027] Specifically, after the first image and the second image are input into the non-rigid registration model, they are first subjected to linear embedding processing and rearrangement processing to convert the images into feature vector sequences. The purpose of the linear embedding processing and the rearrangement processing is to unify the image features of the N-channel D×H×W size into a sequence of D / P×H / P×W / P vectors of length C. Specifically, the purpose of the linear embedding processing is dimensionality reduction, and the purpose of the rearrangement processing is to represent the four-dimensional tensor of channel-depth-row-column as depth×row×column-channel, such that a vector sequence of depth×row×column vectors of length channel is formed.

[0028] In this embodiment, in the global attention processing of the decoding process, the reference image encoder provides q (1, Dr / 32, Hr / 32, Wr / 32, 128), the moving image encoder provides k (1, Dr / 32, Hr / 32, Wr / 32, 128), and the position number corresponding to the moving image feature vector sequence is used as v (1, Dr / 32, Hr / 32, Wr / 32, 3), and the position of the moving image feature vector sequence matching each reference image feature vector sequence is calculated. P4' (1, Dr / 32, Hr / 32, Wr / 32, 3), where Dr represents the number of voxels in the depth direction, Hr represents the number of voxels in the height direction, and Wr represents the number of voxels in the width direction; the position vector sequence P4' is rearranged into (1, 3, Dr / 32, Hr / 32, Wr×32) features, and then upsampled to position information P3 of (1, 3, Dr / 16, Hr / 16, Wr / 16); the position information P3 is re-vectorized to (1, Dr / 16, Hr / 16, Wr / 16, 3) provides the window center position information for the window attention processing of the previous level encoding feature, and then takes each reference image feature vector sequence as q (1, Dr / 6, Hr / 16, Wr / 16, 1, 64), takes the corresponding P3 position information as the center, and takes the moving image feature vector in the 5×5×5 window to form v (1, Dr / 6, Hr / 16, Wr / 16, 125, 64) in the moving image feature. The position label is also taken as v (1, Dr / 6, Hr / 16, Wr / 16, 125, 3) in the 5×5×5 window, and the attention processing in the window is P3' (1, Dr / 16, Hr / 16, Wr / 16, 3), and so on, and more precise matching position information is continuously obtained until the position information (1, 3, Dr / 4, Hr / 4, Wr×4) is obtained. Finally, the moving image matching position information (1, 3, Dr, Hr, Wr) is obtained directly after 4 times upsampling. According to the matching position information provided by the model, the mobile image can be resampled to obtain the result after non-rigid registration.

[0029] In order to further introduce the non-rigid registration model, the processing steps after the image is sent to the network are introduced. It should be noted that although Figure 2 The linear encoding 4×4×4 window size is given, and two independent encoders have four levels of feature output, which are processed by 2, 4, 4, and 4 sliding window transformation modules respectively. However, in actual applications, the linear encoding window size, the number of encoder levels, the number of stacked sliding window transformation modules in each level, and the size of the mutual attention processing window in the decoder can be designed according to the GPU memory size. The processing process specifically includes: 1. The input reference image size is (Dr, Hr, Wr), which is evenly divided into 4×4×4 sub-blocks. The image data in each sub-block is arranged in sequence as a vector of length 64, and a vector output of length 16 is obtained after being processed by the linear transformation layer. In this way, the input reference image can be represented as a (1, Dr / 4, Hr / 4, Wr / 4, 16) tensor R1, which is then superimposed with the learnable position code P0 tensor of the same size; 2. R1 is processed by two consecutive sliding window transformation modules with a window size of 5×5×5 to obtain the first-level reference feature vector output R1'; 3. R1' is reduced to a vector R2 of (1, Dr / 8, Hr / 8, Wr / 8, 32) after the sub-block merging process. This module first rearranges R1' into a tensor of (1, 16, Dr / 4, Hr / 4, Wr / 4), divides the sub-blocks into 2×2×2, and arranges the 8 16-length vectors in the sub-blocks into a vector of length 128. The vector is processed by the linear transformation layer to obtain a vector of length 32, and a total of (Dr / 8, Hr / 8, Wr / 8) vectors of length 32 are obtained, that is, a tensor R2 of size (1, Dr / 8, Hr / 8, Wr / 8, 32) is formed; 4. R2 is processed by four consecutive sliding window transformation modules with a window size of 5×5×5 to obtain the second-level reference feature vector output R2'; 5. R2' is reduced to a vector R3 of (1, Dr / 16, Hr / 16, Wr / 16, 64) after the sub-block merging process. This module first rearranges R2' into a tensor of (1, 32, Dr / 8, Hr / 8, Wr / 8), divides the sub-blocks into 2×2×2, and arranges the 8 32-length vectors in the sub-blocks into a vector of length 256. The vector is processed by the linear transformation layer to obtain a vector of length 64, and a vector R3 of (Dr / 16, Hr / 16, Wr / 16) length 64 is obtained in total; 6. R3 is processed by four consecutive sliding window transformation modules with a window size of 5×5×5 to obtain the third-level reference feature vector output R3'; 7. R3' is reduced to vector R4 of (1, Dr / 32, Hr / 32, Wr / 32, 128) after sub-block merging. 8. R4 is processed by four consecutive sliding window transformation modules with a window size of 5×5×5 to obtain a fourth-level reference feature vector output R4'; 9. The design of the reference image coding network is the same. After the moving image is processed by the moving image coding network, a first-level moving feature vector output M1', a second-level moving feature vector output M2', a third-level moving feature vector output M3', and a fourth-level moving feature vector output M4' can be obtained; 10. Then perform global interactive attention processing and gradually restore the matching position information. The fourth-level reference feature vector output R4' is used as the q input of global interactive attention, the fourth-level mobile feature vector output M4' is used as the k input of global interactive attention, and the fourth-level mobile feature vector output natural position label P4 is used as the v input of global interactive attention to obtain the position P4' of the mobile feature vector output M4' that matches the reference feature vector output R4'; the size of the natural position label sequence P4 is (1, Dr / 32, Hr / 32, Wr / 32, 3), and its label information of (0, 0, 0, 0, :) is (0, 0, 0), the label information of (0, 0, 0, 1, :) is (0, 0, 1), the label information of (0, 0, 1, 0, :) is (0, 1, 0), and the label information of (0, 1, 0, 0, :) is (1, 0, 0), and so on. P4' is the weighted sum of the position information in P4, with a size of (1, Dr / 32, Hr / 32, Wr / 32, 3); the weight is calculated by R4' and M4' through the attention mechanism.

[0030] 11. Then, P4' is rearranged into a vector of (1, 3, Dr / 32, Hr / 32, Wr / 32) and then upsampled by 2×2×2 to obtain a tensor of (1, 3, Dr / 16, Hr / 16, Wr / 16). The tensor is multiplied by 2 and rearranged into a position tensor P3' of (1, Dr / 16, Hr / 16, Wr / 16, 3). 12. According to the position in the third-level mobile feature vector output M3' provided in P3' that matches the third-level reference feature vector output R3' at position (0, 0, 0), take the integer processing result as the center of the window, take the features within the 5×5×5 window of M3', and arrange them in order as a (125, 64) vector. Each position refers to the above operation; a new tensor MW3 can be obtained, whose size is (1, Dr / 16, Hr / 16, Wr / 16, 125, 64); imitating the process of taking the vector within the window and rearranging it on M3, the label information within the window can be taken in the third-level natural position label sequence P3 and rearranged to obtain a new tensor PW3; 13. Rearrange the third-level reference feature vector output R3' to (1, Dr / 16, Hr / 16, Wr / 16, 1, 64) as the q input of the third-level window-interactive attention, use MW3 as the k input of the third-level window-interactive attention, and use PW3 as the v input of the third-level window-interactive attention to obtain the third-level matching position output P3', the size of P3' is (1, Dr / 16, Hr / 16, Wr / 16, 3); 14. Then, P3' is rearranged into a vector of (1, 3, Dr / 16, Hr / 16, Wr / 16) and then upsampled by 2×2×2 to obtain a tensor of (1, 3, Dr / 8, Hr / 8, Wr / 8). The tensor is multiplied by 2 and rearranged into a position tensor P2' of (1, Dr / 8, Hr / 8, Wr / 8, 3). 15. According to the position in the second-level moving feature vector output M2' provided in P2' that matches the second-level reference feature vector output R2' at the position (0, 0, 0), the rounded result is used as the center of the window, and the features within the 5×5×5 window of M2' are taken and arranged in order as a (125, 32) vector. Each position refers to the above operation; a new tensor MW2 can be obtained, and its size is (1, Dr / 8, Hr / 8, Wr / 8, 125, 32); in the same way as the process of taking the vector within the window and rearranging it on M2, the label information within the window can be taken in the second-level natural position label sequence P2 and rearranged to obtain a new tensor PW2; 16. Rearrange the second-level reference feature vector output R2' to (1, Dr / 8, Hr / 8, Wr / 8, 1, 32) as the q input of the second-level window-interactive attention, use MW2 as the k input of the second-level window-interactive attention, and use PW2 as the v input of the second-level window-interactive attention to obtain the second-level matching position output P2', the size of P2' is (1, Dr / 8, Hr / 8, Wr / 8, 3); 17. Then, P2' is rearranged into a vector of (1, 3, Dr / 8, Hr / 8, Wr / 8) and then upsampled by 2×2×2 to obtain a tensor of (1, 3, Dr / 4, Hr / 4, Wr / 4). The tensor is multiplied by 2 and rearranged into a position tensor P1' of (1, Dr / 4, Hr / 4, Wr / 4, 3). 18. According to the position in the first-level moving feature vector output M1' that matches the first-level reference feature vector output R1' at the position (0, 0, 0) provided in P1', the integer processing result is used as the center of the window, and the features in the 5×5×5 window of M1' are taken and arranged in order as a (125, 16) vector. Each position refers to the above operation; a new tensor MW1 can be obtained, whose size is (1, Dr / 4, Hr / 4, Wr / 4, 125, 16). Following the process of taking the vector in the window and rearranging it on M1, the label information in the window can be taken in the first-level natural position label sequence P1 and rearranged to obtain a new tensor PW1; 19. Rearrange the first-level reference feature vector output R1' to (1, Dr / 4, Hr / 4, Wr / 4, 1, 16) as the q input of the first-level window-interactive attention, use MW1 as the k input of the first-level window-interactive attention, and use PW1 as the v input of the first-level window-interactive attention to obtain the first-level matching position output P1', the size of P1' is (1, Dr / 4, Hr / 4, Wr / 4, 3); 20. Then, P1' is rearranged into a vector of (1, 3, Dr / 4, Hr / 4, Wr / 4) and then upsampled by 4×4×4 to obtain a tensor of (1, 3, Dr, Hr, Wr). The tensor is multiplied by 4 and rearranged into a matching position P result of (1, Dr, Hr, Wr, 3).

[0031] It should be noted that the position label sequences P1, P2, P3, and P4 do not need to be learned and can be directly calculated based on the size of the image features. The parameters in other processing modules need to be learned and the best parameters that can achieve non-rigid registration effects are obtained through optimization by the optimizer. After the model training is completed, the model parameters are fixed, and the registration position results can be obtained through the above processing.

[0032] In this embodiment, the parameters of the non-rigid registration model are randomly initialized and optimized under the guidance of the loss function, wherein the loss function is composed of image similarity measurement, labeling constraint and matching position continuity constraint, and the expression is:

[0033] Among them, Loss is expressed as the total loss, α, β, and γ are non-negative real numbers used to control the contribution of different losses to the overall loss. Expressed as image similarity measure loss, Denoted as the label constraint loss of the same organ in the first image and the second image, It is expressed as the matching position continuity constraint loss.

[0034] Specifically, mutual information I(A, B) is a measure of the correlation between two random variables, and can also be used to calculate image similarity. A is the reference image, and B is the moving image deformed by the deformation field. The mutual information calculation formula is:

[0035]

[0036] in, represents the entropy of the reference image, Represents the entropy of the moving image after deformation of the deformation field. They are respectively composed of the unilateral probability density of the image , Calculate and obtain, is the cross entropy, which is the joint probability density of the two Calculated, one-sided probability density It is calculated using the statistical histogram. It is obtained by joint statistical histogram calculation.

[0037] DSC (Dice similarity coefficient) is also called the Dice coefficient, which is used to measure the degree of overlap between two binary images A and B. Among them, A represents the binary image of a certain organ at risk on the reference image, where the organ at risk part is marked as 1 and the other parts are marked as 0; B represents the binary image of the same organ at risk marked on the moving image after deformation by the deformation field, where the organ at risk is marked as 1 and the other parts are marked as 0. The calculation formula of DSC is as follows:

[0038] The higher the overlap ratio between the annotations on the moving image and the annotations on the reference image, the greater the DSC and the better the registration effect. If it is used as loss, it needs to be written as

[0039] It is used to constrain the rationality of the matching position estimation and requires that the deformed moving image should not be folded.

[0040] Use the registration network to calculate the position of each pixel on the reference image The matching position at The Jacobian matrix is:

[0041] Expressed as the x-direction component of the three-dimensional deformation field, Expressed as the y-direction component of the three-dimensional deformation field, It is expressed as the z-direction component of the three-dimensional deformation field. If the deformation field is not folded, the value of the Jacobian determinant needs to be Greater than 0, that is

[0042] In order to ensure that the deformation field does not fold, it is necessary to examine the value of the Jacobian determinant at each pixel position. If the value is greater than zero, the determinant loss The minimum is zero; if the value of the determinant is less than or equal to zero, the loss The maximum value is 1, and the overall loss of the deformation field is the average of the determinant losses of all points.

[0043]

[0044] Step S03: inputting an image to be registered of the same type as the second image into the target model, and outputting a registration result.

[0045] In summary, the deep learning-based non-rigid image registration method in the above-mentioned embodiment of the present invention generates a training set by acquiring a first image of the user and a second image obtained by rigid registration of the first image; establishing a non-rigid registration model, and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model; inputting the image to be registered of the same type as the second image into the target model, and outputting the registration result. Specifically, the non-rigid registration model includes a first encoder, a second encoder and a decoder; the first encoder and the second encoder are both composed of a number of sliding window transformation modules, which are used to respectively calculate feature vector sequences of different scales of the first image and the second image; the decoder is used to determine the position information of voxels on the second image that match each position on the first image according to the feature vector sequence, wherein the feature vector sequence with the smallest size is subjected to global attention processing, and the feature vector sequences of other sizes are subjected to in-window attention processing, which can effectively solve the problem of poor registration effect caused by inconsistent coverage of the body range of the first image and the second image.

[0046] Embodiment 2 In order to verify the feasibility of the non-rigid image registration method based on deep learning in the first embodiment of the present invention, CBCT images and CT images of 186 patients were used as training sets, and CBCT images were used as reference images. After rigid registration of CT images, CBCT images and CT images were uniformly resampled to images of 160×320×320 size to form training samples. Since all samples have been unified in size, batchsize can be set to be greater than 1 during training, which can be determined by the size of the graphics card memory. Then, the following steps are followed: 1. The input reference image size is (160, 320, 320), which is evenly divided into 4×4×4 sub-blocks. The image data in each sub-block is arranged in sequence as a vector of length 64, and processed by the linear transformation layer to obtain a vector output of length 16. In this way, the input reference image can be represented as a (1, 40, 80, 80, 16) tensor R1, which is then superimposed with the learnable position code P0 tensor of the same size; 2. R1 is processed by two consecutive sliding window transformation modules with a window size of 5×5×5 to obtain the first-level reference feature vector output R1'; 3. R1' is reduced to a vector R2 of (1, 20, 40, 40, 32) after the sub-block merging process. This module first rearranges R1' into a tensor of (1, 16, 40, 80, 80), divides the sub-blocks into 2×2×2, and arranges the 8 16-length vectors in the sub-block into a vector of length 128. The vector is processed by the linear transformation layer to obtain a vector of length 32, and a total of (20, 40, 40) vectors of length 32 are obtained, that is, a tensor R2 of size (1, 20, 40, 40, 32) is obtained; 4. R2 is processed by four consecutive sliding window transformation modules with a window size of 5×5×5 to obtain the second-level reference feature vector output R2'; 5. R2' is reduced to a vector R3 of (1, 10, 20, 20, 64) after the sub-block merging process. This module first rearranges R2' into a (1, 32, 20, 40, 40) tensor, divides the sub-blocks into 2×2×2, and arranges the 8 32-length vectors in the sub-block into a vector of length 256. The vector is processed by the linear transformation layer to obtain a vector of length 64, and a total of (10, 20, 20) vectors of length 64 R3 are obtained; 6. R3 is processed by four consecutive sliding window transformation modules with a window size of 5×5×5 to obtain the third-level reference feature vector output R3'; 7. R3' is reduced to vector R4 of (1, 5, 10, 10, 128) after sub-block merging. 8. R4 is processed by four consecutive sliding window transformation modules with a window size of 5×5×5 to obtain a fourth-level reference feature vector output R4'; 9. The design of the reference image coding network is the same. After the moving image is processed by the moving image coding network, a first-level moving feature vector output M1', a second-level moving feature vector output M2', a third-level moving feature vector output M3', and a fourth-level moving feature vector output M4' can be obtained; 10. Then perform global interactive attention processing and gradually restore the matching position information. The fourth-level reference feature vector output R4' is used as the q input of global interactive attention, the fourth-level mobile feature vector output M4' is used as the k input of global interactive attention, and the fourth-level mobile feature vector output natural position label P4 is used as the v input of global interactive attention to obtain the position P4' of the mobile feature vector output M4' that matches the reference feature vector output R4'; the size of the natural position label sequence P4 is (1, Dr / 32, Hr / 32, Wr / 32, 3), and its label information of (0, 0, 0, 0, :) is (0, 0, 0), the label information of (0, 0, 0, 1, :) is (0, 0, 1), the label information of (0, 0, 1, 0, :) is (0, 1, 0), and the label information of (0, 1, 0, 0, :) is (1, 0, 0), and so on. P4' is the weighted sum of the position information in P4, with a size of (1, Dr / 32, Hr / 32, Wr / 32, 3); the weight is calculated by R4' and M4' through the attention mechanism.

[0047] 11. Then, P4' is rearranged into a vector of (1, 3, 5, 10, 10) and then upsampled by 2×2×2 to obtain a tensor of (1, 3, 10, 20, 20). The tensor is multiplied by 2 and rearranged into a position tensor P3' of (1, 10, 20, 20, 3). 12. According to the position in the third-level mobile feature vector output M3' provided in P3' that matches the third-level reference feature vector output R3' at position (0, 0, 0), take the integer processing result as the center of the window, take the features within the 5×5×5 window of M3', and arrange them in order as a (125, 64) vector. Each position refers to the above operation; a new tensor MW3 can be obtained, and its size is (1, 10, 20, 20, 125, 64); imitating the process of taking the vector within the window on M3 and rearranging it, the label information within the window can be taken in the third-level natural position label sequence P3 and rearranged to obtain a new tensor PW3; 13. Rearrange the third-level reference feature vector output R3' to (1, 10, 20, 20, 1, 64) as the q input of the third-level in-window interactive attention module, use MW3 as the k input of the third-level in-window interactive attention module, and use PW3 as the v input of the third-level in-window interactive attention module to obtain the third-level matching position output P3', and the size of P3' is (1, 10, 20, 20, 3); 14. Then, P3' is rearranged into a vector of (1, 3, 10, 20, 20) and then upsampled by 2×2×2 to obtain a tensor of (1, 3, 20, 40, 40). The tensor is multiplied by 2 and rearranged into a position tensor P2' of (1, 20, 40, 40, 3). 15. According to the position in the second-level moving feature vector output M2' provided in P2' that matches the second-level reference feature vector output R2' at the position (0, 0, 0), the rounded result is used as the center of the window, and the features within the 5×5×5 window of M2' are taken and arranged in order as a (125, 32) vector. Each position refers to the above operation; a new tensor MW2 can be obtained, and its size is (1, 20, 40, 40, 125, 32); imitating the process of taking the vector within the window and rearranging it on M2, the label information within the window can be taken in the second-level natural position label sequence P2 and rearranged to obtain a new tensor PW2; 16. Rearrange the second-level reference feature vector output R2' to (1, 20, 40, 40, 1, 32) as the q input of the second-level window-interactive attention, use MW2 as the k input of the second-level window-interactive attention, and use PW2 as the v input of the second-level window-interactive attention to obtain the second-level matching position output P2', the size of P2' is (1, 20, 40, 40, 3); 17. Then, P2' is rearranged into a vector of (1, 3, 20, 40, 40) and then upsampled by 2×2×2 to obtain a tensor of (1, 3, 40, 80, 80). The tensor is multiplied by 2 and rearranged into a position tensor P1' of (1, 40, 80, 80, 3). 18. According to the position in the first-level moving feature vector output M1' that matches the first-level reference feature vector output R1' at the position (0, 0, 0) provided in P1', the rounded result is used as the center of the window, and the features within the 5×5×5 window of M1' are taken and arranged in order as a (125, 16) vector; each position refers to the above operation; a new tensor MW1 can be obtained, and its size is (1, 40, 80, 80, 125, 16); imitating the process of taking the vector in the window and rearranging it on M1, the label information in the window can be taken in the first-level natural position label sequence P1 and rearranged to obtain a new tensor PW1; 19. Rearrange the first-level reference feature vector output R1' to (1, 40, 80, 80, 1, 16) as the q input of the first-level window-interactive attention, use MW1 as the k input of the first-level window-interactive attention, and use PW1 as the v input of the first-level window-interactive attention to obtain the first-level matching position output P1', the size of P1' is (1, 40, 80, 80, 3); 20. Then, P1' is rearranged into a vector of (1, 3, 40, 80, 80) and then upsampled by 4×4×4 to obtain a tensor of (1, 3, 160, 320, 320). The tensor is multiplied by 4 and rearranged into a matching position P result of (1, 160, 320, 320, 3).

[0048] 21. During the training process, Batchsize=2, the initial learning rate is set to 1e-4, and StepLR is used to adjust the learning rate every 5 epochs, with an adjustment ratio of 0.9. StepLR is a learning rate adjuster class in PyTorch, which can adjust the learning rate according to a given step after each epoch. The loss function uses a combination of mutual information similarity loss and continuity loss for training, with a similarity loss ratio of α=10 and a continuity loss ratio of γ=2. The training is performed and the epoch is set to 50.

[0049] 22. After the training is completed, the model parameters are determined, the test data and its annotation information are processed, and various indicators of the reference CBCT image and the registered CT image are analyzed, as well as the degree of overlap between the annotation information on the reference CBCT and the annotation information on the registered CT, to complete the evaluation of the registration results.

[0050] See also Figure 3 , which is a comparison result diagram of the annotation on the CBCT and the annotation on the CT after registration. Specifically, the snow-cyan on the left and right sides are the pelvis, the snow-cyan on the bottom is the rectum, the green in the middle is the sigmoid colon, and the pink and green annotations in the upper middle are the small intestine, wherein the pink is the annotation on the CBCT, and the green is the CT annotation information. According to the matching position information calculated in the embodiment of the present invention, the CT image and the annotation information on the CT are resampled and compared with the annotation information on the CBCT. It is found that: the overlap of the annotation information of the pelvis is very good, the overlap of the annotation information of the rectum is very good, but the overlap of the annotation information of the sigmoid colon and the small intestine is relatively low (the pink and green labels have low overlap).

[0051] Embodiment 3 See also Figure 4 , Figure 4 : is a structural block diagram of a deep learning-based image non-rigid registration system provided in Embodiment 3 of the present invention. The deep learning-based image non-rigid registration system 200 includes: an acquisition module 21, a training module 22 and an input module 23, wherein: An acquisition module 21 is used to acquire a first image of a user and a second image obtained by rigid body registration of the first image to generate a training set; The training module 22 is used to establish a non-rigid registration model, and input the first image and the second image into the non-rigid registration model for training to obtain a target model. After the first image and the second image are input into the non-rigid registration model, they are firstly subjected to linear embedding processing and rearrangement processing to convert the images into feature vector sequences. In addition, the parameters of the non-rigid registration model are randomly initialized and optimized under the guidance of a loss function, wherein the loss function is composed of an image similarity measure, a labeling constraint, and a matching position continuity constraint, and the expression is:

[0052] Among them, Loss is expressed as the total loss, α, β, and γ are non-negative real numbers used to control the contribution of different losses to the overall loss. Expressed as image similarity measure loss, Denoted as the label constraint loss of the same organ in the first image and the second image, It is expressed as the matching position continuity constraint loss; An input module 23, used for inputting an image to be registered of the same type as the second image into the target model, and outputting a registration result; The non-rigid registration model includes a first encoder, a second encoder and a decoder; The first encoder and the second encoder are both composed of a plurality of sliding window transformation modules, which are used to respectively calculate feature vector sequences of different scales of the first image and the second image, wherein a sliding window transformation module group is obtained by combining the sliding window transformation modules, and the sliding window transformation module groups in the first encoder and the second encoder are matched, wherein one end of the output of the sliding window transformation module group is connected to the sliding window transformation module group of the next level through a sub-block merging layer, and the other end is connected to the corresponding interactive attention module, when the sliding window transformation module group is the last level sliding window transformation module group, the output is only connected to the first cross attention module, the first cross attention module is used to perform global attention processing on the feature vector sequence with the smallest size, and the second cross attention module is used to perform in-window attention processing on the feature vector sequences of other sizes, the first cross attention module and the second cross attention module are connected through an upsampling module, and in addition, the inputs of the first cross attention module and the second cross attention module are the feature vector sequence output by the sliding window transformation module group of the corresponding level of the first image, the feature vector sequence output by the sliding window transformation module group of the corresponding level of the second image, and the position label corresponding to the feature vector sequence output by the sliding window transformation module group of the corresponding level of the second image; The decoder is used to determine the position information of voxels on the second image that match each position on the first image based on the feature vector sequence, wherein the feature vector sequence with the smallest size is subjected to global attention processing, and the feature vector sequences of other sizes are subjected to in-window attention processing.

[0053] Embodiment 4 Another aspect of the present invention provides an electronic device, see Figure 5 , shown is an electronic device in Embodiment 4 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor, wherein when the processor 10 executes the computer program 30, the above-mentioned deep learning-based image non-rigid registration method is implemented.

[0054] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0055] The memory 20 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of an electronic device, such as a hard disk of the electronic device. In other embodiments, the memory 20 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Further, the memory 20 may also include both an internal storage unit and an external storage device of the electronic device. The memory 20 may be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or is to be output.

[0056] It should be pointed out that Figure 5 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than those shown in the figure, or combine certain components, or arrange the components differently.

[0057] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned deep learning-based non-rigid image registration method.

[0058] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0059] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0060] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0061] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0062] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. A non-rigid image registration method based on deep learning, characterized in that: The method comprises: Acquire a first image of the user and a second image obtained by rigid body registration of the first image to generate a training set; Establishing a non-rigid registration model, and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model; Inputting an image to be registered of the same type as the second image into the target model, and outputting a registration result; The non-rigid registration model includes a decoder, and a first encoder and a second encoder that are independent of each other; The first encoder and the second encoder are both composed of a plurality of sliding window transformation modules, and are used to respectively calculate feature vector sequences of different scales of the first image and the second image; The decoder is used to determine the position information of voxels on the second image that match each position on the first image based on the feature vector sequence, wherein the feature vector sequence with the smallest size is subjected to global attention processing, and the feature vector sequences of other sizes are subjected to in-window attention processing.

2. The deep learning-based non-rigid image registration method according to claim 1, characterized in that: After being input into the non-rigid registration model, the first image and the second image are firstly subjected to linear embedding processing and rearrangement processing to convert the images into feature vector sequences.

3. The deep learning-based non-rigid image registration method according to claim 2, characterized in that: A sliding window transformation module group is obtained by combining sliding window transformation modules, and the sliding window transformation module groups in the first encoder and the second encoder are matched, wherein one end of the output of the sliding window transformation module group is connected to the next level sliding window transformation module group through a sub-block merging layer, and the other end is connected to the corresponding interactive attention module. When the sliding window transformation module group is the last level sliding window transformation module group, the output is only connected to the first cross attention module, and the first cross attention module is used to perform global attention processing on the feature vector sequence with the smallest size.

4. The method for non-rigid image registration based on deep learning according to claim 3, characterized in that: The second cross-attention module is used to perform in-window attention processing on feature vector sequences of other sizes, and the first cross-attention module and the second cross-attention module are connected through an upsampling module.

5. The method for non-rigid image registration based on deep learning according to claim 4, characterized in that: The inputs of the first cross-attention module and the second cross-attention module are the feature vector sequence output by the sliding window transformation module group of the corresponding level of the first image, the feature vector sequence output by the sliding window transformation module group of the corresponding level of the second image, and the position labels corresponding to the feature vector sequence output by the sliding window transformation module group of the corresponding level of the second image.

6. The method for non-rigid image registration based on deep learning according to claim 5, characterized in that: In the step of establishing a non-rigid registration model and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model, the parameters of the non-rigid registration model are randomly initialized and optimized under the guidance of a loss function, wherein the loss function is composed of an image similarity measure, a labeling constraint, and a matching position continuity constraint, and the expression is: Among them, Loss is expressed as the total loss, α, β, and γ are non-negative real numbers used to control the contribution of different losses to the overall loss. Expressed as image similarity measure loss, Denoted as the label constraint loss of the same organ in the first image and the second image, It is expressed as the matching position continuity constraint loss.

7. A non-rigid image registration system based on deep learning, characterized in that: For implementing the deep learning-based non-rigid image registration method according to any one of claims 1 to 6, the system comprises: An acquisition module, used to acquire a first image of a user and a second image obtained by rigid body registration of the first image, and generate a training set; A training module, used for establishing a non-rigid registration model, and inputting the first image and the second image into the non-rigid registration model for training to obtain a target model; An input module, used for inputting an image to be registered of the same type as the second image into the target model, and outputting a registration result; The non-rigid registration model includes a decoder, and a first encoder and a second encoder that are independent of each other; The first encoder and the second encoder are both composed of a plurality of sliding window transformation modules, and are used to respectively calculate feature vector sequences of different scales of the first image and the second image; The decoder is used to determine the position information of voxels on the second image that match each position on the first image based on the feature vector sequence, wherein the feature vector sequence with the smallest size is subjected to global attention processing, and the feature vector sequences of other sizes are subjected to in-window attention processing.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the deep learning-based non-rigid image registration method as described in any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the deep learning-based non-rigid image registration method as described in any one of claims 1 to 6 is implemented.

Images