Double-stage registration method and device suitable for full-length image of lower limb, terminal and medium

Abstract

The invention provides a two-stage registration method and device suitable for a full-length image of a lower limb, a terminal and a medium. The method comprises the steps of obtaining a to-be-registered CT image and a label of the lower limb, and a target X-ray image and a label of the lower limb; inputting the to-be-registered CT image and the label, and the target X-ray film image and the label into an affine transformation network to generate an affine transformation matrix for performing rough registration on the to-be-registered CT image; and inputting the CT image and the label output in the coarse registration stage, and the target X-ray film image and the label into a deformation field registration network to generate a deformation field for fine registration based on the coarse registration result of the CT image to be registered. According to the method, the deep learning technology is utilized, a brand-new 2D / 3D medical image registration algorithm is constructed, and full-automatic cross-modal registration of lower limb full-length horizontal position CT and standing position X-ray films is achieved for the first time. The research result of the invention can provide effective assistance for preoperative accurate planning, assist the design work of an HTO operation guide plate, and improve the clinical curative effect of early and middle knee OA.

Description

Technical Field

[0001] The present application relates to the technical field of image registration, and in particular to a dual-stage registration method, device, terminal and medium suitable for full-length images of lower limbs. Background Art

[0002] Osteoarthritis (OA) is one of the top ten disabling diseases in my country. It is prone to occur in weight-bearing joints, with the knee joint being the most common. The prevalence of knee arthritis in people over 40 years old in my country is as high as 15.6%, which not only affects the quality of life of patients, but also creates a heavy socioeconomic burden. At present, there is no specific treatment for knee arthritis: non-surgical treatments such as drugs and physical therapy can only relieve joint pain to a certain extent, but they cannot stop or reverse the development of knee arthritis, which eventually leads to patients undergoing knee replacement surgery; artificial joints are expensive and have a limited lifespan, and they still cannot completely replace and restore knee joint function. "Worry-free knees" after knee replacement surgery are still a luxury. Therefore, it is urgent to explore new treatment methods, intervene in the progression of early and middle-stage knee arthritis, and delay or even avoid artificial knee replacement.

[0003] Changes in joint mechanical loads are the main cause of the occurrence and development of knee arthritis. In healthy people, the tibial plateau articular surface is 3° inward relative to the force line of the lower limb, resulting in a much higher load on the medial compartment of the knee joint than on the lateral compartment, and the incidence of arthritis is also significantly increased. On the other hand, joint wear aggravates the deformity of the force line of the knee joint, resulting in increased mechanical loads and further accelerating the progression of knee arthritis. Therefore, high tibial osteotomy can move the force line of the lower limb from the degenerated medial compartment to the normal lateral compartment, reduce the mechanical load of the degenerated compartment, and is expected to break the vicious cycle of "force line deformity-joint degeneration-increased load" in patients with knee arthritis, which is a feasible means of treating early and middle-stage knee arthritis. Previous research results have shown that changing the force line of the lower limb by osteotomy can promote the repair and regeneration of cartilage in the degenerated compartment, and stop or even reverse the development of knee arthritis. However, the efficacy of high tibial osteotomy still has shortcomings such as large individual differences, low satisfaction, and high incidence of complications. Among the many factors that affect the efficacy, how to achieve accurate control of the force line of the lower limb is a key issue to be solved in high tibial osteotomy. Once the intraoperative force line is insufficiently corrected, it is difficult to effectively reduce the mechanical load of the degenerative compartment and interfere with the development of knee arthritis; if the intraoperative force line is overcorrected, the normal compartment will be overloaded, leading to degeneration of the corresponding compartment joint.

[0004] At present, there are many defects in the surgical planning and intraoperative force line judgment of high tibial osteotomy, which seriously limit the surgeon's precise control of the lower limb force line: 1) The two-dimensional surgical planning based on X-rays in the standing position is difficult to match the three-dimensional surgical scene in the supine position during the operation; 2) Traditionally, the surgeon mainly judges the lower limb force line through the naked eye, feel and personal experience during the operation, and the evaluation method is extremely subjective and the accuracy is seriously insufficient; 3) Computer navigation technology can assist the surgeon in judging the lower limb force line and improve the surgical accuracy, but the positioning of anatomical landmarks and their registration with medical images still rely on the surgeon's subjective experience. In addition, its high cost, complex operation process and additional registration time also limit the widespread application of computer surgical navigation technology. Therefore, for this type of surgery, new preoperative planning methods and intraoperative force line judgment technologies should be developed to achieve precise control of the lower limb force line after surgery.

[0005] Deep learning is a machine learning method based on artificial neural networks, which is used to automatically learn and construct data representations. It uses multi-layer neural networks to discover patterns and regularities in data and generate models that contain these patterns and regularities. The core of deep learning is neural networks. In neural networks, inputs are processed by multiple layers (each layer contains multiple neurons) and finally generate outputs. Neural networks establish a mapping relationship from input to output by analyzing the pattern between input and output. Balakrishnan et al. proposed VoxelMorph and constructed a learning-based medical image registration framework. It uses convolutional neural networks to parameterize deformation field functions. The model is trained by minimizing the loss function so that it can align one medical image with another. The model can handle medical images of different types and anatomical structures, and can perform multimodal registration. The main advantages of VoxelMorph are fast speed, can be trained on large data sets, and can handle complex deformation field models. Despite this, existing technologies still have difficulty in solving the optimization problem of registering 2D / 3D anatomical annotations, and cannot make discrete annotation information continuous during training, which affects model performance. Summary of the invention

[0006] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a two-stage registration method, device, terminal and medium suitable for full-length images of the lower limbs, which is used to solve the technical problems that the prior art is still difficult to solve in optimizing the registration of 2D / 3D anatomical annotations, and cannot make the discrete annotation information continuous during the training process, thus affecting the model performance.

[0007] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a two-stage registration method suitable for full-length images of lower limbs, including: obtaining a CT image to be registered and a label, a target X-ray image and a label of the lower limb; inputting the CT image to be registered and the label, the target X-ray image and the label into an affine transformation network to generate an affine transformation matrix for roughly registering the CT image to be registered; inputting the CT image and the label, the target X-ray image and the label output in the coarse registration stage into a deformation field registration network to generate a deformation field for fine registration based on the coarse registration result of the CT image to be registered.

[0008] In some embodiments of the first aspect of the present application, the affine transformation network includes: a first encoder based on a 2D Swin-Transformer module, a second encoder based on a 3D Swin-Transformer module and a corresponding decoder; the first encoder is used to encode the target X-ray image and label; the second encoder is used to encode the CT image and label to be registered.

[0009] In some embodiments of the first aspect of the present application, the first encoder based on the 2D Swin-Transformer module includes: a first encoding layer and four 2D Swin-Transformer layers; wherein the first encoding layer is used to divide the input target X-ray image into non-overlapping blocks and encode them into vectors of specified lengths respectively; the four 2D Swin-Transformer layers are used to generate stepped feature maps of different scales.

[0010] In some embodiments of the first aspect of the present application, the second encoder based on the 3D Swin-Transformer module includes: a second encoding layer and four 3D Swin-Transformer layers; wherein the second encoding layer is used to divide the input CT image to be registered into non-overlapping blocks and encode them into vectors of specified length respectively; the four 3D Swin-Transformer layers are used to generate stepped feature maps of different scales.

[0011] In some embodiments of the first aspect of the present application, a feature fusion module based on a mutual attention mechanism is set between the corresponding network layers of the first encoder and the second encoder; the fused feature map is directly transmitted to the decoder in the affine transformation network, and is transmitted to the decoder in the deformation field registration network in the form of a jump connection.

[0012] In some embodiments of the first aspect of the present application, the feature fusion module based on the mutual attention mechanism is set between the deepest encoder layers of the first encoder and the second encoder.

[0013] In some embodiments of the first aspect of the present application, the decoder in the affine transformation network includes a multi-layer perceptron; the decoder in the deformation field registration network is composed of multiple 3D convolution modules.

[0014] In some embodiments of the first aspect of the present application, the structure of the deformation field registration network includes:

[0015] A feature fusion module based on the mutual attention mechanism is set between the deepest network layers corresponding to the first encoder and the second encoder, and the fused feature map is input into the first 3D convolution module after being processed by the second upsampling module; a feature fusion module based on the mutual attention mechanism is also set between the corresponding second deepest network layers, and the fused feature map is input into the first 3D convolution module for convolution processing; the convolution result output by the first 3D convolution module is input into the second 3D convolution module after passing through the second upsampling module; the convolution result of the second 3D convolution module and the third upsampling module are input into the third 3D convolution module; the convolution result of the third 3D convolution module is input into the fourth 3D convolution module after passing through the fourth upsampling module;

[0016] The target X-ray film image and label are feature-fused (Concat) with the CT image and label to be registered and are respectively input into the downsampling module and the sixth 3D convolution module for processing; the convolution result of the sixth 3D convolution module is input into the fifth 3D convolution module; the output result of the downsampling module is input into the seventh 3D convolution module, and the convolution result of the seventh 3D convolution module is input into the fourth 3D convolution module;

[0017] The fourth 3D convolution module performs convolution processing after receiving the output result of the fourth upsampling module and the output result of the seventh 3D convolution module, and the processing result is transmitted to the fifth 3D convolution module after upsampling processing by the fifth upsampling module; the fifth 3D convolution module also receives the convolution result from the sixth 3D convolution module; the fifth 3D convolution module performs convolution processing on the processing results received from the fifth upsampling module and the sixth 3D convolution module, and the processing result is output as a deformation field through the output module.

[0018] In some embodiments of the first aspect of the present application, the loss function used in the affine transformation network includes:

[0019] L affine =λ sim ·L sim +λ seg ·L seg ;

[0020] L sim =1-LNCC;

[0021]

[0022] L seg =1-Dice(Sigmoid(M reg ),M gt );

[0023]

[0024] Among them, I f represents the target X-ray image, I m represents the CT image to be registered, p represents the pixel coordinates in the image, Ω represents the set of all pixels, N(p) represents the domain of pixel p, q belongs to the neighborhood of pixel p, represents the pixel mean in the neighborhood of pixel p of the CT image to be registered, represents the mean value of pixels in the neighborhood of pixel p of the target X-ray image; M ori 、M reg 、M gt Represent the original segmentation label, the segmentation label after registration, and the true value of the segmentation label respectively; Represents the registration parameters Perform linear interpolation registration on image I; Sum sag represents the sum along the sagittal axis; γ and ε represent constants; Dice represents the loss function; λ sim , seg represents the weight, and the sum of the two is 1.

[0025] In some embodiments of the first aspect of the present application, the loss function used in the deformation field registration network includes:

[0026] L deform =λ sim ·L sim +λ seg ·L seg +λ reg ·L reg ;

[0027] L sim =1-LNCC;

[0028]

[0029] L seg =1-Dice(Sigmoid(M reg ),M gt );

[0030]

[0031]

[0032] Among them, If represents the target X-ray image, I m represents the CT image to be registered, p represents the pixel coordinates in the image, Ω represents the set of all pixels, N(p) represents the domain of pixel p, q belongs to the neighborhood of pixel p, represents the pixel mean in the neighborhood of pixel p of the CT image to be registered, represents the mean value of pixels in the neighborhood of pixel p of the target X-ray image; M ori 、M reg 、M gt Represent the original segmentation label, the segmentation label after registration, and the true value of the segmentation label respectively; Represents the registration parameters Perform linear interpolation registration on image I; Sum sag represents the sum along the sagittal axis; γ and ε represent constants; Dice represents the Dice coefficient; λ sim , seg , reg represents the weight, the sum of the three is 1; φ D represents the registration parameters, Represents the gradient of the registration parameter with respect to the pixel.

[0033] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a two-stage registration device suitable for full-length images of lower limbs, including: an image acquisition module, used to acquire the CT image to be registered and the label, the target X-ray image and the label of the lower limb; a coarse registration module, used to input the CT image to be registered and the label, the target X-ray image and the label into an affine transformation network to generate an affine transformation matrix for coarse registration of the CT image to be registered; a fine registration module, used to input the CT image and the label, the target X-ray image and the label output by the coarse registration module into a deformation field registration network to generate a deformation field for fine registration based on the coarse registration result of the CT image to be registered.

[0034] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the method when executed by a processor.

[0035] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides an electronic terminal, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory so that the terminal executes the method.

[0036] As described above, the present application has the following beneficial effects: The present invention aims to combine artificial intelligence technologies such as deep learning to construct a new 2D / 3D medical image registration technology, transform the supine CT image into a spatial position under the guidance of the lower limb standing X-ray image, and assist in the design of HTO surgical guides. The main contributions of the present invention are as follows:

[0037] (1) The present invention constructs a two-stage neural network for end-to-end 2D / 3D registration of full-length lower limb films, simultaneously processing scanned images and their anatomical structure segmentation labels, from coarse registration of affine transformation to fine registration of deformation field, completing the alignment task of scanned images in different postures step by step. The present invention creatively realizes fully automatic real-time registration based on lower limb CT and X-ray images and segmentation labels. The model has good generalization and can realize rapid image registration in clinical practice without the need for instance-by-instance calculation of traditional algorithms and no waiting time.

[0038] (2) The present invention uses the Swin-Transformer architecture with powerful representation capabilities to process the data of the two modalities separately, extract feature maps of different scales, obtain effective spatial transformation parameters, and achieve accurate image registration. Between the corresponding levels of encoders processing different modalities, the model introduces a mutual attention mechanism to mine the feature correlation information between different modalities, realize the fusion of different modal information, and further improve the registration effect.

[0039] (3) The present invention proposes the SigmoidDice loss function, which makes the segmentation label registration process in the training process continuous and differentiable, so that the anatomical structure information can effectively guide the model training optimization and effectively solve the problem of inconsistent distribution of cross-modal registration data.

[0040] In summary, this invention uses deep learning technology to construct a new 2D / 3D medical image registration algorithm, and for the first time realizes the fully automatic cross-modality registration of lower limb full-length supine CT and standing X-ray films. The research results of this invention will provide effective assistance for accurate preoperative planning, assist in the design of HTO surgical guides, and improve the clinical efficacy of early and mid-stage knee OA. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Shown is a flowchart of a dual-stage registration method suitable for full-length images of lower limbs in one embodiment of the present application.

[0042] Figure 2 Shown is an affine transformation network framework diagram of the first stage in a two-stage registration method in one embodiment of the present application.

[0043] Figure 3 Shown is a schematic diagram of the structure of the Swin-Transformer layer using a 3D image as input in one embodiment of the present application.

[0044] Figure 4 Shown is a schematic diagram of the mutual attention mechanism in one embodiment of the present application.

[0045] Figure 5 Shown is a structural schematic diagram of a deformation field registration network in one embodiment of the present application.

[0046] Figure 6 Shown is a flowchart of a dual-stage registration method suitable for full-length images of lower limbs in one embodiment of the present application.

[0047] Figure 7 Shown is a schematic structural diagram of a dual-stage registration device suitable for full-length images of lower limbs in one embodiment of the present application.

[0048] Figure 8 Shown is a schematic diagram of the structure of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0050] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0051] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0052] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0053] In order to solve the problems in the above-mentioned background technology, the present invention provides a two-stage registration method, system, terminal and medium suitable for full-length films of lower limbs, aiming to construct an end-to-end two-stage network model, and realizes the fully automatic cross-modal registration of supine CT and standing X-ray films of full-length films of lower limbs. In the first stage, the model generates an affine matrix to perform rigid coarse registration of the image, and in the second stage, the deformation field is used to make further fine adjustments to the image. The model uses a sliding window self-attention mechanism with powerful representation capabilities to process images of different modalities (such as CT and X-ray films) respectively, and uses a cross-attention mechanism to realize cross-modal information interaction.

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution in the embodiments of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0055] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0056] <1> Full-length X-ray of lower limbs: used to examine lesions of the hip joint, femur, knee joint, etc. Full-length weight-bearing X-ray of both lower limbs is an AP X-ray of both lower limbs taken when the patient is standing.

[0057] <2> Affine transformation, also known as affine mapping, refers to a linear transformation of a vector space followed by a translation in geometry to transform it into another vector space.

[0058] The embodiment of the present invention provides a dual-stage registration method for full-length film images of lower limbs, a system for the dual-stage registration method for full-length film images of lower limbs, and a storage medium storing an executable program for implementing the dual-stage registration method for full-length film images of lower limbs. As for the implementation of the dual-stage registration method for full-length film images of lower limbs, the embodiment of the present invention will describe an exemplary implementation scenario of the dual-stage registration for full-length film images of lower limbs.

[0059] Before performing HTO surgery, doctors generally obtain the anatomical structure of the patient's lower limbs through supine CT scans and standing X-rays. In order to better simulate the posture of the patient's lower limb bones when the patient is in a standing weight-bearing state, we use the standing X-ray as the target image and align the supine CT image to the standing CT image so that the latter's projection image along the sagittal axis is consistent with the standing X-ray in anatomical structure. Since the patient takes CT and X-rays at different times and has different body postures, the image obtained by directly projecting CT along the sagittal axis has obvious position and posture differences from the X-ray.

[0060] Therefore, the present invention performs coordinate transformation on the image in the three-dimensional space of CT, and then projects the transformed image into the two-dimensional space, and uses the X-ray film as the target to optimize the registration process in the three-dimensional space. The algorithm model designed by the present invention is divided into two stages. First, a rigid affine transformation with a small degree of freedom is performed on the original image to complete a rough registration, and then a fine registration based on the deformation field is completed based on the affine transformation result. Experiments have shown that this two-stage strategy can achieve better results. The network inputs the scanned image and the segmentation map at the same time, generates an affine transformation matrix in the rough registration stage, and generates a deformation field in the fine registration stage.

[0061] like Figure 1 FIG. 2 is a flow chart showing a two-stage registration method for full-length lower limb images in an embodiment of the present invention. The two-stage registration method for full-length lower limb images in this embodiment mainly includes the following steps:

[0062] Step S11: Obtain the CT image to be registered and its label, and the target X-ray image and its label of the lower limb.

[0063] In the embodiment of the present invention, the CT image to be registered and the target X-ray image are two different modal inputs, the CT image to be registered is a 3D image, and the target X-ray image is a 2D image. Therefore, for the two different modal inputs of CT image and X-ray image, the embodiment of the present invention uses two encoders based on the Swin-Transformer network to extract effective features of different scales for registration.

[0064] In order to better fuse the information of the two modalities, the embodiment of the present invention constructs an end-to-end two-stage neural network for 2D / 3D registration of full-length lower limb films, and introduces a cross attention mechanism between the two deepest layers of the two encoders. In the first stage, a multi-layer perceptron (Multiple Layer Perception, MLP) is used as a decoder to generate an affine transformation matrix; in the second stage, the network refers to the U-Net architecture, uses convolutional layers to form a decoder, and generates a deformation field that is exactly the same size as the input. Using the affine matrix or deformation field output by the network, the model can perform spatial transformation on the CT image to complete the registration. The specific content is described in detail below.

[0065] Step S12: inputting the CT image to be registered and its label, the target X-ray film image and its label into an affine transformation network to generate an affine transformation matrix for roughly registering the CT image to be registered.

[0066] It should be understood that affine transformation, also known as affine mapping, refers to a linear transformation of a vector space followed by a translation in geometry to transform it into another vector space. In an embodiment of the present invention, affine transformation can be used to implement operations such as image rotation, scaling, and translation. For example, an inclined image can be converted into a normal image or the image can be scaled by affine transformation. Affine transformation can also be used to implement operations such as image registration and distortion correction. For example, two images at different angles can be registered or a distorted image can be corrected by affine transformation.

[0067] Furthermore, the affine transformation operations in the embodiments of the present invention include, but are not limited to, any one or more combinations of translation transformation operations, rotation transformation operations, scaling transformation operations, and shear transformation operations. Translation transformation operations refer to transformations that move an image in a certain direction; rotation transformation operations refer to transformations that rotate an image around a certain point, which can be represented by an angle; scaling transformations refer to transformations that scale an image in a certain direction, which can be represented by a two-dimensional vector; and shear transformations refer to transformations that shear an image in a certain direction. Affine transformations can be implemented through matrix operations, specifically, the affine transformation is represented as a 3×3 matrix, and then each point in the image is represented as a 3×1 vector, and the transformation is implemented through matrix multiplication.

[0068] Figure 2 The framework diagram of the affine transformation network of the first stage in the two-stage registration method is shown. The affine transformation network includes a first encoder based on a 2D Swin-Transformer module, a second encoder based on a 3D Swin-Transformer module, and a corresponding decoder; the first encoder based on the 2D Swin-Transformer module is used to encode the target X-ray image and label; the second encoder based on the 3D Swin-Transformer module is used to encode the CT image and label to be registered.

[0069] The first encoder based on the 2D Swin-Transformer module includes: a first encoding layer (Embedding I) and four 2D Swin-Transformer layers. The first encoding layer is used to divide the input target X-ray image into non-overlapping blocks and encode them into vectors of specified lengths respectively. The four 2D Swin-Transformer layers are used to generate stepped feature maps of different scales for subsequent registration process.

[0070] The second encoder based on the 3D Swin-Transformer module includes: a second encoding layer (EmbeddingII) and four 3D Swin-Transformer layers. The second encoding layer is used to divide the input CT image to be registered into non-overlapping blocks and encode them into vectors of specified lengths respectively. The four 3D Swin-Transformer layers are used to generate stepped feature maps of different scales for subsequent registration process.

[0071] Furthermore, each Swin-Transformer layer consists of a Swin-Transformer module, a sliding window module, and a patch fusion module. The role of the patch fusion module is to perform downsampling before each stage to reduce the resolution, adjust the number of channels, and form a hierarchical design, while also saving a certain amount of computation.

[0072] by Figure 3 Assume that the size of the feature map is H×W×D and a window of size M×M×M is used, the feature map will be divided into windows. In the first conventional module, the feature map directly completes the self-attention mechanism operation in each window. In the sliding window module, the previously divided feature map window is first translated as a whole by (M / 2, M / 2, M / 2) voxels, and then the self-attention mechanism operation is completed in the new window. After each Swin-Transformer layer completes two self-attention mechanism operations, the feature map is halved in scale and doubled in number of channels through the block fusion layer. The block fusion layer rearranges adjacent pixels (voxels) to different channels to halve the feature scale. The feature map is then normalized by Layer Norm, and then the number of channels is reduced by a linear layer to half the previous number of channels.

[0073] Swin-Transformer extracts information focusing on local features by dividing the feature map into several windows. Through the sliding window mechanism, the network can focus on the interactive information between different windows. The design of the block fusion layer allows the network to achieve downsampling without increasing the amount of computation. The step-by-step network structure design allows the network to converge global information at different scales. These feature maps at different levels are the key to Swin-Transformer's excellent performance in a variety of different tasks.

[0074] More preferably, a feature fusion module based on the mutual attention mechanism is arranged between the two deepest network layers corresponding to the first encoder and the second encoder, so as to obtain a fused feature map based on the mutual attention mechanism; the fused feature map is used as the input of the multi-layer perceptron in the affine transformation network, and the fused feature map is passed to the decoder in the form of a jump connection in the deformation field registration network.

[0075] It is worth noting that, unlike the self-attention mechanism, the query tensor (Q), key tensor (K), and value tensor (V) used in the mutual attention mechanism come from different feature maps. The resulting attention map represents the correlation between the vectors of different feature maps, which can be used for information fusion between different feature maps, such as Figure 4 As shown in Figure 2. The fused feature map is used as the input of the multi-layer perceptron in the affine transformation network, and the features are passed to the decoder in the deformation field registration network in the form of a skip connection. It should be noted that a skip connection refers to skipping certain layers in the neural network and using the output of one layer as the input of the next layer.

[0076] In a more preferred embodiment, considering the limited computing resources, the feature fusion module based on the mutual attention mechanism is only set between the deepest encoder layers, so that the network can extract cross-modal features more efficiently.

[0077] Step S13: Input the CT image and label outputted in the coarse registration stage, and the target X-ray image and label into the deformation field registration network to generate a deformation field for fine registration based on the coarse registration result of the CT image to be registered.

[0078] The structure of the deformation field registration network is as follows: Figure 5 As shown:

[0079] First, a feature fusion module based on the mutual attention mechanism is set between the deepest network layers corresponding to the first encoder and the second encoder, and the fused feature map is input into the first 3D convolution module after being processed by the second upsampling module; a feature fusion module based on the mutual attention mechanism is also set between the corresponding second deepest network layers, and the fused feature map is input into the first 3D convolution module for convolution processing; the convolution result output by the first 3D convolution module is input into the second 3D convolution module after passing through the second upsampling module; the convolution result of the second 3D convolution module and the third upsampling module are input into the third 3D convolution module; the convolution result of the third 3D convolution module is input into the fourth 3D convolution module after passing through the fourth upsampling module.

[0080] Secondly, the target X-ray image and label are feature fused (Concat) with the CT image and label to be registered, and then input into the downsampling module and the sixth 3D convolution module for processing respectively; the convolution result of the sixth 3D convolution module is input into the fifth 3D convolution module; the output result of the downsampling module is input into the seventh 3D convolution module, and the convolution result of the seventh 3D convolution module is input into the fourth 3D convolution module.

[0081] Finally, the fourth 3D convolution module performs convolution processing after receiving the output result of the fourth upsampling module and the output result of the seventh 3D convolution module, and the processing result is transmitted to the fifth 3D convolution module after upsampling processing by the fifth upsampling module; the fifth 3D convolution module also receives the convolution result from the sixth 3D convolution module; the fifth 3D convolution module performs convolution processing on the processing results received from the fifth upsampling module and the sixth 3D convolution module, and the processing result is output as a deformation field through the output module.

[0082] For the convenience of understanding by those skilled in the art, Figure 6 The process of the two-stage registration method suitable for full-length films of lower limbs in the embodiment of the present invention is further explained as follows: First, the CT image to be registered and its label, and the target X-ray film image and its label are input into the affine transformation network to generate the corresponding affine transformation matrix to roughly align the CT image to be registered, and obtain the roughly registered CT image and label. Secondly, the roughly registered CT image and its label, and the target X-ray film image and its label are input into the deformation field registration network for fine registration, and obtain the precisely registered CT image and label. It should be noted that the specific process of the two-stage registration method suitable for full-length films of lower limbs in this example has been described above and will not be repeated here.

[0083] Preferably, the network input designed by the present invention includes an image and a corresponding segmentation label, which correspond to a local normalized correlation coefficient loss function based on image similarity and a loss function SigmoidDice based on anatomical structure coincidence. In addition, for the deformation field registration network in the second stage, the L2 norm of the deformation field gradient is used as a regularization term to constrain the deformation degree of the deformation field.

[0084] Furthermore, an indicator that measures the appearance similarity between the image to be registered and the target image is selected as the loss function. The local normalized correlation coefficient (LNCC) measures the local correlation between the two images and is more stable and less susceptible to interference with the grayscale distribution differences of different images. Its calculation formula is as follows (1):

[0085]

[0086] Where If, ​​Im represent the target image and the image to be registered, respectively, p is the pixel coordinate in the image, Ω represents the set of all pixels, N(p) represents the domain of pixel p, usually a 9×9 domain around the pixel, and I-(p) represents the mean value of the pixels in the neighborhood of the pixel. The value range of LNCC is between 0 and 1, 1 means the maximum correlation between the two images. When used as a loss function, take its opposite and add one, that is, Lsim=1-LNCC.

[0087] Preferably, in the second stage of network training, in order to make the output deformation field as smooth as possible, the embodiment of the present invention uses the L2 norm of the deformation field gradient as a regularization term, and its formula is as follows (2), where p is the voxel coordinate in the deformation field, and Ω represents the set of all voxels.

[0088]

[0089] φ D represents the registration parameters, Represents the gradient of the registration parameter with respect to the pixel.

[0090] Further preferably, since the segmentation label registration generally uses the nearest neighbor interpolation, this makes the network training in the present invention very difficult and the gradient cannot be returned. Therefore, for the segmentation label registration in the training process, the embodiment of the present invention correspondingly designs an algorithm flow and a loss function, which can be called SigmoidDice.

[0091] Specifically, during the training process, the model converts the segmentation label from a single channel to a multi-channel encoding in the form of One-hot and discards the channel representing the background. The network is used to output a registration transformation parameter (affine matrix or deformation field), and linear interpolation is used to align the segmentation label. In the present invention, the target image is two-dimensional, so it is necessary to project the segmentation label after registration. The model adds the voxel values ​​of each channel along the sagittal axis, multiplies the sum by a constant γ and then subtracts a constant ε, and then normalizes it to 0 to 1 through a Sigmoid function to obtain a two-dimensional segmentation label, which can now be used for Dice coefficient calculation to obtain the loss function result. The whole process is shown in the following formula (3), where M ori ,M reg and M gt are the original segmentation label, the segmentation label after registration, and the true value of the segmentation label, respectively. g(I,φ) represents the linear interpolation registration of an image I using the registration parameter φ. Sum sag represents the sum along the sagittal axis.

[0092]

[0093] L seg =1-Dice(Sigmoid(M reg ),M gt ); formula (4)

[0094] Directly projecting the segmentation labels according to the physical process is a discrete process, and the operation is not differentiable, making it difficult to optimize the model. Use One-hot encoding to convert the segmentation labels into binary multi-channel features, and then use the summation method to project. If there are more projection points with the same label in the projection direction, a larger projection value will be obtained. The size of this projection value reflects the credibility of the projection point to be considered as the label. Then, this projection value is linearly transformed, and then the Sigmoid activation function is used to perform nonlinear transformation and normalize it to between 0 and 1. The Dice coefficient can be used to measure the consistency between the projection label and the true label. In this way, the registration process of the segmentation labels during training becomes a continuous and differentiable process, which can realize gradient backpropagation and effectively guide model optimization training.

[0095] For the two-stage network model in the embodiment of the present invention, the network architectures of the two are similar, but the output results are different, so different loss functions need to be used.

[0096] In the first stage of the affine transformation network, the loss functions used include L based on image similarity sim and L based on anatomical structure coincidence seg , as shown in the following formula (4):

[0097] L affine =λ sim ·L sim +λ seg ·L seg ;Formula (5)

[0098] In the second stage of deformation field registration network, the loss function used includes not only L sim and L based on anatomical structure coincidence seg , and also includes L, which constrains the deformation field deformation degree reg , as shown in the following formula (5):

[0099] L deform =λ sim ·L sim +λ seg ·L seg +λ reg ·L reg ;Formula (6)

[0100] like Figure 7 , which shows a schematic diagram of the structure of a dual-stage registration device for lower limb full-length images in an embodiment of the present invention. The dual-stage registration device 700 in an embodiment of the present invention comprises: an image acquisition module 701, a coarse registration module 702, and a fine registration module 703.

[0101] The image acquisition module 701 is used to acquire the CT image to be registered and its label, and the target X-ray image and its label of the lower limb.

[0102] The coarse registration module 702 is used to input the CT image to be registered and the label, the target X-ray film image and the label into the affine transformation network to generate an affine transformation matrix for coarsely registering the CT image to be registered.

[0103] The fine registration module 703 is used to input the CT image and label, the target X-ray film image and label output by the coarse registration module 702 into the deformation field registration network to generate a deformation field for fine registration based on the coarse registration result of the CT image to be registered.

[0104] In some examples, the affine transformation network includes: a first encoder based on a 2D Swin-Transformer module, a second encoder based on a 3D Swin-Transformer module and a corresponding decoder; the first encoder is used to encode the target X-ray image and label; the second encoder is used to encode the CT image and label to be registered.

[0105] In some examples, the first encoder based on the 2D Swin-Transformer module includes: a first encoding layer and four 2D Swin-Transformer layers; wherein the first encoding layer is used to divide the input target X-ray image into non-overlapping blocks and encode them into vectors of specified length respectively; the four 2D Swin-Transformer layers are used to generate stepped feature maps of different scales.

[0106] In some examples, the second encoder based on the 3D Swin-Transformer module includes: a second encoding layer and four 3D Swin-Transformer layers; wherein the second encoding layer is used to divide the input CT image to be registered into non-overlapping blocks and encode them into vectors of specified length respectively; the four 3D Swin-Transformer layers are used to generate stepped feature maps of different scales.

[0107] In some examples, a feature fusion module based on a mutual attention mechanism is set between the corresponding network layers of the first encoder and the second encoder; the fused feature map is directly passed to the decoder in the affine transformation network, and is passed to the decoder in the deformation field registration network in the form of a jump connection.

[0108] In some examples, the feature fusion module based on the mutual attention mechanism is set between the deepest encoder layers of the first encoder and the second encoder.

[0109] In some examples, the decoder in the affine transformation network includes a multi-layer perceptron; and the decoder in the deformation field registration network is composed of multiple 3D convolution modules.

[0110] In some examples, the structure of the deformation field registration network includes: a feature fusion module based on the mutual attention mechanism is set between the deepest network layers corresponding to the first encoder and the second encoder, and the fused feature map is input into the first 3D convolution module after being processed by the second upsampling module; a feature fusion module based on the mutual attention mechanism is also set between the corresponding second deepest network layers, and the fused feature map is input into the first 3D convolution module for convolution processing; the convolution result output by the first 3D convolution module is input into the second 3D convolution module after the second upsampling module; the convolution result of the second 3D convolution module and the third upsampling module are input into the third 3D convolution module; the convolution result of the third 3D convolution module is input into the fourth 3D convolution module after the fourth upsampling module; the target X-ray image and label are compared with the CT image and After feature fusion (Concat) of the labels, they are respectively input into the downsampling module and the sixth 3D convolution module for processing; the convolution result of the sixth 3D convolution module is input into the fifth 3D convolution module; the output result of the downsampling module is input into the seventh 3D convolution module, and the convolution result of the seventh 3D convolution module is input into the fourth 3D convolution module; the fourth 3D convolution module performs convolution processing after receiving the output result of the fourth upsampling module and the output result of the seventh 3D convolution module, and the processing result is transmitted to the fifth 3D convolution module after upsampling processing by the fifth upsampling module; the fifth 3D convolution module also receives the convolution result from the sixth 3D convolution module; the fifth 3D convolution module performs convolution processing on the processing results received from the fifth upsampling module and the sixth 3D convolution module, and the processing result is output as a deformation field through the output module.

[0111] It should be noted that: the two-stage registration device for full-length films of lower limbs provided in the above embodiment only uses the division of the above program modules as an example when performing two-stage registration for full-length films of lower limbs. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the two-stage registration device for full-length films of lower limbs provided in the above embodiment and the two-stage registration method embodiment for full-length films of lower limbs belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0112] The dual-stage registration method for lower limb full-length film images provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the dual-stage registration terminal for lower limb full-length film images, please refer to Figure 8, is an optional hardware structure diagram of a dual-stage registration terminal 800 suitable for full-length film images of lower limbs provided in an embodiment of the present invention. The terminal 800 can be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The dual-stage registration terminal 800 suitable for full-length film images of lower limbs includes: at least one processor 801, a memory 802, at least one network interface 804 and a user interface 806. The various components in the device are coupled together through a bus system 805. It can be understood that the bus system 805 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 805 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 8 In the specification, various buses are labeled as bus systems.

[0113] The user interface 806 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0114] It is understood that the memory 802 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0115] The memory 802 in the embodiment of the present invention is used to store various categories of data to support the operation of the dual-stage registration terminal 800 suitable for full-length film images of lower limbs. Examples of these data include: any executable program used to operate on the dual-stage registration terminal 800 suitable for full-length film images of lower limbs, such as an operating system 8021 and an application 8022; the operating system 8021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 8022 can include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The dual-stage registration method suitable for full-length film images of lower limbs provided in the embodiment of the present invention can be included in the application 8022.

[0116] The method disclosed in the above embodiment of the present invention can be applied to the processor 801, or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 801 or the instruction in the form of software. The above processor 801 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 801 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 801 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0117] In an exemplary embodiment, the dual-stage registration terminal 800 suitable for full-length images of the lower limbs can be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), and complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0118] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0119] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0120] In summary, the present application provides a two-stage registration method, device, terminal and medium suitable for full-length film images of the lower limbs. The present invention uses deep learning technology to construct a new 2D / 3D medical image registration algorithm, and for the first time realizes the full-length supine CT and standing X-ray films of the lower limbs. Automatic cross-modal registration. The research results of the present invention will provide effective assistance for precise preoperative planning, assist in the design of HTO surgical guides, and improve the clinical efficacy of early and mid-stage knee OA. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0121] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A two-stage registration method for full-length lower limb images, characterized in that: include: Obtain the CT image to be registered and its labels, and the target X-ray image and its labels of the lower limbs; Inputting the CT image to be registered and the label, the target X-ray film image and the label into an affine transformation network to generate an affine transformation matrix for roughly registering the CT image to be registered; The CT image and label output from the coarse registration stage, and the target X-ray image and label are input into the deformation field registration network to generate a deformation field for fine registration based on the coarse registration result of the CT image to be registered.

2. The dual-stage registration method for full-length lower limb images according to claim 1, characterized in that: The affine transformation network includes: a first encoder based on a 2D Swin-Transformer module, a second encoder based on a 3D Swin-Transformer module and a corresponding decoder; the first encoder is used to encode the target X-ray image and label; the second encoder is used to encode the CT image and label to be registered.

3. The dual-stage registration method for full-length lower limb images according to claim 2, characterized in that: The first encoder based on the 2D Swin-Transformer module includes: a first encoding layer and four 2D Swin-Transformer layers; wherein the first encoding layer is used to divide the input target X-ray image into non-overlapping blocks and encode them into vectors of specified lengths respectively; and the four 2D Swin-Transformer layers are used to generate stepped feature maps of different scales.

4. The dual-stage registration method for lower limb full-length images according to claim 2, characterized in that: The second encoder based on the 3D Swin-Transformer module includes: a second encoding layer and four 3D Swin-Transformer layers; wherein the second encoding layer is used to divide the input CT image to be registered into non-overlapping blocks and encode them into vectors of specified lengths respectively; and the four 3D Swin-Transformer layers are used to generate stepped feature maps of different scales.

5. The dual-stage registration method for lower limb full-length images according to claim 2, characterized in that: Setting a feature fusion module based on a mutual attention mechanism between corresponding network layers of the first encoder and the second encoder; The fused feature map is directly transmitted to the decoder in the affine transformation network, and is transmitted to the decoder in the deformation field registration network in the form of a skip connection.

6. The dual-stage registration method for lower limb full-length images according to claim 5, characterized in that: The feature fusion module based on the mutual attention mechanism is set between the deepest encoder layer of the first encoder and the second encoder.

7. The dual-stage registration method for lower limb full-length images according to claim 1 or 5, characterized in that: The decoder in the affine transformation network includes a multi-layer perceptron; the decoder in the deformation field registration network is composed of multiple 3D convolution modules.

8. The dual-stage registration method for lower limb full-length images according to claim 2, characterized in that: The structure of the deformation field registration network includes: A feature fusion module based on the mutual attention mechanism is set between the deepest network layers corresponding to the first encoder and the second encoder, and the fused feature map is input into the first 3D convolution module after being processed by the second upsampling module; a feature fusion module based on the mutual attention mechanism is also set between the corresponding second deepest network layers, and the fused feature map is input into the first 3D convolution module for convolution processing; the convolution result output by the first 3D convolution module is input into the second 3D convolution module after passing through the second upsampling module; the convolution result of the second 3D convolution module and the third upsampling module are input into the third 3D convolution module; the convolution result of the third 3D convolution module is input into the fourth 3D convolution module after passing through the fourth upsampling module; The target X-ray film image and label are feature-fused (Concat) with the CT image and label to be registered and are respectively input into the downsampling module and the sixth 3D convolution module for processing; the convolution result of the sixth 3D convolution module is input into the fifth 3D convolution module; the output result of the downsampling module is input into the seventh 3D convolution module, and the convolution result of the seventh 3D convolution module is input into the fourth 3D convolution module; The fourth 3D convolution module performs convolution processing after receiving the output result of the fourth upsampling module and the output result of the seventh 3D convolution module, and the processing result is transmitted to the fifth 3D convolution module after upsampling processing by the fifth upsampling module; the fifth 3D convolution module also receives the convolution result from the sixth 3D convolution module; the fifth 3D convolution module performs convolution processing on the processing results received from the fifth upsampling module and the sixth 3D convolution module, and the processing result is output as a deformation field through the output module.

9. The dual-stage registration method for lower limb full-length images according to claim 1, characterized in that: The loss functions used in the affine transformation network include: L affine =λ sim ·L sim +λ seg ·L seg ; L sim =1-LNCC; L seg =1-Dice(Sigmoid(M reg ),M gt ); Among them, I f represents the target X-ray image, I m represents the CT image to be registered, p represents the pixel coordinates in the image, Ω represents the set of all pixels, N(p) represents the domain of pixel p, q belongs to the neighborhood of pixel p, represents the pixel mean in the neighborhood of pixel p of the CT image to be registered, represents the mean value of pixels in the neighborhood of pixel p of the target X-ray image; M ori 、M reg 、M gt Represent the original segmentation label, the segmentation label after registration, and the true value of the segmentation label respectively; Indicates the registration parameters Perform linear interpolation registration on image I; Sum sag represents the sum along the sagittal axis; γ and ε represent constants; Dice represents the Dice coefficient; λ sim , seg represents the weight, and the sum of the two is 1.

10. The dual-stage registration method for lower limb full-length images according to claim 1, characterized in that: The loss functions used in the deformation field registration network include: L deform =λ sim ·L sim +λ seg ·L seg +λ reg ·L reg ; L sim =1-LNCC; L seg =1-Dice(Sigmoid(M reg ),M gt ); Among them, I f represents the target X-ray image, I m represents the CT image to be registered, p represents the pixel coordinates in the image, Ω represents the set of all pixels, N(p) represents the domain of pixel p, q belongs to the neighborhood of pixel p, represents the pixel mean in the neighborhood of pixel p of the CT image to be registered, represents the mean value of pixels in the neighborhood of pixel p of the target X-ray image; M ori 、M reg 、M gt Represent the original segmentation label, the segmentation label after registration, and the true value of the segmentation label respectively; Represents the registration parameters Perform linear interpolation registration on image I; Sum sag represents the sum along the sagittal axis; γ and ε represent constants; Dice represents the Dice coefficient; λ sim , seg , reg represents the weight, the sum of the three is 1; φ D represents the registration parameters, Represents the gradient of the registration parameter with respect to the pixel.

11. A dual-stage registration device suitable for full-length images of lower limbs, characterized in that: include: An image acquisition module, used to acquire the CT image to be registered and its label, and the target X-ray image and its label of the lower limbs; A coarse registration module, used for inputting the CT image to be registered and the label, the target X-ray film image and the label into an affine transformation network to generate an affine transformation matrix for coarsely registering the CT image to be registered; The fine registration module is used to input the CT image and label, the target X-ray film image and label output by the coarse registration module into the deformation field registration network to generate a deformation field for fine registration based on the coarse registration result of the CT image to be registered.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the dual-stage registration method suitable for lower limb full-length film images as described in any one of claims 1 to 10 is implemented.

13. An electronic terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so as to enable the terminal to perform the dual-stage registration method suitable for lower limb full-length film images as described in any one of claims 1 to 10.

Images