Multi-mode spine image contrast learning registration method

Abstract

The invention discloses a multi-modal spine image contrast learning registration method, and the method comprises the steps: constructing a contrast loss calculation module comprising a feature extractor and a mapper, carrying out the pre-training, and storing the parameters of the contrast loss calculation module obtained through the pre-training; a pre-trained comparison loss calculation module is introduced into the registration network; and a registration network is trained through a comparison loss calculation module. The registration method is suitable for a navigation system of a spinal surgery robot, information of soft and hard tissues of a patient can be integrated to the maximum extent, the preoperative and intraoperative poses of the patient are aligned, and navigation positioning of a mechanical arm is assisted.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a multimodal spinal image comparison learning registration method. Background Technology

[0002] Based on the different imaging capabilities of Magnetic Resonance Imaging (MR) and Cone Beam Computer Tomography (CBCT), MR is used for preoperative scanning to obtain imaging information of the patient's intervertebral discs, nerve roots, and other soft tissues of the spine, while CBCT, due to its flexibility, is used to obtain imaging information of the patient's spinal bone tissue during surgery. In actual surgical scenarios, frequent switching between different modalities can easily lead to misalignment of the observed tissue positions.

[0003] Several traditional tools exist for automatic medical image registration, such as SimpleITK, ANTs(SyN), and NiftyReg. These traditional methods utilize features extracted from the input images to achieve image matching through intuitive explicit features. In addition, existing learning-based automatic medical image registration algorithms, such as VoxelMorph, typically stitch static and moving images together, inputting them into an encoder-decoder neural network to extract features and map them to a spatial transformation for matching.

[0004] The existing solution involves designing and training a deep neural network capable of automatic registration using either CBCT as a fixed image and MRI as a moving image, or vice versa. The process involves two registration steps: first, a rigid affine coarse registration is performed using traditional registration tools to globally align the dimensions of the multimodal images; second, a more flexible deformable registration is performed on the two images after rigid affine transformation registration to align the tissue regions, ultimately achieving precise multimodal image registration of the spine.

[0005] Existing similar neural network architectures such as Figure 1 As shown, if the CBCT image is used as a fixed image, the final registered image is the deformed MRI image; if the MRI image is used as a fixed image, the final registered image is the deformed CBCT image.

[0006] Existing traditional medical image registration tools have significant limitations. Firstly, they involve solving a different optimization problem for each pair of moving and stationary images, leading to redundancy between optimizations and high time costs. Secondly, the objective functions associated with transformation parameters (such as displacement fields or control points) often exhibit nonlinearity, creating a nonconvex optimization dilemma.

[0007] Existing learning-based methods fuse moving and stationary images to simulate single image input and feed the fused image into the model. However, mixing feature extraction and feature matching processes together leads to feature distortion and weak alignment in the fused regions, making it impossible for the network to recognize the one-to-one correspondence between image pairs.

[0008] Existing learning-based methods are trained using hand-designed inter-domain similarity functions. Using pixel-based similarity metrics to model intensity relationships in multimodal image registration tasks can lead to errors in anatomical registration. The repetitiveness and complexity of spinal tissues, as well as the high modal differences between CBCT and MR images, exacerbate this negative impact. Summary of the Invention

[0009] This invention provides a multimodal spinal image contrast learning registration method, which is applicable to the navigation system of spinal surgery robots. It can integrate the information of the patient's soft and hard tissues to the greatest extent and align the patient's preoperative and intraoperative poses to assist the navigation and positioning of the robotic arm.

[0010] To achieve the above objectives, the present invention adopts the following specific technical solution: A multimodal spinal image contrast learning registration method, the registration method comprising: Construct a contrastive loss calculation module that includes a feature extractor and a mapper and pre-train it, then save the parameters of the contrastive loss calculation module obtained from the pre-training. A pre-trained contrast loss calculation module is introduced into the registration network to achieve modality-independent feature representation, which is used to extract high-level semantic feature vectors of moving and stationary images. During the training of the registration network, the parameters of the contrast loss calculation module are frozen so that it does not participate in parameter updates. The registration network is trained using a contrastive loss calculation module: Moving and stationary images, after rigid affine coarse registration, are input into two structurally identical encoders. During the encoding phase, spatial and semantic information of the two modalities is extracted. In the decoder, feature maps at different resolutions are stitched together as supplementary inputs to the fusion layer, and the two images are spatially aligned according to semantics to obtain a displacement field, which is then further used to generate a deformation field. Under the action of the Spatial Transformer Network (STN), the moving image is distorted according to the deformation field to obtain the registered image. The stationary image and the registered image are then input into the contrastive loss calculation module. The module extracts feature vectors from both images, calculates their cosine similarity as a modality-independent similarity metric, and uses this similarity metric as a loss function combined with a regularization term for image registration.

[0011] Furthermore, the usage method of the comparison loss calculation module is as follows: Rigid affine registration is performed on moving and stationary images, and the images of the two modalities are cut into image blocks of the same size. Each pair of image blocks is regarded as an image pair with similar semantic content. The semantic representation in the image blocks is extracted by a feature extractor. The feature map is projected onto a unified embedding space by a mapper to obtain modality-independent feature vectors.

[0012] Furthermore, based on the contrastive learning approach without negative samples, semantic representations in image patches are extracted by a feature extractor, and the feature maps are projected onto a unified embedding space by a mapper.

[0013] Furthermore, the registration network adopts a dual encoder-single decoder network structure. The images of the two modalities are extracted using two dedicated encoders to extract spatial and semantic information respectively, and then stitched and fused at the bottom of the encoder as input to the decoder. In the decoder, feature maps at different resolutions are stitched together as supplementary input fusion layers for spatial information, so that the registration network model can spatially align the two images according to the same semantic organization to obtain the displacement field and further generate the deformation field.

[0014] Furthermore, the pre-training contrastive loss calculation module employs an asymmetric gradient propagation mechanism.

[0015] Furthermore, the asymmetric gradient propagation mechanism is as follows: in the parameter-sharing Siamese network structure, only one side branch is allowed to participate in gradient backpropagation through the predictor, while the feature representation of the other side branch is fixed through an explicit gradient blocking operation, thereby breaking the information degradation of the completely symmetric structure.

[0016] Furthermore, the pre-training process of the contrastive loss calculation module is as follows: a predictor structure is introduced, the feature vector of one mode is input into the predictor, and the feature vector of that mode is learned to fit the feature vector of another mode in the embedding space; by maximizing the cosine similarity between the two, the contrastive loss calculation module is prompted to learn the mode-independent feature vector representation.

[0017] Furthermore, the fixed image is a CBCT image and the moving image is an MR image, or the fixed image is an MR image and the moving image is a CBCT image.

[0018] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: 1. To address the high time cost of traditional medical image registration tools, which require solving a different optimization problem for each pair of moving and stationary images, this invention's registration method utilizes a deep learning network structure to learn global feature representations across the entire dataset. This achieves unified optimization across the entire data domain, eliminating the need for independent iterative calculations for each image pair and significantly improving computational efficiency. Traditional methods often face the challenge of non-convex optimization, potentially leading to local distortions. In contrast, the registration network model constructed in this invention generates a smoother tissue deformation field, effectively preventing local distortions while maintaining registration accuracy. This is particularly important for precise navigation during clinical surgery.

[0019] 2. To address the problem that existing learning-based methods, which fuse moving and fixed images and then input them into the registration network model, can easily lead to feature confusion and information distortion, the registration method of this invention uses dual-channel independent encoders to process fixed and moving images separately, ensuring the pure feature extraction of each image content, and performing accurate feature matching and alignment in subsequent stages, thereby improving the ability to recognize the correspondence between images.

[0020] 3. The pre-training method based on cross-modal image block contrast learning proposed in this invention enables the registration network model to establish semantic consistency between different modalities (such as CBCT and MR), enhances the generalization ability of the registration network model to multimodal data, and helps to solve the registration challenges caused by the repetitiveness and complexity of spinal tissues and the high modal differences between CBCT and MR images.

[0021] 4. The pre-training method used in the registration method of this invention does not rely on negative samples and is trained using image patches as input. Compared with traditional contrastive learning methods, the registration method proposed in this invention removes the requirement for negative samples. In the application scenario of spinal surgery, this avoids the difficulty in constructing positive and negative sample pairs caused by problems such as missing pairwise data and small data volume.

[0022] 5. This invention introduces a registration similarity measurement mechanism based on contrastive learning features into the registration network. It uses the cosine similarity of semantic feature vectors as a metric to guide network parameter updates, thereby achieving more accurate and robust pixel-level image registration at the semantic level. Compared to traditional pixel-based similarity measurement methods, this mechanism more accurately reflects the true correspondence of anatomical structures and reduces registration errors. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of a neural network architecture in the prior art; Figure 2 This is a flowchart of the multimodal spinal image contrast learning registration method of the present invention; Figure 3 This is a schematic diagram of the registration network used in this invention; Figure 4 The training flowchart for the comparison loss calculation module is shown below. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention provides a multimodal spinal image contrast learning registration method, such as... Figure 2 As shown, this registration method is implemented through a registration network, and the architecture of the registration network can adopt... Figure 3 The architecture of the convolutional neural network shown is in Figure 3 In this embodiment, CBCT images are used as stationary images and MR images as moving images. Even if the two are interchanged, excellent registration results can still be guaranteed; that is, the stationary image is a CBCT image and the moving image is an MR image, or vice versa. The registration method includes the following steps: Step 1: Construct a contrastive loss calculation module containing a feature extractor and a mapper, perform pre-training, and save the parameters of the pre-trained contrastive loss calculation module; for example... Figure 4 As shown, the contrastive loss calculation module is used as follows: rigid affine registration is performed on the moving and fixed images of the spine to ensure that tissues at the same location have roughly aligned spatial relationships in both modalities; the images of both modalities are cut into image patches of the same size, and each pair of image patches is considered as an image pair with similar semantic content; based on the contrastive learning idea without negative samples, the semantic representation in the image patches is extracted by a feature extractor, and the feature map is projected onto a unified embedding space by a mapper to obtain modality-independent feature vectors. Based on the contrastive learning idea without negative samples, the feature vectors obtained after feature extraction and mapping of image patches with the same semantic information should have high similarity.

[0026] Step 2: Introduce a pre-trained contrast loss calculation module into the registration network to achieve modality-independent feature representation, which is used to extract high-level semantic feature vectors of moving and stationary images; During the training of the registration network, freeze the parameters of the contrast loss calculation module so that it does not participate in parameter updates. Step 3: Train the registration network using the contrast loss calculation module: (e.g.) Figure 3As shown, the moving and fixed images, which have undergone rigid affine coarse registration, are used as inputs to the registration network model. These images are then input into two encoders with identical structures. During the encoding stage, spatial and semantic information of the two modalities is extracted. In the decoder, feature maps at different resolutions are stitched together as supplementary inputs to the fusion layer, enabling the registration network model to spatially align the two images according to semantics, obtaining a displacement field and further generating a deformation field. Then, under the action of the Spatial Transformer Network (STN), the moving image is distorted according to the deformation field to obtain the registered image. Finally, the fixed image and the registered image are input into the contrast loss calculation module. The contrast loss calculation module extracts the feature vectors of the fixed image and the registered image, calculates their cosine similarity as a modality-independent similarity measure, and uses this similarity measure as a loss function combined with a regularization term for image registration.

[0027] Based on the aforementioned registration method, the registration network adopts a dual-encoder-single-decoder network structure. Two dedicated encoders extract spatial and semantic information from the two image modalities respectively, and these are then concatenated and fused at the bottom of the encoders to serve as input to the decoder. In the decoder, feature maps at different resolutions are concatenated as supplementary input fusion layers for spatial information. This allows the registration network model to spatially align the two images according to semantics, obtaining a displacement field which is then used to generate a deformation field. For the encoder, concatenation and fusion are performed only at the outputs of the bottommost layers (the fifth and tenth convolutional layers). For the decoder, feature maps at different resolutions are concatenated and fused during skip connections, and then used as supplementary input to the decoder.

[0028] Furthermore, to effectively avoid representation collapse during training, the pre-training contrastive loss calculation module employs an asymmetric gradient propagation mechanism. Specifically, in a parameter-sharing Siamese network structure, only one branch is allowed to participate in gradient backpropagation through the predictor, while the feature representation of the other branch is fixed through an explicit gradient blocking operation, thus breaking the information degradation inherent in a completely symmetric structure. For example... Figure 4As shown, the process of pre-training the contrastive loss calculation module using the asymmetric gradient propagation mechanism is as follows: a predictor structure is introduced, which only participates in the pre-training process of this module. The feature vector of one modality is input into the predictor, which learns the fitting of the feature vector of one modality to the feature vector of another modality in the embedding space. By maximizing the cosine similarity between the two, the contrastive loss calculation module learns the modality-independent feature vector representation. That is, the contrastive loss calculation module can convert the image of any modality into a vector form that only cares about space and semantics, thus eliminating the influence of modality differences.

[0029] The contrastive loss calculation module employs a Siamese network structure comprising two parameter-sharing sub-network branches: each branch first extracts semantic representations from image patches using a feature extractor; then, a mapper projects the feature maps onto a unified embedding space to obtain modality-independent feature vectors. Guided by this pre-training strategy, the contrastive loss calculation module can generate semantic feature vectors of the spinal surgical area with good modality independence and region consistency. When applied to the registration network as a similarity metric, this module effectively compares the matching degree of the entire image in the feature space, thereby optimizing the registration results and updating the network parameters.

[0030] The convolutional layers used in the above registration network architecture can be replaced with Transformer blocks or Mamba blocks.

[0031] The aforementioned deep learning-based registration network structure achieves unified optimization of the entire data domain by learning the global feature representation of the entire dataset, eliminating the need for independent iterative calculations for each pair of images, thus significantly improving computational efficiency. At the same time, the constructed registration network model can generate a smoother tissue deformation field, effectively preventing the generation of local distortions while ensuring registration accuracy.

[0032] The image registration method described above is based on dual-channel independent encoding, inputting the fixed image and the moving image into two independent encoders of the registration network, thus avoiding the feature confusion and information distortion problems caused by image fusion in traditional methods. This design enables the network to extract clean feature representations from the two images separately and perform accurate feature matching and alignment in subsequent stages, thereby effectively improving the recognition ability of the correspondence between images and enhancing registration accuracy and robustness.

[0033] The pre-training of the aforementioned registration method is based on cross-modal image patch contrastive learning. A contrastive loss calculation module containing a feature extractor and a mapper is constructed, and pre-trained to enable it to generate modality-independent and semantically consistent spinal region feature representations. During pre-training, rigid affine coarse registration is performed on CBCT and MR images, which are then segmented into semantically aligned image patches. A predictor guides the distribution of feature vectors from different modalities to converge in a unified space, improving the generalization ability of the registration network model for multimodal data.

[0034] The above registration method is based on a registration similarity measurement mechanism using contrastive learning features. It introduces a pre-trained contrastive loss calculation module into the registration network to extract high-level semantic feature vectors of moving and stationary images. It also constructs a similarity measurement function based on the cosine similarity between the two to guide the network parameter update, thereby achieving a more accurate and robust image pixel-level registration effect.

[0035] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

[0036] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal spinal image contrast learning registration method, characterized in that, include: Construct a contrastive loss calculation module that includes a feature extractor and a mapper and pre-train it, then save the parameters of the contrastive loss calculation module obtained from the pre-training. A pre-trained contrast loss calculation module is introduced into the registration network to achieve modality-independent feature representation, which is used to extract high-level semantic feature vectors of moving and stationary images. During the training of the registration network, the parameters of the contrast loss calculation module are frozen so that it does not participate in parameter updates. The registration network is trained using a contrastive loss calculation module: Moving and stationary images, after rigid affine coarse registration, are input into two structurally identical encoders. During the encoding phase, spatial and semantic information of the two modalities is extracted. In the decoder, feature maps at different resolutions are stitched together as supplementary inputs to the fusion layer, and the two images are spatially aligned according to semantics to obtain a displacement field, which is then further used to generate a deformation field. Under the action of the spatial transformation layer, the moving image is distorted according to the deformation field to obtain the registered image. The stationary image and the registered image are then input into the contrastive loss calculation module. The module extracts feature vectors from both images, calculates their cosine similarity as a modality-independent similarity metric, and uses this similarity metric as a loss function combined with a regularization term for image registration.

2. The multimodal spinal image contrast learning registration method as described in claim 1, characterized in that, The usage method of the contrast loss calculation module is as follows: Rigid affine registration is performed on moving and stationary images, and the images of the two modalities are cut into image blocks of the same size. Each pair of image blocks is regarded as an image pair with similar semantic content. The semantic representation in the image blocks is extracted by a feature extractor. The feature map is projected onto a unified embedding space by a mapper to obtain modality-independent feature vectors.

3. The multimodal spinal image contrast learning registration method as described in claim 2, characterized in that, Based on the contrastive learning approach without negative samples, a feature extractor extracts semantic representations from image patches and a mapper projects the feature maps onto a unified embedding space.

4. The multimodal spinal image contrast learning registration method as described in claim 3, characterized in that, The registration network adopts a dual encoder-single decoder network structure. The images of the two modalities are extracted using two dedicated encoders to extract spatial and semantic information respectively, and then stitched and fused at the bottom of the encoder as the input of the decoder. In the decoder, feature maps at different resolutions are stitched together as supplementary input fusion layers for spatial information, enabling the registration network model to spatially align the two images according to semantics, thereby obtaining the displacement field and further generating the deformation field.

5. The multimodal spinal image contrast learning registration method as described in claim 4, characterized in that, The pre-training contrastive loss calculation module employs an asymmetric gradient propagation mechanism.

6. The multimodal spinal image contrast learning registration method as described in claim 5, characterized in that, The asymmetric gradient propagation mechanism is as follows: In the parameter-sharing Siamese network structure, only one side branch is allowed to participate in gradient backpropagation through the predictor, while the feature representation of the other side branch is fixed through an explicit gradient blocking operation, thereby breaking the information degradation of the completely symmetric structure.

7. The multimodal spinal image contrast learning registration method as described in claim 6, characterized in that, The process of pre-training the contrastive loss calculation module is as follows: a predictor structure is introduced, the feature vector of one mode is input into the predictor, and the feature vector of that mode is learned to fit the feature vector of another mode in the embedding space; by maximizing the cosine similarity between the two, the contrastive loss calculation module is prompted to learn the mode-independent feature vector representation.

8. The multimodal spinal image contrast learning registration method as described in any one of claims 1-7, characterized in that, The fixed image is a CBCT image and the moving image is an MR image, or the fixed image is an MR image and the moving image is a CBCT image.

Images