Multi-modal image registration method and device based on deep learning

Abstract

The invention discloses a multi-modal image registration method and equipment based on deep learning. The method comprises the following steps: acquiring a 4DCT image and a plurality of other medical images with different modalities; preprocessing the 4DCT image, determining each phase of a respiratory cycle, and calculating the motion range of different organs in the respiratory cycle; the medical image of each of the other modals is preprocessed; selecting an expiration or inspiration image in one respiratory cycle from the medical images of other modalities; carrying out feature extraction on the selected 4DCT image and other medical images of each modal, and extracting feature points of each image; performing registration on different modal image feature points and 4DCT image feature points by using the trained deep learning model; and rendering the registration result. According to the method, the complex relation between different modal image feature points is learned through the deep learning model, a more accurate registration result can be achieved, registration errors are reduced, and cross-modal image registration precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and specifically relates to a multimodal image registration method and device based on deep learning. Background Art

[0002] There are many different medical image modalities, each of which provides unique information. 4DCT can capture the dynamic changes of organs during the respiratory cycle, kV images are often used for real-time monitoring in radiotherapy, ultrasound images have high resolution for soft tissues, and structured light images and infrared images have advantages in body surface positioning and posture estimation.

[0003] However, due to the differences in imaging principles, resolution, contrast, etc. of different modality images, multimodal image registration becomes a challenging task.

[0004] Traditional image registration methods are usually based on manually extracted features, such as points, lines, and surfaces, and then optimize similarity metrics to find the best transformation parameters. These methods are often computationally intensive, have poor robustness, and are not ideal for the registration of complex deformations and multimodal images. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a multimodal image registration method and device based on deep learning.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] On the one hand, the present invention discloses a multimodal image registration method based on deep learning, comprising:

[0008] Step S1: Acquire 4DCT images and other medical images of different modalities;

[0009] Step S2: preprocessing the 4DCT image, determining each phase of the respiratory cycle, and calculating the range of motion of different organs in the respiratory cycle;

[0010] Step S3: preprocessing the medical images of each other modality;

[0011] Step S4: selecting an image during exhalation or inhalation in one respiratory cycle from the medical images of each other modality;

[0012] Step S5: performing feature extraction on the selected 4DCT image and the medical images of each other modality to extract feature points of each image;

[0013] Step S6: using the trained deep learning model to align the feature points of different modal images with the feature points of the 4DCT image;

[0014] Step S7: Rendering the registration result.

[0015] Based on the above technical solution, the following improvements can be made:

[0016] As a preferred solution, step S1 includes: acquiring 4DCT images and one or more medical images selected from the group consisting of kV images, ultrasound images, structured light images, and infrared images.

[0017] As a preferred solution, step S5 includes: using the SIFT algorithm to perform feature extraction on the selected 4DCT image and other medical images of each modality, and extracting feature points of each image.

[0018] As a preferred solution, the deep learning model is based on a multi-layer perceptron, including: an input layer, multiple hidden layers and an output layer;

[0019] The input layer is used to input feature points of different modal images and 4DCT image feature points;

[0020] Multiple hidden layers are used to perform nonlinear changes and feature extraction on the input data;

[0021] The output layer is used to output the matching results of feature points.

[0022] As a preferred solution, it also includes:

[0023] Step S7: Based on the registration of all time domains of the 4DCT images, a motion association model is established, which specifically includes:

[0024] Step S7.1: registering the images at each time point in the 4DCT image sequence to obtain transformation parameters between adjacent time points;

[0025] Step S7.2: Based on the registration result obtained in step S7.1, a motion association model is fitted using a polynomial function to construct an organ motion association model;

[0026] Step S7.3: When the body surface point cloud data is obtained, it is input into the motion association model to generate the motion deformation field under all modes;

[0027] Step S7.4: Render the generated motion deformation field in the registration result in real time to show the dynamic changes of the organ.

[0028] In addition, on the other hand, the present invention also discloses a multimodal image registration device based on deep learning, comprising:

[0029] An acquisition module, used to acquire 4DCT images and other medical images of different modalities;

[0030] The first preprocessing module is used to preprocess the 4DCT image, determine the various phases of the respiratory cycle, and calculate the motion range of different organs in the respiratory cycle;

[0031] A second preprocessing module, for preprocessing the medical images of each other modality;

[0032] A selection module, used for selecting an image during exhalation or inhalation in one respiratory cycle from medical images of each other modality;

[0033] A feature extraction module is used to extract features from the selected 4DCT image and other medical images of each modality, and extract feature points of each image;

[0034] The registration module is used to register the feature points of different modal images with the feature points of 4DCT images using the trained deep learning model;

[0035] The rendering module is used to render the registration results.

[0036] As a preferred solution, the acquisition module is used to acquire 4DCT images and one or more medical images selected from the group consisting of kV images, ultrasound images, structured light images, and infrared images.

[0037] As a preferred solution, the feature extraction module is used to perform feature extraction on the selected 4DCT image and other medical images of each modality using the SIFT algorithm to extract feature points of each image.

[0038] As a preferred solution, the deep learning model is based on a multi-layer perceptron, including: an input layer, multiple hidden layers and an output layer;

[0039] The input layer is used to input feature points of different modal images and 4DCT image feature points;

[0040] Multiple hidden layers are used to perform nonlinear changes and feature extraction on the input data;

[0041] The output layer is used to output the matching results of feature points.

[0042] As a preferred solution, it also includes:

[0043] The motion association module is used to establish a motion association model based on the registration of all time domains of the 4DCT image, specifically including:

[0044] A registration unit, used for registering the image at each time point in the 4DCT image sequence, and obtaining the transformation parameters between adjacent time points;

[0045] A model building unit is used to fit the motion association model using a polynomial function based on the registration result obtained by the registration unit to build an organ motion association model;

[0046] The motion deformation field generating unit is used to input the body surface point cloud data into the motion association model when the body surface point cloud data is obtained, so as to generate the motion deformation field under all modes;

[0047] The rendering unit is used to render the generated motion deformation field in the registration result in real time to show the dynamic changes of the organ.

[0048] The present invention discloses a multimodal image registration method and device based on deep learning, which has the following beneficial effects:

[0049] First, the present invention learns the complex relationship between feature points of images of different modalities through a deep learning model, which can achieve more accurate registration results, reduce registration errors, and improve the accuracy of image registration across modalities.

[0050] Second, compared with traditional registration methods, the deep learning model of the present invention can quickly perform feature point matching and image transformation, improve the efficiency of registration, and facilitate real-time application in clinical practice.

[0051] Third, the registration method used in the present invention has good universality and generalization, and can achieve cross-modal registration.

[0052] Fourthly, the present invention establishes a motion correlation model, and can generate a multi-modal deformation field and render it in real time only through the body surface point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A flowchart of a multimodal image registration method provided by an embodiment of the present invention.

[0055] Figure 2 A block diagram of a multimodal image registration method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Using ordinal numbers “first,” “second,” “third,” etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0059] In addition, the expression “comprising” an element is an “open” expression, which simply means that corresponding components exist, and should not be interpreted as excluding additional components.

[0060] In order to achieve the purpose of the present invention, a multimodal image registration method and device based on deep learning are provided in some embodiments.

[0061] like Figure 1 As shown, the multimodal image registration method includes:

[0062] Step S1: Acquire 4DCT images and other medical images of different modalities;

[0063] Step S2: preprocessing the 4DCT image, determining each phase of the respiratory cycle, and calculating the range of motion of different organs in the respiratory cycle;

[0064] Step S3: preprocessing the medical images of each other modality;

[0065] Step S4: selecting an image during exhalation or inhalation in one respiratory cycle from the medical images of each other modality;

[0066] Step S5: performing feature extraction on the selected 4DCT image and the medical images of each other modality to extract feature points of each image;

[0067] Step S6: using the trained deep learning model to align the feature points of different modal images with the feature points of the 4DCT image;

[0068] Step S7: Rendering the registration result.

[0069] Each step is described in detail below.

[0070] Step S1 is an acquisition step, which acquires 4DCT images as well as kV images, ultrasound images, structured light images, and infrared images.

[0071] The 4DCT image contains the dynamic change information of the organ in the time dimension. Step S2 preprocesses the 4DCT image.

[0072] By calculating the position change of each voxel at different time points, the various phases of the respiratory cycle are determined. For example, for the 4DCT image of the lungs, the obvious changes in the volume and shape of the lungs during inhalation and exhalation can be observed. By tracking the edge of the lung tissue or the movement of specific markers, the end-inhalation and end-expiration phases in each respiratory cycle can be accurately divided.

[0073] like Figure 2 As shown, in the embodiment of the present invention, a 4DCT image of the lungs is collected, and an ultrasound image of the blood vessels of the liver is collected. In other embodiments, the ultrasound image may also be an image of other parts, but the range is limited to the area associated with the respiratory movement, such as the organs within the 4DCT imaging range.

[0074] On this basis, the range of motion of the organ during the respiratory cycle is further calculated, which can be achieved by measuring the displacement, volume change and other parameters of the organ at different phases. If additional tumor modeling is required, the movement distance of the tumor center can be measured in the 4DCT images of multiple respiratory cycles to determine its range of motion.

[0075] Step S3 preprocesses the medical images of each other modality.

[0076] The preprocessing of kV images specifically includes: geometric correction and grayscale normalization of kV images.

[0077] kV images may have geometric deformation and grayscale inconsistency caused by shooting angles, equipment differences, etc. Therefore, it is necessary to perform geometric correction on the kV imaging system, using known geometric models and calibration parameters to adjust the pixel positions in the image to eliminate image distortion and position information errors caused by equipment or shooting angles.

[0078] Grayscale normalization is aimed at eliminating grayscale differences caused by equipment settings, exposure conditions, etc. By mapping the grayscale value of the image to a uniform range, such as [0,255], images of different kV are comparable.

[0079] For example, a series of kV images acquired during radiotherapy may have geometric deformation and grayscale inconsistency due to slight differences in the patient's position and equipment during each shot. After preprocessing, more accurate and consistent images can be obtained, which is conducive to subsequent registration work.

[0080] The preprocessing of ultrasound images includes:

[0081] Perform one or more of filtering and denoising, histogram equalization, contrast stretching, and enhancing the contrast between tissue and background on the ultrasound image;

[0082] Perform three-dimensional calibration on ultrasound images;

[0083] Segment and process ultrasound images to obtain the range of motion of organs.

[0084] Ultrasound images are often affected by noise and have relatively low resolution. Therefore, it is necessary to perform denoising through the median filter method to improve image quality. And through histogram equalization and contrast stretching, the contrast between tissue and background is enhanced, and the image gradient information of the area of ​​interest in the image is highlighted.

[0085] The acquired ultrasound image is calibrated in three dimensions through the ultrasound phantom, and the pixels in the two-dimensional ultrasound image can be mapped into three-dimensional coordinates.

[0086] The segmentation process is used to extract the blood vessel position and motion range of the liver. The present invention can be but is not limited to using the trained Yolo-v8 for segmentation.

[0087] The structured light image is obtained by a structured light camera and a structured light projector; the infrared light image is obtained by an infrared camera and an infrared projector. The preprocessing of the structured light image and the infrared image specifically includes:

[0088] Through camera calibration, feature extraction and matching, triangulation and other technologies, structured light images and infrared images are reconstructed in three dimensions, converting two-dimensional image information into three-dimensional point cloud data.

[0089] For structured light cameras, structured light is projected onto the human body surface to obtain images at different angles, and then the three-dimensional coordinates of each point are calculated using the principle of feature point matching and triangulation, thereby constructing a three-dimensional point cloud model of the human body surface (the same applies to infrared cameras).

[0090] After reconstruction is completed, Gaussian filtering is performed on the structured light point cloud data and infrared point cloud data to remove noise points and abnormal points.

[0091] The two sets of point cloud data are fused to obtain body surface point cloud data.

[0092] The image is streamlined and the amount of data is reduced through uniform sampling, which removes noise points caused by ambient light interference. At the same time, the amount of data is reduced, making the subsequent registration with other modal images more efficient and accurate.

[0093] After completing the above preprocessing, select an image at a certain motion limit position (at the completion of inhalation or exhalation) in a respiratory cycle from the medical images of each other modality. By aligning the time domain phase, the image difference caused by respiratory motion can be reduced and the accuracy and stability of registration can be improved.

[0094] Step S5 performs feature extraction on the selected 4DCT image and other medical images of each modality using a SIFT (Scale-Invariant Feature Transform) algorithm to extract feature points of each image.

[0095] For each modality selected extreme position image, the process of applying SIFT algorithm to extract feature points is as follows:

[0096] (1) Establishing scale space: By performing Gaussian blurring of different scales on the original image, the scale space of the image is constructed. This enables the algorithm to detect feature points of different sizes.

[0097] (2) Detect key points: In the scale space, by comparing the image differences of adjacent scales, local extreme points are found as key points. These key points are stable under different scales and rotations.

[0098] (3) Determine the direction: For each key point, calculate the gradient direction distribution of its neighborhood pixels and determine a main direction so that the feature point is rotationally invariant.

[0099] (4) Generate feature descriptor: With the key point as the center, extract the local features of the image in its neighborhood and encode them into a 128-dimensional feature descriptor.

[0100] Through the above steps, a large number of SIFT feature points are extracted from the images of each modality, and these feature points are described using feature descriptors. These feature points and descriptors will serve as input for feature matching in subsequent deep learning models.

[0101] Step S6 uses the trained deep learning model to align the feature points of different modal images (such as kV images, ultrasound images, structured light images, infrared images) with the feature points of the 4DCT image.

[0102] The deep learning model is based on a multilayer perceptron (MLP), which includes: an input layer, multiple hidden layers, and an output layer;

[0103] The input layer is used to input feature points of different modal images and 4DCT image feature points;

[0104] Multiple hidden layers are used to perform nonlinear changes and feature extraction on the input data through connections between neurons and activation functions;

[0105] The output layer is used to output the matching results of feature points.

[0106] The number of hidden layers and the number of neurons in each hidden layer can be adjusted and optimized according to the specific problem and data characteristics.

[0107] The input of the model is the SIFT feature point descriptors extracted from the images of different modalities. These descriptors are concatenated or combined into a unified input vector. The output of the model is the matching result of the feature points, indicating which feature points correspond. The possibility of matching is represented by a binary label (0 for no match, 1 for match) or a probability value.

[0108] In order to train the model, a large amount of annotated multimodal image data needs to be collected. The annotation information includes: the real matching relationship between feature points in different modal images. By using these annotated data for supervised learning, the model can learn the complex mapping relationship and matching rules between feature points in different modal images.

[0109] Step S7 renders the registration result. Specifically:

[0110] (1) Coordinate transformation: The images of other modalities (such as kV images, ultrasound images, structured light images, infrared images) are spatially transformed according to the calculated transformation parameters to make them consistent with the spatial coordinate system of the 4D CT image. This involves operations such as linear transformation (such as translation, rotation, scaling) and nonlinear transformation (such as deformation based on thin plate spline functions) to ensure that the corresponding points in the images of different modalities are accurately matched in spatial position.

[0111] (2) Image fusion: Using volume rendering technology, the registered multimodal images are fused and displayed in three-dimensional space.

[0112] (3) Visualization effect optimization: To improve the visualization effect, parameters such as color mapping, transparency, and lighting can be adjusted to more clearly present the information of different modal images and the relationship between them.

[0113] For example, the 4DCT image can be displayed as a semi-transparent gray, while the images of other modalities can be overlaid in bright colors to highlight the lesion area or structure of interest.

[0114] (4) Multimodal interpolation: Since the frame rates of the modalities are different, the present invention uses a linear interpolation function to align the frame rates of images of different modalities with the 4D CT image.

[0115] Through the above rendering steps, the accuracy and effectiveness of multimodal registration can be tested.

[0116] Furthermore, the multimodal image registration method of the present invention further includes:

[0117] Step S7: Based on the registration of all time domains of the 4DCT images, a motion association model is established, which specifically includes:

[0118] Step S7.1: registering the images at each time point in the 4DCT image sequence to obtain transformation parameters between adjacent time points;

[0119] Step S7.2: Based on the registration result obtained in step S7.1, a motion association model is fitted using a polynomial function to construct an organ motion association model;

[0120] Step S7.3: When the body surface point cloud data is obtained, it is input into the motion association model to generate the motion deformation field under all modes;

[0121] Step S7.4: Render the generated motion deformation field in real time in the registration result of step S7.1 to show the dynamic changes of the organ.

[0122] The motion correlation model refers to the deformation matrix of an object in different time domains, which is obtained based on 4DCT processing.

[0123] Since the data is mostly sparsely distributed in the time domain, such as 5 Hz, polynomial fitting can interpolate the time domain into a continuous sine wave to map the respiratory movement curve, where the peaks and troughs represent the range of organ movement.

[0124] In step S7.3, the motion deformation field refers to: the motion deformation field refers to the deformation matrix, and the deformation matrix can be multiplied with the image under the modality to obtain the theoretical morphology of the organ in the time domain. Different modalities have independent motion deformation conversion matrices for the modality.

[0125] Through the above rendering steps, doctors can be provided with an intuitive and comprehensive three-dimensional view, helping them to better understand and analyze the patient's condition and develop more accurate diagnosis and treatment plans.

[0126] In addition, an embodiment of the present invention further discloses a multimodal image registration device based on deep learning, comprising:

[0127] An acquisition module, used to acquire 4DCT images and other medical images of different modalities;

[0128] The first preprocessing module is used to preprocess the 4DCT image, determine the various phases of the respiratory cycle, and calculate the motion range of different organs in the respiratory cycle;

[0129] A second preprocessing module, for preprocessing the medical images of each other modality;

[0130] A selection module, used for selecting an image during exhalation or inhalation in one respiratory cycle from medical images of each other modality;

[0131] A feature extraction module is used to extract features from the selected 4DCT image and other medical images of each modality, and extract feature points of each image;

[0132] The registration module is used to register the feature points of different modal images with the feature points of 4DCT images using the trained deep learning model;

[0133] The rendering module is used to render the registration results.

[0134] Furthermore, the acquisition module is used to acquire 4DCT images and one or more medical images selected from kV images, ultrasound images, structured light images, and infrared images.

[0135] Furthermore, the feature extraction module is used to perform feature extraction on the selected 4DCT image and other medical images of each modality using the SIFT algorithm to extract feature points of each image.

[0136] Furthermore, the deep learning model is based on a multi-layer perceptron, which includes: an input layer, multiple hidden layers, and an output layer;

[0137] The input layer is used to input feature points of different modal images and 4DCT image feature points;

[0138] Multiple hidden layers are used to perform nonlinear changes and feature extraction on the input data;

[0139] The output layer is used to output the matching results of feature points.

[0140] Furthermore, it also includes:

[0141] The motion association module is used to establish a motion association model based on the registration of all time domains of the 4DCT image, specifically including:

[0142] A registration unit, used for registering the image at each time point in the 4DCT image sequence, and obtaining the transformation parameters between adjacent time points;

[0143] A model building unit is used to fit the motion association model using a polynomial function based on the registration result obtained by the registration unit to build an organ motion association model;

[0144] The motion deformation field generating unit is used to input the body surface point cloud data into the motion association model when the body surface point cloud data is obtained, so as to generate the motion deformation field under all modes;

[0145] The rendering unit is used to render the generated motion deformation field in the registration result in real time to show the dynamic changes of the organ.

[0146] In this embodiment, the specific content of the multimodal image registration device based on deep learning is similar to the content of the multimodal image registration method based on deep learning disclosed in the above embodiment, and will not be repeated here.

[0147] The present invention discloses a multimodal image registration method and device based on deep learning, which has the following beneficial effects:

[0148] First, the present invention learns the complex relationship between feature points of images of different modalities through a deep learning model, which can achieve more accurate registration results, reduce registration errors, and improve the accuracy of image registration across modalities.

[0149] Second, compared with traditional registration methods, the deep learning model of the present invention can quickly perform feature point matching and image transformation, improve the efficiency of registration, and facilitate real-time application in clinical practice.

[0150] Third, the registration method used in the present invention has good universality and generalization, and can achieve cross-modal registration.

[0151] Fourthly, the present invention establishes a motion correlation model, and can generate a multi-modal deformation field and render it in real time only through the body surface point cloud.

[0152] The present invention is adaptable to multiple modalities and can process images of multiple modalities such as 4DCT images, kV images, ultrasound images, structured light images and infrared images, making full use of the advantages of images of different modalities to provide more comprehensive information for medical diagnosis and treatment.

[0153] The present invention has strong robustness, is highly robust to noise, deformation, partial occlusion and the like in images, and can work stably in complex clinical environments.

[0154] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention, can take the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0155] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it. They cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal image registration method based on deep learning, characterized in that: include: Step S1: Acquire 4DCT images and other medical images of different modalities; Step S2: preprocessing the 4DCT image, determining each phase of the respiratory cycle, and calculating the range of motion of different organs in the respiratory cycle; Step S3: preprocessing the medical images of each other modality; Step S4: selecting an image during exhalation or inhalation in one respiratory cycle from the medical images of each other modality; Step S5: performing feature extraction on the selected 4DCT image and the medical images of each other modality to extract feature points of each image; Step S6: using the trained deep learning model to align the feature points of different modal images with the feature points of the 4DCT image; Step S7: Rendering the registration result.

2. The multimodal image registration method according to claim 1, characterized in that: The step S1 includes: acquiring a 4DCT image and one or more medical images selected from the group consisting of a kV image, an ultrasound image, a structured light image, and an infrared image.

3. The multimodal image registration method according to claim 1, characterized in that: The step S5 includes: performing feature extraction on the selected 4DCT image and other medical images of each modality using the SIFT algorithm to extract feature points of each image.

4. The multimodal image registration method according to claim 1, characterized in that: The deep learning model is based on a multi-layer perceptron, including: an input layer, multiple hidden layers and an output layer; The input layer is used to input feature points of different modal images and 4DCT image feature points; The multiple hidden layers are used to perform nonlinear changes and feature extraction on the input data; The output layer is used to output the matching results of the feature points.

5. The multimodal image registration method according to any one of claims 1 to 4, characterized in that: Also includes: Step S7: Based on the registration of all time domains of the 4DCT images, a motion association model is established, which specifically includes: Step S7.1: registering the images at each time point in the 4DCT image sequence to obtain transformation parameters between adjacent time points; Step S7.2: Based on the registration result obtained in step S7.1, a motion association model is fitted using a polynomial function to construct an organ motion association model; Step S7.3: When the body surface point cloud data is obtained, it is input into the motion association model to generate the motion deformation field under all modes; Step S7.4: Render the generated motion deformation field in the registration result in real time to show the dynamic changes of the organ.

6. A multimodal image registration device based on deep learning, characterized in that: include: An acquisition module, used to acquire 4DCT images and other medical images of different modalities; The first preprocessing module is used to preprocess the 4DCT image, determine the various phases of the respiratory cycle, and calculate the motion range of different organs in the respiratory cycle; A second preprocessing module, for preprocessing the medical images of each other modality; A selection module, used for selecting an image during exhalation or inhalation in one respiratory cycle from medical images of each other modality; A feature extraction module is used to extract features from the selected 4DCT image and other medical images of each modality, and extract feature points of each image; The registration module is used to register the feature points of different modal images with the feature points of 4DCT images using the trained deep learning model; The rendering module is used to render the registration results.

7. The multimodal image registration device according to claim 6, characterized in that: The acquisition module is used to acquire 4DCT images and one or more medical images selected from kV images, ultrasound images, structured light images, and infrared images.

8. The multimodal image registration device according to claim 6, characterized in that: The feature extraction module is used to perform feature extraction on the selected 4DCT image and other medical images of each modality using the SIFT algorithm to extract feature points of each image.

9. The multimodal image registration device according to claim 6, characterized in that: The deep learning model is based on a multi-layer perceptron, including: an input layer, multiple hidden layers and an output layer; The input layer is used to input feature points of different modal images and 4DCT image feature points; The multiple hidden layers are used to perform nonlinear changes and feature extraction on the input data; The output layer is used to output the matching results of the feature points.

10. The multimodal image registration device according to any one of claims 6 to 9, characterized in that: Also includes: The motion association module is used to establish a motion association model based on the registration of all time domains of the 4DCT image, specifically including: A registration unit, used for registering the image at each time point in the 4DCT image sequence, and obtaining the transformation parameters between adjacent time points; A model building unit is used to fit the motion association model using a polynomial function based on the registration result obtained by the registration unit to build an organ motion association model; The motion deformation field generating unit is used to input the body surface point cloud data into the motion association model when the body surface point cloud data is obtained, so as to generate the motion deformation field under all modes; The rendering unit is used to render the generated motion deformation field in the registration result in real time to show the dynamic changes of the organ.

Images