Virtual dynamic three-dimensional image fusion method and system based on pre-registration

Abstract

The present application discloses a virtual dynamic three-dimensional image fusion method and system based on pre-registration, wherein the method includes: obtaining a static three-dimensional image of a target area and a dynamic three-dimensional CBCT image including at least one respiratory cycle; dividing the dynamic three-dimensional CBCT image into respiratory phases; obtaining the corresponding three-dimensional image frame on each phase node; sequentially registering the multiple three-dimensional image frames obtained by division with the static three-dimensional image to obtain a single-frame deformation field; collecting the patient's respiratory signal in real time during surgery; interpolating the single-frame deformation field based on the respiratory signal to generate a complete deformation field within the entire respiratory cycle in real time; applying the interpolated complete deformation field to the static three-dimensional image to make the static three-dimensional image virtually dynamic. By pre-registering the respiratory phase node image before surgery and interpolating based on the real-time respiratory phase during surgery, the amount of calculation during real-time fusion during surgery is reduced, thereby achieving ultra-low latency image fusion.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a virtual dynamic three-dimensional image fusion method and system based on pre-registration. Background Art

[0002] With the rapid development of medical image technology, the registration and fusion technology of medical images has important theoretical research and clinical application value in lesion localization, guiding neurosurgery, radiotherapy planning, and checking treatment effects. Medical image registration and fusion technology can provide accurate guidance for surgery in the preoperative planning stage. By fusing medical images of different modalities, doctors can have a more comprehensive understanding of the location, size, and morphology of lesions, which helps to formulate a more scientific surgical plan. Preoperative images usually have high resolution and detailed anatomical information, while intraoperative images provide real-time surgical vision. For example, in tumor resection surgery, doctors can accurately determine the location, range, and structure of surrounding tissues of tumors by fusing preoperative MRI images with intraoperative CT images, avoid accidental injury to healthy tissues, and ensure the safety and effectiveness of surgery. However, since dynamic three-dimensional CBCT images in the prior art are generally reconstructed from two-dimensional CBCT images, the reconstruction process takes a long time and real-time imaging cannot be achieved. During intraoperative navigation, there is a certain lag in image registration, which may cause errors in the surgical process. Summary of the invention

[0003] In order to solve the above technical problems, this application provides a virtual dynamic three-dimensional image fusion method and system based on pre-registration, which reduces the amount of calculation during real-time fusion during surgery by pre-registering the respiratory phase node images before surgery and interpolating based on the real-time respiratory phase during surgery, thereby achieving ultra-low latency image fusion. This application is suitable for clinical surgical navigation, real-time diagnosis, dynamic monitoring and other scenarios, further improving efficiency and safety.

[0004] Specifically, the technical solution of this application is as follows:

[0005] In a first aspect, the present application discloses a virtual dynamic three-dimensional image fusion method based on pre-registration, comprising the following steps:

[0006] Acquire a static three-dimensional image of a target area, and acquire a dynamic three-dimensional CBCT image of the target area; the dynamic three-dimensional CBCT image includes at least one respiratory cycle;

[0007] The dynamic three-dimensional CBCT image is divided into respiratory phases based on a preset division density; a corresponding three-dimensional image frame is obtained at each phase node; and a plurality of the three-dimensional image frames obtained by division are sequentially registered with the static three-dimensional image to obtain a single-frame deformation field of each three-dimensional image frame relative to the static three-dimensional image;

[0008] During the operation, the patient's breathing signal is collected in real time; based on the breathing signal, the single-frame deformation field is interpolated and calculated to generate a complete deformation field in the entire breathing cycle in real time;

[0009] The complete deformation field obtained by interpolation is applied to the static three-dimensional image to make the static three-dimensional image virtually dynamic.

[0010] In some implementations, the step of sequentially registering the divided three-dimensional image frames with the static three-dimensional image to obtain a single-frame deformation field of each three-dimensional image frame relative to the static three-dimensional image comprises the following steps:

[0011] Using a rigid registration algorithm, the three-dimensional image frame at the target phase node is preliminarily aligned with the static three-dimensional image to eliminate large-scale displacement and rotation;

[0012] Use non-rigid registration algorithm to finely align images; generate single-frame deformation fields at target phase nodes;

[0013] Each of the single-frame deformation fields is stored and numbered.

[0014] In some embodiments, the real-time acquisition of the patient's respiratory signal during the operation comprises the following steps:

[0015] Using a respiratory sensor to collect the patient's respiratory signal in real time;

[0016] Or, real-time dynamic two-dimensional CBCT images of the patient are acquired, and the respiratory signal is extracted from the real-time dynamic two-dimensional CBCT images.

[0017] In some implementations, the interpolation calculation of the single-frame deformation field based on the respiratory signal to generate a complete deformation field in the entire respiratory cycle in real time comprises the following steps:

[0018] Using a linear interpolation method, between any two adjacent phase nodes, a value of an unknown point is estimated by a linear function;

[0019] Or, using a weighted interpolation method, adding weights to the plurality of phase nodes that have been divided to estimate the value of the unknown point;

[0020] Alternatively, use an interpolation method based on a machine learning model, which uses a machine learning algorithm to learn the relationship between data to achieve interpolation.

[0021] In some embodiments, the virtual dynamic three-dimensional image fusion method based on pre-registration further includes: aligning and fusing the generated virtual dynamic three-dimensional image with the intraoperative real-time dynamic two-dimensional CBCT image.

[0022] In some embodiments, the method of aligning and fusing the generated virtual dynamic three-dimensional image with the intraoperative real-time dynamic two-dimensional CBCT image comprises the following steps:

[0023] Setting weights for the virtual dynamic three-dimensional image and the real-time dynamic two-dimensional CBCT image respectively;

[0024] The images of different modalities are aligned at the same spatial position and pixel-superimposed; the details of the superimposed images are ensured through image post-processing technology.

[0025] In a second aspect, the present application further discloses a virtual dynamic three-dimensional image fusion system based on pre-registration, comprising:

[0026] An image acquisition module, used to acquire a static three-dimensional image of a target area; and also used to acquire a dynamic three-dimensional CBCT image of the target area; the dynamic three-dimensional CBCT image includes at least one respiratory cycle;

[0027] A phase division module, used to divide the dynamic three-dimensional CBCT image into respiratory phases based on a preset division density; and obtain a corresponding three-dimensional image frame at each phase node;

[0028] A pre-registration module, used for sequentially registering the divided multiple three-dimensional image frames with the static three-dimensional image to obtain a single-frame deformation field of each three-dimensional image frame relative to the static three-dimensional image;

[0029] Respiration acquisition module, used to collect patient breathing signals in real time during surgery;

[0030] An interpolation calculation module, used to perform interpolation calculation on the single-frame deformation field based on the respiratory signal, and generate a complete deformation field in the entire respiratory cycle in real time;

[0031] The dynamic generation module is used to apply the complete deformation field obtained by interpolation to the static three-dimensional image to make the static three-dimensional image virtually dynamic.

[0032] In some embodiments, the pre-registration module specifically includes:

[0033] A rigid registration submodule, using a rigid registration algorithm, performs preliminary alignment of the three-dimensional image frame at the target phase node with the static three-dimensional image to eliminate large-scale displacement and rotation;

[0034] The non-rigid registration submodule uses a non-rigid registration algorithm to finely align images;

[0035] A deformation field calculation submodule is used to generate a single-frame deformation field on a target phase node;

[0036] The storage submodule is used to store and number each of the single-frame deformation fields.

[0037] In some embodiments, the virtual dynamic three-dimensional image fusion system based on pre-registration further includes: an image fusion module for aligning and fusing the generated virtual dynamic three-dimensional image with the intraoperative real-time dynamic two-dimensional CBCT image.

[0038] In some embodiments, the virtual dynamic three-dimensional image fusion system based on pre-registration further includes: a parameter setting module for pre-setting the division density of the respiratory phase division; and for automatically correcting the deformation field interpolation parameters according to real-time image changes.

[0039] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0040] 1. The pre-registration and deformation field interpolation of this application greatly reduce the amount of real-time calculation, achieve ultra-low latency of image fusion, and meet the needs of real-time monitoring during surgery. The interpolation method replaces the high-consumption registration optimization algorithm and significantly reduces the calculation time of image fusion.

[0041] 2. This application collects the patient's respiratory signal in real time through a respiratory sensor or a respiratory phase extraction method; based on the extracted respiratory signal, the single-frame deformation field is interpolated to ensure the synchronization of the generated image and the respiratory state. The generated virtual dynamic three-dimensional image can quickly respond to changes in the respiratory state, ensuring that the image update during the operation is timely and reliable.

[0042] 3. The virtual dynamic three-dimensional image fusion method provided in this application only requires a small amount of real-time CBCT image data, reduces the need for repeated image acquisition, optimizes the image acquisition and processing process, significantly reduces the patient's radiation dose, reduces the risk of radiation exposure, and improves clinical safety.

[0043] 4. This application can comprehensively utilize the advantages of different modal images, including CT, MRI and CBCT, through image fusion. CT provides high spatial resolution of bone structure, MRI provides high soft tissue contrast, and CBCT provides real-time dynamic monitoring. The combination of the three improves the comprehensiveness and accuracy of the image. The fused image provides high-resolution anatomical structure and dynamic functional information, enhancing the clinical application value of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The preferred implementation scheme will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of the present application.

[0045] Figure 1A flowchart of the steps of an embodiment of a virtual dynamic three-dimensional image fusion method based on pre-registration of the present application;

[0046] Figure 2 A flowchart of another embodiment of a virtual dynamic three-dimensional image fusion method based on pre-registration according to the present application;

[0047] Figure 3 A flowchart of another embodiment of a virtual dynamic three-dimensional image fusion method based on pre-registration according to the present application;

[0048] Figure 4 This is a structural block diagram of an embodiment of a virtual dynamic three-dimensional image fusion system based on pre-registration of the present application. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0050] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections.

[0051] In order to simplify the drawings, only the parts related to the invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the specific implementation methods of the present application will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0053] Preoperative images usually have high resolution and detailed anatomical information. Patients are usually required to undergo medical imaging scans before surgery, and the area of ​​interest is segmented and modeled on the computer, and the optimal surgical path is planned; while intraoperative images provide a real-time surgical field of view. During surgery, intraoperative image registration can achieve real-time positioning and navigation of surgical instruments.

[0054] By registering preoperative images with intraoperative images, detailed preoperative anatomical information can be combined with real-time intraoperative images. Doctors can see the location of the pre-planned path in the patient's body and its relative position to the target lesion in real time, so as to perform the surgery more accurately and help doctors locate lesions and important anatomical structures more accurately according to the pre-planned surgical path, thereby improving the accuracy and safety of the surgery. For example, in tumor resection surgery, doctors can accurately determine the location, range and structure of the surrounding tissues of the tumor through the registered images, avoid accidental injury to healthy tissues, and ensure the safety and effectiveness of the surgery.

[0055] This real-time navigation system can significantly improve the efficiency and safety of surgery, reduce operation time and radiation exposure of patients. Image registration can help doctors understand the location, size and morphology of lesions more comprehensively before surgery, so as to develop more reasonable surgical paths and plans.

[0056] The image registration technology in the existing technology generally directly registers the static image with path planning before the operation with the dynamic three-dimensional image during the operation; however, the dynamic three-dimensional image during the operation is generally obtained by rotating the X-ray source and the detector to scan around the target object, collect a series of two-dimensional projection data, and then reconstruct the three-dimensional image using these data; the dynamic three-dimensional image reconstruction process takes a long time and cannot achieve real-time imaging, which results in the inability to register the static image before the operation with the real-time dynamic three-dimensional image during the operation, resulting in a lag in registration. This may cause errors in the surgical process.

[0057] If the intraoperative dynamic two-dimensional dynamic image is used for registration with the preoperative image, the spatial information of the registered image is limited, which is difficult to meet the needs of complex surgery. In addition, during the image acquisition process of the target area in the patient's body, the target area will deform with the patient's breathing frequency. The preoperative dynamic image obtained by registering the preoperative static image with the intraoperative real-time dynamic three-dimensional image needs to be consistent with the patient's real-time breathing cycle frequency. Assuming that the breathing cycle is different from the image acquisition cycle, the image needs to be re-collected when the patient's breathing rhythm changes, which increases the radiation dose and the time of the operation.

[0058] In order to address the deficiencies of the 4DCT imaging method in the prior art in terms of real-time and low dose, the present application provides a method for realizing ultra-low latency virtual dynamic three-dimensional image data fusion by using pre-registration technology, thereby realizing real-time, low-dose, high-quality multimodal image fusion in clinical applications, and improving the efficiency and accuracy of clinical diagnosis and surgical navigation.

[0059] Reference Manual Attached Figure 1 , an embodiment of a virtual dynamic three-dimensional image fusion method based on pre-registration provided by the present application comprises the following steps:

[0060] S100, acquiring a static three-dimensional image of a target area, and acquiring a dynamic three-dimensional CBCT image of the target area; the dynamic three-dimensional CBCT image includes at least one respiratory cycle.

[0061] Specifically, the static three-dimensional image is generally collected before surgery, including: CT images, MRI images, or X-ray images of the patient's target area. The static three-dimensional image collected before surgery can provide high-resolution anatomical structure information to help doctors plan the surgical path and plan in detail. Of course, the static three-dimensional image can also be collected after surgery, and the virtual dynamicization of the image collected after surgery can simulate the treatment effect. Medical images of different modalities (such as CT, MRI, X-ray, etc.) each have different advantages. Through registration technology, these images of different modalities can be fused together, and more comprehensive anatomical and functional information can be provided. For example, registering the preoperative MRI image with the intraoperative CBCT image can make up for the problem of insufficient anatomical details of the CBCT image and improve the efficiency and safety of the operation. Relevant technicians should select different types of image data as the static three-dimensional image to be virtualized and dynamicized according to different processing requirements.

[0062] The dynamic three-dimensional CBCT image is collected before the formal operation of the operation. Since the reconstruction of the three-dimensional CBCT image takes time, the dynamic three-dimensional CBCT image cannot be obtained in real time, but can only be obtained with a delay. The dynamic three-dimensional CBCT image collected in this application includes at least one respiratory cycle, so as to extract the respiratory phase information therefrom.

[0063] The reconstruction of three-dimensional CBCT images is the process of converting two-dimensional CBCT projection data into three-dimensional images. The common three-dimensional CBCT image reconstruction can use the FDK (Feldkamp-Davis-Kress) algorithm, which is a classic three-dimensional cone-beam CT (CBCT) image reconstruction algorithm. It is based on filtered back projection technology and can generate high-quality three-dimensional images. Alternatively, PMF-STINR (Prior Model-Free Spatiotemporal Implicit Neural Representation) technology can be used. PMF-STINR technology is a machine learning-based technology for the reconstruction of dynamic CBCT images. PMF-STINR uses a joint image reconstruction and registration method to reconstruct dynamic CBCT images from conventional 3D CBCT scans.

[0064] Optionally, in one implementation of this embodiment, at S110, a static three-dimensional image of the target area is obtained; at S120, a dynamic three-dimensional CBCT image of the target area is obtained; and thereafter, the process further includes: S130, denoising and standardizing the image data, specifically, applying a denoising algorithm (such as Gaussian filtering, wavelet denoising) and standardization processing to unify the intensity range and resolution of images of different modalities to ensure the consistency of the image data.

[0065] S200, dividing the dynamic three-dimensional CBCT image into respiratory phases based on a preset division density, obtaining a corresponding three-dimensional image frame at each phase node, and sequentially registering the multiple three-dimensional image frames obtained by division with the static three-dimensional image to obtain a single-frame deformation field of each three-dimensional image frame relative to the static three-dimensional image.

[0066] Specifically, the dynamic CBCT image is divided into several respiratory phases, each phase corresponds to a static three-dimensional image, representing different respiratory states. The phase division is based on a preset division density. The division density refers to the number of phase nodes obtained within a respiratory cycle. For example, if the division density is set to 4 and the time / length of a respiratory cycle is 1, the phase node positions obtained by division are: 1 / 4; 2 / 4; 3 / 4; 1.

[0067] Preferably, in some other implementations of this embodiment, the patient's breathing habits can be intelligently analyzed in advance, and key phase nodes that can represent the patient's breathing characteristics can be selected in a more personalized way. The corresponding three-dimensional image frame at each phase node is then registered with the static three-dimensional image to generate a deformation field for each phase.

[0068] S300 collects the patient's breathing signals in real time during surgery.

[0069] Specifically, the respiratory signal is collected in real time during surgery, and a corresponding respiratory signal extraction device or a method for real-time extraction from intraoperative images can be used.

[0070] Optionally, the patient's respiratory signal is collected in real time with the help of a respiratory sensor; or, the current respiratory phase information is directly obtained with the help of a respiratory gating device to ensure signal accuracy and real-time performance. Respiratory sensors include chest strap sensors, nasal oxygen cannula sensors, optical respiratory sensors, pressure sensing technology, piezoelectric respiratory sensors, and other types.

[0071] In some other implementations of this embodiment, when conditions such as a respiratory signal extraction device are not available, the respiratory signal can also be extracted from the real-time dynamic two-dimensional CBCT image by acquiring the patient's real-time dynamic two-dimensional CBCT image. Specifically, the dynamic two-dimensional CBCT image does not require a three-dimensional reconstruction process, so it can be acquired in real time. For example, the Amsterdam Shround method or other respiratory signal extraction techniques are used to estimate the respiratory signal from the real-time CBCT projection data. More preferably, the airborne CBCT lung projection data obtained by clinical scanning is obtained, the diaphragm is extracted using grayscale transformation, the state of breathing is determined according to the spatial position of the diaphragm, and the real-time respiratory signal is extracted.

[0072] S400, based on the breathing signal, interpolating the single-frame deformation field to generate a complete deformation field in the entire breathing cycle in real time.

[0073] Specifically, in step S200, the deformation field on each phase node has been obtained by using the pre-registration technology. Assuming that any two adjacent phase nodes are n and n+1, according to the current respiratory phase, the deformation field interpolation is determined to be between the anchor phases n and n+1. Interpolation algorithms such as linear interpolation, weighted interpolation, and interpolation methods based on machine learning models can generate a complete deformation field in the entire respiratory cycle in real time with a small amount of calculation.

[0074] S500, applying the complete deformation field obtained by interpolation to the static three-dimensional image to make the static three-dimensional image virtually dynamic.

[0075] Specifically, the complete deformation field obtained by interpolation is applied to a static three-dimensional image (such as a static CT or MRI image) to make the static three-dimensional image virtually dynamic, thereby generating a real-time dynamic three-dimensional image.

[0076] This embodiment aims to solve the technical problem in the prior art that the three-dimensional CBCT image reconstruction process is time-consuming and cannot achieve real-time imaging, resulting in a lag in image registration. By pre-registering the respiratory phase node images before surgery and performing interpolation based on the real-time respiratory phase during surgery, the amount of calculation during real-time fusion during surgery is reduced, thereby achieving ultra-low latency image fusion.

[0077] Reference Manual Attached Figure 2 , another embodiment of the virtual dynamic three-dimensional image fusion method based on pre-registration is disclosed in the present application. Based on an embodiment of the above method, step S200 is shown, dividing the dynamic three-dimensional CBCT image into respiratory phases based on a preset division density; obtaining the corresponding three-dimensional image frame on each phase node; and sequentially registering the multiple three-dimensional image frames obtained by division with the static three-dimensional image to obtain a single-frame deformation field of each three-dimensional image frame relative to the static three-dimensional image. Specifically, the following steps are included:

[0078] S210, dividing the dynamic three-dimensional CBCT image into respiratory phases based on a preset division density; and acquiring a three-dimensional image frame corresponding to each phase node.

[0079] S220, using a rigid registration algorithm, preliminarily aligning the three-dimensional image frame on the target phase node with the static three-dimensional image to eliminate large-scale displacement and rotation.

[0080] Specifically, a mutual information-based rigid registration algorithm (such as feature point-based rigid registration or mutual information-based rigid registration method) is used to preliminarily align the CBCT image of each phase with the static CT / MRI image to eliminate large-scale displacement and rotation.

[0081] S230, finely aligning the images using a non-rigid registration algorithm; generating a single-frame deformation field at a target phase node.

[0082] Specifically, an advanced non-rigid registration algorithm (such as DeedsBCV, ConvexAdam, TransMorph, etc.) is used to generate the deformation field of each phase and finely align the images. It does not matter what registration method is used, as long as the accuracy requirement is met.

[0083] Currently, the algorithms that are more suitable for multimodal lung registration are:

[0084] S231, using the DeedsBCV algorithm to achieve non-rigid registration. DeedsBCV: DeedsBCV (DEformable Image registration with Explicit Deep Evolving Deformation for Biomechanical and Computational Vision) is a non-rigid registration algorithm that combines optical flow methods and multi-scale optimization strategies. Its main principles include:

[0085] Optical flow estimation: The displacement vector of each pixel from the source image to the target image is estimated through the principle of optical flow. Specifically, the similarity measure between images is minimized, such as mutual information (MI) or normalized cross correlation (NCC).

[0086] Multi-scale optimization: A pyramid strategy is used to gradually optimize the deformation field from low resolution to high resolution. At each scale, the large-scale deformation is estimated first, and then the local deformation is refined to improve the efficiency and accuracy of registration.

[0087] Deformation field smoothing: Regularization terms are introduced to ensure the smoothness and continuity of the deformation field and avoid unnatural deformations.

[0088] S232, using the ConvexAdam algorithm to achieve non-rigid registration. ConvexAdam: ConvexAdam is a non-rigid registration method based on a dual optimization framework and an adaptive optimizer (such as Adam). Its main principles include:

[0089] Convex optimization: The non-rigid registration problem is modeled as a convex optimization problem, which usually includes a data fidelity term and a deformation field regularization term to ensure the convexity of the objective function to avoid local optimal solutions.

[0090] Adam optimizer: In the convex optimization framework, the Adam optimizer is used to update the gradient of the deformation field parameters. Adam combines momentum and adaptive learning rate mechanisms to converge quickly in complex optimization spaces.

[0091] Automatic hyperparameter selection: Use automated methods to select hyperparameters in the optimization process, such as learning rate and regularization coefficient, to improve the adaptability and efficiency of the algorithm.

[0092] S232, uses the latest deep learning algorithms to achieve non-rigid registration.

[0093] With the development of deep learning technology, more and more non-rigid registration methods based on deep neural networks have been proposed. These methods use the powerful feature extraction and modeling capabilities of neural networks to achieve more efficient and accurate registration.

[0094] In some implementations of this embodiment, a deep learning algorithm such as TransMorph is used. TransMorph is a non-rigid registration framework based on the Transformer architecture. By introducing the self-attention mechanism, the network's ability to capture long-distance dependencies is enhanced. Its main features include: Self-attention mechanism: It can effectively capture long-distance structural correspondences in the image and improve the registration accuracy. Multi-head attention: Enhance feature representation capabilities and adapt to different types of deformations. Combining CNN and Transformer: Utilize the local feature extraction capabilities of CNN and the global context modeling capabilities of Transformer to achieve more comprehensive deformation field generation.

[0095] In some other implementations of this embodiment, deep learning algorithms such as CycleMorph are used. CycleMorph is an unsupervised non-rigid registration method based on a generative adversarial network (GAN). By introducing a cycle consistency loss, it achieves bidirectional registration between the source image and the target image, ensuring the reversibility and consistency of the deformation field. Its main features include: Cycle consistency: through the registration process from source to target and then back to source, the reversibility and logical consistency of the deformation field are ensured. Adversarial training: using a discriminator to improve the authenticity and naturalness of the generated deformation field. Unsupervised learning: no paired deformation field labels are required, reducing the dependence on labeled data.

[0096] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0097] S240, storing and numbering each of the single-frame deformation fields.

[0098] Specifically, the deformation field data calculated and generated by each phase node is stored as an anchor phase deformation field and numbered from 1 to N for use in real-time image generation.

[0099] Another embodiment of the virtual dynamic three-dimensional image fusion method based on pre-registration is disclosed in the present application. Based on an embodiment of the above method, step S400 is shown, based on the respiratory signal, interpolating the single-frame deformation field to generate a complete deformation field in the entire respiratory cycle in real time; specifically, the following sub-steps are included:

[0100] In an implementation of this embodiment, S400 includes S410, using a linear interpolation method to estimate the value of an unknown point between any two adjacent phase nodes through a linear function.

[0101] Specifically, the linear weight calculation is performed according to the current phase position to generate the current deformation field. Linear interpolation is one of the more common interpolation methods. Its basic idea is to estimate the value of an unknown point by a linear function between two known data points.

[0102] In another implementation of this embodiment, S400 includes S420, using a weighted interpolation method to add weights to the plurality of divided phase nodes to estimate the value of the unknown point.

[0103] Specifically, according to the nonlinear changes of the respiratory signal, the interpolation weights are adjusted to adapt to different respiratory stages. Weighted interpolation is to consider the influence of different known data points on the unknown point values ​​during the interpolation process. Each known point has a weight, and the size of the weight reflects the importance of the point in estimating the unknown point value. The advantage of weighted interpolation is that the weights can be flexibly adjusted according to the actual situation, so that the interpolation results are more in line with the distribution of the actual data. The disadvantage is that the method for determining the weights needs to be selected according to the specific problem. If the weights are not set reasonably, the interpolation results may be inaccurate.

[0104] In another implementation of this embodiment, S400 includes S430, using an interpolation method based on a machine learning model, using a machine learning algorithm to learn the relationship between data, thereby achieving interpolation.

[0105] Specifically, advanced interpolation methods based on machine learning models, such as S431, Gaussian process regression (GPR) algorithm: Gaussian process model is used to predict the deformation field, generating a smooth deformation field with uncertainty estimation. GPR models the correlation of data through kernel functions.

[0106] S432, U-Net-based deep learning interpolation network algorithm: Using the U-Net architecture, the convolutional neural network automatically learns the details of the deformation field to improve the interpolation accuracy and continuity of the deformation field. The loss function of U-Net usually includes L2 loss and regularization terms.

[0107] S433, CNN-based adaptive interpolation algorithm: Combines convolutional neural networks to extract image features and generate adaptive interpolation parameters to further improve the accuracy and detail performance of the deformation field. This method extracts features through multi-layer convolution operations and generates interpolation parameters through fully connected layers. The optimization objectives also include data fidelity and regularization terms.

[0108] Reference Manual Attached Figure 3 Another embodiment of a virtual dynamic three-dimensional image fusion method based on pre-registration in the present application, based on any of the above embodiments, further includes step S600, aligning and fusing the generated virtual dynamic three-dimensional image with the intraoperative real-time dynamic two-dimensional CBCT image.

[0109] In one implementation of this embodiment, the step S600 specifically includes:

[0110] S610, weights are set for the virtual dynamic three-dimensional image and the real-time dynamic two-dimensional CBCT image. Specifically, weighted superposition is performed according to the importance and weight of each modality image to highlight their respective advantages. For example, CBCT performs well in spatial resolution, but its grayscale resolution is low, and the imaging effect of soft tissue, blood vessels and other structures is not as good as CT and MRI. CBCT has low soft tissue contrast, which limits its effectiveness in some clinical applications.

[0111] S620, aligning the images of different modalities at the same spatial position and superimposing pixels; and ensuring the details of the superimposed images through image post-processing technology.

[0112] Specifically, different modality images are superimposed at the same spatial position to enhance the three-dimensional sense and detail expression of the image.

[0113] In another implementation of this embodiment, the step S600 specifically includes: S630, realizing the alignment and fusion of virtual dynamic three-dimensional images and intraoperative real-time dynamic two-dimensional CBCT images based on a deep learning fusion algorithm. Specifically, a deep learning architecture such as a convolutional neural network (CNN) and a generative adversarial network (GAN) is used to train a fusion model to achieve high-quality multimodal image fusion and improve the clarity and detail of the fused image. Taking GAN as an example, its loss function includes the adversarial loss of the generator and the discriminator.

[0114] Based on any of the above embodiments, it also includes real-time presentation of the fused multimodal image through a dedicated display device terminal (such as a multi-screen display, a virtual reality device, an augmented reality device), supporting multi-angle viewing and dynamic interaction.

[0115] Specifically, dynamic parameter adjustment refers to real-time adjustment of image display parameters (such as contrast, brightness, and viewing angle) to ensure the best display effect.

[0116] Interactive operation means allowing doctors to adjust image display and fusion parameters through gestures, voice, etc. to improve operational convenience.

[0117] Automatic correction refers to automatically correcting the deformation field interpolation parameters according to real-time image changes to ensure the continuity and stability of image fusion.

[0118] Based on the same technical concept, the present application also discloses a virtual dynamic three-dimensional image fusion system based on pre-registration, which can be used to implement any of the above-mentioned virtual dynamic three-dimensional image fusion methods based on pre-registration. Specifically, an embodiment of a virtual dynamic three-dimensional image fusion system based on pre-registration of the present application is as shown in the attached specification. Figure 4 As shown, including:

[0119] The image acquisition module is used to acquire a static three-dimensional image of a target area; and is also used to acquire a dynamic three-dimensional CBCT image of the target area; the dynamic three-dimensional CBCT image includes at least one respiratory cycle.

[0120] Specifically, the static three-dimensional image is generally collected before surgery, including: CT images, MRI images, or X-ray images of the patient's target area. The static three-dimensional image collected before surgery can provide high-resolution anatomical structure information to help doctors plan the surgical path and plan in detail. Of course, the static three-dimensional image can also be collected after surgery, and the virtual dynamicization of the image collected after surgery can simulate the treatment effect. Medical images of different modalities (such as CT, MRI, X-ray, etc.) each have different advantages. Relevant technical personnel should select different types of image data as the static three-dimensional image to be virtually dynamicized according to different processing requirements.

[0121] The dynamic three-dimensional CBCT image is collected before the formal operation of the operation. Since the reconstruction of the three-dimensional CBCT image takes time, the dynamic three-dimensional CBCT image cannot be obtained in real time, but can only be obtained with a delay. The dynamic three-dimensional CBCT image collected in this application includes at least one respiratory cycle, so as to extract the respiratory phase information therefrom.

[0122] The phase division module is used to divide the dynamic three-dimensional CBCT image into respiratory phases based on a preset division density; and obtain the corresponding three-dimensional image frame at each phase node.

[0123] Specifically, the dynamic CBCT image is divided into several respiratory phases, each phase corresponds to a static three-dimensional image, representing different respiratory states. The phase division is based on a preset division density. The division density refers to the number of phase nodes obtained within a respiratory cycle. For example, if the division density is set to 4 and the time / length of a respiratory cycle is 1, the phase node positions obtained by division are: 1 / 4; 2 / 4; 3 / 4; 1.

[0124] Preferably, in some other implementations of this embodiment, the patient's breathing habits can be intelligently analyzed in advance, and key phase nodes that can represent the patient's breathing characteristics can be selected in a more personalized way. The corresponding three-dimensional image frame at each phase node is then registered with the static three-dimensional image to generate a deformation field for each phase.

[0125] The pre-registration module is used to sequentially register the divided multiple three-dimensional image frames with the static three-dimensional image to obtain a single-frame deformation field of each three-dimensional image frame relative to the static three-dimensional image.

[0126] The optional pre-registration module specifically includes: a rigid registration submodule, which uses a rigid registration algorithm to preliminarily align the three-dimensional image frame on the target phase node with the static three-dimensional image to eliminate large-scale displacement and rotation.

[0127] The non-rigid registration submodule uses a non-rigid registration algorithm to finely align images.

[0128] The deformation field calculation submodule is used to generate a single-frame deformation field on the target phase node.

[0129] The storage submodule is used to store and number each of the single-frame deformation fields.

[0130] The breathing acquisition module is used to collect the patient's breathing signals in real time during surgery.

[0131] Specifically, the respiratory signal is collected in real time during surgery, and a corresponding respiratory signal extraction device or a method for real-time extraction from intraoperative images can be used.

[0132] Optionally, the patient's respiratory signal is collected in real time with the help of a respiratory sensor; or, the current respiratory phase information is directly obtained with the help of a respiratory gating device to ensure signal accuracy and real-time performance. Respiratory sensors include chest strap sensors, nasal oxygen cannula sensors, optical respiratory sensors, pressure sensing technology, piezoelectric respiratory sensors, and other types.

[0133] The interpolation calculation module is used to perform interpolation calculation on the single-frame deformation field based on the breathing signal, and generate a complete deformation field in the entire breathing cycle in real time.

[0134] Specifically, in the pre-registration module, the deformation field at each phase node has been obtained by using the pre-registration technology. Assume that any two adjacent phase nodes are n and n+1; according to the current respiratory phase, determine that the deformation field interpolation is between the anchor phases n and n+1. Interpolation algorithms such as linear interpolation, weighted interpolation, and interpolation methods based on machine learning models can generate a complete deformation field in the entire respiratory cycle in real time with a small amount of calculation.

[0135] The dynamic generation module is used to apply the complete deformation field obtained by interpolation to the static three-dimensional image to make the static three-dimensional image virtually dynamic.

[0136] Specifically, the complete deformation field obtained by interpolation is applied to a static three-dimensional image (such as a static CT or MRI image) to make the static three-dimensional image virtually dynamic, thereby generating a real-time dynamic three-dimensional image.

[0137] The present application provides another embodiment of a virtual dynamic three-dimensional image fusion system based on pre-registration. On the basis of the above system embodiment, the virtual dynamic three-dimensional image fusion system based on pre-registration also includes: an image fusion module, which is used to align and fuse the generated virtual dynamic three-dimensional image with the real-time dynamic two-dimensional CBCT image during surgery.

[0138] Specifically, image alignment and fusion can integrate medical image information from different time points and different modalities. This integrated image contains richer anatomical, pathological and functional information, which helps doctors diagnose diseases more accurately and discover some details that may be overlooked in a single image. Medical images of different modalities have their own advantages. For example, CT has a better imaging effect on bone structures, and MRI has clearer imaging of soft tissue structures. Aligning and fusing preoperative multimodal images with intraoperative images can provide doctors with more comprehensive and rich anatomical and pathological information, overcome the technical challenges faced by single image guidance, and make surgical navigation systems more accurate and efficient.

[0139] More preferably, based on any one of the above embodiments, the virtual dynamic three-dimensional image fusion system based on pre-registration also includes: a parameter setting module, used to pre-set the division density of the respiratory phase division; and also used to automatically correct the deformation field interpolation parameters according to real-time image changes.

[0140] Specifically, the parameter setting module is also used to realize dynamic parameter adjustment, interactive operation and automatic correction. Dynamic parameter adjustment refers to real-time adjustment of image display parameters (such as contrast, brightness, and viewing angle) to ensure the best display effect. Interactive operation refers to allowing doctors to adjust image display and fusion parameters through gestures, voice, etc. to improve the convenience of operation. Automatic correction refers to automatically correcting the deformation field interpolation parameters according to real-time image changes to ensure the continuity and stability of image fusion.

[0141] The present application discloses a virtual dynamic three-dimensional image fusion system based on pre-registration. Through pre-registration and deformation field interpolation, it effectively solves the shortcomings of existing 4DCT technology in terms of real-time and low dose, and significantly improves the real-time and accuracy of clinical image fusion. This method not only reduces the computational delay of image fusion and meets the needs of real-time monitoring during surgery, but also reduces the patient's radiation dose and improves clinical safety by optimizing the image acquisition and processing process. It combines the multimodal imaging advantages of CT, MRI and CBCT to provide doctors with richer and more accurate imaging information, enhance the efficiency and reliability of clinical diagnosis and surgical navigation, and has broad application prospects and significant clinical value.

[0142] The virtual dynamic three-dimensional image fusion method and system based on pre-registration of the present application have the same technical concept, and the technical details of the embodiments of the two are applicable to each other. In order to reduce repetition, they will not be repeated here.

[0143] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned program modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program units or modules to complete all or part of the functions described above. The program modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a processing unit, and the above-mentioned integrated unit can be implemented in the form of hardware or in the form of software program units. In addition, the specific names of the program modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0144] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

Claims

1. A virtual dynamic three-dimensional image fusion method based on pre-registration, characterized in that: The steps include: Acquire a static three-dimensional image of a target area, and acquire a dynamic three-dimensional CBCT image of the target area; the dynamic three-dimensional CBCT image includes at least one respiratory cycle; Divide the dynamic three-dimensional CBCT image into respiratory phases based on a preset division density; and obtain a corresponding three-dimensional image frame at each phase node; Sequentially registering the divided 3D image frames with the static 3D image to obtain a single-frame deformation field of each 3D image frame relative to the static 3D image; Collect patient breathing signals in real time during surgery; Based on the breathing signal, interpolation calculation is performed on the single-frame deformation field to generate a complete deformation field in the entire breathing cycle in real time; Applying the complete deformation field obtained by interpolation to the static three-dimensional image to make the static three-dimensional image virtually dynamic; The step of sequentially registering the divided three-dimensional image frames with the static three-dimensional image to obtain a single-frame deformation field of each three-dimensional image frame relative to the static three-dimensional image comprises the following steps: Using a rigid registration algorithm, the three-dimensional image frame at the target phase node is preliminarily aligned with the static three-dimensional image to eliminate large-scale displacement and rotation; Use non-rigid registration algorithm to finely align images; generate single-frame deformation fields at target phase nodes; Each of the single-frame deformation fields is stored and numbered.

2. A virtual dynamic three-dimensional image fusion method based on pre-registration as claimed in claim 1, characterized in that: The real-time acquisition of the patient's respiratory signal during the operation comprises the following steps: Using a respiratory sensor to collect the patient's respiratory signal in real time; Or, real-time dynamic two-dimensional CBCT images of the patient are acquired, and the respiratory signal is extracted from the real-time dynamic two-dimensional CBCT images.

3. The virtual dynamic three-dimensional image fusion method based on pre-registration according to claim 1, characterized in that: The single-frame deformation field is interpolated and calculated based on the breathing signal to generate a complete deformation field in the entire breathing cycle in real time; The steps include: Using a linear interpolation method, between any two adjacent phase nodes, a value of an unknown point is estimated by a linear function; Or, using a weighted interpolation method, adding weights to the plurality of phase nodes that have been divided to estimate the value of the unknown point; Alternatively, use an interpolation method based on a machine learning model, which uses a machine learning algorithm to learn the relationship between data to achieve interpolation.

4. A virtual dynamic three-dimensional image fusion method based on pre-registration according to any one of claims 1 to 3, characterized in that: Also includes: The generated virtual dynamic three-dimensional image is aligned and fused with the real-time dynamic two-dimensional CBCT image during the operation.

5. The virtual dynamic three-dimensional image fusion method based on pre-registration according to claim 4, characterized in that: The method of aligning and fusing the generated virtual dynamic three-dimensional image with the intraoperative real-time dynamic two-dimensional CBCT image comprises the following steps: Setting weights for the virtual dynamic three-dimensional image and the real-time dynamic two-dimensional CBCT image respectively; The images of different modalities are aligned at the same spatial position and pixel-superimposed; the details of the superimposed images are ensured through image post-processing technology.

6. A virtual dynamic three-dimensional image fusion system based on pre-registration, characterized in that: include: An image acquisition module, used to acquire a static three-dimensional image of a target area; Also used to obtain dynamic three-dimensional CBCT images of the target area; The dynamic three-dimensional CBCT image includes at least one respiratory cycle; A phase division module, used to divide the dynamic three-dimensional CBCT image into respiratory phases based on a preset division density; and obtain a corresponding three-dimensional image frame at each phase node; A pre-registration module, used for sequentially registering the divided multiple three-dimensional image frames with the static three-dimensional image to obtain a single-frame deformation field of each three-dimensional image frame relative to the static three-dimensional image; Respiration acquisition module, used to collect patient breathing signals in real time during surgery; An interpolation calculation module, used to perform interpolation calculation on the single-frame deformation field based on the respiratory signal, and generate a complete deformation field in the entire respiratory cycle in real time; A dynamic generation module, used for applying the complete deformation field obtained by interpolation to the static three-dimensional image to make the static three-dimensional image virtually dynamic; Wherein, the pre-registration module specifically includes: A rigid registration submodule, using a rigid registration algorithm, performs preliminary alignment of the three-dimensional image frame at the target phase node with the static three-dimensional image to eliminate large-scale displacement and rotation; The non-rigid registration submodule uses a non-rigid registration algorithm to finely align images; A deformation field calculation submodule is used to generate a single-frame deformation field on a target phase node; The storage submodule is used to store and number each of the single-frame deformation fields.

7. A virtual dynamic three-dimensional image fusion system based on pre-registration as claimed in claim 6, characterized in that: Also includes: The image fusion module is used to align and fuse the generated virtual dynamic three-dimensional image with the real-time dynamic two-dimensional CBCT image during the operation.

8. The virtual dynamic three-dimensional image fusion system based on pre-registration according to claim 6, characterized in that: Also includes: A parameter setting module, used for presetting the division density of the respiratory phase division; It is also used to automatically correct deformation field interpolation parameters according to real-time image changes.

Images