Multimodal image real-time registration and fusion system and method for cardiac interventional surgery

Abstract

This invention discloses a multimodal image real-time registration and fusion system and method for cardiac interventional surgery, comprising: a multimodal image acquisition module, an image data transmission module, an intraoperative twin image registration and analysis platform, an algorithm calculation module, a parameter adjustment module, and an image fusion output module. After acquiring image data, the multimodal image acquisition module sends it to the intraoperative twin image registration and analysis platform via the image data transmission module. The platform incorporates a cardiac deformation adaptive optical flow algorithm, a multi-resolution hierarchical mapping algorithm, and a large-deformation cardiac image registration algorithm. The algorithm calculation module performs registration calculations, and the parameter adjustment module optimizes the fusion parameters based on the calculation results. Finally, the image fusion output module generates and outputs the fused image. This invention achieves real-time and accurate multimodal image registration and fusion, providing reliable image support for cardiac interventional surgery and improving surgical accuracy and safety.

Description

Technical Field

[0001] This invention relates to the field of cardiac surgical image registration technology, and in particular to a multimodal image real-time registration and fusion system and method for cardiac interventional surgery. Background Technology

[0002] In interventional cardiac surgery, surgeons rely on clear and precise imaging information to determine the location of cardiac structures, lesions, and instrument manipulation pathways. However, single-modal imaging cannot fully present the dynamic physiological characteristics and anatomical details of the heart. Therefore, combining multimodal imaging has become a key direction for improving surgical precision. As interventional cardiac surgery becomes more minimally invasive and refined, the requirements for real-time imaging and registration accuracy have significantly increased. Specific technologies are needed to achieve efficient registration and fusion of different modalities of imaging, providing coherent and reliable imaging support for intraoperative decision-making. Currently, interventional cardiac surgery involves diverse types of imaging data, and the heart is in a state of continuous dynamic deformation. Traditional image processing methods are insufficient to meet the needs of real-time registration and fusion during surgery. There is an urgent need to construct systems and methods adapted to the dynamic characteristics of the heart, integrating multiple algorithms and dedicated analysis platforms to overcome the technical bottlenecks in the intraoperative application of multimodal imaging.

[0003] Existing technologies for multimodal image processing in cardiac interventional surgery have two significant drawbacks: First, existing image registration algorithms are mostly designed for static or small-deformation scenarios, failing to adequately adapt to the continuous and complex dynamic deformation characteristics of the heart. This leads to registration deviations when processing images of large cardiac deformations, making it difficult to accurately capture structural changes at different stages of cardiac motion, resulting in registration results that do not meet the requirements for precise intraoperative operations. Second, existing technologies lack a dedicated integrated image processing platform for cardiac interventional surgery. Algorithm computation, parameter adjustment, and image fusion are independent processes, resulting in low efficiency in data transmission and collaborative processing between modules. This makes it impossible to achieve real-time registration and fusion of multimodal images, and parameter adjustment lacks deep integration with the surgical scenario, making it difficult to dynamically optimize parameters according to the image requirements of different surgical stages, thus affecting image output quality and surgical assistance effects. Summary of the Invention

[0004] In order to overcome the shortcomings and deficiencies of the existing technology, the present invention provides a multimodal image real-time registration and fusion system and method for cardiac interventional surgery.

[0005] The technical solution adopted in this invention is a multimodal image real-time registration and fusion system for cardiac interventional surgery, comprising: a multimodal image acquisition module for acquiring dynamic image data of different modalities during cardiac interventional surgery, the output of which is connected to the input of an image data transmission module; an image data transmission module for sending the acquired multimodal image data to an intraoperative twin image registration and analysis platform according to a preset transmission protocol, the output of which is connected to the input of the intraoperative twin image registration and analysis platform; and an intraoperative twin image registration and analysis platform, which incorporates a cardiac deformation adaptive optical flow algorithm, a multi-resolution hierarchical mapping algorithm, and a large deformation cardiac image registration algorithm, and receives the transmitted image data. The algorithm processing is initiated afterward, and its output is connected to the input of the algorithm operation module and the parameter adjustment module, respectively. The algorithm operation module calls three algorithms in the intraoperative twin image registration and analysis platform to perform registration operations on the image data, generate intermediate registration data, and its output is connected to the input of the parameter adjustment module. The parameter adjustment module adjusts various parameters of the real-time registration and fusion of multimodal images according to the intermediate data generated by the algorithm operation module, and its output is connected to the input of the image fusion output module. The image fusion output module performs fusion processing on the registered image data according to the adjusted parameters and outputs the fused real-time image. This module receives the parameter data transmitted by the parameter adjustment module.

[0006] Furthermore, the expression for the cardiac deformation adaptive optical flow algorithm is as follows: ,in, Coordinates at time t The optical flow vector of the cardiac image at that location. Coordinates at time t The grayscale value of the image at that location, coordinates The horizontal and vertical offsets, For time increments, Coordinates at time t Image grayscale gradient at that location, It is the inverse of the image gray-level gradient autocorrelation matrix.

[0007] Furthermore, the expression for the multi-resolution hierarchical mapping algorithm is: Lower coordinate ( The mapping value at ) for The size of the weight matrix elements, For the first Coordinates at layer resolution The image value at the location is s, where s is the resolution scaling factor and n is the dimension of the weight matrix.

[0008] Furthermore, the expression for the large-deformation cardiac image registration algorithm is as follows: ,in, coordinates The deformation registration transformation function at the location, , respectively, are the deformation coefficients in the x and y directions. Let M be the deformation basis functions in the x and y directions, respectively, and M be the number of deformation basis functions.

[0009] Furthermore, the image registration evaluation model expression of the intraoperative twin image registration analysis platform is as follows: Where E is the registration evaluation value, To evaluate the weighting coefficients, For reference image With the image to be registered The sum of the squared differences, For reference image With the image to be registered The normalized cross-correlation coefficient, Hausdorff For reference image With the image to be registered The distance to Hausdorf.

[0010] Furthermore, the parameter optimization model expression for real-time registration and fusion of multimodal images is as follows: ,in, Let P be the optimized registration and fusion parameters, where P is the set of parameters to be optimized. To optimize weights, The fused image feature value corresponding to parameter P. To fuse image feature values ​​for the target, Let P be the variance of the fused image feature values.

[0011] Furthermore, the intraoperative twin image registration and analysis platform includes: an image data receiving unit, which receives multimodal image data sent by the image data transmission module, parses the data format, extracts basic information such as image resolution, frame rate, and grayscale range, and transmits the parsed information to the algorithm calling unit; an algorithm calling unit, which, based on the received basic image information, determines the cardiac interventional surgery stage corresponding to the current image data, selects an appropriate algorithm combination from three built-in algorithms, and sends an algorithm calling instruction to the algorithm operation unit; a parameter storage unit, which stores the historical optimal parameters of the three algorithms under different cardiac interventional surgery scenarios and the standard parameter range of multimodal image fusion, receives the adjusted parameters fed back by the parameter adjustment module, and updates the stored content; and a calculation result evaluation unit, which receives the registration intermediate data generated by the algorithm operation module, analyzes the data using preset evaluation indicators, and sends the evaluation results to the parameter adjustment module to provide a basis for parameter adjustment.

[0012] Furthermore, the algorithm operation module includes: a data preprocessing unit, which performs noise filtering on the image data transmitted from the intraoperative twin image registration and analysis platform, uses adaptive filtering to remove random noise from the images, preserves detailed information about the cardiac structure, and transmits the processed data to the algorithm operation unit; an algorithm operation unit, which, according to the instructions of the intraoperative twin image registration and analysis platform, sequentially starts the cardiac deformation adaptive optical flow algorithm, the multi-resolution hierarchical mapping algorithm, and the large deformation cardiac image registration algorithm to perform step-by-step operations on the preprocessed image data and generate intermediate registration data; an operation process monitoring unit, which monitors the data throughput, operation time, and data integrity in real time during the algorithm operation process, and sends an abnormal signal to the intraoperative twin image registration and analysis platform when an operation abnormality occurs, and simultaneously suspends the current operation; and an intermediate data storage unit, which classifies and stores the intermediate registration data generated by the algorithm operation unit, and establishes a data index according to time sequence and operation steps to facilitate the parameter adjustment module to quickly retrieve the required data.

[0013] Furthermore, the parameter adjustment module includes: a parameter receiving unit, which receives intermediate registration data transmitted by the algorithm calculation module and evaluation results sent by the intraoperative twin image registration analysis platform, integrates the data, and extracts feature information related to parameter adjustment; a parameter calculation unit, which calculates the initially adjusted parameter values ​​based on the extracted feature information and the basic parameters of multimodal image real-time registration and fusion, using a combination of linear interpolation and nonlinear correction; a parameter verification unit, which substitutes the initially adjusted parameter values ​​into a preset parameter verification model to determine whether the parameters are within a reasonable range and whether they meet the image registration and fusion requirements of the current cardiac interventional surgery; and a parameter output unit, which transmits the verified parameter values ​​to the image fusion output module and simultaneously feeds back the parameter adjustment results to the parameter storage unit of the intraoperative twin image registration analysis platform for storage.

[0014] A real-time multimodal image registration and fusion method for cardiac interventional surgery, applied to a real-time multimodal image registration and fusion system for cardiac interventional surgery, includes: First, acquiring dynamic image data of different modalities during the cardiac interventional surgery through a multimodal image acquisition module, and transmitting the acquired data to an image data transmission module; Second, the image data transmission module encapsulates the acquired image data according to a preset transmission protocol and sends the encapsulated data to an intraoperative twin image registration and analysis platform; Third, after receiving the data, the intraoperative twin image registration and analysis platform activates its built-in cardiac deformation adaptive optical flow algorithm, multi-resolution hierarchical mapping algorithm, and large-deformation cardiac image registration algorithm, while simultaneously processing the data... The data is transmitted to the algorithm processing module and the parameter adjustment module respectively. In the fourth step, the algorithm processing module calls three algorithms to perform registration operations on the received image data, generating intermediate registration data, which is then transmitted to the parameter adjustment module. In the fifth step, the parameter adjustment module adjusts various parameters of the real-time registration and fusion of multimodal images based on the intermediate registration data and the evaluation results of the intraoperative twin image registration analysis platform, and transmits the adjusted parameters to the image fusion output module. In the sixth step, the image fusion output module performs pixel-level fusion processing on the registered image data based on the adjusted parameters, generating and outputting the fused real-time image, while simultaneously feeding the fusion result back to the intraoperative twin image registration analysis platform for parameter optimization reference.

[0015] Beneficial Effects: This invention proposes a multimodal image real-time registration and fusion system and method for cardiac interventional surgery. The multimodal image acquisition module, image data transmission module, and intraoperative twin image registration and analysis platform are integrated. The intraoperative twin image registration and analysis platform incorporates a cardiac deformation adaptive optical flow algorithm, a multi-resolution hierarchical mapping algorithm, and a large-deformation cardiac image registration algorithm. This allows for targeted adaptation to the continuous dynamic deformation characteristics of the heart, avoiding registration errors caused by existing algorithms that are only applicable to static or small-deformation scenarios. It accurately captures structural changes in the heart at different stages of motion, meeting the registration results required for precise intraoperative operations. Simultaneously, the system integrates the algorithm calculation module, parameter adjustment module, and image fusion output module into a unified intraoperative twin image registration and analysis platform. The block establishes a collaborative processing mechanism through data transmission, improving data transmission and processing efficiency and enabling real-time registration and fusion of multimodal images. Furthermore, the parameter adjustment module, combined with the registration intermediate data generated by the algorithm calculation module and dynamically optimized parameters for the surgical scenario, addresses the issues of independent modules and disconnect between parameter adjustment and the scenario in existing technologies. In addition, the method employs a step-by-step process: a multimodal image acquisition module acquires data, an image data transmission module transmits data, an intraoperative twin image registration and analysis platform calls the algorithm, an algorithm calculation module performs calculations, a parameter adjustment module adjusts parameters, and an image fusion output module outputs the fused image. This further ensures the real-time performance and accuracy of image processing, providing coherent and reliable image support for cardiac interventional surgery and contributing to improved surgical precision and safety. Attached Figure Description

[0016] Figure 1 This is a diagram showing the system module composition of the present invention;

[0017] Figure 2 This is a flowchart of the method steps of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] like Figure 1 As shown, a multimodal image real-time registration and fusion system for cardiac interventional surgery includes:

[0020] The multimodal image acquisition module is used to acquire dynamic image data of different modalities during cardiac interventional surgery. The output of this module is connected to the input of the image data transmission module.

[0021] Specifically, the multimodal image acquisition module serves as the system's data input source, acquiring dynamic image data of different modalities during cardiac interventional surgery. Its implementation adapts to the image acquisition needs of the surgical scenario. In terms of hardware, it adopts image acquisition equipment with high frame rate and high resolution characteristics, with specific parameters set at a frame rate of 30-60 frames / second and a resolution of 1920×1080 pixels. It supports the acquisition of multiple modalities such as CT, MRI, and DSA. During acquisition, the synchronous control module realizes the time-series coordination of multiple devices to ensure the consistency of different modal images in the time dimension. The acquired image data is stored in the DICOM standard format, and the data size of a single frame image is controlled within the range of 5-10MB to avoid data redundancy affecting subsequent transmission efficiency. This module provides complete and accurate raw data support for subsequent registration and fusion. During implementation, the accuracy of the collected data needs to be ensured through equipment calibration. The calibration cycle is set before each operation for each device. The calibration indicators include grayscale accuracy and geometric distortion rate. The grayscale accuracy error must be ≤2% and the geometric distortion rate must be ≤0.5%. After the data is collected, it is directly transmitted to the image data transmission module. Seamless data connection is achieved through the hardware interface between modules. The interface transmission rate is ≥1000Mbps to ensure real-time data transmission without delay.

[0022] The image data transmission module is used to send the acquired multimodal image data to the intraoperative twin image registration and analysis platform according to the preset transmission protocol. Its output end is connected to the input end of the intraoperative twin image registration and analysis platform.

[0023] Specifically, the image data transmission module is responsible for data transfer between the multimodal image acquisition module and the intraoperative twin image registration and analysis platform. During implementation, a dedicated transmission link based on the TCP / IP protocol is used. The transmission protocol has been customized and optimized to improve transmission efficiency while ensuring data integrity. Specific parameters are set as follows: data packet size 1024-2048 bytes; data packet verification using the CRC32 checksum algorithm; and error rate controlled within 10⁻⁻⁶. 6 The transmission bandwidth is dynamically adjusted based on the amount of data collected, with a minimum guaranteed bandwidth of ≥50Mbps and a maximum expandable bandwidth of 1000Mbps to meet the continuous image data transmission needs during surgery. The implementation of this module requires the initial setup and testing of the transmission link. Testing includes transmission latency and packet loss rate, with a transmission latency of ≤100ms and a packet loss rate of ≤0.1%. After successful testing, the data transmission process is initiated. During transmission, the collected DICOM format image data undergoes lightweight encapsulation. The encapsulated data header includes metadata information such as modality identifier, acquisition time, and device number, facilitating rapid parsing by the intraoperative twin image registration and analysis platform. The module's output is connected to the intraoperative twin image registration and analysis platform's input via fiber optic or gigabit Ethernet. Connection stability is ensured through link redundancy design; in the event of a primary link failure, a backup link can switch within 50ms, ensuring uninterrupted data transmission and achieving efficient and reliable transmission of raw image data to the analysis platform, laying the foundation for subsequent algorithm processing.

[0024] The intraoperative twin image registration and analysis platform has built-in cardiac deformation adaptive optical flow algorithm, multi-resolution hierarchical mapping algorithm and large deformation cardiac image registration algorithm. After receiving the transmitted image data, it starts algorithm processing, and its output end is connected to the input end of the algorithm operation module and the parameter adjustment module respectively.

[0025] Specifically, the intraoperative twin image registration and analysis platform is the core control and analysis unit of the system. It incorporates a cardiac deformation adaptive optical flow algorithm, a multi-resolution hierarchical mapping algorithm, and a large-deformation cardiac image registration algorithm. The hardware uses a high-performance server configured with a CPU with 8 cores or more, 32GB or more of memory, and a GPU with 16GB or more of video memory to ensure algorithm computation efficiency. The platform runs on a customized Linux operating system with a system response time of ≤500ms. During implementation, the platform first receives encapsulated data sent by the image data transmission module. It extracts image metadata and pixel information through the built-in data parsing module, with parsing time ≤10ms / frame. Then, it automatically determines the appropriate algorithm combination based on the image modality and surgical stage (such as the puncture stage or stent implantation stage), with an algorithm call response time of ≤200ms. After calling the algorithm, it sends computation instructions to the algorithm computation module and simultaneously transmits basic parameter information to the parameter adjustment module. The parameter transmission adopts a structured data format, with data transmission time ≤50ms. The platform also features a result evaluation function, with evaluation indicators including registration error and data integrity. The registration error evaluation threshold is set to ≤1mm. When the evaluation result does not meet the requirements, a re-calculation instruction is sent to the algorithm calculation module, and the reference data of the parameter adjustment module is updated simultaneously. This module achieves integrated control of algorithm invocation, data analysis, and instruction issuance, ensuring the coordinated operation of all modules in the system. Its implementation requires regular algorithm performance optimization, with an optimization cycle of quarterly and an optimization goal of improving computational efficiency by 5%-10%, ensuring adaptability to the needs of different surgical scenarios.

[0026] The algorithm operation module calls three algorithms within the intraoperative twin image registration and analysis platform to perform registration operations on the image data, generate intermediate registration data, and connects the output end to the input end of the parameter adjustment module.

[0027] Specifically, the algorithm processing module is responsible for executing the registration tasks issued by the intraoperative twin image registration and analysis platform. The hardware uses a dedicated computing chip with a processing frequency of ≥2GHz, a processing cache of ≥8MB, and supports parallel processing. The processing time for a single frame image is ≤300ms. During implementation, the module first receives image data and algorithm instructions transmitted from the intraoperative twin image registration and analysis platform. After receiving the data, it enters the preprocessing stage. The preprocessing uses adaptive filtering technology to remove image noise. The size of the filtering window is dynamically adjusted according to the image grayscale distribution, ranging from 3×3 to 7×7 pixels. After filtering, the signal-to-noise ratio of the image is improved to over 30dB. Then, according to the instructions, three algorithms are started sequentially for step-by-step calculation. During the calculation, a block processing strategy is adopted, dividing the image into 16×16 pixel sub-blocks. The sub-block calculation order is executed from left to right and from top to bottom. After each sub-block calculation is completed, the intermediate result is stored in binary format with a storage rate of ≥50MB / s. The computation status is monitored in real time during the operation, with monitoring indicators including computation temperature and progress. The computation temperature is controlled within the range of -5℃ to 70℃. When the temperature exceeds 65℃, the cooling fan is activated, and the fan speed increases linearly with the temperature, with a maximum speed of 5000 rpm. The computation progress is transmitted in real time to the intraoperative twin image registration and analysis platform via a progress feedback module, with a feedback frequency of once per second. After the computation is completed, intermediate registration data is generated, with a data volume of 1.5-2 times that of the original image data. The intermediate data is then transmitted to the parameter adjustment module using a compressed transmission method with a compression rate of 30%-50% and a transmission time of ≤200ms. This module generates intermediate registration data through precise computation, providing data support for parameter adjustment. The computation accuracy needs to be verified monthly using a standard image dataset, with a verification error of ≤0.8mm to ensure the accuracy of the computation results.

[0028] The parameter adjustment module adjusts various parameters of real-time registration and fusion of multimodal images based on the intermediate data generated by the algorithm calculation module. The output end is connected to the input end of the image fusion output module.

[0029] Specifically, the parameter adjustment module dynamically adjusts the registration fusion parameters of multimodal images in real time based on the registration intermediate data generated by the algorithm calculation module. The module hardware adopts an embedded controller with a processing frequency of ≥1GHz, memory of ≥4GB, and parameter adjustment response time of ≤250ms. During implementation, the module first receives the registration intermediate data transmitted by the algorithm calculation module and the evaluation results sent by the intraoperative twin image registration analysis platform. After receiving the data, it is integrated and processed. During integration, the data is classified according to data type (such as grayscale feature data and deformation feature data). After classification, feature information related to parameter adjustment is extracted. The extracted indicators include grayscale mean and deformation amplitude. The grayscale mean is calculated over the entire image area, and the deformation amplitude is calculated with an accuracy of ≤0.1mm. Subsequently, based on the fundamental parameters of multimodal image registration and fusion (such as fusion weights and resolution matching coefficients), preliminary adjustment parameters are calculated using a combination of linear interpolation and nonlinear correction. The linear interpolation step size is set to 0.01, and the nonlinear correction employs a polynomial fitting method with a fitting order of 2-3. After the preliminary parameters are calculated, they are substituted into a preset parameter verification model. The verification model compares the theoretical fusion effect corresponding to the parameters with the actual surgical requirements to determine whether the parameters are reasonable. The verification time is ≤100ms. If the parameters are unreasonable, they are recalculated, with the number of recalculations ≤3. The parameters that pass verification are stored in XML format and then transmitted to the image fusion output module at a transmission rate ≥200Mbps. Simultaneously, the adjustment results are fed back to the parameter storage unit of the intraoperative twin image registration and analysis platform. The feedback data includes parameter comparison information before and after adjustment. This module ensures the quality of image fusion through precise parameter adjustment. Parameter adjustment accuracy testing is required every six months. The test uses standard images of different modalities, and the test index is the similarity between the fused image and the standard image. The similarity must be ≥90% to ensure that the adjustment effect meets the surgical requirements.

[0030] The image fusion output module performs fusion processing on the registered image data according to the adjusted parameters and outputs the fused real-time image. This module receives parameter data transmitted by the parameter adjustment module.

[0031] Specifically, the image fusion output module, as the system's result output unit, is responsible for fusing and outputting the registered image data based on the parameters transmitted by the parameter adjustment module. The hardware utilizes a high-performance image processing card, supporting multi-channel output with an adjustable output resolution ranging from 1280×720 to 3840×2160 pixels. The output frame rate is consistent with the acquisition frame rate, at 30-60 frames per second. During implementation, the module first receives the adjustment parameters transmitted by the parameter adjustment module and the registered image data transmitted by the algorithm calculation module. After receiving the data, it parses the adjustment parameters, including fusion weight allocation and pixel fusion rules. The fusion weight allocation accuracy is ≤0.001, and the pixel fusion rules are determined based on the image modal characteristics. For example, when fusing CT and MRI images, a weighted average rule based on grayscale values ​​is used, with the weight coefficients executed according to the adjustment parameter settings. Subsequently, pixel-level fusion processing is initiated. During processing, fusion calculations are performed sequentially according to the image pixel coordinates, with each pixel's fusion time ≤1μs. During the fusion process, edge enhancement technology is used to improve the clarity of the heart structure edges. The enhancement intensity is controlled by the edge enhancement coefficient in the adjustment parameters, with a coefficient range of 0.5-1.5. The resulting real-time image after fusion is output simultaneously via HDMI and DVI interfaces. The brightness and contrast of the output signal can be adjusted via a hardware knob, with adjustment ranges of 0-100% for both brightness and contrast. The output image has 256 gray levels and a color reproduction error of ≤5%. The fusion result is stored as a video file in H.265 encoding at a bitrate of 10-20 Mbps. The storage path is automatically named according to the surgery date and patient ID for easy future tracing. This module provides surgeons with clear and accurate fused images to assist in surgical decisions. Before each surgery, an output quality check is required, including checks on image sharpness and color accuracy, to ensure the output image meets clinical observation needs.

[0032] Preferably, the expression for the cardiac deformation adaptive optical flow algorithm is: ,in, Coordinates at time t The optical flow vector of the cardiac image at that location. Coordinates at time t The grayscale value of the image at that location, coordinates The horizontal and vertical offsets, For time increments, Coordinates at time t Image grayscale gradient at that location, It is the inverse of the image gray-level gradient autocorrelation matrix.

[0033] Specifically, the cardiac deformation adaptive optical flow algorithm is used to capture the dynamic deformation features of images during cardiac interventional surgery. Key technical parameters need to be set in conjunction with the physiological characteristics of cardiac motion during implementation. The time increment is set to 0.02-0.05 seconds to match the frame rate of image acquisition during surgery, ensuring accurate reflection of the positional changes within each heartbeat cycle. The monitoring range for lateral and longitudinal offsets is set to 0-5 mm, covering the typical motion amplitude of the heart in the surgical scenario, avoiding optical flow vector calculation errors due to insufficient offset monitoring. The Sobel operator is used to calculate the image grayscale gradient, with an operator template size of 3×3. During calculation, the image grayscale values ​​are processed pixel-by-pixel, with a range of 0-255, conforming to the grayscale standards of conventional medical images. During the inverse matrix calculation of the image grayscale gradient autocorrelation matrix, the matrix dimension remains consistent with the image resolution. For example, for a 1920×1080 pixel image, the matrix dimension is set to 2×2. LU decomposition is used to improve computational efficiency, with decomposition time controlled within 0.01-0.03 seconds. This algorithm provides dynamic deformation basis for subsequent image registration by accurately calculating optical flow vectors. During implementation, gradient calculation accuracy calibration is required before each batch of calculations. The calibration uses a standard dynamic image dataset to ensure that the grayscale gradient calculation error is ≤1% and the optical flow vector calculation error is ≤0.1 mm, thus ensuring the applicability of the algorithm in different surgical scenarios and meeting the high-precision requirements of cardiac interventional surgery for capturing dynamic deformation of images.

[0034] Preferably, the expression for the multi-resolution hierarchical mapping algorithm is: Lower coordinate ( The mapping value at ) for The size of the weight matrix elements, For the first Coordinates at layer resolution The image value at the location is s, where s is the resolution scaling factor and n is the dimension of the weight matrix.

[0035] Specifically, the multi-resolution hierarchical mapping algorithm is used to achieve accurate mapping between images of different resolutions. During implementation, the resolution levels and related parameters need to be set according to the image modal characteristics. The resolution level k is set to 3-5 levels, progressively increasing from low to high resolution. The resolution scaling factor for each level is set to 0.5-0.8 to ensure that mapping efficiency is improved while preserving image details. For example, the first level resolution is 80% of the original image, the second level is 80% of the first level, and so on. The weight matrix dimension n is set to 3 or 5, and the matrix element values ​​are determined according to the Gaussian distribution, ranging from 0.01 to 0.2, ensuring that the weight allocation conforms to the spatial correlation of image pixels. For example, in a 3×3 weight matrix, the weight of the central element is set to 0.2, and the weights of the surrounding elements decrease from the center to the edge, decreasing to 0.01. Image value extraction must correspond to the resolution level. During extraction, bilinear interpolation is used to supplement the pixel values, with interpolation accuracy controlled within 0.5 gray levels to avoid image information loss due to resolution scaling. This algorithm addresses the issue of inconsistent resolution in multimodal images, laying the foundation for subsequent registration. During implementation, error detection is required for the mapping results at each level. The detection metric is the structural similarity of the images before and after mapping, which must be ≥95%. At the same time, the mapping time for each level is controlled to be ≤0.05 seconds to ensure that the real-time requirements of surgery are met and to adapt to the collaborative application of different modal images in surgery.

[0036] Preferably, the expression for the large-deformation cardiac image registration algorithm is: ,in, coordinates The deformation registration transformation function at the location, , respectively, are the deformation coefficients in the x and y directions. Let M be the deformation basis functions in the x and y directions, respectively, and M be the number of deformation basis functions.

[0037] Specifically, the large-deformation cardiac image registration algorithm is used to handle images with large deformations that may occur during cardiac interventional surgery. Implementation requires setting parameters based on the range of cardiac deformation and the algorithm's computational efficiency. The number of deformation basis functions is set to 10-20, and multinomial basis functions are selected to ensure coverage of deformation characteristics in different parts of the heart. For example, for deformation in the ventricular region, the number of basis functions can be set to 20 to improve deformation fitting accuracy. The deformation coefficients in the x and y directions range from -0.5 to 0.5, and the coefficient adjustment step size is set to 0.01. Precise control of deformation registration is achieved through gradual coefficient optimization, avoiding registration oscillations caused by excessive coefficient adjustments. The deformation registration transformation function is calculated iteratively, with 5-10 iterations. After each iteration, the registration error is calculated, and the error threshold is set to ≤0.3 mm. Iteration stops when the error meets the threshold requirement, ensuring registration accuracy while controlling computation time. This algorithm breaks through the limitations of traditional algorithms in adapting to images with small deformations, achieving accurate registration of cardiac images with large deformations. During implementation, a standard dataset simulating large cardiac deformations is used for algorithm verification. The verification includes different deformation scenarios (deformation range of 2-5 mm) to ensure that the registration error is ≤0.5 mm in each scenario and the computation time is ≤0.1 seconds. This meets the needs of fast and accurate registration of images with large deformations during surgery and provides doctors with accurate references for the location of cardiac structures.

[0038] Preferably, the image registration evaluation model expression of the intraoperative twin image registration analysis platform is as follows: Where E is the registration evaluation value, To evaluate the weighting coefficients, For reference image With the image to be registered The sum of the squared differences, For reference image With the image to be registered The normalized cross-correlation coefficient, Hausdorff For reference image With the image to be registered The distance to Hausdorf.

[0039] Specifically, the image registration evaluation model of the intraoperative twin image registration analysis platform is used to quantitatively evaluate the registration effect. During implementation, evaluation parameters need to be set according to the surgical requirements for registration accuracy. Among the evaluation weighting coefficients, the weighting coefficient corresponding to the sum of squared differences is set to 0.3-0.5, the weighting coefficient corresponding to the normalized cross-correlation coefficient is set to 0.3-0.4, and the weighting coefficient corresponding to the Hausdorff distance is set to 0.2-0.3. By reasonably allocating weights, it is ensured that the evaluation results can comprehensively reflect the gray-level similarity and structural similarity of the images. The calculation range of the sum of squared differences is the entire image area. During calculation, the squared differences between the corresponding pixel gray-level values ​​of the reference image and the image to be registered are summed, and the result ranges from 0 to 10. 6 The smaller the value, the higher the grayscale similarity of the images. The normalized cross-correlation coefficient is calculated using a sliding window method, with a window size of 11×11 pixels and a coefficient range of -1 to 1. The closer the value is to 1, the higher the structural similarity of the images. The Hausdorff distance is calculated for key structural contours in the images (such as the edges of heart vessels), with the distance unit being millimeters and a value range of 0-5 millimeters. The smaller the value, the higher the structural contour matching degree. This model provides an objective quantitative standard for the registration effect. During implementation, an evaluation value threshold of ≤50 needs to be set. When the evaluation value exceeds the threshold, the platform is triggered to re-perform the registration calculation. At the same time, the calculation time of the evaluation model is controlled to ≤0.03 seconds to ensure that it does not affect the real-time performance of the surgery and to provide a reliable evaluation basis for subsequent parameter adjustments.

[0040] The preferred expression for the parameter optimization model of real-time registration and fusion of multimodal images is: ,in, Let P be the optimized registration and fusion parameters, where P is the set of parameters to be optimized. To optimize weights, The fused image feature value corresponding to parameter P. To fuse image feature values ​​for the target, Let P be the variance of the fused image feature values.

[0041] Specifically, the multimodal image real-time registration and fusion parameter optimization model is used to select the optimal registration and fusion parameters. During implementation, the parameters need to be set in conjunction with the surgical image quality requirements and computational efficiency. In the optimization weights, the weight coefficient corresponding to eigenvalue deviation is set to 0.6-0.7, and the weight coefficient corresponding to eigenvalue variance is set to 0.3-0.4, prioritizing the consistency between the fused image feature values ​​and the target feature values ​​while controlling the fluctuation range of feature values. The set of parameters to be optimized includes fusion weights, resolution matching coefficients, and edge enhancement coefficients. The fusion weights range from 0 to 1, the resolution matching coefficients range from 0.8 to 1.2, and the edge enhancement coefficients range from 0.5 to 1.5. The parameter adjustment step size is set to 0.01 to ensure the precision of parameter optimization. The extraction of fused image feature values ​​includes grayscale mean, contrast, and edge intensity. The grayscale mean ranges from 50 to 200, the contrast ranges from 10 to 50, and the edge intensity ranges from 0 to 100. The target fused image feature values ​​need to be pre-set according to the surgical modality (e.g., CT and MRI fusion), based on the optimal clinical image quality standards. The variance of the feature values ​​is calculated using the sliding window method, with a window size of 5×5 pixels and a variance range of 0 to 100. The smaller the value, the more stable the feature value. This model achieves intelligent optimization of registration and fusion parameters. During implementation, the number of parameter optimization iterations is set to 3-5 times, with each iteration taking ≤0.08 seconds. After optimization, the quality of the fused image needs to be verified. Quality indicators include structural similarity ≥90% and edge clarity improvement ≥20%, ensuring that the optimized parameters can generate high-quality fused images that meet the clinical observation needs of cardiac interventional surgery.

[0042] Preferably, the intraoperative twin image registration and analysis platform includes: an image data receiving unit, which receives multimodal image data sent by the image data transmission module, parses the data format, extracts basic information such as image resolution, frame rate, and grayscale range, and transmits the parsed information to the algorithm calling unit; an algorithm calling unit, which, based on the received basic image information, determines the cardiac interventional surgery stage corresponding to the current image data, selects an appropriate algorithm combination from three built-in algorithms, and sends an algorithm calling instruction to the algorithm operation unit; a parameter storage unit, which stores the historical optimal parameters of the three algorithms under different cardiac interventional surgery scenarios and the standard parameter range of multimodal image fusion, receives the adjusted parameters fed back by the parameter adjustment module, and updates the stored content; and a calculation result evaluation unit, which receives the registration intermediate data generated by the algorithm operation module, analyzes the data using preset evaluation indicators, and sends the evaluation results to the parameter adjustment module to provide a basis for parameter adjustment.

[0043] Specifically, the intraoperative twin image registration and analysis platform comprises four units, each working collaboratively to process image data and manage algorithms. Implementation requires clearly defining the technical parameters and operating procedures for each unit. The image data receiving unit has a processing rate set to ≥100MB / s. When parsing basic image information, the resolution recognition accuracy must cover the range of 1280×720 to 3840×2160 pixels, the frame rate recognition range is 5-60 frames / second, the grayscale resolution error is ≤2 grayscale levels, and the data transmission latency after parsing is ≤50ms. The algorithm calling unit has a response time ≤100ms for judging the surgical stage. When selecting algorithm combinations, it considers the matching degree between the image modality and the surgical stage, with a matching degree threshold set to ≥85%. The transmission rate for sending instructions to the algorithm processing unit is ≥10Mbps. The parameter storage unit has a storage capacity ≥1TB. Historical best parameters are stored categorized by surgical type with 100% accuracy. The update response time for parameters after adjustment is ≤30ms. Data storage reliability is ensured through a RAID5 array, with a data loss rate ≤10⁻. 9 The evaluation metrics used in the calculation result evaluation unit include registration error and data integrity. The registration error evaluation accuracy is ≤0.1mm, the data integrity verification pass rate must be ≥99.9%, and the time taken to transmit the evaluation result to the parameter adjustment module is ≤40ms. This platform achieves multi-module collaborative control and data coordination. During implementation, performance tests of each unit must be conducted regularly, once a month. Test metrics include processing speed, response time, and storage reliability to ensure that the parameters of each unit meet the set standards, guaranteeing the stable operation of the platform in cardiac interventional surgery and providing core management support for image registration and fusion.

[0044] Preferably, the algorithm operation module includes: a data preprocessing unit, which performs noise filtering on the image data transmitted from the intraoperative twin image registration and analysis platform, uses adaptive filtering to remove random noise in the images, preserves detailed information of the heart structure, and transmits the processed data to the algorithm operation unit; an algorithm operation unit, which, according to the instructions of the intraoperative twin image registration and analysis platform, sequentially starts the cardiac deformation adaptive optical flow algorithm, the multi-resolution hierarchical mapping algorithm, and the large deformation cardiac image registration algorithm to perform step-by-step operations on the preprocessed image data and generate intermediate registration data; an operation process monitoring unit, which monitors the data throughput, operation time, and data integrity in real time during the algorithm operation process, and sends an abnormal signal to the intraoperative twin image registration and analysis platform when an operation abnormality occurs, and simultaneously suspends the current operation; and an intermediate data storage unit, which classifies and stores the intermediate registration data generated by the algorithm operation unit, and establishes a data index according to the time sequence and operation steps to facilitate the parameter adjustment module to quickly retrieve the required data.

[0045] Specifically, the algorithm operation module comprises four units, each performing image data processing and computation step-by-step. Implementation requires clearly defined technical parameters and operational specifications. The data preprocessing unit uses an adaptive filtering window size that can be dynamically adjusted within the range of 3×3 to 7×7 pixels. The filtering rate is ≥50 frames / second, the noise filtering efficiency must be ≥90%, the signal-to-noise ratio of the processed image is improved to ≥35dB, and the data transmission delay to the algorithm operation unit is ≤20ms. The algorithm operation unit initiates the three algorithms sequentially with an interval ≤10ms. The step-by-step computation rate for the preprocessed image is ≥30 frames / second, the error control for each step is ≤0.2mm, and the amount of intermediate registration data generated is 1.2-1.8 times the original data volume. The computation process monitoring unit monitors data throughput with an accuracy ≤1MB / s, computation time monitoring error ≤10ms, data integrity verification frequency is once per frame, the response time for sending signals in case of an anomaly is ≤10ms, and the execution delay for pausing the current computation is ≤5ms. The intermediate data storage unit has a storage rate of ≥80MB / s, a time precision of indexing in chronological order of ≤1ms, an accuracy of classification by calculation steps of ≥99.9%, and a response time of retrieval of data by the parameter adjustment module of ≤30ms. This module enables precise calculation and process control of image data. Daily unit function verification is required during implementation, using a standard image dataset to ensure that filtering effects, calculation accuracy, monitoring reliability, and storage efficiency all meet parameter requirements, providing high-quality intermediate data for subsequent parameter adjustments.

[0046] Preferably, the parameter adjustment module includes: a parameter receiving unit, which receives intermediate registration data transmitted by the algorithm calculation module and evaluation results sent by the intraoperative twin image registration analysis platform, integrates the data, and extracts feature information related to parameter adjustment; a parameter calculation unit, which calculates the initially adjusted parameter values ​​based on the extracted feature information and the basic parameters of real-time multimodal image registration and fusion, using a combination of linear interpolation and nonlinear correction; a parameter verification unit, which substitutes the initially adjusted parameter values ​​into a preset parameter verification model to determine whether the parameters are within a reasonable range and whether they meet the image registration and fusion requirements of the current cardiac interventional surgery; and a parameter output unit, which transmits the verified parameter values ​​to the image fusion output module and simultaneously feeds back the parameter adjustment results to the parameter storage unit of the intraoperative twin image registration analysis platform for storage.

[0047] Specifically, the parameter adjustment module comprises four units, each working collaboratively to adjust and verify parameters. During implementation, the technical parameters and operating standards for each unit must be clearly defined. The parameter receiving unit receives data at a rate ≥60MB / s, with a classification processing delay ≤20ms during data integration. The accuracy of feature extraction includes a grayscale feature error ≤1 grayscale level, a deformation feature error ≤0.1mm, and a feature information extraction completeness rate ≥99.8%. The parameter calculation unit uses a linear interpolation step size of 0.001, a polynomial fitting order of 2-4 for nonlinear correction, a response time ≤80ms for calculating preliminary parameters, and a parameter calculation error ≤0.01. The parameter verification unit uses a cross-validation model with a verification coverage ≥95%, an accuracy of ≤0.02 in determining the reasonable range of parameters, an accuracy rate of ≥99% in determining whether the parameters meet surgical requirements, and a maximum of 3 recalculations, with each recalculation taking ≤50ms. The parameter output unit transmits parameters at a rate ≥20Mbps with a transmission delay ≤15ms. When feeding back the adjustment results to the parameter storage unit of the intraoperative twin image registration and analysis platform, the data format compatibility is ≥99%, and the feedback response time is ≤25ms. This module achieves precise optimization of registration and fusion parameters. During implementation, parameter adjustment accuracy tests need to be conducted weekly using image data from different surgical scenarios to ensure that the parameter adjustments meet the surgical image quality requirements and provide optimal parameter support for image fusion output.

[0048] The cardiac deformation adaptive optical flow algorithm of this invention is used to capture the dynamic deformation features of images during cardiac interventional surgery. By calculating the optical flow vector of cardiac images at different times and coordinates, it reflects the motion trajectory and deformation state of the cardiac structure. In terms of implementation, the algorithm is based on image grayscale values. First, it obtains the image grayscale value at coordinates (x, y) at time t, then calculates the grayscale difference after coordinate shifts between adjacent time points, and combines this with the time increment to obtain the grayscale change rate. Simultaneously, it calculates the image grayscale gradient at that coordinate, and corrects the grayscale change rate using the inverse matrix of the grayscale gradient autocorrelation matrix, finally obtaining the optical flow vector. This algorithm accurately quantifies the dynamic deformation of the heart, providing dynamic motion data support for subsequent image registration and avoiding registration deviations caused by continuous cardiac motion. It overcomes the limitations of traditional optical flow algorithms in adapting to complex cardiac deformation, enabling real-time and accurate capture of subtle movements and large-amplitude deformations of the heart during surgery. This ensures that image registration always closely matches the actual cardiac motion state, providing doctors with accurate image references reflecting the dynamic structure of the heart and helping to improve the accuracy of cardiac interventional surgery.

[0049] The multi-resolution hierarchical mapping algorithm in this invention addresses the problem of resolution inconsistency in multimodal images. By constructing multi-level resolution mapping relationships, it unifies image data of different resolutions to a consistent resolution dimension, achieving multimodal image collaboration at the resolution level. In implementation, the algorithm first sets a resolution level k and a scaling factor s. Starting from the original image (level k-1), it performs weighted calculations on the pixels of the original image using an n×n weight matrix, with the elements of the weight matrix allocated according to pixel spatial correlation. Then, it adjusts the resolution of the weighted image according to the scaling factor to obtain the image mapping value at the k-th resolution, progressively completing the multi-resolution mapping layer by layer. This algorithm eliminates registration barriers caused by resolution differences in multimodal images, enabling registration operations on different modal images under the same resolution benchmark. It breaks through the technical bottleneck of resolution incompatibility in multimodal images, preserving image detail information through hierarchical mapping while improving the collaboration of multimodal image data. This lays the foundation for accurate calculations in subsequent large-deformation registration algorithms, ensuring that multimodal images can be effectively fused to comprehensively present cardiac structural information.

[0050] This invention presents a large-deformation cardiac image registration algorithm specifically designed for accurate registration of images with significant deformation during cardiac interventional surgery. By constructing a deformation registration transformation function, the algorithm corrects the coordinates of large-deformation cardiac images, ensuring structural consistency between the deformed image and the reference image. In terms of implementation, the algorithm centers on coordinate transformation. First, it sets the deformation basis functions and corresponding coefficients in the x and y directions. Then, it constructs the deformation registration transformation function through summation. The coordinates of the image to be registered are substituted into the transformation function to calculate the corrected coordinates, thus achieving image deformation correction. The number and coefficients of the deformation basis functions are dynamically adjusted according to the magnitude of cardiac deformation to ensure coverage of different deformation scenarios. This algorithm addresses the problem of insufficient adaptability of traditional registration algorithms to large-deformation images, achieving accurate registration of cardiac images under large deformation conditions. It ensures that image registration maintains high accuracy even during significant cardiac movements (such as ventricular contraction and relaxation), avoiding structural misalignment caused by deformation, providing surgeons with continuous and accurate cardiac structural images, and reducing surgical risks caused by image registration errors.

[0051] The intraoperative twin image registration and analysis platform of this invention is a control platform integrating algorithm invocation, data processing, and result evaluation. It simulates the intraoperative image processing flow of cardiac interventional surgery, integrating data reception, algorithm control, parameter storage, and result evaluation into a single platform. It incorporates a cardiac deformation adaptive optical flow algorithm, a multi-resolution hierarchical mapping algorithm, and a large-deformation cardiac image registration algorithm. In terms of implementation, the platform first parses the basic information of multimodal images through the image data receiving unit, then the algorithm invocation unit selects the appropriate algorithm combination according to the surgical stage and issues calculation instructions; the parameter storage unit saves historical optimal parameters and standard parameter ranges; the calculation result evaluation unit analyzes intermediate registration data and provides feedback on the evaluation results; and simultaneously, the algorithm calculation module and parameter adjustment module are linked to achieve data interaction. This platform coordinates the collaborative operation of all modules in the system, realizing full-process control of image data from reception and processing to evaluation. Breaking away from the limitations of independent modules and poor coordination in traditional image processing, a "central system" for real-time intraoperative image processing is constructed to ensure efficient connection between algorithm operation, parameter adjustment, and image fusion. This provides stable and reliable control support for real-time registration and fusion of multimodal images, and promotes the integrated and intelligent development of cardiac interventional surgery image processing.

[0052] like Figure 2 As shown, a real-time multimodal image registration and fusion method for cardiac interventional surgery is described. This method is applied to a real-time multimodal image registration and fusion system for cardiac interventional surgery, and includes: First, acquiring dynamic image data of different modalities during the cardiac interventional surgery through a multimodal image acquisition module, and transmitting the acquired data to an image data transmission module; Second, the image data transmission module encapsulates the acquired image data according to a preset transmission protocol, and sends the encapsulated data to an intraoperative twin image registration and analysis platform; Third, after receiving the data, the intraoperative twin image registration and analysis platform activates its built-in cardiac deformation adaptive optical flow algorithm, multi-resolution hierarchical mapping algorithm, and large-deformation cardiac image registration algorithm, while simultaneously... The data is transmitted to the algorithm calculation module and the parameter adjustment module respectively. In the fourth step, the algorithm calculation module calls three algorithms to perform registration calculations on the received image data, generating intermediate registration data, and transmits the intermediate data to the parameter adjustment module. In the fifth step, the parameter adjustment module adjusts the parameters of real-time registration and fusion of multimodal images based on the intermediate registration data and the evaluation results of the intraoperative twin image registration analysis platform, and transmits the adjusted parameters to the image fusion output module. In the sixth step, the image fusion output module performs pixel-level fusion processing on the registered image data based on the adjusted parameters, generates a fused real-time image and outputs it, and simultaneously feeds back the fusion result to the intraoperative twin image registration analysis platform for parameter optimization reference.

[0053] A multimodal image real-time registration and fusion system and method for cardiac interventional surgery is proposed. The system comprises a multimodal image acquisition module, an image data transmission module, and an intraoperative twin image registration and analysis platform. The intraoperative twin image registration and analysis platform incorporates a cardiac deformation adaptive optical flow algorithm, a multi-resolution hierarchical mapping algorithm, and a large-deformation cardiac image registration algorithm. This enables precise matching of the continuous and complex dynamic deformation features of the heart, avoiding registration errors caused by existing algorithms that are only applicable to static or small-deformation scenarios. Through the synergistic effect of these three algorithms, structural changes in the heart at different stages of motion can be accurately captured, ensuring that the registration results meet the requirements of precise intraoperative operation. This solves the problem of insufficient registration accuracy in existing technologies due to the inability of algorithms to adapt to dynamic cardiac deformation.

[0054] This system and method exhibit significant advantages in module collaboration and parameter optimization, effectively addressing the shortcomings of existing technologies where modules are independent and parameter adjustments are disconnected. The system integrates the algorithm computation module, parameter adjustment module, and image fusion output module into a unified intraoperative twin image registration and analysis platform. These modules form a highly efficient collaborative processing mechanism through data transmission, significantly improving data transmission and processing efficiency and enabling real-time registration and fusion of multimodal images. Simultaneously, the parameter adjustment module dynamically optimizes various parameters for real-time multimodal image registration and fusion by combining intermediate registration data generated by the algorithm computation module with the specific surgical scenario. This avoids the problems of independent modules and disconnected parameter adjustments from the surgical scenario found in existing technologies, further ensuring image output quality, providing consistent and reliable image support for cardiac interventional surgery, and contributing to improved overall surgical outcomes.

[0055] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," "link," and "fix" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0056] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal image real-time registration and fusion system for cardiac interventional surgery, characterized in that, include: The multimodal image acquisition module is used to acquire dynamic image data of different modalities during cardiac interventional surgery. Its output is connected to the input of the image data transmission module. The image data transmission module sends the acquired multimodal image data to the intraoperative twin image registration and analysis platform according to a preset transmission protocol. Its output is connected to the input of the intraoperative twin image registration and analysis platform. The intraoperative twin image registration and analysis platform incorporates a cardiac deformation adaptive optical flow algorithm, a multi-resolution hierarchical mapping algorithm, and a large-deformation cardiac image registration algorithm. After receiving the transmitted image data, it initiates algorithm processing, and its output is connected to the algorithm's computational... The module and parameter adjustment module are connected at their input ends; the algorithm calculation module calls three algorithms within the intraoperative twin image registration and analysis platform to perform registration calculations on the image data, generating intermediate registration data, and its output end is connected to the input end of the parameter adjustment module; the parameter adjustment module adjusts various parameters of the real-time registration and fusion of multimodal images based on the intermediate data generated by the algorithm calculation module, and its output end is connected to the input end of the image fusion output module; the image fusion output module performs fusion processing on the registered image data according to the adjusted parameters, outputting the fused real-time image, and this module receives the parameter data transmitted by the parameter adjustment module.

2. The multimodal image real-time registration and fusion system for cardiac interventional surgery according to claim 1, characterized in that, The expression for the cardiac deformation adaptive optical flow algorithm is: ,in, Coordinates at time t The optical flow vector of the cardiac image at that location. Coordinates at time t The grayscale value of the image at that location, coordinates The horizontal and vertical offsets, For time increments, Coordinates at time t Image grayscale gradient at that location, It is the inverse of the image gray-level gradient autocorrelation matrix.

3. The multimodal image real-time registration and fusion system for cardiac interventional surgery according to claim 1, characterized in that, The expression for the multi-resolution hierarchical mapping algorithm is: lower coordinate ( The mapping value at ) for The size of the weight matrix elements, For the first Coordinates at layer resolution The image value at the location is s, where s is the resolution scaling factor and n is the dimension of the weight matrix.

4. The multimodal image real-time registration and fusion system for cardiac interventional surgery according to claim 1, characterized in that, The expression for the large-deformation cardiac image registration algorithm is as follows: ,in, coordinates The deformation registration transformation function at the location, , respectively, are the deformation coefficients in the x and y directions. Let M be the deformation basis functions in the x and y directions, respectively, and M be the number of deformation basis functions.

5. The multimodal image real-time registration and fusion system for cardiac interventional surgery according to claim 1, characterized in that, The image registration evaluation model expression of the intraoperative twin image registration analysis platform is as follows: Where E is the registration evaluation value, To evaluate the weighting coefficients, For reference image With the image to be registered The sum of the squared differences, For reference image With the image to be registered The normalized cross-correlation coefficient, Hausdorff For reference image With the image to be registered The distance to Hausdorf.

6. The multimodal image real-time registration and fusion system for cardiac interventional surgery according to claim 1, characterized in that, The parameter optimization model expression for real-time registration and fusion of multimodal images is as follows: ,in, Let P be the optimized registration and fusion parameters, where P is the set of parameters to be optimized. To optimize weights, The fused image feature value corresponding to parameter P. To fuse image feature values ​​for the target, Let P be the variance of the fused image feature values.

7. The multimodal image real-time registration and fusion system for cardiac interventional surgery according to claim 1, characterized in that, The intraoperative twin image registration and analysis platform includes: an image data receiving unit, which receives multimodal image data sent by the image data transmission module, parses the data format, extracts basic information such as image resolution, frame rate, and grayscale range, and transmits the parsed information to the algorithm calling unit; an algorithm calling unit, which determines the cardiac interventional surgery stage corresponding to the current image data based on the received basic image information, selects an appropriate algorithm combination from three built-in algorithms, and sends an algorithm calling instruction to the algorithm operation unit; a parameter storage unit, which stores the historical optimal parameters of the three algorithms under different cardiac interventional surgery scenarios and the standard parameter range of multimodal image fusion, receives the adjusted parameters fed back by the parameter adjustment module, and updates the stored content; and a calculation result evaluation unit, which receives the registration intermediate data generated by the algorithm operation module, analyzes the data using preset evaluation indicators, and sends the evaluation results to the parameter adjustment module to provide a basis for parameter adjustment.

8. The multimodal image real-time registration and fusion system for cardiac interventional surgery according to claim 1, characterized in that, The algorithm operation module includes: a data preprocessing unit, which filters noise from the image data transmitted by the intraoperative twin image registration and analysis platform, uses adaptive filtering to remove random noise from the images, preserves detailed information about the heart structure, and transmits the processed data to the algorithm operation unit; the algorithm operation unit, according to the instructions of the intraoperative twin image registration and analysis platform, sequentially starts the cardiac deformation adaptive optical flow algorithm, the multi-resolution hierarchical mapping algorithm, and the large deformation cardiac image registration algorithm to perform step-by-step operations on the preprocessed image data and generate intermediate registration data; an operation process monitoring unit, which monitors the data throughput, operation time, and data integrity in real time during the algorithm operation process, and sends an abnormal signal to the intraoperative twin image registration and analysis platform when an operation abnormality occurs, and suspends the current operation; and an intermediate data storage unit, which classifies and stores the intermediate registration data generated by the algorithm operation unit, and establishes a data index according to time sequence and operation steps to facilitate the parameter adjustment module to quickly retrieve the required data.

9. The multimodal image real-time registration and fusion system for cardiac interventional surgery according to claim 1, characterized in that, The parameter adjustment module includes: a parameter receiving unit, which receives intermediate registration data transmitted by the algorithm calculation module and evaluation results sent by the intraoperative twin image registration analysis platform, integrates the data, and extracts feature information related to parameter adjustment; a parameter calculation unit, which calculates the initially adjusted parameter values ​​based on the extracted feature information and the basic parameters of real-time multimodal image registration and fusion, using a combination of linear interpolation and nonlinear correction; a parameter verification unit, which substitutes the initially adjusted parameter values ​​into a preset parameter verification model to determine whether the parameters are within a reasonable range and whether they meet the image registration and fusion requirements of the current cardiac interventional surgery; and a parameter output unit, which transmits the verified parameter values ​​to the image fusion output module and simultaneously feeds back the parameter adjustment results to the parameter storage unit of the intraoperative twin image registration analysis platform for storage.

10. A method for real-time registration and fusion of multimodal images for interventional cardiac surgery, characterized in that, This method is applied to the multimodal image real-time registration and fusion system for cardiac interventional surgery as described in claim 1, comprising: First, acquiring dynamic image data of different modalities during cardiac interventional surgery through a multimodal image acquisition module, and transmitting the acquired data to an image data transmission module; Second, the image data transmission module encapsulates the acquired image data according to a preset transmission protocol, and sends the encapsulated data to an intraoperative twin image registration and analysis platform; Third, after receiving the data, the intraoperative twin image registration and analysis platform activates its built-in cardiac deformation adaptive optical flow algorithm, multi-resolution hierarchical mapping algorithm, and large deformation cardiac image registration algorithm, and simultaneously transmits the data to the algorithms respectively. The process involves six steps: First, the algorithm calculation module calls three algorithms to perform registration calculations on the received image data, generating intermediate registration data, which is then transmitted to the parameter adjustment module. Second, based on the intermediate registration data and the evaluation results from the intraoperative twin image registration analysis platform, the parameter adjustment module adjusts various parameters for real-time registration and fusion of multimodal images, transmitting the adjusted parameters to the image fusion output module. Third, the image fusion output module performs pixel-level fusion processing on the registered image data based on the adjusted parameters, generating and outputting the fused real-time image. Simultaneously, the fusion result is fed back to the intraoperative twin image registration analysis platform for parameter optimization reference.

Images