Multi-modal medical image heterogeneous signal synchronous fusion method and device, system

Abstract

The application discloses a multimodal medical image heterogeneous signal synchronous fusion method and device and system, comprising the following steps: S1, acquiring a DSA video stream and an IVUS video stream; S2, performing cross-modal medical image registration based on a space-time Transformer according to the DSA video stream and the IVUS video stream, and realizing medical image heterogeneous signal synchronous fusion. The technical scheme of the application solves the problems of image separation, inaccurate positioning, system delay and the like in the prior art, and improves the precision and safety of interventional surgery.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method, apparatus, and system for synchronous fusion of heterogeneous signals in multimodal medical images. Background Technology

[0002] Currently, intravascular ultrasound (IVUS) and digital subtraction angiography (DSA) are two commonly used imaging techniques in interventional procedures. IVUS provides cross-sectional images of the vessel wall, showing the luminal structure and plaque composition; DSA provides two-dimensional or three-dimensional path images of the vessel, showing blood flow and vessel contours. In existing technologies, IVUS and DSA devices typically operate independently, requiring physicians to observe the two images separately during the procedure and perform "image fusion" in their minds. This has the following drawbacks: 1. Image separation: Doctors need to switch between different screens or devices, which is cumbersome and can easily lead to increased cognitive load; 2. Inaccurate positioning: The position of the IVUS catheter in DSA images is difficult to accurately correspond, especially in areas without obvious anatomical landmarks; 3. Lack of real-time fusion: Existing fusion systems are mostly post-processing systems and cannot provide fused images in real-time or near real-time during surgery; 4. Low system integration: Most systems do not achieve deep integration at the hardware level, and data synchronization relies on common interfaces, resulting in high latency.

[0003] Currently, the proposed solutions include integrating IVUS with other imaging modalities (such as OCT and CTA) or registering IVUS with DSA. However, there are still significant shortcomings in terms of recording time, algorithm accuracy, system architecture, and user interaction, resulting in low clinical applicability. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a method, apparatus, and system for synchronous fusion of heterogeneous signals in multimodal medical images.

[0005] To achieve the above objectives, the present invention provides the following solution: A method for synchronous fusion of heterogeneous signals in multimodal medical images includes: Step S1: Obtain the DSA video stream and IVUS video stream; Step S2: Based on the DSA video stream and IVUS video stream, perform cross-modal medical image registration based on spatiotemporal Transformer to achieve synchronous fusion of heterogeneous medical image signals.

[0006] Preferably, in step S1, the DSA video stream and IVUS video stream are acquired through a protocol-aware acquisition card.

[0007] Preferably, in step S2, the DSA video stream and IVUS video stream are timestamped uniformly at the FPGA layer and synchronized at the GPU layer via CUDA Event.

[0008] The present invention also provides a device for synchronous fusion of heterogeneous signals in multimodal medical images, comprising: The first processing module is used to acquire DSA video streams and IVUS video streams; The second processing module performs cross-modal medical image registration based on the DSA video stream and IVUS video stream, realizing synchronous fusion of heterogeneous medical image signals.

[0009] Preferably, the first processing module acquires DSA video stream and IVUS video stream through a protocol-aware acquisition card.

[0010] Preferably, the second processing module timestamps the DSA video stream and IVUS video stream uniformly at the FPGA layer and synchronizes them at the GPU layer via CUDA Event.

[0011] The present invention also provides a multimodal medical image heterogeneous signal synchronous fusion system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes the multimodal medical image heterogeneous signal synchronous fusion method when executed by the processor.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention can stably and accurately fuse IVUS, DSA, OCT, and ECG images from different brands on the same screen in real time, solving problems such as image separation, inaccurate positioning, and system latency in existing technologies, thereby improving the accuracy and safety of interventional procedures. Attached Figure Description

[0013] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of the method for synchronous fusion of heterogeneous signals in multimodal medical images according to an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] Example 1 like Figure 1 As shown, this invention provides a method for synchronous fusion of heterogeneous signals in multimodal medical images, comprising: Step S1: Obtain the DSA video stream and IVUS video stream; Step S2: Based on the DSA video stream and IVUS video stream, perform cross-modal medical image registration based on spatiotemporal Transformer to achieve synchronous fusion of heterogeneous medical image signals.

[0018] In one embodiment of the present invention, in step S1, DSA video stream and IVUS video stream are acquired through a protocol-aware acquisition card. The protocol-aware acquisition card incorporates device fingerprint recognition logic to automatically identify the brand of the access device (GE / Siemens / Philips) and dynamically load the protocol parsing firmware, avoiding manual configuration.

[0019] The signal acquisition and recognition logic of the protocol-aware acquisition card is a closed-loop automated process of "sensing-recognition-adaptation-acquisition": When a medical device is connected, the acquisition card first senses the presence of the signal and evaluates its quality through the physical layer detection circuit. Then, it starts a multi-dimensional fingerprint recognition engine to extract device identity information in five dimensions in parallel: EDID features, temporal fingerprint, auxiliary data mode, color space characteristics, and electrical signature. This information is then fuzzily matched with the built-in fingerprint databases of brands such as GE, Siemens, and Philips. Once the device brand is identified (confidence > 85%), the system immediately loads the corresponding brand's dedicated protocol parsing firmware from Flash. Through FPGA local reconfiguration technology, it activates a specific private protocol parser, color conversion LUT, and synchronization mode, ultimately establishing an acquisition channel optimized for the device. This enables complete capture of video stream, metadata, and DICOM tags. The entire process requires no manual intervention and completes the fully automated adaptation from physical access to stable acquisition within 3 seconds.

[0020] As one embodiment of the present invention, in step S2, cross-modal medical image registration based on spatiotemporal Transformer captures long-distance spatiotemporal dependencies between different modal images through a self-attention mechanism, thereby achieving end-to-end implicit registration from the pixel level to the semantic level.

[0021] Cross-modal medical image registration based on spatiotemporal Transformer achieves pixel-level to semantic-level alignment through a three-stage end-to-end process of "spatiotemporal encoding - cross-modal interaction - implicit registration": First, DSA video stream and IVUS sequence are input into modality-specific spatiotemporal encoders, and local spatial features are extracted through 3D convolution. Then, the temporal Transformer models the inter-frame dynamic evolution to generate embedded features carrying spatiotemporal context information. Subsequently, the feature sequences of the two modalities are concatenated and fed into the cross-modal Transformer. A bidirectional cross-attention mechanism is used to establish a long-distance dependency between the DSA projection view and the IVUS cross-sectional view. At the same time, differentiable dynamic time warping (DTW) is used to achieve soft temporal alignment at different frame rates, so that the network implicitly learns the correspondence of anatomical structures in the attention weights. Finally, the fused spatiotemporal features are regressed by the registration head to obtain rigid or non-rigid transformation parameters, or the deformation field is directly output, completing the cross-modal registration from the original pixels to high-level semantics in an end-to-end manner without the need for explicit feature extraction, similarity measurement and iterative optimization steps in traditional methods.

[0022] Furthermore, in step S2, the DSA video stream and IVUS video stream are timestamped uniformly at the FPGA layer and synchronized at the GPU layer through CUDA Event to ensure that the display timing difference is less than 1 frame (16ms).

[0023] The DSA-IVUS cross-modal time synchronization mechanism based on an FPGA-GPU heterogeneous architecture achieves sub-millisecond precision through a three-stage pipeline of "hardware timestamp marking - host time alignment - device event synchronization": First, at the FPGA acquisition card level, each DSA and IVUS video stream is equipped with an independent hardware timestamp counter (based on IEEE 1588). The PTP (or Free-Running Counter) embeds a 64-bit nanosecond-level timestamp into the frame metadata or the blanking region of the pixel row at the rising edge of the frame valid signal (DE). Simultaneously, it ensures clock synchronization between multiple cards through the GPS / BeiDou disciplined clock or PTP slave clock on the acquisition card. Subsequently, after being transferred to the host memory via PCIeDMA, the CPU parses the hardware timestamp in the frame metadata, establishes a time mapping table for the two video streams, identifies timing deviations caused by frame rate differences, and calculates the optimal time alignment window. Finally, at the GPU level, the CUDAEvent mechanism is used to create a device-side synchronization point. After asynchronously copying the DSA and IVUS frame data to the video memory, the cudaStreamWaitEvent is used to block the processing kernels of the two streams at the specified event and wait until the data of both sides is ready before starting the forward inference of the spatiotemporal Transformer. This eliminates the asynchronous problems caused by PCIe transmission jitter and GPU scheduling latency, ensuring the spatiotemporal consistency of the cross-modal registration network input.

[0024] This invention supports multimodal signal compatibility. It can acquire high-quality medical signals from different brands and video formats by recognizing the signal protocol. The same framework can be extended to OCT, DSA, IVUS three-modal and even more modal fusion, and can be widely used in catheterization laboratories established at different times.

[0025] Example 2 The present invention also provides a device for synchronous fusion of heterogeneous signals in multimodal medical images, comprising: The first processing module is used to acquire DSA video streams and IVUS video streams; The second processing module performs cross-modal medical image registration based on the DSA video stream and IVUS video stream, realizing synchronous fusion of heterogeneous medical image signals.

[0026] As one embodiment of the present invention, the first processing module acquires DSA video stream and IVUS video stream through a protocol-aware acquisition card.

[0027] As one embodiment of the present invention, the second processing module timestamps the DSA video stream and IVUS video stream uniformly at the FPGA layer and synchronizes them at the GPU layer through CUDA Event.

[0028] Example 3 The present invention also provides a multimodal medical image heterogeneous signal synchronous fusion system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes the multimodal medical image heterogeneous signal synchronous fusion method when executed by the processor.

[0029] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for synchronous fusion of heterogeneous signals in multimodal medical images, characterized in that, include: Step S1: Obtain the DSA video stream and IVUS video stream; Step S2: Based on the DSA video stream and IVUS video stream, perform cross-modal medical image registration based on spatiotemporal Transformer to achieve synchronous fusion of heterogeneous medical image signals.

2. The method for synchronous fusion of heterogeneous signals in multimodal medical images as described in claim 1, characterized in that, In step S1, the DSA video stream and IVUS video stream are acquired through a protocol-aware capture card.

3. The method for synchronous fusion of heterogeneous signals in multimodal medical images as described in claim 1, characterized in that, In step S2, the DSA video stream and IVUS video stream are timestamped uniformly at the FPGA layer and synchronized at the GPU layer via CUDA Event.

4. A device for synchronous fusion of heterogeneous signals in multimodal medical images, characterized in that, include: The first processing module is used to acquire DSA video streams and IVUS video streams; The second processing module performs cross-modal medical image registration based on the DSA video stream and IVUS video stream, thereby achieving synchronous fusion of heterogeneous medical image signals.

5. The multimodal medical image heterogeneous signal synchronous fusion device as described in claim 4, characterized in that, The first processing module acquires DSA and IVUS video streams through a protocol-aware acquisition card.

6. The multimodal medical image heterogeneous signal synchronous fusion device as described in claim 5, characterized in that, The second processing module timestamps the DSA and IVUS video streams at the FPGA layer and synchronizes them at the GPU layer via CUDA Events.

7. A multimodal medical image heterogeneous signal synchronous fusion system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program, when executed by the processor, performs the multimodal medical image heterogeneous signal synchronous fusion method as described in any one of claims 1-3.

Images