A cardiovascular multimodal fusion analysis method, system, device and storage medium

Abstract

The present invention belongs to the field of medical equipment, and specifically relates to a cardiovascular multimodal fusion analysis method, system, device and storage medium, wherein the method steps include: acquiring at least two diagnostic images; obtaining corresponding control signals and corresponding image information according to the at least two diagnostic images; obtaining corresponding fused images according to the at least two diagnostic images, the corresponding control signals and the corresponding image information; integrating the corresponding fused images to obtain a fused image; and also providing a system and device using the method. The method provided by the present invention and the system and device using the method have the beneficial effect of being easy to operate, and can enable doctors to diagnose and treat diseases efficiently.

Description

Technical Field

[0001] The present invention belongs to the field of medical equipment, and specifically relates to a cardiovascular multimodal fusion analysis method, system, device and storage medium. Background Art

[0002] Coronary angiography is a common and effective method for diagnosing coronary atherosclerotic heart disease (CHD). It is a relatively safe and reliable invasive diagnostic technique and is considered the "gold standard" for diagnosing CHD. Coronary angiography can observe the degree of stenosis of coronary artery lesions and whether there is collateral circulation, etc. However, because it can only display the lumen contour, it cannot display vascular lesions or stent adhesion, and is easily affected by vascular bending, overlap, and projection angle. Currently common imaging diagnostic methods for coronary heart disease have their advantages and disadvantages, and each imaging device is an independent device. Even if multiple devices are used at the same time, it is impossible to accurately perform a comprehensive and real-time analysis of the same blood vessel segment. Summary of the invention

[0003] In order to solve the above problems existing in the prior art, the present invention provides a cardiovascular multimodal fusion analysis method, system, device and storage medium. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0004] A cardiovascular multimodal fusion analysis method, comprising:

[0005] Acquire at least two diagnostic images;

[0006] Obtaining a corresponding control signal and corresponding image information according to the at least two diagnostic images;

[0007] Obtaining a corresponding fused image according to the at least two diagnostic images, the corresponding control signal and the corresponding image information;

[0008] The corresponding fused images are integrated to obtain a fused image.

[0009] In one embodiment of the present invention, obtaining a corresponding control signal and corresponding image information according to the at least two diagnostic images includes:

[0010] Comparing the at least two diagnostic images with preset thresholds corresponding to the at least two diagnostic images to obtain corresponding control signals;

[0011] Corresponding image information is obtained according to the at least two diagnostic images and the corresponding control signal.

[0012] In one embodiment of the present invention, the control signal includes a start control signal and a stop control signal.

[0013] The present invention also provides a cardiovascular multimodal fusion analysis system, comprising:

[0014] An image acquisition module, used for acquiring at least two diagnostic images;

[0015] A data processing module is used to obtain a corresponding control signal and corresponding image information according to the at least two diagnostic images; to obtain a corresponding fused image according to the at least two diagnostic images, the corresponding control signal and the corresponding image information; and to integrate the corresponding fused images to obtain a fused image.

[0016] In one embodiment of the present invention, a display module is further included, for displaying the fused image.

[0017] In one embodiment of the present invention, the data processing module includes:

[0018] a control signal generating unit, configured to compare the at least two diagnostic images with preset thresholds corresponding to the at least two diagnostic images to obtain a corresponding control signal;

[0019] An image capture unit is used to obtain corresponding image information according to the at least two diagnostic images and the corresponding control signal.

[0020] The present invention also provides a cardiovascular multimodal fusion analysis device, comprising:

[0021] Memory for storing computer programs;

[0022] a processor, the processor being connected to the memory and configured to execute a computer program stored in the memory;

[0023] A display, used for displaying the signal output by the processor;

[0024] When the processor is used to execute the program stored in the memory, the method steps according to any one of claims 1 to 4 are implemented.

[0025] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method steps described in any of the above embodiments are implemented.

[0026] Beneficial effects of the present invention:

[0027] The present invention performs image recognition analysis on the acquired diagnostic image, identifies special marks on the diagnostic image to obtain control signals and image information, and then processes the acquired corresponding diagnostic image according to the control signal and image information to obtain a fused image with the same time start and end point, and integrates the fused image to obtain a fused image. This method can fuse images by directly acquiring diagnostic images, and can fuse data from various devices without changing the original device physics or communication protocol; through image recognition technology, the start and end points of image acquisition are automatically determined without human operation, making the operation easier; by fusing various diagnostic images, doctors can analyze the location of lesions on the same screen, and integrating multiple images allows doctors to diagnose diseases efficiently.

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of a cardiovascular multimodal fusion analysis method provided by an embodiment of the present invention;

[0030] Figure 2 It is a flow chart of the fusion of IVUS diagnostic images and DSA diagnostic images in a cardiovascular multimodal fusion analysis method provided by an embodiment of the present invention;

[0031] Figure 3 A fused image obtained by a cardiovascular multimodal fusion analysis method provided in an embodiment of the present invention based on an IVUS diagnostic image and a DSA diagnostic image;

[0032] Figure 4 It is a flow chart of a cardiovascular multimodal fusion analysis method for fusing OCT diagnostic images with DSA diagnostic images provided by an embodiment of the present invention;

[0033] Figure 5 A fused image obtained by a cardiovascular multimodal fusion analysis method provided in an embodiment of the present invention based on an OCT diagnostic image and a DSA diagnostic image;

[0034] Figure 6 It is a structural block diagram of a cardiovascular multimodal fusion analysis system provided by an embodiment of the present invention;

[0035] Figure 7 It is a structural block diagram of a cardiovascular multimodal fusion analysis device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0037] See also Figure 1 , Figure 1 : is a flowchart of a cardiovascular multimodal fusion analysis method provided by an embodiment of the present invention, comprising:

[0038] Acquire at least two diagnostic images;

[0039] Obtaining a corresponding control signal and corresponding image information according to the at least two diagnostic images;

[0040] Obtaining a corresponding fused image according to the at least two diagnostic images, the corresponding control signal and the corresponding image information;

[0041] The corresponding fused images are integrated to obtain a fused image.

[0042] The present invention performs image recognition analysis on at least two acquired diagnostic images, identifies special marks on the diagnostic images to obtain control signals and image information, and then processes the acquired corresponding diagnostic images according to the control signals and image information to obtain a fused image with a consistent time start and end point, and integrates the fused images to obtain a fused image. This method can fuse images by directly acquiring diagnostic images, and can fuse image data from various devices without changing the original device physics or communication protocol; the image recognition technology can automatically determine the start and end points of image acquisition without the need for human operation, making the operation easier; by fusing various diagnostic images, doctors can analyze and diagnose the location of lesions on the same screen.

[0043] In one embodiment of the present invention, obtaining a corresponding control signal and corresponding image information according to the at least two diagnostic images includes:

[0044] Comparing the at least two diagnostic images with preset thresholds corresponding to the at least two diagnostic images to obtain corresponding control signals;

[0045] Corresponding image information is obtained according to the at least two diagnostic images and the corresponding control signal.

[0046] In one embodiment of the present invention, the control signal includes a start control signal and a stop control signal.

[0047] Specifically, a start mark and an end mark are provided on the diagnostic image, and the image of the corresponding area on the diagnostic image is compared according to a preset threshold, and a start control signal and an end control signal are obtained according to the comparison result. When at least two diagnostic images are captured and recorded by the control signal, the blood vessels corresponding to the diagnostic images can be obtained.

[0048] Furthermore, a fused image of the corresponding blood vessel area on the diagnostic image can be obtained through a start control signal and an end control signal; the fused image is integrated and processed so that the fused image is displayed on the same screen, and the doctor makes a diagnosis by observing the fused images of various related areas displayed on the same screen, so that the doctor can make a more convenient and rapid diagnosis and analysis, thereby improving the doctor's work efficiency.

[0049] In one embodiment of the present invention, see Figure 2 , Figure 2 This is a flow chart of the fusion of IVUS diagnostic images and DSA diagnostic images in a cardiovascular multimodal fusion analysis method provided by an embodiment of the present invention, taking the fusion of intravascular ultrasound (IVUS) images and coronary angiography (DSA) images as an example:

[0050] First, the acquired IVUS diagnostic image is processed using an optical character recognition (OCR) algorithm based on maximum stable extreme region (MSER) ​​and non-maximum suppression (NMS) to determine the acquisition prompt on the IVUS diagnostic image; when the algorithm detects the appearance of an acquisition prompt on the IVUS diagnostic image (taking the Boco IVUS device as an example, "Caputre Mode Imaging" will appear on the diagnostic image), it means that the IVUS device has imaged, and at this time the DSA start control signal begins to control the acquisition of the DSA diagnostic image;

[0051] Then, when a pull-back prompt appears on the IVUS diagnostic image (taking the Boco IVUS device as an example, "Caputre Mode Recording" will appear on the diagnostic image), it means that the IVUS device starts to pull back. At this time, the DSA termination control signal controls the stop of the acquisition of the DSA diagnostic image, thereby obtaining the first group of DSA images, and the starting point information of the first group of DSA images is obtained based on the first group of DSA images; at the same time, an IVUS start control signal is generated, and the IVUS start control signal starts to control the acquisition of the IVUS diagnostic image;

[0052] Secondly, when the pull-back prompt on the IVUS diagnostic image disappears, an IVUS termination control signal is generated, and the acquisition of the IVUS diagnostic image is stopped according to the IVUS termination control signal to obtain the IVUS fusion image, and the DSA start control signal controls the acquisition of the DSA diagnostic image. When the acquired DSA diagnostic image reaches the preset image time, the acquisition of the DSA diagnostic image is stopped to obtain the second group of DSA images, and the termination point information of the second group of DSA is obtained according to the second group of DSA images;

[0053] Thirdly, the starting point information of the first group of DSA images and the ending point information of the second group of DSA images are marked on the acquired DSA diagnostic image to obtain a DSA fusion image;

[0054] Finally, the IVUS fusion image and the DSA fusion image are integrated by frame registration so that each frame of the IVUS fusion image corresponds to a fixed position on the DSA image to obtain the fused image. At this time, the corresponding frames of the DSA fusion image and the IVUS fusion image at the time node can be read by selecting the time node on the time axis. For details, see Figure 3 .

[0055] Furthermore, when the acquisition prompt or the pull-back prompt appears on the IVUS diagnostic image and disappears, a prompt for injecting contrast medium will also appear to prompt the doctor to inject contrast medium into the patient.

[0056] In one embodiment of the present invention, see Figure 4 , Figure 4 This is a flowchart of a cardiovascular multimodal fusion analysis method for fusing OCT diagnostic images with DSA diagnostic images provided by an embodiment of the present invention, taking the fusion of optical coherence tomography (OCT) and coronary angiography (DSA) images as an example:

[0057] First, the acquired OCT diagnostic image is processed. When the absolute value of the pixel difference between the two frames before and after the OCT diagnostic image at a fixed position is greater than the preset threshold, it means that the OCT starts the pull-back operation. At this time, the DSA start control signal controls the acquisition of the DSA diagnostic image to obtain the starting point information of the DSA fusion image.

[0058] Then, when the pixel sum of the current image of the OCT diagnostic image at a fixed position is greater than a given threshold, it means that the OCT diagnostic image starts to be automatically replayed, and at this time, the acquisition of the DSA diagnostic image is stopped to obtain the DSA fusion image and the termination point information of the DSA fusion image; at the same time, an OCT start control signal is generated, and the OCT start control signal controls the acquisition of the OCT diagnostic image; after the OCT diagnostic image is replayed for a cycle, an OCT termination control signal is generated, and the OCT termination control signal terminates the acquisition of the OCT diagnostic image to obtain the OCT fusion image;

[0059] Secondly, the DSA fusion image is marked according to the starting point information and the ending point information of the DSA fusion image, and the coordinates of the pull-back starting point and the pull-back ending point are obtained in turn to obtain the DSA fusion image, and then the OCT pull-back path is obtained;

[0060] Finally, the DSA fusion image and the OCT fusion image are integrated through frame registration so that each frame of the OCT fusion image corresponds to a fixed position on the DSA image to obtain the fused image. At this time, the corresponding frames of the DSA fusion image and the OCT fusion image at the time node can be read by selecting the time node on the time axis. For details, see Figure 5 .

[0061] Furthermore, during the automatic playback of OCT diagnostic images, the data processing module will automatically determine whether a cycle has been played back (the start to end of the playback of the OCT diagnostic image is a cycle). After a cycle ends, the acquisition of the OCT diagnostic image will stop.

[0062] In other embodiments, the fusion of multiple cardiovascular diagnostic images can also be achieved simultaneously.

[0063] Furthermore, the image recognition processing method used in this embodiment is to process the acquired IVUS diagnostic image through an optical character recognition (OCR) algorithm by identifying special prompts, but in other embodiments, it can be through including but not limited to color features, structural features, manual features (HoG) and deep learning-based features.

[0064] It should also be noted that, in other embodiments, fusion operations of more than one diagnostic image may be implemented, and the fusion process steps are the same as the method steps of the above embodiment, which will not be repeated here.

[0065] The present invention also provides a cardiovascular multimodal fusion analysis system, comprising:

[0066] An image acquisition module, used for acquiring at least two diagnostic images;

[0067] A data processing module is used to obtain a corresponding control signal and corresponding image information according to the at least two diagnostic images; to obtain a corresponding fused image according to the at least two diagnostic images, the corresponding control signal and the corresponding image information; and to integrate the corresponding fused images to obtain a fused image.

[0068] In one embodiment of the present invention, a display module is further included, for displaying the fused image.

[0069] In one embodiment of the present invention, see Figure 6 , Figure 6 : is a structural block diagram of a cardiovascular multimodal fusion analysis system provided by an embodiment of the present invention, wherein the data processing module comprises:

[0070] a control signal generating unit, configured to compare the at least two diagnostic images with preset thresholds corresponding to the at least two diagnostic images to obtain a corresponding control signal;

[0071] An image capture unit is used to obtain corresponding image information according to the at least two diagnostic images and the corresponding control signal.

[0072] See also Figure 7 , Figure 7 This is a structural block diagram of a cardiovascular multimodal fusion analysis device provided by an embodiment of the present invention, including:

[0073] Memory for storing computer programs;

[0074] a processor, the processor being connected to the memory and configured to execute a computer program stored in the memory;

[0075] A display, used for displaying the signal output by the processor;

[0076] When the processor is used to execute the program stored in the memory, the method steps described in any of the above embodiments are implemented.

[0077] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method steps described in any of the above embodiments are implemented.

[0078] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A cardiovascular multimodal fusion analysis method, characterized in that: include: Acquire at least two diagnostic images; Obtaining a corresponding control signal and corresponding image information according to the at least two diagnostic images; Specifically, an optical character recognition algorithm based on maximum stable extreme value area and non-maximum suppression is used to perform image processing on the acquired IVUS diagnostic image to determine the acquisition prompt on the IVUS diagnostic image; when the algorithm detects that an acquisition prompt appears on the IVUS diagnostic image, the IVUS device is imaged, and the DSA start control signal starts to control the acquisition of the DSA diagnostic image; when a pull-back prompt appears on the IVUS diagnostic image, the DSA end control signal controls the stop of the acquisition of the DSA diagnostic image, thereby obtaining a first group of DSA images, and the starting point information of the first group of DSA images is obtained based on the first group of DSA images; an IVUS start control signal is generated, and the IVUS start control signal starts to control the acquisition of the IVUS diagnostic image; when the pull-back prompt on the IVUS diagnostic image disappears, an IVUS end control signal is generated, and the acquisition of the IVUS diagnostic image is stopped according to the IVUS end control signal to obtain an IVUS fusion image; the DSA start control signal controls the acquisition of the DSA diagnostic image, and when the acquired DSA diagnostic image reaches a preset image time, the acquisition of the DSA diagnostic image is stopped to obtain a second group of DSA images, and the end point information of the second group of DSA images is obtained based on the second group of DSA images; Obtaining a corresponding fused image according to the at least two diagnostic images, the corresponding control signal and the corresponding image information; specifically, marking the starting point information of the first group of DSA images and the ending point information of the second group of DSA images on the collected DSA diagnostic image to obtain a DSA fused image; Integrating the corresponding fused images to obtain a fused image; Specifically, the IVUS fusion image and the DSA fusion image are integrated by frame registration so that each frame of the IVUS fusion image corresponds to a fixed position on the DSA image to obtain a fused image. At this time, the corresponding frames of the DSA fusion image and the IVUS fusion image at the time node can be read by selecting a time node on the time axis.

2. The cardiovascular multimodal fusion analysis method according to claim 1, characterized in that: Obtaining a corresponding control signal and corresponding image information according to the at least two diagnostic images includes: Comparing the at least two diagnostic images with preset thresholds corresponding to the at least two diagnostic images to obtain corresponding control signals; Corresponding image information is obtained according to the at least two diagnostic images and the corresponding control signal.

3. The cardiovascular multimodal fusion analysis method according to claim 2, characterized in that: The control signal includes: a start control signal and a stop control signal.

4. A cardiovascular multimodal fusion analysis system, characterized in that: For implementing the cardiovascular multimodal fusion analysis method according to any one of claims 1 to 3, the system comprises: An image acquisition module, used for acquiring at least two diagnostic images; A data processing module is used to obtain a corresponding control signal and corresponding image information according to the at least two diagnostic images; to obtain a corresponding fused image according to the at least two diagnostic images, the corresponding control signal and the corresponding image information; and to integrate the corresponding fused images to obtain a fused image.

5. The cardiovascular multimodal fusion analysis system according to claim 4, characterized in that: The device also includes a display module for displaying the fused image.

6. The cardiovascular multimodal fusion analysis system according to claim 4, characterized in that: The data processing module comprises: a control signal generating unit, configured to compare the at least two diagnostic images with preset thresholds corresponding to the at least two diagnostic images to obtain a corresponding control signal; The image capture unit is used to obtain corresponding image information according to the at least two diagnostic images and the corresponding control signal.

7. A cardiovascular multimodal fusion analysis device, characterized in that: include: Memory for storing computer programs; a processor, the processor being connected to the memory and configured to execute a computer program stored in the memory; A display, used for displaying the signal output by the processor; When the processor is used to execute the program stored in the memory, the steps of the method according to any one of claims 1 to 3 are implemented.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images