Fusion angiography method, registration method and device of endoluminal images and angiographic images

Abstract

This application relates to the field of image information processing technology and provides a fusion contrast imaging method, a registration method and apparatus for intracavitary images and contrast images. The fusion contrast imaging method includes: acquiring contrast images and intracavitary images; fusing the contrast images and intracavitary images based on the correspondence between contrast markers and intracavitary markers to obtain a fusion contrast imaging result. This application links contrast images and intracavitary images through contrast markers and intracavitary markers, enabling intracavitary image information and contrast image information to correspond and be fused to obtain a fusion contrast imaging result, thereby overcoming the problem that traditional contrast images contain only a single type of information.

Description

Technical Field

[0001] This application belongs to the field of image information processing technology, and in particular relates to a fusion imaging method, a registration method and apparatus for intracavitary images and imaging images. Background Technology

[0002] Contrast imaging is a very important auxiliary medical examination method, often used to locate lesions. However, traditional contrast imaging contains limited information and has the following drawbacks.

[0003] (1) Traditional angiography data can only be used to observe lesions from a single angle at the same time. If the location of the lesion needs to be accurately determined, the angiography imaging angle needs to be switched, making it difficult to directly and comprehensively observe the location of the lesion.

[0004] (2) Traditional angiography images can only locate the area of ​​stenosis in the cavity, but it is difficult to analyze the cause of stenosis, thus making it impossible to select effective treatment methods and achieve precise PCI treatment.

[0005] (3) Traditionally acquired contrast images have low resolution and cannot be used to identify lesions, plaque types, and other information, making it difficult to perform effective tissue analysis.

[0006] Therefore, how to provide a more comprehensive method for displaying information has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0007] This application provides a method and apparatus for fusion contrast imaging, intracavitary images, and registration of contrast images, which can solve the problem that traditional contrast images contain only a single type of information.

[0008] In a first aspect, embodiments of this application provide a fusion imaging method, including:

[0009] Acquire contrast images and intracavitary images;

[0010] Based on the correspondence between contrast markers and intracavitary markers, the contrast images and intracavitary images are fused to obtain a fused contrast image result;

[0011] The contrast markers are designated points and / or designated areas on the contrast image;

[0012] The intracavitary markings are as follows:

[0013] The designated points and / or designated regions on the intracavitary image; or...

[0014] The set points and / or set areas on the intracavitary image acquisition device;

[0015] The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space.

[0016] In one possible implementation of the first aspect, the fusion imaging result is the result of displaying the intracavitary image information on the imaging image.

[0017] In one possible implementation of the first aspect, the contrast marker is a set point and / or a set area on the intracavitary image acquisition device in the contrast image; the intracavitary marker is a set point and / or a set area on the intracavitary image acquisition device used to determine the intracavitary image acquisition angle.

[0018] In one possible implementation of the first aspect, the step of fusing the contrast image and the intracavitary image based on the correspondence between contrast markers and intracavitary markers to obtain a fused contrast image result includes:

[0019] The acquisition angle of the intracavitary image is determined based on the intracavitary image and the intracavitary markings in the intracavitary image;

[0020] Based on the correspondence between the contrast markers and the intracavitary markers, the acquisition angle of the intracavitary image is determined to be the acquisition area on the contrast image;

[0021] The information from the intracavitary image is fused with the information from the acquisition area to obtain a fused imaging result.

[0022] The above method provides more specific steps for matching intracavitary images and contrast images, enabling more reliable fusion of intracavitary image information and contrast image information.

[0023] In one possible implementation of the first aspect, the contrast image is a standard contrast image;

[0024] The step of fusing information from the intracavitary image with information from the acquisition area to obtain a fused imaging result includes:

[0025] For the set to be mapped, the mapping relationship between image point clouds is calculated; the set to be mapped includes the standard contrast images and non-standard contrast images; the non-standard contrast images refer to one or more specified contrast images that are not the standard contrast images;

[0026] Based on the mapping relationship, the position of the contrast marker in the non-standard contrast image is determined in the standard contrast image;

[0027] Based on the correspondence between the contrast marker of the i-th non-standard contrast image and the intracavitary marker of the i-th intracavitary image, and the position of the contrast marker of the i-th non-standard contrast image on the standard contrast image, the i-th acquisition area corresponding to the acquisition angle of the i-th intracavitary image on the standard contrast image is determined; where i is a positive integer not greater than the number of non-standard contrast images; the intracavitary marker of the i-th intracavitary image and the contrast marker of the i-th non-standard contrast image have a correspondence.

[0028] For all selectable values ​​of i, the information on the i-th intracavitary image is fused with the information of the i-th acquisition area on the standard contrast image to obtain the fused contrast result.

[0029] In one possible implementation of the first aspect,

[0030] Before the step of calculating the mapping relationship between image point clouds for the set to be mapped, the method further includes:

[0031] Intraoperative angiography images are acquired in real time to obtain the j-th intraoperative angiography image, and the j-th intraoperative angiography image is determined as the standard angiography image; where j is a positive integer;

[0032] After the step of fusing the information from the i-th intracavitary image with the information from the i-th acquisition region of the standard contrast image for all selectable values ​​of i to obtain the fused contrast result, the method further includes:

[0033] Intraoperative angiography images are acquired in real time to obtain the (j+k)th intraoperative angiography image, and the (j+k)th intraoperative angiography image is determined as the standard angiography image; where k is the set interval frame number;

[0034] Repeat the step of fusing the information on the intracavitary image with the information of the acquisition area to obtain the fused imaging result.

[0035] In one possible implementation of the first aspect, both the angiographic image and the intraluminal image are acquired for the vascular cavity; the fused angiographic result includes vascular information obtained by running a vascular model with the intraluminal image as input; the vascular model is a semantic segmentation model trained based on vascular samples and vascular labels;

[0036] The vascular information includes any one or a combination of vascular calcification information, endovascular stent information, and vascular lumen cross-sectional area information.

[0037] In one possible implementation of the first aspect, both the angiographic image and the intraluminal image are acquired for the cardiovascular cavity; the fused angiographic result includes the fractional coronary flow reserve and / or the intraluminal image light decay index extracted from the intraluminal image.

[0038] In one possible implementation of the first aspect, after the step of fusing the contrast image and the intracavitary image based on the correspondence between the contrast markers and the intracavitary markers to obtain the fused contrast image result, the method further includes:

[0039] Display the contrast image from the fused contrast imaging result on a designated image display device;

[0040] After the step of displaying the contrast image from the fused contrast imaging result on the designated image display device, the method further includes:

[0041] Upon receiving the first instruction, based on the information type in the first instruction, one or a combination of the following will be superimposed on the image displayed on the image display device: the vascular calcification information, the vascular stent information, and the vascular lumen cross-sectional area information.

[0042] If the second instruction is received, then based on the information type in the second instruction, the coronary artery flow reserve fraction and / or intraluminal image light decay index are superimposed on the image displayed on the image display device.

[0043] Secondly, embodiments of this application provide a method for registering intracavitary images and contrast images, including:

[0044] Acquire a set of contrast-enhanced images and a set of intracavitary images; the set of contrast-enhanced images includes a first contrast-enhanced image and a second contrast-enhanced image.

[0045] In the set of intracavitary images, the intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the first contrast image is identified as the first intracavitary image, and the intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the second contrast image is identified as the second intracavitary image; the set of intracavitary images includes at least one interval intracavitary image; the interval intracavitary image is an intracavitary image acquired within an interval period; the interval period refers to the time period between the acquisition time of the first intracavitary image and the acquisition time of the second intracavitary image;

[0046] The cleavage angiography markers of the marking path are determined, and based on the one-to-one correspondence between the intracavitary markers of the cleavage intracavitary images and the cleavage angiography markers, the information of the cleavage intracavitary images is registered with the information of the set standard angiography images; the marking path is the path between the position corresponding to the angiography marker of the first angiography image and the position corresponding to the angiography marker of the second angiography image on the standard angiography image.

[0047] The ratio of the marker path lengths on both sides of the cleavage angiography marker is the same as the ratio of the interval time period lengths on both sides of the acquisition time of the image within the cleavage cavity.

[0048] The contrast markers are designated points and / or designated areas on the contrast image;

[0049] The intracavitary markings are as follows:

[0050] The designated points and / or designated regions on the intracavitary image; or...

[0051] The set points and / or set areas on the intracavitary image acquisition device;

[0052] The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space.

[0053] In one possible implementation of the second aspect, the contrast marker is a set point and / or a set area on the intracavitary image acquisition device in the contrast image; the intracavitary marker is a set point and / or a set area on the intracavitary image acquisition device used to determine the intracavitary image acquisition angle.

[0054] In one possible implementation of the second aspect, the step of registering the information of the intracavitary image with the information of a set standard contrast image based on the one-to-one correspondence between the intracavitary image's intracavitary marker and the septal angiography marker includes:

[0055] For the set to be mapped, the mapping relationship between image point clouds is calculated; the set to be mapped includes the first contrast image, the second contrast image, and the standard contrast image.

[0056] Based on the mapping relationship, the positions of the contrast markers in the first contrast image and the second contrast image are determined in the standard contrast images.

[0057] In the standard contrast-enhanced image, the marker path is determined based on the contrast marker positions of the first contrast-enhanced image and the second contrast-enhanced image.

[0058] In one possible implementation of the second aspect, the step of calculating the mapping relationship between image point clouds for the set to be mapped includes:

[0059] For the set to be mapped, point cloud matching is performed based on the ICP algorithm to obtain the mapping relationship between point clouds of any two frames of angiography images in the set to be mapped.

[0060] In one possible implementation of the second aspect, the step of determining the marker path in the standard contrast-enhanced image based on the contrast marker positions of the first contrast-enhanced image and the second contrast-enhanced image includes:

[0061] The standard contrast-enhanced image is input into the cavity model to obtain the cavity pixel set in the standard contrast-enhanced image; the cavity model is trained with cavity samples and labels, and is a model that obtains the cavity pixel set by taking the cavity contrast-enhanced image as input.

[0062] In the standard contrast-enhanced images, taking the position of the contrast marker on the first frame of the contrast-enhanced image set as the source point and the position of the contrast marker on the last frame of the contrast-enhanced image set as the endpoint, and using the cavity pixel set as a constraint, the shortest path model is run to obtain the marker path;

[0063] The shortest path model is a model that takes the source point and the destination point as input to obtain the shortest path between the source point and the destination point.

[0064] In one possible implementation of the second aspect, the shortest path model is based on Dijkstra's algorithm.

[0065] In one possible implementation of the second aspect, the cavity model is a model trained based on the Swin-Unet network.

[0066] In one possible implementation of the second aspect, the training loss function of the cavity model is the GDL Loss function.

[0067] Thirdly, embodiments of this application provide a fusion imaging apparatus, comprising:

[0068] The image acquisition module is used to acquire contrast images and intracavitary images;

[0069] The fusion module is used to fuse the contrast image and the intracavitary image based on the correspondence between the contrast markers and the intracavitary markers to obtain a fused contrast image result;

[0070] The contrast markers are designated points and / or designated areas on the contrast image;

[0071] The intracavitary markings are as follows:

[0072] The designated points and / or designated regions on the intracavitary image; or...

[0073] The set points and / or set areas on the intracavitary image acquisition device;

[0074] The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space.

[0075] Fourthly, embodiments of this application provide a registration device for intracavitary images and contrast images, comprising:

[0076] The image acquisition module is used to acquire a set of contrast images and a set of intracavitary images; the set of contrast images includes a first contrast image and a second contrast image.

[0077] An end-matching module is used to determine, within the intracavitary image set, an intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the first contrast image as a first intracavitary image, and an intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the second contrast image as a second intracavitary image; the intracavitary image set includes at least one interval intracavitary image; the interval intracavitary image is an intracavitary image acquired within an interval time period; the interval time period refers to the time period between the acquisition time of the first intracavitary image and the acquisition time of the second intracavitary image;

[0078] The interval matching module is used to determine the interval angiography markers of the marking path, and based on the one-to-one correspondence between the intracavitary markers of the interval intracavitary images and the interval angiography markers, to register the information of the interval intracavitary images with the information of a set standard angiography image; the marking path is the path between the position corresponding to the angiography marker of the first angiography image and the position corresponding to the angiography marker of the second angiography image on the standard angiography image.

[0079] The ratio of the marker path lengths on both sides of the cleavage angiography marker is the same as the ratio of the interval time period lengths on both sides of the acquisition time of the image within the cleavage cavity.

[0080] The contrast markers are designated points and / or designated areas on the contrast image;

[0081] The intracavitary markings are as follows:

[0082] The designated points and / or designated regions on the intracavitary image; or...

[0083] The set points and / or set areas on the intracavitary image acquisition device;

[0084] The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space.

[0085] Fifthly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the fusion imaging method described in any one of the first aspects or the registration method of intracavitary images and imaging images described in any one of the second aspects.

[0086] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the fusion imaging method described in any one of the first aspects or the registration method for intracavitary images and imaging images described in any one of the second aspects.

[0087] In a seventh aspect, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the fusion imaging method described in any one of the first aspects or the registration method of intracavitary images and imaging images described in any one of the second aspects.

[0088] It is understood that the beneficial effects of the third to seventh aspects mentioned above can be referred to the beneficial effects of the solutions in the first or second aspects mentioned above. For details, please refer to the corresponding embodiments, which will not be repeated here. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 This is a schematic flowchart of a fusion imaging method provided in an embodiment of this application;

[0091] Figure 2 This is a schematic flowchart of a registration method for intracavitary images and contrast images provided in an embodiment of this application;

[0092] Figure 3 This is a schematic diagram of the technical process of the multimodal angiography system provided in the embodiments of this application;

[0093] Figure 4 This is a schematic diagram of an application scenario provided in the embodiments of this application;

[0094] Figure 5 This is a schematic diagram of the workflow of the angiography and intracavitary image registration system provided in the embodiments of this application;

[0095] Figure 6 This is a schematic diagram of the cavity model network structure provided in the embodiments of this application;

[0096] Figure 7 This is an example diagram of the imaging image display interface provided in the embodiments of this application;

[0097] Figure 8This is an example diagram of the calcification information display interface provided in the embodiments of this application;

[0098] Figure 9 This is an example diagram of the stent information and lumen area display interface provided in the embodiments of this application;

[0099] Figure 10 This is an example diagram of the VFR display interface provided in the embodiments of this application;

[0100] Figure 11 This is an example diagram of the optical decay index display interface provided in the embodiments of this application;

[0101] Figure 12 This is an abstract example diagram of the display screen provided in the embodiments of this application;

[0102] Figure 13 This is a schematic diagram of the fusion imaging device provided in the embodiments of this application;

[0103] Figure 14 This is a schematic diagram of the structure of the registration device for intracavitary images and contrast images provided in the embodiments of this application;

[0104] Figure 15 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application;

[0105] Figure 16 This is a schematic diagram of the proposed placement location of the stent, as marked by the doctor, provided in an embodiment of this application.

[0106] Figure 17 This is a schematic diagram showing the effect of the doctor-marked stent placement position on a frame of the dynamic fusion angiography result, as provided in the embodiments of this application.

[0107] Reference numerals: Image acquisition module 1301; Fusion module 1302; Set acquisition module 1401; End matching module 1402; Interval matching module 1403; Terminal device 15; Processor 1501; Memory 1502; Computer program 1503. Detailed Implementation

[0108] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0109] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0110] like Figure 1 As shown in the figure, this application provides a fusion imaging method, including:

[0111] Step 102: Obtain contrast images and intracavitary images;

[0112] Step 104: Based on the correspondence between the contrast markers and the intracavitary markers, fuse the contrast image and the intracavitary image to obtain the fused contrast result;

[0113] The contrast markers are designated points and / or designated areas on the contrast image;

[0114] The intracavitary markings are as follows:

[0115] The designated points and / or designated regions on the intracavitary image; or...

[0116] The set points and / or set areas on the intracavitary image acquisition device;

[0117] The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space.

[0118] As an example, not a limitation. Figure 4 This illustrates an optional application scenario for this embodiment and subsequent embodiments: intracavitary images (i.e., intracavitary images) are acquired through an intracavitary imaging device. After the intracavitary image acquisition module of the multimodal DSA (digital subtraction angiography) device acquires the intracavitary images, the intracavitary image data analysis module analyzes them and sends the intracavitary images and their analysis results to the angiography image-intracavitary image fusion module. The angiography image-intracavitary image fusion module fuses the intracavitary images and their analysis results with the angiography images originating from the angiography image acquisition module, and then displays them through the multimodal angiography display module.

[0119] In this optional application scenario, the execution entity of this embodiment and subsequent embodiments can be a DSA device, or a processor located in the DSA device (i.e., calling each module through the processor).

[0120] In one optional implementation, the angiographic image is a digital subtraction angiography image acquired by a DSA device, and the intravascular image is an intravascular ultrasound image (IVUS) and / or an optical coherence tomography (OCT) image.

[0121] It is worth noting that in this embodiment, both the intracavitary image and the angiographic image are acquired based on a specific part of a preset cavity, such as the coronary artery cavity or the intestinal cavity. It can be understood that the image acquisition objects of the intracavitary image and the angiographic image in this embodiment overlap to at least a portion, and the fusion angiography is performed on this overlapping area.

[0122] In an optional implementation, between steps 102 and 104, there is a step of preprocessing the contrast image and the intracavitary image.

[0123] In the case where the angiography marker is a set point on the angiography image, the angiography marker is a point in the cavity. An example of a point angiography marker is an end of a device catheter used to acquire intracavitary images on the angiography image.

[0124] In the case where the angiography marker is a defined area on the angiography image, the angiography marker is an area in the cavity, and the projection of this area on the angiography image may be a line segment, that is, the area may be part of a cross-section of the cavity. An example of an area angiography marker is a cross-section of a device catheter used to acquire intracavitary images on the angiography image.

[0125] It is worth noting that, in this embodiment, although the contrast marker and the contrast image have a one-to-one correspondence, that is, according to the above-mentioned contrast marker setting scheme, there is only one contrast marker in a single contrast image (but the contrast marker may be a set point and a set area, or either of the two), the position of contrast marker a in contrast image A relative to the cavity is fixed and objectively exists. Therefore, it can be understood that any contrast image B that is consistent with the imaging area of ​​contrast image A has a position that corresponds one-to-one with contrast marker a. Furthermore, the position that corresponds one-to-one with contrast marker a in contrast image B and contrast marker a in contrast image A can be mapped to the same point and / or the same area in real physical space. This same point and / or the same area in real physical space may correspond to a certain point or a certain area in the cavity.

[0126] The beneficial effects of this embodiment are as follows:

[0127] By linking contrast images and intracavitary images through contrast markers and intracavitary markers, the intracavitary image information and contrast image information can be correlated and fused to obtain fused contrast imaging results. This overcomes the problem that traditional contrast images contain only one type of information. Based on this fused contrast imaging result, the location of lesions can be directly and comprehensively observed, the area of ​​stenosis in the cavity can be located and the cause of stenosis can be analyzed, and information such as lesion and plaque type can be identified, enabling effective tissue analysis.

[0128] Furthermore, there are two feasible ways to implement the fusion in step 104.

[0129] The first approach is to use an intracavitary image as a base and add information from the contrast image to that intracavitary image to obtain a fused contrast result.

[0130] The second approach is to use an imaging image as a base and add information from the intracavitary image to that imaging image to obtain a fused imaging result.

[0131] The first approach is equivalent to combining angiographic images and intravascular images. During the diagnostic process, the intravascular images are mainly observed to identify the characteristics of the blood vessel lumen, stents, etc. Then, the angiographic images and intravascular images are registered, and finally, the angiographic images are used to achieve treatment (such as PCI treatment).

[0132] However, the first method has the following objective drawbacks:

[0133] 1. Registration and fusion systems are susceptible to noise interference, making it difficult to achieve accurate registration between angiographic images and intracavitary images, which affects auxiliary diagnosis and treatment (such as auxiliary diagnosis and treatment of PCI).

[0134] 2. Existing registration and fusion methods only achieve a one-to-one correspondence between the angiographic image and the intracavitary image location, as well as the correspondence between lesion information and tissue feature information. It still requires the surgeon to rely on subjective judgment and has high requirements for the surgeon's clinical experience. It is not suitable for beginners. Moreover, the existing registration is offline registration and can only register intracavitary image data. In the actual treatment process, only dynamic angiographic images can be used for treatment. At this time, it is difficult to apply the previous registration information to the treatment process based on dynamic angiographic images. Therefore, it will reduce the guiding role of the existing fusion accuracy in surgery (such as PCI surgery).

[0135] Therefore, based on these two issues, in a more preferred solution:

[0136] The fusion imaging result is the result of displaying the intracavitary image information on the imaging image.

[0137] In other words, the precise localization and treatment of lesions need to be achieved on contrast images. However, the first method only uses intracavitary images to achieve tissue analysis of the lesion, which is difficult to enhance the information on the contrast images. The registration of intracavitary images and contrast images depends on the surgeon's subjective judgment. The registration accuracy cannot be guaranteed, and it is difficult to reach the level of effectively guiding the surgeon in the diagnosis and treatment of lesions.

[0138] Therefore, this preferred approach, which focuses on angiography images, provides an integrated, multimodal angiography imaging system that directly fuses and displays vascular tissue information onto the angiography images, thereby enabling dynamic, real-time treatment based on angiography images.

[0139] The beneficial effects of this better solution are:

[0140] By displaying intracavitary images, which are relatively concentrated in a local area of ​​the cavity, on a relatively overall angiographic image, doctors can be provided with more intuitive information during the operation, which is conducive to improving the efficiency of the operation.

[0141] According to the above embodiments, in this embodiment:

[0142] The contrast markers are set points and / or set areas on the intracavitary image acquisition device in the contrast image; the intracavitary markers are set points and / or set areas on the intracavitary image acquisition device used to determine the intracavitary image acquisition angle.

[0143] As an example, not a limitation, the acquisition of intracavitary images is not yet complete at the time of angiography. Therefore, if an intracavitary image acquisition device (such as a laparoscopic catheter) is present in the angiography image, using a set point on the intracavitary image acquisition device, such as one end of the laparoscopic catheter, as an angiography marker can effectively reduce the difficulty of determining the angiography marker. Similarly, using the same set point, i.e., the set point on the intracavitary image acquisition device (such as one end of the laparoscopic catheter), as the intracavitary marker of the intracavitary image, the intracavitary marker of the intracavitary image can be uniquely determined through the acquisition perspective of the intracavitary image, which can also effectively reduce the difficulty of determining the intracavitary marker.

[0144] Based on this, for contrast images and intracavitary images acquired at the same time, the contrast markers and intracavitary markers have a mapping relationship with the same point in real space. Alternatively, it can be understood that the contrast markers and intracavitary markers are mappings of a point in real space to the two different images, the contrast image and the intracavitary image. Through this point in real space, the positional relationship of information in the intracavitary image and the contrast image can be matched relatively easily, thereby achieving fusion.

[0145] The beneficial effects of this embodiment are as follows:

[0146] By matching the acquisition perspective of the intracavitary image with the intracavitary image acquisition device directly displayed in the contrast imaging, the matching of intracavitary image and contrast imaging image can be achieved more efficiently, making the fusion of intracavitary image information and contrast imaging image information easier.

[0147] According to any of the above embodiments, in this embodiment:

[0148] The step of fusing the contrast image and the intracavitary image based on the correspondence between contrast markers and intracavitary markers to obtain the fused contrast image result includes:

[0149] The acquisition angle of the intracavitary image is determined based on the intracavitary image and the intracavitary markings in the intracavitary image;

[0150] Based on the correspondence between the contrast markers and the intracavitary markers, the acquisition angle of the intracavitary image is determined to be the acquisition area on the contrast image;

[0151] The information from the intracavitary image is fused with the information from the acquisition area to obtain a fused imaging result.

[0152] The beneficial effects of this embodiment are as follows:

[0153] It provides more specific matching steps for intracavitary images and contrast images, enabling more reliable fusion of intracavitary image information and contrast image information.

[0154] According to any of the above embodiments, in this embodiment:

[0155] The contrast-enhanced images are standard contrast-enhanced images;

[0156] The step of fusing information from the intracavitary image with information from the acquisition area to obtain a fused imaging result includes:

[0157] For the set to be mapped, the mapping relationship between image point clouds is calculated; the set to be mapped includes the standard contrast images and non-standard contrast images; the non-standard contrast images refer to one or more specified contrast images that are not the standard contrast images;

[0158] Based on the mapping relationship, the position of the contrast marker in the non-standard contrast image is determined in the standard contrast image;

[0159] Based on the correspondence between the contrast marker of the i-th non-standard contrast image and the intracavitary marker of the i-th intracavitary image, and the position of the contrast marker of the i-th non-standard contrast image on the standard contrast image, the i-th acquisition area corresponding to the acquisition angle of the i-th intracavitary image on the standard contrast image is determined; where i is a positive integer not greater than the number of non-standard contrast images; the intracavitary marker of the i-th intracavitary image and the contrast marker of the i-th non-standard contrast image have a correspondence.

[0160] For all selectable values ​​of i, the information on the i-th intracavitary image is fused with the information of the i-th acquisition area on the standard contrast image to obtain the fused contrast result.

[0161] In an optional implementation, the method of this embodiment can be applied to a surgical scenario, in which:

[0162] The acquisition of angiographic images is real-time and continuous. For example, during the surgical period from the start to the end of the surgery, angiographic images are continuously acquired in real time at the angiographic frequency.

[0163] Intracavitary image acquisition is a single-shot acquisition covering the cavity of interest, for example, from the start of the surgery to the end of the intracavitary acquisition period (obviously, the end of the intracavitary acquisition is earlier than the end of the surgery). The acquisition is performed using an intracavitary frequency through a laparoscopic catheter moving at an intracavitary speed. The area between the starting and ending points of the intracavitary catheter's movement is the cavity of interest. During the acquisition process using intracavitary frequency, the acquisition position of each intracavitary image is different (it can be understood that the acquisition position is related to the movement speed of the laparoscopic catheter, the acquisition time, and the starting position). The cavity position corresponding to the information displayed in each intracavitary image is not completely consistent.

[0164] In this implementation, at least two steps are required to fuse information from the intracavitary image into the contrast image.

[0165] The first step is to select an imaging image as the basis for fusion;

[0166] The second step is to determine the corresponding region (i.e., the acquisition area) on the contrast image for the information in each intracavitary image.

[0167] In the first step, considering that the contrast images are acquired in real time, the heartbeat and breathing of the person receiving the surgery and contrast, as well as the slight vibrations of the environment, will cause the cavity pixels in different contrast images to shift, rotate or deform. The fusion process relies on the strict matching of contrast markers and intracavitary markers, and such shifts, rotations or deformations will lead to information distortion.

[0168] For example, assuming there is a correspondence between the contrast marker x in contrast image X and the intracavitary marker y in intracavitary image Y, then fusing the information from intracavitary image Y into contrast image X will not cause information distortion. Conversely, when fusing the information from intracavitary image Y into standard contrast image Z, during the process of synchronously mapping the position of contrast marker x in contrast image X to the position of contrast marker x' in standard contrast image Z, due to the translation, rotation, or deformation of the cavity pixels in standard contrast image Z relative to contrast image X, the position of contrast marker x relative to the cavity in contrast image X is different from the position of contrast marker x' relative to the cavity in standard contrast image Z. This causes information that should have been fused to position A in cavity B to be fused to position B in cavity, resulting in information distortion. Although this distortion is minor, any minor information distortion is not negligible in surgical or diagnostic procedures.

[0169] In other words, changes in the cavity pixel positions of the contrast image (standard contrast image) as the basis for fusion relative to other contrast images may lead to information distortion. To solve this problem, this embodiment calculates the mapping relationship between image point clouds for the set to be mapped, and uses this mapping relationship to correspond the cavity positions in different contrast images.

[0170] Based on the example above, this embodiment modifies the process of "synchronously mapping the position of contrast marker x in contrast image X to the standard contrast image Z as contrast marker x'" as follows:

[0171] The process of synchronously mapping the position of the contrast marker x in the contrast image X relative to the cavity pixel (or the set of cavity pixels, hereinafter the same) to the standard contrast image Z to obtain the contrast marker x' whose position relative to the cavity pixel in the standard contrast image Z is the same as the position of the contrast marker x in the contrast image X relative to the cavity pixel.

[0172] This solved the problem.

[0173] In the second step, for a specific intracavitary image, there exists a one-to-one corresponding contrast image, that is, the contrast markers and intracavitary markers of the two have a corresponding relationship. Through the first step, the point cloud mapping relationship between the contrast image and the standard contrast image is also known. Therefore, we can first determine the position (i.e., acquisition area) of the information of the specific intracavitary image on the one-to-one corresponding contrast image, and then determine the position (i.e., acquisition area) of the information of the specific intracavitary image on the standard contrast image through the point cloud mapping relationship.

[0174] The beneficial effects of this embodiment are as follows:

[0175] By calculating the mapping relationship between image point clouds and determining the acquisition area based on the mapping relationship, it is possible to effectively reduce the cavity position deviation that may exist between multiple contrast images due to breathing or heartbeat, and further enable the information of multiple cavity images to be more accurately fused into a single standard contrast image.

[0176] According to any of the above embodiments, in this embodiment:

[0177] Before the step of calculating the mapping relationship between image point clouds for the set to be mapped, the method further includes:

[0178] Intraoperative angiography images are acquired in real time to obtain the j-th intraoperative angiography image, and the j-th intraoperative angiography image is determined as the standard angiography image; where j is a positive integer;

[0179] After the step of fusing the information from the i-th intracavitary image with the information from the i-th acquisition region of the standard contrast image for all selectable values ​​of i to obtain the fused contrast result, the method further includes:

[0180] Intraoperative angiography images are acquired in real time to obtain the (j+k)th intraoperative angiography image, and the (j+k)th intraoperative angiography image is determined as the standard angiography image; where k is the set interval frame number;

[0181] Repeat the step of fusing the information on the intracavitary image with the information of the acquisition area to obtain the fused imaging result.

[0182] Furthermore, the previous embodiment provided how to eliminate information distortion caused by changes in cavity pixels in standard contrast images and other contrast images. However, in actual surgical applications, there is also the issue of the real-time nature of contrast images, i.e., the surgery is dynamic and the cavity is also dynamic. In this case, the scheme of using a certain standard contrast image as the fusion basis obviously has the risk of not being able to update information in a timely manner, especially the contrast information. Therefore, this embodiment provides a standard contrast image that is updated in real time.

[0183] In practical applications, the time spent in the fusion step may be longer than the acquisition interval between two adjacent contrast images, making it impractical to update each real-time acquired contrast image to a standard contrast image. To address this issue, the real-time acquired contrast images can be updated as standard contrast images at set time intervals. For example, the 3rd, 6th, 9th, 12th, 15th frames, etc., can be updated as standard contrast images in sequence.

[0184] The beneficial effects of this embodiment are as follows:

[0185] By updating standard angiography images in real time, the system provides doctors with real-time intracavitary image fusion information during the procedure, avoiding misjudgments caused by inconsistencies between earlier acquired angiography images and the current angiography image information.

[0186] According to any of the above embodiments, in this embodiment:

[0187] like Figure 7 As shown, after the step of fusing the contrast image and the intracavitary image based on the correspondence between the contrast markers and the intracavitary markers to obtain the fused contrast image result, the method further includes:

[0188] The contrast image from the fused contrast imaging result is displayed on the designated image display device.

[0189] The beneficial effects of this embodiment are as follows:

[0190] By displaying angiographic images, doctors are given more concise information before the operation, and this provides a good foundation for displaying detailed information of each part according to the doctor's instructions later.

[0191] According to any of the above embodiments, in this embodiment:

[0192] Both the angiography image and the intraluminal image are acquired for the vascular cavity; the fused angiography result includes vascular information obtained by running a vascular model with the intraluminal image as input; the vascular model is a semantic segmentation model trained based on vascular samples and vascular labels;

[0193] The vascular information includes any one or a combination of vascular calcification information, endovascular stent information, and vascular lumen cross-sectional area information.

[0194] This embodiment, based on the automatic identification of vascular tissue features from intravascular imaging, enhances multimodal angiography images, and mainly includes:

[0195] 1) Enhancing angiographic images with calcification identification information. Vascular calcification is one of the manifestations of atherosclerosis and is an important reference for doctors in clinical diagnosis. Semantic segmentation technology is applied to intravascular images to identify and obtain vascular calcification information, which is then enriched into the angiographic images. For example... Figure 8 As shown, the multimodal contrast-enhanced images provide a clear overview of the distribution of calcifications in the patient. Doctors can further view calcification information at a specific point in the image by manipulating the mouse; the upper right corner of the screen displays the thickness, angle, and area of ​​the calcification.

[0196] 2) Enhanced angiography images using information on metal stent identification, lumen area, and lumen thickness. Adding information on the metal stent and lumen identified in the intraluminal images to the angiography images not only assists the surgeon in accurately selecting stent placement but also shows the stent's position after implantation, facilitating subsequent stent posterior expansion and aiding in postoperative stent analysis. For example... Figure 9 As shown, the green curve is mapped proportionally to the size of the intraluminal lumen on the multimodal angiography image, and it displays the lumen area and minimum lumen area at the current mouse position. The yellow line segment indicates the location of the minimum lumen on the angiography image, and the blue line segment marks the lumen area reference point to help doctors determine the stent placement location. Doctors can also use the mouse to manually modify the position of the marked point and set a new lumen area reference point. The grid area in the figure marks the existing stent locations in the angiography image, facilitating postoperative follow-up for patients.

[0197] The beneficial effects of this embodiment are as follows:

[0198] By performing intermediate processing on the intracavitary images, namely extracting vascular information from the cavity images using a vascular model, more comprehensive information is provided for the fusion angiography results.

[0199] According to any of the above embodiments, in this embodiment:

[0200] Both the angiographic images and the intravascular images are acquired from the cardiovascular cavity; the fused angiographic results include the fractional coronary flow reserve and / or the intravascular image light decay index extracted from the intravascular images.

[0201] In an optional implementation, the fractional flow reserve (FFR) is a value obtained by fitting intravascular images, i.e., VFR (Virtual Flow Reserve).

[0202] This embodiment enhances multimodal angiography images based on intracavitary image computation information, mainly including:

[0203] 1) Virtual Flow Reserve (VFR) for Enhanced Angiographic Images. The fractional flow reserve (FFR) of the coronary arteries, as a functional assessment indicator, plays a crucial role in guiding treatment strategies for coronary artery disease. Using intravascular imaging to fit an FFR value, termed VFR, enhances angiographic image information. For example... Figure 7As shown, VFR information is displayed on the angiography image according to color levels. Not only can you view the VFR value throughout the entire pull-back process, but you can also drag the mouse to view the VFR value at any position in the intracavitary image pull-back segment, thereby effectively determining whether the patient needs stent treatment.

[0204] 2) Intracavitary imaging light decay index, enhancing contrast images. The light decay coefficient of the image, automatically calculated through a single intracavitary image pull-back scan, can be used to observe the instability of plaques, locate unstable plaque regions, and thus provide guidance for lesion assessment in patients. Displaying the light decay index on contrast images allows for continuous observation of changes in plaque stability, helping physicians provide more precise medical guidance to patients from a holistic perspective. Contrast images in this modality include... Figure 8 As shown, by moving the cursor, the doctor can view the light decay index corresponding to a certain point in the blood vessel cavity.

[0205] The beneficial effects of this embodiment are as follows:

[0206] By adding the coronary flow reserve fraction and / or intravascular image light decay index obtained through further calculations to the fusion angiography results, more comprehensive information is provided.

[0207] According to any of the above embodiments, in this embodiment:

[0208] After the step of displaying the contrast image from the fused contrast imaging result on the designated image display device, the method further includes:

[0209] Upon receiving the first instruction, based on the information type in the first instruction, one or a combination of the following will be superimposed on the image displayed on the image display device: the vascular calcification information, the vascular stent information, and the vascular lumen cross-sectional area information.

[0210] For example, upon receiving a first instruction indicating a type of vascular calcification, the image displayed by the image display device can be generated by... Figure 7 Become Figure 8 Upon receiving the first instruction containing stent information and vascular lumen cross-sectional area information, the image displayed by the image display device can be generated by... Figure 7 Become Figure 9 ;

[0211] If the second instruction is received, then based on the information type in the second instruction, the coronary artery flow reserve fraction and / or intraluminal image light decay index are superimposed on the image displayed on the image display device.

[0212] For example, upon receiving a second instruction of type VFR, the image displayed by the image display device can be determined by... Figure 7 Become Figure 10Upon receiving a second instruction with a light attenuation type, the image displayed by the image display device can be... Figure 7 Become Figure 11 ;

[0213] Figure 12 An abstract schematic diagram is shown showing an image display device displaying an image after receiving a first instruction or a second instruction;

[0214] in:

[0215] Unfilled enclosed lines represent abstract angiographic images of blood vessels;

[0216] The enclosed lines filled with diagonal lines represent the information superimposed and displayed after receiving the first or second instruction, i.e., the information area. It's worth noting that although... Figure 12 The information area in the image only covers a small part of the blood vessel, but in practical applications, the information area can also cover most or all of the blood vessel (e.g., the information area used to display the light decay index or VFR).

[0217] The triangular marker refers to the cursor that is moved by the doctor or other operator;

[0218] The detailed information display area at the cursor location is an area that displays detailed parameters of the cursor location based on the information type of the first or second instruction. For example, after receiving a first instruction with the type of vascular calcification, the detailed information display area at the cursor location can display the calcification thickness, calcification angle, calcification area, etc. at the cursor location.

[0219] The beneficial effects of this embodiment are as follows:

[0220] Based on instructions from doctors or other operators, accurately displaying the data that doctors or other operators need and are concerned about in relatively concise imaging information can provide a more efficient data basis for determining and adjusting surgical plans.

[0221] This application also provides a method for registering intracavitary images and contrast images. The method for registering intracavitary images and contrast images and the method for fusing contrast images provided in this application are based on the same inventive concept and can be used in combination. For example, the registration method for registering intracavitary images and contrast images can be used to register intracavitary images and contrast images, and then fusing contrast images can be performed based on the registered intracavitary images and contrast images to obtain the fusing contrast image result.

[0222] Specifically, such as Figure 2 As shown, this embodiment provides a method for registering intracavitary images and contrast images, including:

[0223] Step 202: Obtain a set of contrast images and a set of intracavitary images; the set of contrast images includes a first contrast image and a second contrast image.

[0224] As an example and not a limitation, in step 202, the angiographic image is a digital subtraction angiography image acquired by a DSA device, and the intravascular image is an intravascular ultrasound image (IVUS) and / or an optical coherence tomography (OCT) image.

[0225] It is worth noting that in some application scenarios, the set of contrast images consists of multiple frames of contrast images acquired continuously. However, the first and second contrast images are not necessarily consecutively acquired adjacent frames. That is, there may be one or more frames of contrast images between the first and second contrast images.

[0226] Step 204: In the set of intracavitary images, determine the intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the first contrast image as the first intracavitary image, and determine the intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the second contrast image as the second intracavitary image;

[0227] In an optional implementation, the one-to-one correspondence in step 204 can be understood as:

[0228] The intracavitary image set includes a first intracavitary image, a second intracavitary image, and at least one septal intracavitary image; the acquisition time of the first contrast image and the first intracavitary image is a first time, and the acquisition time of the second contrast image and the second intracavitary image is a second time; the acquisition time of the septal intracavitary image is between the first time and the second time.

[0229] The intracavitary image set includes at least one interval intracavitary image; the interval intracavitary image is an intracavitary image acquired within an interval period; the interval period refers to the time period between the acquisition time of the first intracavitary image and the acquisition time of the second intracavitary image;

[0230] In a preferred embodiment, the acquisition frequency of intracavitary images is higher than that of contrast images, resulting in some special intracavitary images. These special intracavitary images do not have a one-to-one correspondence with the contrast images acquired at the same time. However, even if there is no one-to-one correspondence with the contrast images, these special intracavitary images still need to be registered to a certain contrast image, which is an important problem that this embodiment can solve.

[0231] Step 206: Determine the cleavage angiography markers of the marking path, and based on the one-to-one correspondence between the intracavitary markers of the cleavage intracavitary images and the cleavage angiography markers, register the information of the cleavage intracavitary images with the information of the set standard angiography images; the marking path is the path between the position corresponding to the angiography marker of the first angiography image and the position corresponding to the angiography marker of the second angiography image on the standard angiography image.

[0232] Specifically, the standard contrast image in step 206 can be either an element in the set of contrast images or the most recently acquired contrast image.

[0233] Furthermore, the step of registering the information of the intracavitary image with the information of the set standard contrast image can be either registering the contrast image information to the intracavitary image or registering the intracavitary image information to the contrast image.

[0234] The ratio of the marker path lengths on both sides of the cleavage angiography marker is the same as the ratio of the interval time period lengths on both sides of the acquisition time of the image within the cleavage cavity.

[0235] It is understandable:

[0236] The cleavage angiography marker divides the marking path into two parts, denoted as the first path (from the starting point of the marking path to the cleavage angiography marker) and the second path (from the angiography marker to the ending point of the marking path). The ratio of the lengths of the marking paths on both sides of the cleavage angiography marker is the ratio of the first path to the second path.

[0237] The acquisition time of the image inside the interval cavity divides the interval time period into two parts, denoted as the left time period (from the acquisition time of the first image inside the cavity to the acquisition time of the image inside the interval cavity) and the right time period (from the acquisition time of the image inside the interval cavity to the acquisition time of the second image inside the cavity). The ratio of the length of the interval time period on both sides of the acquisition time of the image inside the interval cavity is the ratio of the left time period to the right time period.

[0238] The contrast markers are designated points and / or designated areas on the contrast image;

[0239] The intracavitary markings are as follows:

[0240] The designated points and / or designated regions on the intracavitary image; or...

[0241] The set points and / or set areas on the intracavitary image acquisition device;

[0242] The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space.

[0243] In one optional implementation, the execution entity of this embodiment and subsequent embodiments may be a DSA device, a processor located in the DSA device (i.e., calling each module through the processor), or a computer processor.

[0244] The beneficial effects of this embodiment are as follows:

[0245] By linking contrast-enhanced images and intracavitary images through contrast markers and intracavitary markers, the intracavitary image information and contrast-enhanced image information can be matched efficiently. On this basis, the septal contrast markers corresponding to the septal intracavitary images are further determined by the marker path, so that the septal intracavitary images can be effectively matched to the standard contrast-enhanced images even when there are no strictly corresponding contrast-enhanced images.

[0246] According to the above embodiment of the registration method between intracavitary images and contrast images, in this embodiment:

[0247] The contrast markers are set points and / or set areas on the intracavitary image acquisition device in the contrast image; the intracavitary markers are set points and / or set areas on the intracavitary image acquisition device used to determine the intracavitary image acquisition angle.

[0248] As an example, not a limitation, the acquisition of intracavitary images is not yet complete at the time of angiography. Therefore, if an intracavitary image acquisition device (such as a laparoscopic catheter) is present in the angiography image, using a set point on the intracavitary image acquisition device, such as one end of the laparoscopic catheter, as an angiography marker can effectively reduce the difficulty of determining the angiography marker. Similarly, using the same set point, i.e., the set point on the intracavitary image acquisition device (such as one end of the laparoscopic catheter), as the intracavitary marker of the intracavitary image, the intracavitary marker of the intracavitary image can be uniquely determined through the acquisition perspective of the intracavitary image, which can also effectively reduce the difficulty of determining the intracavitary marker.

[0249] Based on this, for contrast images and intracavitary images acquired at the same time, the contrast markers and intracavitary markers have a mapping relationship with the same point in real space. Alternatively, it can be understood that the contrast markers and intracavitary markers are mappings of a point in real space to the two different images, the contrast image and the intracavitary image. Through this point in real space, the positional relationship of information in the intracavitary image and the contrast image can be matched relatively easily, thereby achieving registration.

[0250] The beneficial effects of this embodiment are as follows:

[0251] By matching the acquisition perspective of the intracavitary image with the intracavitary image acquisition device directly displayed in the contrast imaging, the matching of intracavitary image and contrast imaging image can be achieved more efficiently, making the fusion of intracavitary image information and contrast imaging image information easier.

[0252] According to any embodiment of the above-described registration method for intracavitary images and contrast images, in this embodiment:

[0253] The step of registering the information of the intracavitary image with the information of a set standard contrast image based on the one-to-one correspondence between the intracavitary image and the septal angiography mark includes:

[0254] For the set to be mapped, the mapping relationship between image point clouds is calculated; the set to be mapped includes the first contrast image, the second contrast image, and the standard contrast image.

[0255] In an alternative implementation, the standard contrast image is either the first contrast image itself or the second contrast image itself.

[0256] Based on the mapping relationship, the positions of the contrast markers in the first contrast image and the second contrast image are determined in the standard contrast images.

[0257] In the standard contrast-enhanced image, the marker path is determined based on the contrast marker positions of the first contrast-enhanced image and the second contrast-enhanced image.

[0258] In an optional implementation, the method of this embodiment can be applied to a surgical scenario, in which:

[0259] The acquisition of angiographic images is real-time and continuous. For example, during the surgical period from the start to the end of the surgery, angiographic images are continuously acquired in real time at the angiographic frequency.

[0260] Intracavitary image acquisition is a single-shot acquisition covering the cavity of interest, for example, from the start of the surgery to the end of the intracavitary acquisition period (obviously, the end of the intracavitary acquisition is earlier than the end of the surgery). The acquisition is performed using an intracavitary frequency through a laparoscopic catheter moving at an intracavitary speed. The area between the starting and ending points of the intracavitary catheter's movement is the cavity of interest. During the acquisition process using intracavitary frequency, the acquisition position of each intracavitary image is different (it can be understood that the acquisition position is related to the movement speed of the laparoscopic catheter, the acquisition time, and the starting position). The cavity position corresponding to the information displayed in each intracavitary image is not completely consistent.

[0261] In this embodiment, at least two steps are required to register the information in the intracavitary image with the contrast image.

[0262] The first step is to select an imaging image as the registration basis;

[0263] The second step is to determine the corresponding region (i.e., the acquisition area) on the contrast image for the information in each intracavitary image.

[0264] In the first step, considering that the contrast images are acquired in real time, the heartbeat and breathing of the person receiving the surgery and contrast, as well as the slight vibrations of the environment, will cause the cavity pixels in different contrast images to shift, rotate or deform. The fusion process relies on the strict matching of contrast markers and intracavitary markers, and such shifts, rotations or deformations will lead to information distortion.

[0265] For example, assuming there is a correspondence between the contrast marker x in contrast image X and the intracavitary marker y in intracavitary image Y, then fusing the information from intracavitary image Y into contrast image X will not cause information distortion. Conversely, when fusing the information from intracavitary image Y into standard contrast image Z, during the process of synchronously mapping the position of contrast marker x in contrast image X to the position of contrast marker x' in standard contrast image Z, due to the translation, rotation, or deformation of the cavity pixels in standard contrast image Z relative to contrast image X, the position of contrast marker x relative to the cavity in contrast image X is different from the position of contrast marker x' relative to the cavity in standard contrast image Z. This causes information that should be registered to position A in cavity B to be registered to position B, resulting in information distortion. Although this distortion is minor, any minor information distortion is not negligible in surgical or diagnostic procedures.

[0266] In other words, changes in the cavity pixel positions of the imaging image (standard imaging image) used as the basis for registration relative to other imaging images may lead to information distortion. To solve this problem, this embodiment calculates the mapping relationship between image point clouds for the set to be mapped, and uses this mapping relationship to match the cavity positions in different imaging images.

[0267] Based on the example above, this embodiment modifies the process of "synchronously mapping the position of contrast marker x in contrast image X to the standard contrast image Z as contrast marker x'" as follows:

[0268] The process of synchronously mapping the position of the contrast marker x in the contrast image X relative to the cavity pixel (or the set of cavity pixels, hereinafter the same) to the standard contrast image Z to obtain the contrast marker x' whose position relative to the cavity pixel in the standard contrast image Z is the same as the position of the contrast marker x in the contrast image X relative to the cavity pixel.

[0269] This solved the problem.

[0270] In the second step, for a specific intracavitary image, there exists a one-to-one corresponding contrast image, that is, the contrast markers and intracavitary markers of the two have a corresponding relationship. Through the first step, the point cloud mapping relationship between the contrast image and the standard contrast image is also known. Therefore, we can first determine the position (i.e., acquisition area) of the information of the specific intracavitary image on the one-to-one corresponding contrast image, and then determine the position (i.e., acquisition area) of the information of the specific intracavitary image on the standard contrast image through the point cloud mapping relationship.

[0271] The beneficial effects of this embodiment are as follows:

[0272] By calculating the mapping relationship between image point clouds and determining the acquisition area based on the mapping relationship, it is possible to effectively reduce the cavity position deviation that may exist between multiple contrast images due to breathing or heartbeat, and further enable the information of multiple cavity images to be more accurately fused into a single standard contrast image.

[0273] According to any embodiment of the above-described registration method for intracavitary images and contrast images, in this embodiment:

[0274] The step of calculating the mapping relationship between image point clouds for the set to be mapped includes:

[0275] For the set to be mapped, point cloud matching is performed based on the ICP algorithm to obtain the mapping relationship between point clouds of any two frames of angiography images in the set to be mapped.

[0276] The beneficial effects of this embodiment are as follows:

[0277] Considering that the imaging data acquisition process takes about 2-3 seconds, the cavity deformation is relatively small, and the operation is mainly rotation and translation, the ICP algorithm can fit the mentioned rotation and translation well without consuming too much computing resources.

[0278] According to any embodiment of the above-described registration method for intracavitary images and contrast images, in this embodiment:

[0279] The step of determining the marker path in the standard contrast-enhanced image based on the contrast marker positions of the first contrast-enhanced image and the second contrast-enhanced image includes:

[0280] The standard contrast-enhanced image is input into the cavity model to obtain the cavity pixel set in the standard contrast-enhanced image; the cavity model is trained with cavity samples and labels, and is a model that obtains the cavity pixel set by taking the cavity contrast-enhanced image as input.

[0281] In the standard contrast-enhanced images, taking the position of the contrast marker on the first frame of the contrast-enhanced image set as the source point and the position of the contrast marker on the last frame of the contrast-enhanced image set as the endpoint, and using the cavity pixel set as a constraint, the shortest path model is run to obtain the marker path;

[0282] The shortest path model is a model that takes the source point and the destination point as input to obtain the shortest path between the source point and the destination point.

[0283] The beneficial effects of this embodiment are as follows:

[0284] By introducing a cavity pixel set, the determination of the marking path becomes more scientific and accurate, and the registration accuracy between subsequent intracavitary images and standard contrast images is further improved.

[0285] According to any embodiment of the above-described registration method for intracavitary images and contrast images, in this embodiment:

[0286] The shortest path model is based on Dijkstra's algorithm.

[0287] The beneficial effects of this embodiment are as follows:

[0288] By employing Dijkstra's algorithm, the marker path can be determined more quickly and effectively, thereby improving the efficiency of registration.

[0289] According to any embodiment of the above-described registration method for intracavitary images and contrast images, in this embodiment:

[0290] The cavity model is a model trained based on the Swin-Unet network.

[0291] The beneficial effects of this embodiment are as follows:

[0292] By employing the Swin-Unet network, semantic segmentation of cavity pixels can be performed more efficiently and accurately, providing a better foundation for confirming the marked path.

[0293] According to any embodiment of the above-described registration method for intracavitary images and contrast images, in this embodiment:

[0294] The training loss function for the cavity model is the GDL Loss function.

[0295] The beneficial effects of this embodiment are as follows:

[0296] By using the GDL Loss function as the loss function for model training, the problem of cavity class imbalance in imaging images during speech segmentation can be effectively solved.

[0297] The following will provide a cross-use embodiment combining the above-mentioned fusion imaging method and the registration method of intracavitary images and contrast images.

[0298] This embodiment provides a multimodal angiography method based on intravascular cardiovascular images.

[0299] The overall design concept of this embodiment is as follows: Figure 3 As shown, firstly, angiographic images and intravascular images are acquired simultaneously. Utilizing the synchronous sampling characteristic of angiographic and intravascular images, registration and fusion of the two images are achieved. Semantic segmentation and data processing techniques are applied to automatically identify vascular tissue features from the intravascular images. Based on the matching relationship between the angiographic and intravascular images, these vascular tissue features are added to the angiographic images to enhance them, resulting in multimodal angiographic images. This allows for more accurate and efficient lesion identification. Surgeons can also mark certain points on the angiographic images, such as the placement points of metal stents, rotational atherectomy, and balloons. Through the Iterative Closest Point (ICP) algorithm, the marked points provided by the surgeon can be displayed on the angiographic images in real time during the procedure, thereby achieving precise PCI surgical treatment.

[0300] The main steps of multimodal contrast imaging technology are:

[0301] (1) Synchronous acquisition of contrast images and intracavitary images: Since the optional execution subject in this embodiment is to realize an integrated, multimodal contrast imaging system, it is necessary to use DSA to acquire intracavitary images (the images acquired by the DSA device are contrast images) and preprocess the intracavitary images to obtain intracavitary images to be registered and fused. By aligning the contrast data and intracavitary image data on the time axis, contrast images and intracavitary images acquired at the same time are obtained for subsequent analysis and processing.

[0302] (2) Registration System Implementation of Angiographic Images and Intravascular Images: A feasible registration method for angiographic images and intravascular images involves the doctor selecting a frame of the angiographic image and marking a mark point, along with its corresponding vascular portion. However, during the mark point tracking process, it is easily affected by noise in the angiographic image, and once an error occurs in the mark point, subsequent points are even more difficult to detect, resulting in low registration and fusion accuracy, making it difficult to guide PCI surgery. Therefore, this embodiment proposes a new registration and fusion algorithm to improve the accuracy of registration and fusion without increasing user interaction. The specific method is as follows: Based on a set of acquired angiographic images and corresponding intravascular images, the doctor first marks the catheter retraction mark point in the intravascular image of the first and last frames of the angiographic image (referred to as the starting mark point and the ending mark point, respectively); then, to address the issue of changes in vascular position and angle in different frames of the angiographic image, the point cloud matching ICP iterative nearest point algorithm is used to map the starting mark point and the ending mark point onto other angiographic images, facilitating the matching of angiographic images and intravascular images. Then, a specific frame of the angiography image is selected, and semantic segmentation technology is applied to identify the vascular path. On the vascular path generated by semantic segmentation, Dijkstra's shortest path algorithm is applied to calculate the shortest path from the starting mark point to the ending mark point. Under the premise of a constant image sampling frequency and uniform catheter retraction within the patient, the calculated shortest path is segmented, and each frame of the intraluminal image corresponds to each segmented point. The overall process is as follows: Figure 5 As shown.

[0303] The specific steps of the imaging and intracavitary image registration system are as follows:

[0304] 1) Manually mark the position of the intracavitary imaging probe on the first and last frames of the contrast image set, and record them as the start mark point and the end mark point.

[0305] 2) During angiography, the position of blood vessels in the image changes slightly due to the patient's heartbeat. To concentrate the patient's diagnostic information onto a single angiography image, the ICP algorithm is used to register point clouds from different frames of angiography images. Since the angiography image data acquisition process takes approximately 2-3 seconds, and the vascular deformation is relatively small, involving mostly rotation and translation operations, the ICP point cloud matching algorithm is suitable. The ICP formula is shown below, p s p t Let R and t represent the source point cloud and the target point cloud, respectively, and R and t represent the rotation matrix and translation distance.

[0306]

[0307] Considering that vascular deformation variables in angiographic images accumulate continuously over half a patient's respiratory cycle, using only the first and last frames would introduce bias due to the cumulative deformation. However, sequential frame-by-frame ICP point cloud mapping is computationally intensive, and any deviation in the intermediate point cloud mapping will affect subsequent point cloud mapping calculations. Therefore, for 40 frames of angiographic images, ICP point cloud mapping at 3-frame intervals effectively controls the degree of vascular deformation and the probability of cumulative point cloud mapping bias while keeping computationally low.

[0308] 3) Given the start and end marker points and the point cloud mapping relationship between angiographic images, a semantic segmentation model is used to identify vascular regions in the angiographic images. The model used in the patent is Swin-unet, and the network architecture is as follows: Figure 6 As shown, it mainly consists of four parts: encoder, decoder, bottleneck, and skip connection. The basic unit of the network model is the swin transformer block. For 512... 2 The imaging image was divided into three feature representation parts, namely 128 2 64 2 and 32 2 Through 128 2 and 64 2 The feature map is processed by two skip connection operations, which effectively reduces the loss of image spatial information caused by patch merging downsampling.

[0309] Meanwhile, to address the imbalance of blood vessel categories in angiographic images during semantic segmentation, GDL Loss was selected as the swin-unet loss function. The formula is shown below.

[0310]

[0311]

[0312] Where l represents the category, n represents the position of the pixel in the image, and r ln p represents the true pixel category at position n. ln w represents the corresponding predicted probability value. l This represents the weight of each category. The weights w are assigned based on the statistical frequency of each category in the imaging images. l By using the GDL Loss function, the impact of data class imbalance during the training of the semantic segmentation network is effectively reduced.

[0313] 4) Semantically segment and identify pixels in the angiography image as vascular regions, and denote them as nodes in the graph. Spatially adjacent vascular pixels have paths with a distance of 1. Map the start and end mark points onto a certain angiography image through point cloud mapping, and use them as the source and end points of Dijkstra's algorithm, respectively. Use the shortest path from the start point to the end point as the moving path of the endovascular imaging catheter mark point.

[0314] 5) Because the sampling frequency of the contrast imaging and intracavitary images remains constant, and the speed at which the intracavitary imaging catheter moves within the body remains constant during acquisition, and both the sampling time interval t and the speed are constant, the distance the intracavitary imaging catheter travels within the body is s = vt, and the path length remains constant. Therefore, by equally dividing the movement path of the mark point, the corresponding point of any intracavitary image on a certain contrast imaging image can be obtained.

[0315] The beneficial effects of this embodiment are as follows:

[0316] 1. A registration and fusion system based on contrast images and intracavitary images improves registration and fusion accuracy without changing the complexity of user interaction.

[0317] 2. The application of intravascular imaging enables an integrated, multimodal angiography system that directly displays vascular tissue on the angiographic images, achieving rapid and accurate diagnosis of lesions.

[0318] 3. During the surgery, the angiographic image markers are dynamically registered in real time, providing precise guidance for clinical treatment.

[0319] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0320] The following example uses PCI surgery for patients with coronary heart disease as an application scenario. Combined with any of the above embodiments (hereinafter referred to as fusion method and registration method according to the main body of the embodiment), a complete embodiment of using fusion method and / or registration method to guide PCI surgery is provided.

[0321] At least some steps of this PCI application embodiment can be combined with... Figure 3 The steps shown correspond to...

[0322] As an example and not a limitation, in this embodiment, the object of PCI surgery is referred to as the patient, the performer of PCI surgery is referred to as the doctor, the angiographic images are acquired by DSA equipment (hereinafter referred to as intracardiac angiographic images), and the intracardiac images are optical computed tomography images of intracardiac tubular arteries based on OCT (hereinafter referred to as OCT images).

[0323] In this embodiment, before performing PCI surgery, it is necessary to simultaneously acquire intracardiac angiography images and OCT images of the patient.

[0324] It is worth noting that the simultaneous acquisition of intracardiac angiography images and OCT images is a preferred implementation method. In fact, through the above-mentioned fusion method, angiography images and intracardiac images acquired asynchronously can also be registered and fused to obtain a fused angiography result. That is, by mapping the intracardiac markers in the intracardiac image to the mapping points in real space, the position of the angiography markers is determined. Furthermore, the corresponding region of intracardiac image information in the angiography image is determined by the angiography markers and intracardiac markers. Then, the intracardiac image information is fused into the region to obtain the fused angiography result.

[0325] However, in this embodiment, the synchronous acquisition scheme eliminates the minute displacements and deformations of the coronary arteries caused by heartbeat, respiration or other factors by synchronously acquiring angiographic images and intracardiac images, which can make the fusion angiography results more accurate and the registration more precise.

[0326] The time period during which the patient's intracardiac angiography and OCT images are simultaneously acquired is designated as the first time period (in some optional implementations, the first time period is defined as the time period during which the asynchronously acquired OCT images, i.e., the intracardiac images are acquired); the time period during which the patient undergoes PCI surgery is designated as the second time period.

[0327] The first and second time periods can be either continuous or discretely spaced. That is, the acquisition of intracardiac angiography and OCT images can be either acquired before PCI surgery and then immediately used for PCI surgical guidance, or acquired for doctors to conduct detailed analysis (e.g., consultation) before being used for PCI surgical guidance.

[0328] During the first phase, in the process of acquiring intracardiac angiography and OCT images, registration and / or fusion methods can be applied in real time to obtain fused angiography results, or the registration and / or fusion methods can be applied after the acquisition is completed to obtain fused angiography results.

[0329] As an example and not a limitation, in this embodiment, after the first time period ends, that is, after the acquisition is completed, the doctor will screen the acquired OCT images to exclude OCT images that are not of good guidance for PCI surgery, and obtain effective OCT images as one of the bases for subsequent fusion.

[0330] It is worth noting that in this embodiment, the sequence number of the valid OCT images needs to be retained. For example, if 10 OCT images are acquired in the first time period, and the 2nd to 8th OCT images have poor guiding significance for PCI surgery, then the 1st, 9th, and 10th OCT images are retained as the 1st, 9th, and 10th valid OCT images, respectively, instead of the 1st, 2nd, and 3rd valid OCT images. The purpose of retaining the sequence number information of the valid OCT images is that when applying the above registration method, the determination of the interval angiography marker depends on the sequence number of the valid OCT images.

[0331] In this embodiment, after the first time period ended, the doctor also provided additional doctor annotation information (e.g., Figure 15 The proposed placement of the support shown serves as one of the bases for subsequent fusion.

[0332] In an optional implementation, one of the contrast images acquired during the first time period should be selected as the standard contrast image. After the first time period, the above-mentioned fusion method and / or registration method should be performed to fuse the effective OCT image and the doctor's annotation information into the standard contrast image (referred to as the first standard contrast image in this implementation) to obtain the first fused contrast image result.

[0333] Subsequently, during the second time period, real-time angiography images were still acquired for the patient so that doctors could have a more immediate understanding of the changes in the patient's coronary arteries. At this time, since the fusion angiography results were based on the first standard angiography image, there may be differences in the position of the vascular lumen between the first standard angiography image and the real-time angiography image acquired during the second time period due to heartbeat, breathing or other reasons. In other words, the guiding value for PCI surgery is limited.

[0334] Therefore, in this embodiment, each time an angiography image is acquired in real time during the second time period, it is updated to a standard angiography image (referred to as the second standard angiography image in this embodiment), and the OCT information and doctor annotation information in the first fusion angiography result are updated to the second standard angiography image to obtain the second fusion angiography result.

[0335] Specifically, in this embodiment, an optional method for updating the second fusion imaging result is as follows:

[0336] Using the point cloud mapping step in the above fusion method and / or registration method, the mapping relationship between the point cloud in the first standard contrast image and the point cloud in the second standard contrast image is determined;

[0337] Based on the mapping relationship between the point cloud in the first standard contrast image and the point cloud in the second standard contrast image, the OCT information and doctor's annotation information in the first fusion contrast image result are mapped to the corresponding points or point cloud in the second standard contrast image to obtain the second fusion contrast image result.

[0338] In another optional implementation, both the contrast images and OCT images acquired in the first time period are used as the basis for fusion. The real-time contrast images acquired in the second time period are determined as the standard contrast images. The above fusion method and / or registration method are (repeatedly) executed to obtain the fused contrast image result. It can be understood that the fused contrast image result in this implementation is a dynamic fused contrast image result that updates the OCT image information and doctor annotation information in the first time period to the contrast images acquired in the second time period in real time.

[0339] Furthermore, in the two embodiments described above, the dynamic fusion angiography result is obtained by fusing the angiography images acquired in each frame within the second time period. As angiography images are continuously acquired, the fusion angiography result is also continuously fused and displayed. From the doctor's perspective, the dynamic display of the fusion angiography result, based on the real-time updated fusion angiography result (including the current status of the patient's coronary arteries, the OCT information from the first time period, and the doctor's annotation information, where, taking the doctor's annotation information for the proposed stent placement location as an example, is as follows: Figure 16 (As shown) Adjust the surgical strategy in real time to achieve better surgical results.

[0340] In a preferred embodiment, when the frame rate of the contrast imaging images is high, such that the time required to perform the above-mentioned fusion method and / or registration method is greater than the interval between adjacent frames of the contrast imaging images (or, the time required to perform point cloud mapping is greater than the interval between adjacent frames of the contrast imaging images), the effect of dynamically displaying the fused contrast imaging results may not be satisfactory. In this case, the contrast imaging images acquired in real time can be updated to standard contrast imaging images at intervals of a set number of frames. For example, with an interval of 3 frames, the 1st, 4th, 7th, 10th... frames of the contrast imaging images acquired in real time in the second time period will be updated to standard contrast imaging images, and the method of this embodiment will be executed to overcome the above-mentioned problems.

[0341] Corresponding to the fusion imaging method described in the above embodiments, Figure 13 A structural block diagram of the fusion imaging apparatus provided in an embodiment of this application is shown. For ease of explanation, only the parts relevant to the embodiment of this application are shown. (Refer to...) Figure 13 The device includes:

[0342] Image acquisition module 1301 is used to acquire contrast images and intracavitary images;

[0343] The fusion module 1302 is used to fuse the contrast image and the intracavitary image based on the correspondence between the contrast markers and the intracavitary markers to obtain a fused contrast image result;

[0344] The contrast markers are designated points and / or designated areas on the contrast image;

[0345] The intracavitary markings are as follows:

[0346] The designated points and / or designated regions on the intracavitary image; or...

[0347] The set points and / or set areas on the intracavitary image acquisition device;

[0348] The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space.

[0349] Corresponding to the registration method of intracavitary images and contrast images described in the above embodiments, Figure 14 This diagram illustrates the structural block diagram of a registration device for intracavitary images and contrast images provided in an embodiment of this application. For ease of explanation, only the parts relevant to the embodiments of this application are shown. (Refer to...) Figure 14 The device includes:

[0350] The image acquisition module 1401 is used to acquire a set of contrast images and a set of intracavitary images; the set of contrast images includes a first contrast image and a second contrast image.

[0351] The end-matching module 1402 is used to determine, within the intracavitary image set, an intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the first contrast image as a first intracavitary image, and an intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the second contrast image as a second intracavitary image; the intracavitary image set includes at least one interval intracavitary image; the interval intracavitary image is an intracavitary image whose acquisition time is within an interval period; the interval period refers to the time period between the acquisition time of the first intracavitary image and the acquisition time of the second intracavitary image;

[0352] The interval matching module 1403 is used to determine the interval angiography markers of the marking path, and based on the one-to-one correspondence between the intracavitary markers of the interval intracavitary images and the interval angiography markers, to register the information of the interval intracavitary images with the information of the set standard angiography images; the marking path is the path between the position corresponding to the angiography marker of the first angiography image and the position corresponding to the angiography marker of the second angiography image on the standard angiography image.

[0353] The ratio of the marker path lengths on both sides of the cleavage angiography marker is the same as the ratio of the interval time period lengths on both sides of the acquisition time of the image within the cleavage cavity.

[0354] The contrast markers are designated points and / or designated areas on the contrast image;

[0355] The intracavitary markings are as follows:

[0356] The designated points and / or designated regions on the intracavitary image; or...

[0357] The set points and / or set areas on the intracavitary image acquisition device;

[0358] The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space.

[0359] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0360] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0361] This application embodiment also provides a terminal device 15, which includes: at least one processor 1501, a memory 1502, and a computer program 1503 stored in the memory and executable on the at least one processor 1501. When the processor 1501 executes the computer program 1503, it implements the steps in any of the above-described method embodiments.

[0362] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0363] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0364] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0365] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of fusion imaging, characterized by, include: Acquire contrast images and intracavitary images; Based on the correspondence between contrast markers and intracavitary markers, the contrast images and intracavitary images are fused to obtain a fused contrast image result; The contrast markers are designated points and / or designated areas on the contrast image; The intracavitary markings are as follows: The set points and / or set areas on the intracavitary image; or, The set points and / or set areas on the intracavitary image acquisition device; The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space; The step of fusing the contrast image and the intracavitary image based on the correspondence between contrast markers and intracavitary markers to obtain the fused contrast image result includes: The acquisition angle of the intracavitary image is determined based on the intracavitary image and the intracavitary markings in the intracavitary image; Based on the correspondence between the contrast markers and the intracavitary markers, the acquisition angle of the intracavitary image is determined to be the acquisition area on the contrast image; The information from the intracavitary image is fused with the information from the acquisition area to obtain a fused imaging result.

2. The fusion imaging method as described in claim 1, characterized in that: The fusion imaging result is the result of displaying the intracavitary image information on the imaging image; or... The contrast markers are designated points and / or designated areas on the intracavitary image acquisition device in the contrast image; the intracavitary markers are designated points and / or designated areas on the intracavitary image acquisition device used to determine the intracavitary image acquisition angle. or, Both the angiographic images and the intraluminal images are acquired targeting the vascular cavity; the fused angiographic result includes vascular information obtained by running a vascular model with the intraluminal images as input; the vascular model is a semantic segmentation model trained based on vascular samples and vascular labels; the vascular information includes any one or a combination of any of the following: vascular calcification information, intravascular stent information, and vascular lumen cross-sectional area information; or... Both the angiographic images and the intraluminal images are acquired from the vascular cavity; the fused angiographic results also include the coronary artery flow reserve fraction and / or intraluminal image light decay index extracted from the intraluminal images.

3. The fusion imaging method as described in claim 1, characterized in that: The contrast image is a standard contrast image; the step of fusing the information on the intracavitary image with the information of the acquisition area to obtain the fused contrast image result includes: For the set to be mapped, the mapping relationship between image point clouds is calculated; the set to be mapped includes the standard contrast images and non-standard contrast images; the non-standard contrast images refer to one or more specified contrast images that are not the standard contrast images; Based on the mapping relationship, the position of the contrast marker in the non-standard contrast image is determined in the standard contrast image; Based on the correspondence between the contrast marker of the i-th non-standard contrast image and the intracavitary marker of the i-th intracavitary image, and the position of the contrast marker of the i-th non-standard contrast image on the standard contrast image, the i-th acquisition area corresponding to the acquisition angle of the i-th intracavitary image on the standard contrast image is determined; where i is a positive integer not greater than the number of non-standard contrast images; the intracavitary marker of the i-th intracavitary image and the contrast marker of the i-th non-standard contrast image have a correspondence. For all selectable values ​​of i, the information on the i-th intracavitary image is fused with the information of the i-th acquisition area on the standard contrast image to obtain the fused contrast result. Alternatively, before the step of calculating the mapping relationship between image point clouds for the set to be mapped, the method further includes: Intraoperative angiography images are acquired in real time to obtain the j-th intraoperative angiography image, and the j-th intraoperative angiography image is determined as the standard angiography image; where j is a positive integer; After the step of fusing the information from the i-th intracavitary image with the information from the i-th acquisition region of the standard contrast image for all selectable values ​​of i to obtain the fused contrast result, the method further includes: Intraoperative angiography images are acquired in real time to obtain the (j+k)th intraoperative angiography image, and the (j+k)th intraoperative angiography image is determined as the standard angiography image; where k is the set interval frame number; Repeat the step of fusing the information on the intracavitary image with the information of the acquisition area to obtain the fused imaging result; Alternatively, after the step of fusing the contrast image and the intracavitary image based on the correspondence between the contrast markers and the intracavitary markers to obtain the fused contrast image result, the method further includes: Display the contrast image from the fused contrast imaging result on a designated image display device; Alternatively, after the step of displaying the contrast image from the fused contrast imaging result on a designated image display device, the method further includes: Upon receiving the first instruction, based on the information type in the first instruction, the image display device will overlay and display any one or a combination of multiple of the following information: vascular calcification information, endovascular stent information, and vascular lumen cross-sectional area information. If the second instruction is received, then based on the information type in the second instruction, the coronary artery flow reserve fraction and / or intravascular image light decay index are superimposed on the image displayed on the image display device.

4. A method of registering an endoluminal image and an angiographic image, characterized by, include: Obtain the set of contrast images and the set of intracavitary images; The set of contrast images includes a first contrast image and a second contrast image; In the set of intracavitary images, the intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the first contrast image is identified as the first intracavitary image, and the intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the second contrast image is identified as the second intracavitary image; the set of intracavitary images includes at least one interval intracavitary image; the interval intracavitary image is an intracavitary image acquired within an interval period; the interval period refers to the time period between the acquisition time of the first intracavitary image and the acquisition time of the second intracavitary image; The cleavage angiography markers of the marking path are determined, and based on the one-to-one correspondence between the intracavitary markers of the cleavage intracavitary images and the cleavage angiography markers, the information of the cleavage intracavitary images is registered with the information of the set standard angiography images; the marking path is the path between the position corresponding to the angiography marker of the first angiography image and the position corresponding to the angiography marker of the second angiography image on the standard angiography image. The ratio of the marker path lengths on both sides of the cleavage angiography marker is the same as the ratio of the interval time period lengths on both sides of the acquisition time of the image within the cleavage cavity. The contrast markers are designated points and / or designated areas on the contrast image; The intracavitary markings are as follows: The set points and / or set areas on the intracavitary image; or, The set points and / or set areas on the intracavitary image acquisition device; The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space; The step of registering the information of the intracavitary image with the information of a set standard contrast image based on the one-to-one correspondence between the intracavitary image and the septal angiography mark includes: For the set to be mapped, the mapping relationship between image point clouds is calculated; the set to be mapped includes the first contrast image, the second contrast image, and the standard contrast image. Based on the mapping relationship, the positions of the contrast markers in the first contrast image and the second contrast image are determined in the standard contrast images. In the standard contrast-enhanced image, the marker path is determined based on the contrast marker positions of the first contrast-enhanced image and the second contrast-enhanced image.

5. The registration method for intracavitary images and contrast images as described in claim 4, characterized in that: The step of calculating the mapping relationship between image point clouds for the set to be mapped includes: For the set to be mapped, point cloud matching is performed based on the ICP algorithm to obtain the mapping relationship between the point clouds of any two frames of angiography images in the set to be mapped. Alternatively, the step of determining the marker path in the standard contrast-enhanced image based on the contrast marker positions of the first contrast-enhanced image and the second contrast-enhanced image includes: The standard contrast-enhanced image is input into the cavity model to obtain the cavity pixel set in the standard contrast-enhanced image; the cavity model is trained with cavity samples and labels, and is a model that obtains the cavity pixel set by taking the cavity contrast-enhanced image as input. In the standard contrast-enhanced images, taking the position of the contrast marker on the first frame of the contrast-enhanced image set as the source point and the position of the contrast marker on the last frame of the contrast-enhanced image set as the endpoint, and using the cavity pixel set as a constraint, the shortest path model is run to obtain the marker path; The shortest path model is a model that takes the source point and the destination point as input to obtain the shortest path between the source point and the destination point.

6. The registration method for intracavitary images and contrast images as described in claim 5, characterized in that: The shortest path model is based on Dijkstra's algorithm; or, The cavity model is a model trained based on the Swin-Unet network; or... The training loss function for the cavity model is the GDL Loss function.

7. The registration method for intracavitary images and contrast images as described in any one of claims 4 to 6, characterized in that, The contrast markers are set points and / or set areas on the intracavitary image acquisition device in the contrast image; the intracavitary markers are set points and / or set areas on the intracavitary image acquisition device used to determine the intracavitary image acquisition angle.

8. A fusion imaging device, characterized in that, include: The image acquisition module is used to acquire contrast images and intracavitary images; The fusion module is used to fuse the contrast image and the intracavitary image based on the correspondence between the contrast markers and the intracavitary markers to obtain a fused contrast image result; The contrast markers are designated points and / or designated areas on the contrast image; The intracavitary markings are as follows: The set points and / or set areas on the intracavitary image; or, The set points and / or set areas on the intracavitary image acquisition device; The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space; The fusion module is specifically used for: The acquisition angle of the intracavitary image is determined based on the intracavitary image and the intracavitary markings in the intracavitary image; Based on the correspondence between the contrast markers and the intracavitary markers, the acquisition angle of the intracavitary image is determined to be the acquisition area on the contrast image; The information from the intracavitary image is fused with the information from the acquisition area to obtain a fused imaging result.

9. A registration device for intracavitary images and contrast images, characterized in that, include: The image set acquisition module is used to acquire the angiography image set and the intracavitary image set; The set of contrast images includes a first contrast image and a second contrast image; An end-matching module is used to determine, within the intracavitary image set, an intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the first contrast image as a first intracavitary image, and an intracavitary image whose intracavitary marker corresponds one-to-one with the contrast marker of the second contrast image as a second intracavitary image; the intracavitary image set includes at least one interval intracavitary image; the interval intracavitary image is an intracavitary image acquired within an interval time period; the interval time period refers to the time period between the acquisition time of the first intracavitary image and the acquisition time of the second intracavitary image; The interval matching module is used to determine the interval angiography markers of the marking path, and based on the one-to-one correspondence between the intracavitary markers of the interval intracavitary images and the interval angiography markers, to register the information of the interval intracavitary images with the information of a set standard angiography image; the marking path is the path between the position corresponding to the angiography marker of the first angiography image and the position corresponding to the angiography marker of the second angiography image on the standard angiography image. The ratio of the marker path lengths on both sides of the cleavage angiography marker is the same as the ratio of the interval time period lengths on both sides of the acquisition time of the image within the cleavage cavity. The contrast markers are designated points and / or designated areas on the contrast image; The intracavitary markings are as follows: The set points and / or set areas on the intracavitary image; or, The set points and / or set areas on the intracavitary image acquisition device; The contrast markers and intracavitary markers that have a corresponding relationship can be mapped to the same point and / or the same region in real space; The interval matching module is specifically used for: For the set to be mapped, the mapping relationship between image point clouds is calculated; the set to be mapped includes the first contrast image, the second contrast image, and the standard contrast image. Based on the mapping relationship, the positions of the contrast markers in the first contrast image and the second contrast image are determined in the standard contrast images. In the standard contrast-enhanced image, the marker path is determined based on the contrast marker positions of the first contrast-enhanced image and the second contrast-enhanced image.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the fusion imaging method as described in any one of claims 1 to 3 or the registration method of intracavitary image and imaging image as described in any one of claims 4 to 7.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the fusion imaging method as described in any one of claims 1 to 3 or the registration method of intracavitary images and imaging images as described in any one of claims 4 to 7.

Images