Methods and apparatus for stent visualization
By using cascaded space transformer networks and neural network technology with subtraction operations, the problem of clear visualization of stents during stent placement was solved, enabling fast and accurate stent image generation and improving the efficiency of the stent placement process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNITED IMAGING INTELLIGENCE CO LTD
- Filing Date
- 2022-12-26
- Publication Date
- 2026-06-26
AI Technical Summary
Current technologies make it difficult to clearly visualize stents during placement, especially as they are affected by heartbeat, respiration, and stent movement. Traditional methods require iterative online optimization, which is slow and cannot meet clinical needs.
By employing cascaded spatial transformer networks (STN0 and STN1) and subtraction operations, a neural network can separate the scaffolded layer and the unscaffolded layer without online optimization, generating clear scaffolded images.
It enables fast and accurate visualization of the stent, reduces the impact on stent movement and background movement, and improves the efficiency of the stent placement process.
Smart Images
Figure CN116309044B_ABST
Abstract
Description
Technical Field
[0001] The aspects of the disclosed embodiments generally relate to stent positioning and placement, and more specifically to enhanced stent visualization during the stent placement process. Background Technology
[0002] Stent placement is a common practice to prevent arteries from becoming completely blocked. During stent placement, a stent is delivered through a catheter to the blocked area of the artery, and then inflated using a balloon. Fluoroscopy images, such as X-ray images, are used to help surgeons monitor the stent and balloon during the stent placement procedure.
[0003] While unique markers are typically placed on the balloon to facilitate stent placement, clearly visualizing the stent remains challenging. Movements such as heartbeat, breathing, and stent movement will affect the clarity of stent visualization in fluoroscopic images or videos. Clearly enhancing stent images from fluoroscopic videos can be helpful in several ways, including, for example, monitoring balloon expansion, identifying stent deployment or underexpansion, or observing stent fracture.
[0004] Traditional methods align multiple N-fluoroscopic (or cinematic) images based on detected balloon markers, then obtain the stent image by averaging the aligned images. In some scenarios, balloon markers are detected, stent motion is estimated, and based on the estimated stent motion, a stent-free layer is separated using an online optimization process. The stent-free layer is the final image presented to the surgeon. However, these methods for stent visualization typically require iterative online optimization until the stent-free layer results converge. Estimating the motion of the stent-free layer during online optimization is slow and cannot meet clinical needs.
[0005] Therefore, it is desirable to provide devices and methods for solving at least some of the aforementioned problems. Summary of the Invention
[0006] The disclosed embodiments relate to a device for enhancing visualization of a support structure. This and other advantages of the disclosed embodiments are substantially as shown in at least one drawing and / or described in conjunction with that drawing.
[0007] According to a first aspect, the disclosed embodiments provide an apparatus for visualizing a stent during stent placement. In one embodiment, the apparatus includes a processor configured to: transform a first stent image to an image space using a first spatial transformer network (STN0) to generate a first transformed stent image; generate a new background image from the first transformed stent image; and transform the new background image to a background image space using a second spatial transformer network (STN1) to generate a stent-free background image (B). k); Use the second spatial transformer network (STN1) to transform the scaffold-free background image (B) k Transform the image to image space; generate a scaffold image in image space from the transformed scaffold-free background image; and transform the scaffold image in image space to scaffold image space to generate a clear scaffold image for scaffold visualization (S). k The disclosed embodiments use neural networks to separate scaffolded and unscaffolded layers without online optimization. Scaffolded and unscaffolded motions are not estimated online in an explicit manner.
[0008] In a possible implementation, the first scaffold image is an image from an image frame sequence, and the hardware processor is further configured to generate a clear scaffold image (S) based on image frames in the image frame sequence. k ).
[0009] In a possible implementation, the hardware processor is further configured to use an averaging layer to generate unsupported background images from a second spatial transformer network (STN1) for image frames in an image frame sequence based on the transformed new background image, and to use an averaging layer to generate sharp supported images from a first spatial transformer network (STN0) for image frames in an image frame sequence based on the transformed supported images. Aspects of the disclosed embodiments enable processing of all images in an image sequence.
[0010] In a possible implementation, the hardware processor is also configured to use the balloon marker location as input to the first spatial transformer network (STN0) to generate a first transduced stent image.
[0011] In a possible implementation, the hardware processor is also configured to use a subtraction operation to generate a new background image from the first transformed scaffold image, wherein the input to the subtraction operation is the corresponding image from the image sequence.
[0012] In a possible implementation, the separated background image is used as input to a second spatial transformer network to transform the new background image to the background image space.
[0013] In a possible implementation, the corresponding image from the image sequence is used as input to a second spatial transformer network (STN1) to transform the scaffold-free background image (B) into a vector image. k Transform to image space.
[0014] In a possible implementation, the hardware processor is configured to use a subtraction operation to generate a scaffolded image in image space from a transformed scaffolded background image, wherein the input to the subtraction operation is a corresponding image from an image sequence.
[0015] In a possible implementation, the hardware processor is also configured to use the balloon marker location as input to the first spatial transformer network (STN0) to generate clear stent images.
[0016] In a possible implementation, the first space transformer network and the second space transformer network form a cascaded space transformer network.
[0017] According to a second aspect, the disclosed embodiments provide a network for scaffold visualization enhancement. In one embodiment, the network includes at least two spatial transformation networks. One spatial transformation network is used to transform the image based on an affine transformation computed from corresponding points.
[0018] Another spatial transformation network is used to align one image with another.
[0019] According to a third aspect, the disclosed embodiments relate to a method in a neural network. In one embodiment, in a first operation, a stent image is fed to a first spatial transformation network, which is given a corresponding balloon label as other input. The result is fed to a subtraction operator. Another input to the subtraction operator is a corresponding image from an image sequence. The result of the subtraction operator is fed to a second spatial transformation network, which is given a separated background image as input. These steps are repeated for all available frames, and then the result is fed to an averaging layer to generate a stent-free background image.
[0020] In a possible implementation, in the second operation, the result from the previous averaging layer (the background image without a support) is then fed to a second spatial transformation network, which takes the corresponding frame image as another input. The result is then fed to a subtraction layer, which takes the corresponding frame image as another input. The result is then fed to a first spatial transformation network, which takes the balloon marker as another input. These steps are repeated for all available frames, and the result is fed to the averaging layer. The result is the support image.
[0021] According to a fourth aspect, the disclosed embodiments provide a method for enhancing stent visualization during stent placement. In one embodiment, the method includes: using a hardware processor to generate a clear stent image and a stent-free background image from image frames of an image frame sequence. The method further includes: configuring the hardware processor to: transform the first stent image of the image frame sequence to an image space of the image frame sequence using a first spatial transformer network (STN0) to generate a first transformed stent image; generate a new background image from the first transformed stent image; and transform the new background image to a background image space using a second spatial transformer network (STN1) to generate a stent-free background image (B). k ); Use the second spatial transformer network (STN1) to transform the scaffold-free background image (B) kTransform the image space of the image frame sequence to the image space of the transformed unsupported background image; generate a support image in the image space of the image frame sequence from the transformed unsupported background image; and transform the support image in the image space to the support image space to generate a clear support image (S). k Repeat this process for all available image frames.
[0022] According to the fifth aspect, the disclosed embodiments provide a computer program product having a non-transitory computer-readable medium having computer-implemented instructions stored thereon, which, when executed by a computer, cause the computer to perform the methods and processes described herein.
[0023] These and other aspects, embodiments, and advantages of the exemplary embodiments will become apparent from the examples described herein in conjunction with the accompanying drawings. However, it should be understood that the specification and drawings are intended for illustrative purposes only and not as limiting definitions of the disclosed invention. Further aspects and advantages of the invention will be set forth in the following description and will be apparent in part from the description, or may be learned by practice of the invention. Attached Figure Description
[0024] In the following detailed sections of this disclosure, the invention will be described in more detail with reference to exemplary embodiments shown in the accompanying drawings, in which:
[0025] Figure 1 This is a schematic block diagram of an exemplary device incorporating aspects of the disclosed embodiments.
[0026] Figures 2A to 2C An exemplary input image of the device according to the disclosed embodiment is shown.
[0027] Figure 3A An example of a bracket image generated by the device of the disclosed embodiment is illustrated.
[0028] Figure 3B An example of a bracketless background image generated by the device of the disclosed embodiment is illustrated.
[0029] Figure 4 This is a schematic block diagram of an exemplary network structure of a device incorporating aspects of the disclosed embodiments.
[0030] Figure 5 This is a schematic block diagram of an exemplary network structure of a device incorporating aspects of the disclosed embodiments.
[0031] Figure 6 This is a schematic block diagram of an exemplary spatial transformer network structure incorporating aspects of the disclosed embodiments.
[0032] Figure 7 Examples Figure 6 An embodiment of the internal structure of an exemplary spatial transformer network.
[0033] Figure 8 This is a schematic block diagram of an exemplary spatial transformer network incorporating aspects of the disclosed embodiments.
[0034] Figure 9 Examples Figure 6 An embodiment of the internal structure of an exemplary spatial transformer network.
[0035] Figure 10 This is a flowchart of an exemplary method incorporating aspects of the disclosed embodiments. Detailed Implementation
[0036] The following detailed description illustrates exemplary aspects of the disclosed embodiments and how they can be implemented. Although some modes of performing aspects of the disclosed embodiments have been disclosed, those skilled in the art will recognize that other embodiments for performing or practicing aspects of the disclosed embodiments are also possible.
[0037] Figure 1 This is a schematic block diagram of an exemplary device 100 for enhancing stent visualization according to aspects of the disclosed embodiments. Aspects of the disclosed embodiments generally relate to providing clear stent image visualization using a cascaded spatial transform or transformer network configured to iteratively generate stent images S. k And the background image B without support k Where k indicates the number of cascaded networks. According to an aspect of the disclosed embodiment, the mapping function f is configured to map input 102 (i.e., input X-ray image I) to... n and marked coordinates M ref M n Mapped to output 104 (i.e., scaffold image S) k And the background image B without support k In one embodiment, the function f is generated by a neural network (such as...). Figure 1 The example neural network 110 is implemented.
[0038] like Figure 1 As an example, the output 104 of device 100 typically includes two single images: a bracket image S. k And the background image B without support k . Support image S k This is an enhanced image of the scaffold relative to a clear background, such as an anatomical structure without ribs and lungs. Scaffold-free background image B k Excluding the support. As will generally be understood, there is an X-ray image I. n , support image S k and background image Bk Many associated movements. These movements may include, but are not limited to, heart movement, respiratory movement, chest cavity movement, and lung movement. Typically, the stent may move with any one or more of these movements. Aspects of the disclosed embodiments are configured to correlate these movements with the generated stent image S. k And the background image B without support k Separation.
[0039] like Figure 1 As shown, input 102 includes the image sequence I0, I1,...I n , among which, I n It's a frame image. Figure 1 In the example, the image sequence I0, I1,...I n The image in the image is an X-ray image. Figures 2A to 2C An example of X-ray image acquisition at different time points is illustrated. Although only three images are illustrated in this example, aspects of the disclosed embodiments are not limited thereto. In alternative embodiments, any suitable number of frame images, in addition to including three images, can be used for the image input sequence I0, I1,...I n .
[0040] Figures 2A to 2C The exemplary frame image includes a catheter (in one embodiment, the catheter is an artery) and balloon markers 204, 208 associated with the stent 206. In this example, balloon markers 204, 208 are represented as black dots for ease of visualization. It should be understood that in conventional image sequences, balloon markers 204, 208 may be represented as blank spaces, white dots, or circles.
[0041] Figure 3A The image S shows a clear image of the support structure. k An example. As used herein, the term "scaffold image" typically refers to a single image used for visualization. Scaffold image S k These are typically the images that a physician or doctor wants to see during fluoroscopy or stent placement procedures. Figure 3A In the image, for ease of visualization, bracket 302 is shown against a white background. Original input image I n Other anatomical structures were removed.
[0042] The term "background image" refers to a single image excluding the support frame. Figure 3B Image B without support frame is shown. k Examples.
[0043] Figure 4 An example of a cascaded network structure 400 incorporating aspects of a disclosed embodiment is illustrated. Figure 4In the example, the cascaded space transformer networks STN0, STN1...STN(k-1) are configured to predict S k and B k Where k is at least 1. The inputs to the spatial transformer networks STN0 and STN1 include the coordinates of two balloon markers M. n (where n is the image frame index) and the fixed frame index M for the balloon marker. ref (It is selected from {0,1,...,N-1}). The spatial transformer networks STN0, STN1...STN(k-1) can be different network structures, the same network structure with different weights, or the same network structure with the same weights.
[0044] It relies on three main coordinate systems. These coordinate systems include the X-ray image sequence I0-I... n The various X-ray images I n coordinate system, support image S k The coordinate system and background image B k The coordinate system. According to aspects of the disclosed embodiments, in Figure 4 In the example, the first spatial transformation network STN0 is configured to transform the scaffold image S k The coordinate system and X-ray image I n The image is transformed between coordinate systems. The second spatial transformation network STN1 is used to transform the background image B. k The coordinate system and X-ray image I n Transform the image between coordinate systems.
[0045] exist Figure 4 In the illustrated exemplary cascaded network structure 400, spatial transformation network STN0 and spatial transformation network STN1 are used twice. The first spatial transformation network STN0 is configured to transform the image based on an affine transformation computed from corresponding points. The second spatial transformation network STN1 is used to align one image with another. (Refer to...) Figure 5 Furthermore, network 400 may also include subtraction layers or operations and averaging layers or operations.
[0046] Figure 5 An example is illustrated for separating the scaffold image S at iteration k according to the disclosed embodiments. k and background image B k An exemplary network architecture 500 is provided. Network architecture 500 typically includes four sub-networks. These sub-networks include a first spatial transformation network or layer STN0, a second spatial transformation network or layer STN1, subtraction operation layers 504 and 514, and average pooling layers 510 and 520. Although the subtraction operation layers 504 and 514 and the average pooling layers 510 and 520 are... Figure 5The example is shown to include two different network structures, but aspects of the disclosed embodiments are not limited thereto. In alternative embodiments, the subtraction operation layers 504, 514 and the average pooling layers 510, 520 may each include a single network structure.
[0047] like Figure 5 For example, the scaffold image S k-1 Marking position M n and mark reference position M ref It is the input of the first spatial transformation network STN0. Figure 5 Different layers of the first spatial transformer network STN0 shown are configured to transform the scaffold image S k-1 Transformed into X-ray sequences I0, I1, ... I n One after another, the original images I n The coordinate system. This transformation produces an "aligned image," which typically means a scaffold image S. k-1 Compared with the original X-ray image I n Coordinate system alignment or mapping.
[0048] The aligned image or result 502 of the first spatial transformer layer STN0 is used to find a new unsupported background image B. k .exist Figure 5 In the example, for the illustrated iteration, the result 502 of the first spatial transformer network STN0 and the image frame I1 are processed in subtraction operation 504. The result 506, or the output of subtraction operation 504, is a new scaffold-free background image.
[0049] Then, the second spatial transformer network STN1 is used to transform the new unsupported background image 506 to the previous unsupported background image B. k-1 The new scaffold-free background image remains in the coordinate system of the original X-ray image (also known as the "original image space"). The result of subtraction operation 504 is 506, and the scaffold-free background image B is... k-1 It is the input of the second spatial transformer network STN1.
[0050] In this example, the result 508 of the second spatial transformer network STN1 is the new scaffold-free background image B. k .like Figure 5 As shown, the new background image B k Based on average pooling layer 510.
[0051] Then, the spatial transformer network STN1 is used a second time. After the average pooling layer 510, the order of the first transformer network STN0 and the second transformer network STN1 is reversed. In this example, it is a new unsupported background image B of a single image. kThe image is transformed back to the original image (i.e., image I) by the spatial transformer network STN1. n The coordinate system of ).
[0052] The result 512 of the second spatial transformer network STN1 is the transformation back to the original image I. n A new unsupported background image in the coordinate system. In this example, the result 512 of the second spatial transformer network STN1 and image frame I0 are processed in the subtraction layer or operation 514. The result 516 is the scaffolded image.
[0053] Then, the first spatial transformer network STN0 is used a second time to transform the stent image 516 from the X-ray image space to the stent image space. In this example, the result 516, along with the balloon marker fixed frame index M, is... ref The two balloon marker positions M0 are the inputs to the first spatial transformer network STN0.
[0054] In this example, the result 518 of the first spatial transformation network STN0 is processed by the averaging layer 520. The output of the average pooling layer 520 is the scaffold image S. k .
[0055] In one embodiment, when k=1, the unsupported background image B 0 It can be initialized as a black image, containing zeros. The scaffold image S 0 It can be initialized as image sequence I0 to I n One of the images.
[0056] For example, in one embodiment, the bracket image S 0 It can be an image sequence I0...I n The first image I0 is used. During the inference phase, the input can be directly fed into the network structure 500. Then, the scaffold image S is obtained. k To enable better visualization.
[0057] The aspects of the disclosed embodiments are not limited to specific network structures. Figure 5 The image order in the illustrated sequence is merely exemplary. In alternative embodiments, any suitable image order sequence may be used. Figure 5 The vertical operations illustrated are independent of the image order. Furthermore, the number of images n and the number of cascaded images k can be variables.
[0058] Figure 6 An example of a first spatial transformer network STN0 incorporating aspects of a disclosed embodiment is illustrated. The first spatial transformer network STN0 is used to transform an input image from spatial coordinates 2 to spatial coordinates 1. For example, as... Figure 5As shown, in the first use of the spatial transformation network STN0, spatial coordinate 1 is the location M of the two balloon markers. n Spatial coordinate 2 is the fixed frame index M of the balloon marker. ref In the second use of the spatial transformation network STN0, spatial coordinate 1 is M. ref And spatial coordinate 2 is M n .
[0059] Figure 7 An exemplary internal structure of the first spatial transformer network STN0 is illustrated. In this example, the positioning network 702 predicts the affine transformation based on spatial coordinates 2 and 1. θ The localization network 702 can be any suitable affine transformation estimator. In one embodiment, the localization network 702 is a neural network. Training samples can be built, and supervised learning can be used to train the network. In one embodiment, the neural network is a fully connected network. Figure 7 The grid generator shown is used to resample the input image based on the predicted affine transformation parameters.
[0060] Figure 8 An example of a second spatial transformer network (STN1) is illustrated. As shown in this example, the second spatial transformer network (STN1) is used to align input image 1 to the coordinate system of input image 2, thereby producing a new or transformed image 1. (See again...) Figure 5 For example, in the first use of the second spatial transformer network STN1, STN1 compares the unsupported background image 506 with the unsupported background image B. k-1 Align the coordinate system. In the second use, the unsupported background image B... k With the original image frame I n Align the coordinate systems.
[0061] Figure 9 An example of a second spatial transformer network (STN1) using a spatial transformer network structure is illustrated. In this example, a grid generator is followed by a convolutional neural network (CNN) 902 to resample the input image 1. This produces a new or transformed image 1 that is aligned with the coordinates of the input image 2.
[0062] In one embodiment, a first spatial transformer network STN0 and a second spatial transformer network STN1 can be trained separately. For the second spatial transformer network STN1, supervised or unsupervised learning can be used to train the network. For unsupervised learning, the loss is based on a comparison between the transformed image 1 and the input image 2. For supervised learning, a gold standard for the transformed image 1 is required, which can be generated from any suitable image registration algorithm.
[0063] Figure 10An embodiment of a process incorporating aspects of a disclosed embodiment is illustrated. For example... Figure 10 As shown, the process or computer-implemented method includes: using a hardware processor to generate a clear scaffold image and a scaffold-free background image from image frames of an image frame sequence. In one embodiment, a first scaffold image of the image frame sequence is transformed 1002 to the image space of the image frame sequence using a first spatial transformer network (STN0) to generate a first transformed scaffold image. In one embodiment, the corresponding balloon marker position is used as the input to the first spatial transformer network.
[0064] A new background image 1004 is generated from the first transformed scaffold image. In one embodiment, a subtraction operation is used, where the corresponding image frame from the image frame sequence is used as input to the subtraction operation.
[0065] The new background image is transformed to the background image space using a second spatial transformer network (STN1) to generate a scaffold-free background image (B). k In one embodiment, the separated background image is the input to a second spatial transformer network (STN1). In another embodiment, these steps are repeated for all available frames, and the results are processed in an averaging layer to generate an unsupported background image.
[0066] The second spatial transformer network (STN1) is used to transform the unsupported background image (B) k The image space of the 1008 image is transformed to the image space of the image frame sequence. The corresponding frame image from the image frame sequence is the input of the second spatial transformer network (STN1). In this example, the second spatial transformer network (STN1) is used twice.
[0067] A 1010 scaffold image is generated from a transformed scaffold-free background image in the image space of an image frame sequence. In one embodiment, a subtraction operation is used, wherein the corresponding image frame from the image frame sequence is the input to the subtraction operation.
[0068] The result of the subtraction operation is fed into a first spatial transformer network (STN0), where the generated stent image is transformed 1012 to the stent image space. In one embodiment, the balloon marker location is the input to the first spatial transformer network. These steps are repeated for all available image frames, and an averaging layer is used to generate a clear stent image S. k .
[0069] like Figure 1 For example, device 100 includes at least a processor 106, a memory 108, and a neural network 110. Processor 106 is communicatively coupled to memory 108 and neural network 110. In one embodiment, processor 106 is configured to acquire X-ray image sequences (I0, I1, ..., I...). n) and balloon marker location or location data M ref M n , which serves as the input to neural network 110.
[0070] The output of neural network 110 is a clear scaffold image S. k And the background image B without support k The function "f" is implemented through the neural network 110 combined with the operation of the processor 106.
[0071] Device 100 includes appropriate logic, circuitry, interfaces, and / or code configured to perform and execute the processes described herein. Examples of device 100 may include, but are not limited to, application servers, web servers, database servers, file servers, cloud servers, or combinations thereof.
[0072] Processor 106 includes appropriate logic, circuitry, interfaces, and / or code configured to process multiple images (or sequences of image frames) using neural network 110. Processor 106 is configured to respond to and process instructions from driving device 100. Examples of processor 106 include, but are not limited to, microprocessors, microcontrollers, complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, or any other type of processing circuitry. Optionally, processor 106 may be one or more separate processors, processing devices, and various elements associated with the processing devices that may be shared by other processing devices. Additionally, one or more separate processors, processing devices, and elements are arranged in various architectures to respond to and process instructions from driving device 100. In one embodiment, processor 106 is a hardware processor configured to execute machine-readable instructions to perform the processes generally described herein.
[0073] In one embodiment, neural network 110 refers to an artificial neural network configured to receive input, compress the input, and decompress the compressed input to generate an output such that the generated output resembles the received input. In other words, neural network 110 is used to reduce the size of input data to a smaller representation, and can reconstruct the original data from the compressed data whenever the original data is needed.
[0074] In one aspect, the disclosed embodiments include a training phase and an operation phase. During the training phase, a neural network 110 is trained using training data so that the neural network 110 can perform a specific, intended function during the operation phase. The processor 106 is configured to perform unsupervised or semi-supervised training of the neural network 110 using the training data. In unsupervised training of the neural network 110, unlabeled training data is used to train the neural network 110. Furthermore, in semi-supervised training of the neural network 110, a smaller amount of labeled training data and a larger amount of unlabeled training data are used to train the neural network 110.
[0075] Also refer to Figure 1 The memory 108 may include appropriate logic, circuitry, interfaces, and / or code, and may be configured to store instructions executable by the processor 106. The memory 108 is also configured to store data 108 as generally described herein. The memory 108 may also be configured to store the operating system and associated applications of the device 100, including the neural network 110. Examples of implementations of the memory 108 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), hard disk drive (HDD), flash memory, and / or secure digital storage (SD) cards. Non-transitory computer-readable storage media may include, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof.
[0076] The disclosed embodiments use a neural network to separate scaffolded and unscaffolded layers without online optimization. Scaffolded and unscaffolded motions do not need to be explicitly estimated online. A mapping function f maps the input, sequence of images, and marker coordinates into two single image outputs. Function f is implemented entirely via a neural network. Therefore, the method disclosed herein is faster than conventional optimization-based methods. Furthermore, the network is trained on a large amount of data and is significantly more robust than non-learning-based methods.
[0077] The various embodiments and variations of the aforementioned device 100, with necessary modifications, are applicable to this method. The method described herein is computationally efficient and does not impose a processing burden on the processor 106.
[0078] Modifications to embodiments of the disclosed embodiments described above are possible without departing from the scope of the embodiments described herein. Expressions such as “comprising,” “incorporating,” “having,” and “are” are used to describe and claim aspects of the disclosed embodiments and are intended to be interpreted in a non-exclusive manner, allowing for the presence of additional items, components, or elements not explicitly described. References to the singular are also interpreted to refer to the plural.
[0079] Therefore, while the essential novel features of the invention applied to exemplary embodiments thereof have been shown, described, and pointed out, it should be understood that those skilled in the art may make various omissions, substitutions, and changes in the form and details of the illustrated devices and methods, as well as in their operation, without departing from the spirit and scope of the currently disclosed invention. Furthermore, it is clearly contemplated that all combinations of those elements that perform substantially the same function in substantially the same manner to achieve the same result are within the scope of the invention. Moreover, it should be recognized that structures and / or elements shown and / or described in connection with any disclosed form or embodiment of the invention may be incorporated into any other disclosed or described or suggested form or embodiment as a general matter of design choice.
Claims
1. A device for visualizing a support structure, the device including a hardware processor configured to: The first stent image is transformed to the X-ray image space using a first spatial transformer network to generate a first transformed stent image in the X-ray image space, wherein... The first transformed support image is aligned with the X-ray image coordinate system in the X-ray image space; A new, unsupported background image in the X-ray image space is generated based on the first transformed scaffold image and the X-ray image. The new scaffold-free background image in the X-ray image space is transformed to the background image space using a second spatial transformer network. The new scaffold-free background image in the X-ray image space and the image frames in the background image space are used as the input of the second spatial transformer network. The input of the first average pooling layer is the new scaffold-free background image transformed to the background image space, so as to generate a new scaffold-free background image in the background image space as the output of the first average pooling layer. The second spatial transformer network is used to transform the supportless background image in the new background image space to the X-ray image space, resulting in the transformed supportless background image in the X-ray image space. A scaffold image in the X-ray image space is generated based on the transformed scaffold-free background image and the X-ray image; as well as The first spatial transformer network is used to transform the stent image in the X-ray image space to the stent image space, and the stent image in the stent image space is used as the input of the second averaging layer to generate a clear stent image for stent visualization as the output of the second averaging layer.
2. A method for visualizing a scaffold, comprising: The method of generating a sharp scaffold image and a scaffold-free background image from image frames of an image frame sequence using a hardware processor includes: using the hardware processor to: The first stent image is transformed to the X-ray image space using a first spatial transformer network to generate a first transformed stent image in the X-ray image space, wherein the first transformed stent image is aligned with the X-ray image coordinate system in the X-ray image space. A new, unsupported background image in the X-ray image space is generated based on the first transformed scaffold image and the X-ray image. The new scaffoldless background image in the X-ray image space is transformed to the background image space using a second spatial transformer network. The new scaffoldless background image in the X-ray image space and the image frames in the background image space are used as the input of the second spatial transformer network. The input of the first average pooling layer is the new scaffoldless background image transformed to the background image space, so as to generate the scaffoldless background image in the new background image space, which is used as the output of the first average pooling layer. The second spatial transformer network is used to transform the scaffoldless background image in the new background image space to the X-ray image space, resulting in the transformed scaffoldless background image in the X-ray image space. Generate a support image in the X-ray image space based on the transformed supportless background image and the X-ray image; and The first spatial transformer network is used to transform the stent image in the X-ray image space to the stent image space, and the stent image in the stent image space is used as the input of the second averaging layer to generate the clear stent image as the output of the second averaging layer.
3. The method according to claim 2, wherein, The method further includes: using the balloon marker position of the stent as input to the first spatial transformer network to generate the first transformed stent image in the X-ray image space.
4. The method according to claim 2, wherein, The method further includes: taking the corresponding X-ray image of the image frame sequence and the first transformed scaffold image in the X-ray image space as input to the subtraction operation, so as to obtain the new scaffold-free background image in the X-ray image space as the output of the subtraction operation.
5. The method according to claim 2, wherein, The method further includes: using a separated background image as input to the second spatial transformer network to transform the new scaffold-free background image in the X-ray image space to the background image space, wherein the method further includes: using a corresponding image from the image frame sequence as input to the second spatial transformer network to transform the new scaffold-free background image in the X-ray image space to the background image space of the image frame sequence.
6. The method according to claim 2, wherein, The method further includes: taking the corresponding X-ray image of the image frame sequence and the transformed unsupported background image in the X-ray image space as input to the subtraction operation, so as to obtain the support image in the X-ray image space as the output of the subtraction operation.
7. The method according to claim 2, wherein, The method further includes generating a clear image of the stent by using the balloon marker position of the stent as input to the first spatial transformer network.
8. The method according to claim 2, wherein, The method further includes: using the balloon marker positions of the stent as input to the second spatial transformer network to generate the stent-free background image, wherein the order of the balloon marker positions input to the second spatial transformer network is reversed relative to the order of the balloon marker positions input to the first spatial transformer network.
9. A computer program product comprising a non-transitory computer-readable medium having machine-readable instructions stored thereon, the machine-readable instructions, when executed by a computer, causing the computer to perform the method according to any one of claims 2-8.