Medical image processing apparatus and medical image processing method

Abstract

Perform 2D / 3D alignment with high precision. [Solution] The medical image processing apparatus according to the embodiment includes a processing circuit. The processing circuit generates a first map representing the estimated location of a medical device within an anatomical region based on a 2D (two-dimensional) image acquired by a medical imaging device, which represents the medical device within an anatomical region of a patient or other subject. The processing circuit generates a second map representing the expected location of the medical device within an anatomical region based on volumetric imaging data representing the anatomical region and a plurality of alignment parameters that define the pose of the medical imaging device. The processing circuit compares the first map and the second map with each other. The processing circuit updates the plurality of alignment parameters based on the comparison between the first map and the second map.

Description

Technical Field

[0001] The present invention relates to a medical image processing apparatus and a medical image processing method.

Background Art

[0002] 2D / 3D registration is a technique for estimating the spatial relationship between a three-dimensional (3D) structure, such as volumetric imaging data, a model of a medical instrument, or other 3D structures, and its two-dimensional (2D) image. For example, 2D / 3D registration can be used in medical procedures to align volumetric imaging data, which represents an anatomical region or structure that can be obtained by other X-ray scanners such as computed tomography (CT), magnetic resonance scanners, ultrasonic scanners, or other medical scanners, with a 2D image representing the anatomical region or structure. Using 2D / 3D registration, an optimal geometric transformation for aligning the representation of the 3D structure with the 2D image can be obtained. This is also called an optimization problem.

[0003] An example of a medical procedure that requires 2D / 3D registration is transcatheter aortic valve implantation. To align volumetric imaging data obtained by a CT scanner with a 2D image, an X-ray image simulated based on the volumetric imaging data is generated. This X-ray simulation image is also called a digitally reconstructed radiograph (DRR). The pose of the X-ray source of the medical imaging device used to obtain the 2D image, such as the position and angle, can be estimated by comparing the 2D image and the DRR. Using an optimization process, the degrees of freedom, which may include six degrees of freedom, can be determined, and the pose of the X-ray source can be determined.

[0004] The 2D images representing anatomical regions or structures may be part of a live image sequence acquired by a medical imaging technique such as fluoroscopy or another medical imaging method. These 2D images may also be referred to as intraoperative 2D images. As is often the case during medical procedures, if the field of view of the 2D image shrinks, 2D / 3D alignment will become difficult and / or unclear. Manual alignment of the X-ray simulation image and the 2D image will be necessary, which will be extremely time-consuming. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] U.S. Patent Application Publication No. 2008 / 247621 [Overview of the project] [Problems that the invention aims to solve]

[0006] One of the problems that the embodiments disclosed herein and in the drawings aim to solve is to perform 2D / 3D alignment with high precision. However, the problems that the embodiments disclosed herein and in the drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0007] The medical image processing apparatus according to the embodiment includes a processing circuit. The processing circuit generates a first map representing the estimated location of a medical device within an anatomical region based on a 2D (two-dimensional) image acquired by a medical imaging device, which represents the medical device within an anatomical region of a patient or other subject. The processing circuit generates a second map representing the expected location of the medical device within an anatomical region based on volumetric imaging data representing the anatomical region and a plurality of alignment parameters that define the pose of the medical imaging device. The processing circuit compares the first map and the second map with each other. The processing circuit updates the plurality of alignment parameters based on the comparison between the first map and the second map. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram of a medical image processing device according to the embodiment. [Figure 2] Figure 2 is a flowchart showing an overview of the process in the embodiment. [Figure 3A] Figure 3A shows a 2D image representing an anatomical region of a patient or other subject. [Figure 3B] Figure 3B shows a 2D image representing the anatomical region shown in Figure 3A. [Figure 3C] Figure 3C shows a 2D image representing the difference between the 2D image in Figure 3A and the 2D image in Figure 3B. [Figure 4] Figure 4 shows a graph representing the mutual information between the 2D image in Figure 3A and the 2D image in Figure 3B. [Figure 5A] Figure 5A shows a 2D image of Figure 3B. [Figure 5B] Figure 5B shows a 2D image of Figure 5A with the background removed. [Figure 5C] Figure 5C shows an exemplary first map generated based on the 2D image in Figure 5B. [Figure 6A] Figure 6A shows a 2D image representing a cross-sectional view of volumetric imaging data representing an anatomical region. [Figure 6B] Figure 6B shows a 2D image generated by rendering the volumetric imaging data represented by the 2D image in Figure 6A. [Figure 6C] FIG. 6C shows an image representing a cross-sectional view of volumetric imaging data representing an anatomical region having aortic segmentation. [Figure 6D] FIG. 6D shows a 2D image representing the anatomical region after segmentation. [Figure 6E] FIG. 6E shows the 2D image of FIG. 3A including the outline of the aorta. [Figure 6F] FIG. 6F shows an exemplary distance map generated based on the 2D image shown in FIG. 6D. [Figure 7] FIG. 7 shows an exemplary spline representing the predicted location and / or route of a medical device within volumetric imaging data representing an anatomical region. [Figure 8A] FIG. 8A shows a 2D image representing a simulation of a medical device. [Figure 8B] FIG. 8B shows a 2D image representing a simulation of an anatomical region. [Figure 8C] FIG. 8C shows a 2D image representing the difference between the 2D image of FIG. 8A and the 2D image of FIG. 8B. [Figure 8D] FIG. 8D shows a 2D image representing a simulation of an anatomical region with the size or dimension of the simulation of the anatomical region enlarged. [Figure 8E] FIG. 8E shows a 2D image representing the difference between the 2D image of FIG. 8A and the 2D image of FIG. 8D. [Figure 9A] FIG. 9A shows a 2D image representing an anatomical region of a patient or other subject. [Figure 9B] FIG. 9B shows a 2D image representing an anatomical region. MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, embodiments of a medical image processing apparatus and a medical image processing method will be described in detail with reference to the drawings.

[0010] One embodiment provides a medical image processing apparatus including a processing circuit. The processing circuit generates a first map representing an estimated location of a medical device within an anatomical region of a patient or other subject, the first map being generated based on a two-dimensional (2D) image acquired by a medical imaging device, the 2D image representing the medical device within the anatomical region, and generates a second map representing a predicted location of the medical device within the anatomical region, the second map being generated based on volumetric imaging data representing the anatomical region and a plurality of alignment parameters, the plurality of alignment parameters defining a pose of the medical imaging device, compares the first map with the second map, and updates the plurality of alignment parameters based on the comparison between the first map and the second map.

[0011] The processing circuit may initialize an alignment of at least one of the medical imaging device and / or a 2D image sequence having the volumetric imaging data, for example, based on a plurality of updated alignment parameters. The 2D image sequence may at least include the 2D image and / or one or more subsequent 2D images representing the medical device within the anatomical region.

[0012] The predicted location of the medical device within the anatomical region may be part of planning data. The processing circuit may generate the second map based on the planning data.

[0013] The processing circuit may project the planning data onto one or more 2D images, for example, based on a plurality of updated alignment parameters. The one or more 2D images may at least include the 2D image and / or one or more subsequent 2D images representing the medical device within the anatomical region.

[0014] The processing circuit may determine the planning data based on a medical procedure to be performed.

[0015] The aforementioned planning data may be defined by the user.

[0016] The planning data may include at least one of the following: the route of the medical device within the anatomical region, and / or the location of an anatomical structure.

[0017] The aforementioned medical procedure to be performed may include transcatheter aortic valve implantation. The aforementioned anatomical region may include the aorta of the patient or other subject.

[0018] The processing circuit may perform at least one of the following steps, one or more times: generating the first map, generating the second map, comparing the first map with the second map, and / or updating the plurality of alignment parameters based on the comparison between the first map and the second map.

[0019] In the first iteration, the plurality of alignment parameters may include a plurality of initial alignment parameters. In each subsequent iteration, the plurality of alignment parameters may include a plurality of updated alignment parameters. The plurality of updated alignment parameters may include at least one alignment parameter that is updated relative to the corresponding previous alignment parameter. In each subsequent iteration, the processing circuit may generate the second map based on the plurality of updated alignment parameters.

[0020] The processing circuit may detect the medical device in the 2D image. The processing circuit may use filtering methods and / or machine learning methods to detect the medical device in the 2D image.

[0021] The processing circuit may segment the anatomical region within the volumetric imaging data. The processing circuit may project the segmented anatomical region onto the image plane to generate, for example, an image representing the segmented anatomical region. The segmented anatomical region may be projected onto the image plane based on the plurality of alignment parameters.

[0022] The processing circuit may, for example, generate a binary image representing the anatomical region after the segment based on the generated image. The processing circuit may enlarge the size or dimensions of the anatomical region after the segment. The processing circuit may generate a distance map based on the generated image.

[0023] The processing circuit may generate the second map based on one or more properties of the medical device.

[0024] The 2D image may include a fluorescence fluoroscopy image. The field of view of the 2D image may be reduced to focus on the anatomical region, or it may be zoomed in.

[0025] The aforementioned anatomical region may include arteries, veins, or organs of the patient or other subject. The organs may include cylindrical or tubular shapes.

[0026] One embodiment provides a medical image processing method comprising: generating a first map representing the estimated location of a medical device within an anatomical region of a patient or other subject, the first map being generated based on a two-dimensional (2D) image acquired by a medical imaging device, the 2D image representing the medical device within the anatomical region; generating a second map representing the expected location of the medical device within the anatomical region, the second map being generated based on volumetric imaging data representing the anatomical region and a plurality of alignment parameters, the plurality of alignment parameters defining the pose of the medical imaging device; comparing the first map and the second map with each other; and updating the plurality of alignment parameters based on the comparison between the first map and the second map.

[0027] A medical image processing device 10 according to an embodiment is schematically shown in Figure 1. The medical image processing device 10 includes a computing device 12, which may be provided in the form of a personal computer (PC) or a workstation. In this embodiment, the computing device 12 is connected to a scanner 14, for example, via a data storage unit 16. However, in other embodiments, it would be preferable that the medical image processing device is not connected to or coupled to any scanner.

[0028] The medical image processing device 10 further comprises one or more display screens 18 and one or more input devices 20 such as a computer keyboard, mouse, or trackball.

[0029] In this embodiment, the scanner 14 is a computed tomography (CT) scanner. However, in other embodiments, it would be preferable that the scanner be another medical scanner, such as a magnetic resonance scanner, an ultrasound scanner, or another medical scanner. The scanner 14 is configured to generate volumetric image data representing anatomical regions of a patient or other subject.

[0030] Volumetric image data contains multiple voxels arranged in a three-dimensional (3D) grid. Each voxel has an associated voxel value. The voxel value represents a measurement of a physical parameter. For example, in a CT scan, the voxel value represents the opacity of the voxel to X-rays, i.e., the X-ray stopping power. The X-ray stopping power is measured in Hounsfield units (HUs) (unit volume mass), which are closely correlated with density.

[0031] In this embodiment, the volumetric image dataset obtained by the scanner 14 is stored in the data storage unit 16 and then provided to the computing device 12. In an alternative embodiment, the volumetric image dataset may be supplied from a remote data storage unit (not shown). The data storage unit 16 or the remote data storage unit may comprise any suitable form of memory storage unit.

[0032] In this embodiment, the computing device 12 is connected to the medical imaging device 22. The medical imaging device 22 is configured to acquire a first two-dimensional (2D) image representing a medical device within an anatomical region. The medical device may also be referred to as a tool, instrument, or interventional object.

[0033] In this embodiment, the first 2D image is part of an image sequence, such as a live image sequence representing an anatomical region. For example, the image sequence, such as the first 2D image, is acquired by fluoroscopy or other medical imaging techniques. The first 2D image may also be referred to as a frame of the image sequence of the anatomical region. The medical imaging device 22 is configured to perform fluoroscopy, for example, to acquire an image sequence of an anatomical region. The image sequence may be part of a live video representing a medical device within the anatomical region. The medical imaging device 22 comprises a radiation source 22a, such as an X-ray source, and a detector 22b, such as a fluorescent screen. For example, the medical imaging device 22 is provided in the form of a fluoroscope. During use, a patient or other subject is placed between the X-ray source and the fluorescent screen. During use, X-rays emitted by the X-ray source pass through the patient or other subject. The X-rays attenuate as they pass through different tissues of the patient or other subject's body. The X-rays that have passed through the patient or other subject's body are detected on the fluorescent screen. As X-rays passing through the body of a patient or other subject interact with atoms on a screen via the photoelectric effect, an image is generated on the screen. The first 2D image may be called an image in surgery. The image sequence may also be called a 2D medical image sequence. In this embodiment, the pose of the radiation source 22a relative to the detector 22b is fixed, such as its position and orientation. For example, the radiation source 22a and the detector 22b may be mounted on a support such as an arm, C-arm, or other support in a fixed pose relative to each other. In other embodiments, it would be preferable that the configuration or arrangement of the medical imaging device differs. For example, in such other embodiments, the medical imaging device may include another radiation source and / or another detector, and / or at least one of the radiation source and detector may be movable relative to at least the other.

[0034] The computing device 12 includes a processing circuit 24 for data processing. The processing circuit 24 comprises a central processing unit (CPU) and a graphical processing unit (GPU). The processing circuit 24 provides processing resources for automatically or semi-automatically processing volumetric image datasets and / or medical image datasets. In other embodiments, the data to be processed may include any image data, and may not be medical image data.

[0035] In this embodiment, the computing device 12 includes a rendering circuit 26 configured to generate a second 2D image from volumetric image data representing an anatomical region. For example, the processing circuit 24 may include the rendering circuit 26. In this embodiment, the second 2D image includes a digitally reconstructed simulation image representing an anatomical region. The second 2D image may be called a planning image. However, in other embodiments, it would be preferable that the second 2D image include an image projected using a different projection and / or simulation method.

[0036] In this embodiment, the processing circuit 24 includes a display circuit 28 configured to display a first and / or second 2D image to the user on a display screen 18.

[0037] In this embodiment, circuits 24, 26, and 28 are each implemented on a CPU and / or GPU by a computer program having computer-readable instructions executable to perform one or more operations of the medical image processing apparatus 10 and / or medical image processing method of the embodiment described in the specification. In other embodiments, the circuits may be implemented as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0038] Furthermore, the computing device 12 includes a hard drive, an operating system including RAM, ROM, a data bus, various device drivers, and other PC components including hardware devices such as a graphics card. Such components are not shown in Figure 1 for clarity.

[0039] Figure 2 is a flowchart showing an overview of the process in the embodiment.

[0040] In stage 30, the processing circuit 24 generates a first map representing the location of a medical device within an anatomical region of the patient or other subject. The location of the medical device within an anatomical region may also be called the likely location within the anatomical region. The processing circuit 24 is configured to generate a first map based on the first 2D image described above. The first map may also be called a tool likelihood map, tool likelihood image, or first generated image. The medical device may include a catheter, a probe such as an ultrasound probe, a stent, a clip, a replacement aortic valve, and / or another medical device.

[0041] In stage 32, the processing circuit 24 generates a second map representing the expected location of the medical device within an anatomical region. The second map may be understood as a simulation of the first 2D image, for example, the location of the medical device within an anatomical region. The expected location of the medical device within an anatomical region is part of planning data, such as preoperative planning data. The planning data may further include the location of anatomical structures, and routes such as access routes, intervention routes, or other routes for the medical device within an anatomical region, and / or other information about the medical procedure or medical intervention to be performed. In some embodiments, the processing circuit 24 determines the planning data based on the medical procedure to be performed. For example, the processing circuit 24 determines the planning data based on a standardized plan for the medical procedure. In other embodiments, the user defines the planning data. The medical procedure may be called an intervention. The second map may also be called a tool probability map or image, probability map or image, 2D probability map or image, or second generated image. Generating the second map may help solve the optimization problems described above. Planning data can also be called planning information, such as pre-operative planning information.

[0042] The processing circuit 24 generates a second map based on volumetric imaging data representing anatomical regions, as will be described in detail below. For example, the rendering circuit 26 generates a second map based on volumetric image data received from the scanner 14 or the data storage unit 16.

[0043] In this embodiment, the volumetric imaging data includes volumetric imaging data acquired by a CT scanner. The volumetric imaging data may be referred to as a medical image volume (CT) or a 3D CT volume. In other embodiments, it would be preferable that the volumetric imaging data be acquired by any other suitable scanner.

[0044] The processing circuit 24 further generates a second map based on a plurality of alignment parameters. In this embodiment, the alignment parameters determine the pose of the medical imaging device 22.

[0045] In this embodiment, the medical imaging device 22 is movable relative to an anatomical region of a patient or other subject. The medical imaging device 22 has multiple degrees of freedom. For example, the medical imaging device 22 has six degrees of freedom, such as three translational degrees of freedom and three rotational degrees of freedom. The medical imaging device 22 is movable in the x, y, and z directions, for example, in the x-axis, y-axis, and z-axis directions of a three-dimensional coordinate system. The medical imaging device 22 is rotatable about the x, y, and z axes. The pose of the medical imaging device 22 can be characterized by six coordinates that define the position about the x, y, and z axes in a three-dimensional coordinate system, and each rotation about the x, y, and z axes. The rotation of the medical imaging device 22 may also be called the orientation or angle with respect to at least one of the x, y, and z axes. The alignment parameters may also be called 2D / 3D alignment parameters. In some embodiments, it would be preferable that the alignment parameters determine the pose of part of the medical imaging device. For example, a part of the medical imaging device 22 includes a radiation source 22a, such as an X-ray source. In such an embodiment, the features described above may be applied to the radiation source 22a.

[0046] In stage 34, the processing circuit 24 compares the first map and the second map with each other. For example, the processing circuit 24 compares the first map and the second map with each other based on one or more image metrics, such as one or more similarity metrics. For example, the image metrics include negative normalized cross-correlation, cross-information, or other suitable image metrics. The processing circuit 24 determines one or more similarities or one equivalence between the first map and the second map. This may enable deambiguation of the alignment between the anatomical region represented by the volumetric imaging data and the anatomical region represented by the first 2D image. In this way, the comparison between the first map and the second map may enable deambiguation of the optimization problem described above and / or enable the optimization of the alignment parameters. For example, the estimation of the pose of the medical imaging device 22 may be facilitated by comparing the estimated location of the medical device within the anatomical region with the predicted location of the medical device within the anatomical region.

[0047] In stage 36, the processing circuit 24 updates or optimizes the alignment parameters based on a comparison between the first map and the second map. The processing circuit 24 may use an optimization algorithm or process. To optimize or update the alignment parameters, for example, to minimize the image metric, the optimization algorithm or process may include a machine learning algorithm or process such as a slope descent algorithm or any other suitable machine learning algorithm.

[0048] The processing circuit repeats stages 30 to 36 once or multiple times.

[0049] In some embodiments, in each iteration, the processing circuit 24 repeats stages 32-36 once or more times until the image metric is minimized. For example, stages 32-36 may be repeated once or more times using the same first map.

[0050] In some embodiments, the processing circuit 24 repeats stages 30 to 36. For example, in one or more subsequent iterations, stages 30 to 36 may be repeated for one or more subsequent 2D images in the image sequence. In each iteration, the processing circuit 24 generates a first map based on a first 2D image in the image sequence and performs stages 32 to 36 based on the first map. As described above, stages 32 to 36 may be repeated one or more times using the same first map.

[0051] In the above embodiment, in the first iteration, the multiple alignment parameters include a plurality of initial alignment parameters in stage 32. The processing circuit 24 may determine the initial alignment parameters from a graph representing mutual information between the first 2D image and the second 2D image, or from another image representing volumetric imaging data. The plurality of initial alignment parameters may be called coarse analysis alignment parameters. In each subsequent iteration, the multiple alignment parameters include a plurality of updated alignment parameters in stage 32. The plurality of updated alignment parameters include at least one alignment parameter that has been updated relative to the corresponding previous alignment parameter. In each subsequent iteration, the processing circuit 24 generates a second map based on the plurality of updated alignment parameters. This may enable the second map to be aligned to the medical device in the first 2D image that needs updating and / or improvement. The generation of the second map based on the updated alignment parameters may be called updating the second map.

[0052] In stage 38, the processing circuit 24 initializes the alignment of the medical imaging device 22 to the volumetric imaging data based on a plurality of updated alignment parameters. This also initializes the alignment of one or more subsequent first 2D images from an image sequence, such as the first 2D image and / or image sequence, to the volumetric imaging data based on the plurality of updated alignment parameters. The alignment of subsequent first 2D images from a 2D image sequence to the volumetric imaging data may be constrained to alignment parameters that are the same as or similar to the updated alignment parameters. The processing circuit 24 stores the plurality of updated alignment parameters as constraints for the alignment of the medical imaging device 22 and / or the first 2D images to the volumetric imaging data, for example, in the memory storage unit of the computing device 12.

[0053] Figure 3A shows a 2D image representing an anatomical region of a patient or other subject. This 2D image is a digitally reconstructed simulation image generated from volumetric image data representing the anatomical region. The anatomical region includes the aorta of the patient or other subject's heart. However, in other embodiments, it would be preferable that the anatomical region includes another anatomical structure or region. For example, in other embodiments, the anatomical region may include another artery, vein, organ, or another anatomical structure of the patient or other subject. In such other embodiments, the organ may include cylindrical or tubular structures. For example, the organ may include the esophagus, rectum, etc. The term anatomical structure may be understood as including an organ or any other anatomical structure of the patient's or subject's body. The anatomical structure may be part of the anatomical region.

[0054] The 2D image shown in Figure 3A is a digitally reconstructed simulation image generated from volumetric image data representing an anatomical region. In Figure 3A, the bones of the patient's or other subject's skeleton are dominant in the 2D image, making it difficult to identify the features of the aorta. The bones of the skeleton are indicated by reference numeral 40 in Figure 3A. The 2D image shown in Figure 3A represents a planning image containing planning data for a medical procedure to be performed. In this embodiment, the medical procedure to be performed includes transcatheter aortic valve implantation (TAVI) procedure. In this embodiment, the medical device is accessed via the aorta. However, in other embodiments, it would be preferable that the medical procedure to be performed include different medical procedures. For example, in other embodiments, the medical procedure may include vascular interventional procedures, ultrasound procedures such as prostate / rectal ultrasound, or endoscopic ultrasound. The 2D image shown in Figure 3A can be considered as the second 2D image described above.

[0055] Figure 3B shows a 2D image representing the anatomical region shown in Figure 3A. The 2D image shown in Figure 3B was acquired using the fluorescence fluoroscopy method described above. In the 2D image shown in Figure 3B, the bones 40 of the patient's or other subject's skeleton are visible. In this embodiment, a part of the medical device 42, which is in the form of a catheter, is also visible in the 2D image shown in Figure 3B. The 2D image shown in Figure 3B can be considered the first 2D image described above.

[0056] In Figures 3A and 3B, the field of view is reduced. Reducing the field of view in a 2D image can make 2D / 3D alignment extremely difficult. However, reducing the field of view may be necessary to focus on or zoom in on anatomical regions. In this embodiment, the field of view reduction is to focus on the aorta, not on the skeleton of the patient or other subject.

[0057] Figure 3C shows a 2D image representing the difference between the second 2D image shown in Figure 3A and the first 2D image shown in Figure 3B. The 2D image shown in Figure 3C was obtained by subtracting the first 2D image shown in Figure 3B from the second 2D image shown in Figure 3A. In the 2D image shown in Figure 3C, it can be seen that only the bones 40 of the skeleton and the medical device 42 are clearly visible. Therefore, alignment of the medical imaging device 22 with the volumetric imaging data may be difficult and / or unclear.

[0058] Figure 4 shows a graph representing the mutual information between the 2D image shown in Figure 3A and the 2D image shown in Figure 3B. The graph shown in Figure 4 may also be called a similarity graph. The graph shown in Figure 4 was generated by moving a virtual X-ray detector, which will be described in detail later, in at least one direction. In this embodiment, the virtual X-ray detector was moved in the z direction, indicated by the y-axis of the graph in Figure 4. The z direction represents the degrees of freedom of the virtual X-ray detector. Each time the virtual X-ray detector is moved, a digitally reconstructed simulation image is generated from volumetric image data representing the anatomical region. This simulation image is compared with a 2D image representing the anatomical region obtained using the fluorescence fluoroscopy method described above. Based on this comparison, an image metric is determined. In this embodiment, the image metric includes mutual information. It can be seen that many minute 44 representing the bones 40 of the skeleton shown in Figures 3A-3C are present in Figure 4. The repetition of the bones 40 of the skeleton shown in Figures 3A-3C can obscure the alignment between corresponding anatomical structures in Figures 3A and 3B, making 2D / 3D alignment difficult. Even if the virtual X-ray detector is moved in only one direction, 2D / 3D alignment may be difficult in this example, so only one degree of freedom of the virtual X-ray detector is changed.

[0059] For example, 2D / 3D alignment may be used to bring planning data represented by the 2D image shown in Figure 3A and intervention data represented by the image shown in Figure 3B into the same coordinate frame. The process shown in Figure 2 may enable optimization of the alignment parameters that determine the pose of the medical imaging device 22. This may enable a more robust initialization of the alignment of the medical imaging device 22 to volumetric imaging data. This, in turn, may enable a more robust alignment of the volumetric imaging data to the image sequence.

[0060] Figures 5A-5C illustrate the steps for generating the first map, as described with respect to stage 30 shown in Figure 2. The 2D image shown in Figure 5A corresponds to the first 2D image shown in Figure 3B. The 2D image shown in Figure 5B shows the first 2D image shown in Figure 5A with the background removed. The 2D image shown in Figure 5C shows an exemplary first map generated based on the first 2D image shown in Figure 5B.

[0061] In this embodiment, the processing circuit 24 detects the medical device 42 in a first 2D image representing an anatomical region. The processing circuit 24 detects the location of the medical device 42 in the first 2D image representing the anatomical region using a filtering technique. For example, the processing circuit 24 may apply one or more filters to the first 2D image. In this embodiment, a filter was applied to the first 2D image shown in Figure 5A. The filter may be called a classical filter. The filter includes a smoothing filter such as a median filter, an anisotropic diffusion filter, or any other suitable smoothing filter. The processing circuit 24 removes the background of the first 2D image by applying a smoothing filter. In this embodiment, a smoothing filter was applied to the first 2D image shown in Figure 5A. The filtered first 2D image is shown in Figure 5B.

[0062] The filter may further include a vesselness filter, such as a Frangi vesselness filter or any other suitable vesselness filter. The processing circuit 24 applies the vesselness filter to the first 2D image to detect the medical device 42 in the first 2D image, for example, representing an anatomical region. The processing circuit 24 extracts the medical device 42 from the first 2D image to generate a first map. In this embodiment, the medical device 42 was extracted from the first 2D image shown in Figure 5A to generate the first map 46 shown in Figure 5C. As can be seen from Figure 5C, the first map represents the estimated location of the medical device 42 within the anatomical region.

[0063] In other embodiments, the processing circuit uses a machine learning model, such as a convolutional neural network (CNN), to detect medical devices within a first 2D image and / or to extract medical devices from the first 2D image.

[0064] Figures 6A to 6F illustrate the step of generating a second map, as described with respect to stage 32 shown in Figure 2.

[0065] Figure 6A shows a 2D image representing a cross-sectional view of volumetric imaging data representing an anatomical region. Figure 6B shows a 2D image generated by rendering the volumetric imaging data represented by the 2D image in Figure 6A. In this embodiment, Figure 6B shows a digital reconstruction simulation image (DRR) of the volumetric image data representing the anatomical region shown in Figure 6A. The DRR can be understood as a simulation or composite image generated from the source volumetric image data. The rendering circuit 26 generates the DRR from the volumetric image data representing the anatomical region by, for example, casting multiple rays 48 with fixed initial energy from a virtual light source 50 through the volumetric image data. In this embodiment, the volumetric image data includes CT data, the multiple rays include multiple X-rays 48, and the virtual light source 50 includes a virtual X-ray source. The multiple X-rays 48 attenuate as they move through the CT data, and when the X-rays 48 are incident on the detector 52, the energy of the attenuated X-rays 48 is measured. In this embodiment, the detector 52 is provided in the form of a virtual X-ray detector. The detector 52 represents the image plane. The energy absorbed by the CT data is determined and converted into pixel values. For example, the rendering circuit 26 can calculate the pixel value at the location where the X-ray entered the detector 52 as a weighted average of the values ​​of the voxels through which the X-ray passed, for example using the Siddon method. In other embodiments, it would be preferable to use a different rendering process to generate a 2D image from volumetric imaging data representing an anatomical region. The other rendering process may include more complex physical-based processing, such as the DeepDRR process described in Unberath M. et al in “DeepDRR - A Catalyst for Machine Learning in Fluoroscopy-guided Procedures” (March 2018), arXiv:1803.08606 or any other suitable rendering process.

[0066] Figure 6C shows an image representing a cross-sectional view of volumetric imaging data representing an anatomical region. In this figure, the aorta 54 is segmented. For example, in this embodiment, the processing circuit 24 segments the anatomical region including the aorta. The processing circuit 24 uses a segmentation method such as machine learning or a convolutional neural network (CNN) based method, or another segmentation method, to segment the anatomical region. The processing circuit 24 may segment the anatomical region automatically. The term "segment" may be used interchangeably with the term "mask".

[0067] In some embodiments, during the segmentation of an anatomical region, the processing circuit 24 sets the values ​​of each voxel adjacent to or surrounding the anatomical region to zero. The processing circuit maintains the values ​​of each voxel that are part of the anatomical region, such as the measured Hounsfield unit values.

[0068] In some embodiments, during the segmentation of an anatomical region, the processing circuit sets the value of each voxel adjacent to or surrounding the anatomical region to zero. The processing circuit sets the value of each voxel that is part of the anatomical region to a predetermined value such as 1.

[0069] The processing circuit 24 projects the segmented anatomical region onto the image plane to generate a 2D image representing the segmented anatomical region. In some embodiments, the processing circuit 24 uses the generated 2D image as a second map. In other embodiments, the processing circuit 24 further processes the generated 2D image to generate a second map.

[0070] The processing circuit 24 projects the segmented anatomical region onto the image plane based on a plurality of alignment parameters. The processing circuit 24 projects the segmented anatomical region using the DRR method described above. Figure 6D shows a 2D image representing the segmented anatomical region. The segmented anatomical region in Figure 6D is projected without modification. For example, in some embodiments, where the processing circuit sets the values ​​of each voxel adjacent to or surrounding the anatomical region to zero and sets the values ​​of each voxel that is part of the anatomical region to a predetermined value, when the anatomical region is segmented, the generated 2D image representing the anatomical region may be a binary 2D image such as the 2D image shown in Figure 6D.

[0071] In some embodiments, the processing circuit 24 applies a threshold to the 2D image representing the segmented anatomical region. For example, in embodiments where the processing circuit maintains the values ​​of each voxel that are part of the anatomical region, once the anatomical region is segmented, the processing circuit 24 applies a threshold to the generated 2D image representing the segmented anatomical region. This may generate a 2D binary image representing the segmented anatomical region, such as the 2D image shown in Figure 6D. For example, the threshold may be a pixel value that divides the pixels of the 2D image representing the segmented anatomical region into at least a first and a second portion. The first portion of the pixel includes one or more pixels having a pixel value higher than the threshold. The second portion of the pixel includes one or more pixels having a pixel value lower than the threshold. For example, the threshold may be a pixel value greater than zero. For example, pixels that are part of the segmented anatomical region may be part of the first portion of the pixel. The processing circuit 24 may assign a predetermined value, such as 1, to each pixel in the first portion of the pixel. As a result, a 2D binary image may be generated.

[0072] In this embodiment, the processing circuit 24 generates an outline 56 of the segmented anatomical region based on the generated 2D binary image representing the segmented anatomical region. The processing circuit 24 projects the outline 56 of the segmented anatomical region onto the generated 2D image, as shown in Figure 6D.

[0073] Figure 6E shows a second 2D image shown in Figure 3A, including an outline 56 of a post-segmented anatomical region, such as the post-segmented aorta. The processing circuit 24 projects the outline of the post-segmented anatomical region onto the second 2D image. The processing circuit 24 further projects the outline 56 of the anatomical region onto the second 2D image based on a number of alignment parameters. The outline 56 of the anatomical region may be projected onto the second 2D image to indicate the location of the anatomical region and / or the expected location of the medical device.

[0074] As described above, in some embodiments, the processing circuit 24 further processes the generated 2D images representing the segmented anatomical regions. For example, the processing circuit 24 generates a distance map based on the generated 2D images representing the segmented anatomical regions. In such embodiments, the processing circuit generates the distance map using the distance from the center of the segmented anatomical region. Figure 6F shows an exemplary distance map generated based on the generated 2D images representing the segmented anatomical regions shown in Figure 6D. In this embodiment, the distance map represents a second map.

[0075] As will be explained in detail below, additionally or alternatively, the size or dimensions of the anatomical region after segmentation may change, for example, by becoming larger.

[0076] In other embodiments, the processing circuit 24 uses a 2D image representing the segmented anatomical region, such as shown in Figure 6D, as a second map. The calculation of pixel values ​​at each location where X-rays enter the detector 52 is similar in nature to the distance transformation described above. This allows the 2D image representing the segmented anatomical region to be used as a second map without requiring further processing.

[0077] In some embodiments, the processing circuit 24 generates a second map based on one or more properties of the medical device. For example, the processing circuit considers the properties of the medical device when determining the expected location of the medical device within an anatomical region. For example, the processing circuit 24 encodes one or more properties of the medical device in the second map. The properties of the medical device may include the minimum curvature of the medical device and / or another property of the medical device. The properties of the medical device may also be called mechanical features.

[0078] As described above, in some embodiments, the planning data is defined by the user. For example, the user may manually annotate the expected location and / or route of the medical device within volumetric imaging data representing an anatomical region. The user may manually annotate the planning data within volumetric imaging data. The expected location and / or route of the medical device may be annotated as a spline or spline function in volumetric imaging data representing an anatomical region. Alternatively, the expected location and / or route may be annotated using a point annotation method or tool or another suitable annotation method or tool.

[0079] Figure 7 shows an exemplary spline representing the expected location and / or route of a medical device in volumetric imaging data representing an anatomical region. In the example shown in Figure 7, the spline 55 is piecewise defined by three polynomials to interpolate between points P0 and P2. However, in other embodiments, it would be preferable to define the spline by four or more polynomials or two or fewer polynomials.

[0080] In such embodiments, the processing circuit 24 defines 3D regions in volumetric imaging data representing planning data. For example, the processing circuit 24 defines 3D regions such as cylindrical 3D regions along splines. The processing circuit 24 segments or extrudes the 3D regions from the volumetric imaging data. The features described above regarding anatomical region segmentation may also be applied to 3D region segmentation.

[0081] The processing circuit 24 projects the segmented or extruded 3D region onto the image plane to generate a second map. In this embodiment, the segmented or extruded 3D region replaces the segmented anatomical region described above. Thus, the processing circuit 24 uses and / or processes the segmented or extruded 3D region in the same way as the segmented anatomical region described above.

[0082] In use, there will be inherent uncertainty in the alignment between the expected and estimated locations of medical devices within the anatomical region. This can be due to the fact that medical devices can be located anywhere within the anatomical region, and / or that they may move between the acquisition of volumetric imaging data and the first 2D image due to, for example, patient respiration or cardiac movement. The generation of the second map described here may allow for this inherent uncertainty in alignment. As mentioned above, the size or dimensions of the anatomical region after segmentation, or the 3D region after segmentation or extrudement, may be increased. This will improve the flexibility of the alignment between the expected and estimated locations of medical devices within the anatomical region. This will be further explained below with respect to Figures 8A-8E.

[0083] Figure 8A shows a 2D image representing a simulation 42a of the medical device. This simulation 42a of the medical device is given in the form of a guidewire.

[0084] Figure 8B shows a 2D image representing a simulation of an anatomical region. The simulation of the anatomical region is given in the form of the aorta simulation 54a.

[0085] Figure 8C shows a 2D image representing the difference between the 2D image shown in Figure 8A and the 2D image shown in Figure 8B. The 2D image shown in Figure 8C was obtained by subtracting the 2D image shown in Figure 8B from the 2D image shown in Figure 8A. From Figure 8C, it can be seen that the guidewire and the aorta are not aligned, which may affect the accuracy of 2D / 3D alignment. For example, if the capture range of the image metric is narrow, the range of initial alignment parameters for optimization or updating by the processing circuit 24 may be small. As a result, the alignment parameters for initializing 2D / 3D alignment may be inaccurate.

[0086] Figure 8D shows a 2D image representing a simulation of an anatomical region with enlarged size or dimensions. As described above, in some embodiments, the processing circuit 24 enlarges the size or dimensions of the segmented anatomical region, or the 3D region after segmentation or extrusion. For example, the size or dimensions of the segmented anatomical region, or the 3D region after segmentation or extrusion, may be enlarged before projecting the segmented anatomical region onto the image plane. Alternatively, the size or dimensions of the segmented anatomical region, or the 3D region after segmentation or extrusion, may be enlarged after projecting the segmented anatomical region onto the image plane. For example, the processing circuit 24 may further process the generated 2D image representing the segmented anatomical region, or the 3D region after segmentation or extrusion, by enlarging the size or dimensions of the segmented anatomical region, or the 3D region after segmentation or extrusion. In this embodiment, the processing circuit enlarges an anatomical region, such as the aorta simulation 54a. The expansion of the size or dimensions of an anatomical region after segmentation may also be called the expansion of the anatomical region after segmentation.

[0087] Figure 8E shows a 2D image representing the difference between the 2D image shown in Figure 8A and the 2D image shown in Figure 8D. The 2D image shown in Figure 8E was obtained by subtracting the 2D image shown in Figure 8D from the 2D image shown in Figure 8A. Figure 8E shows that the extent of the aorta has been expanded by extending the aortic simulation 54a. For example, by increasing the size or dimensions of the anatomical region, the capture range of the image metrics will be expanded compared to the example shown in Figure 8C, for example. This may expand the range of initial alignment parameters for optimization or updating in the processing circuit 24. As a result, a more robust and / or improved 2D / 3D alignment initialization will be obtained.

[0088] In some embodiments, the processing circuit 24 projects planning data onto one or more first 2D images and / or second 2D images based on a plurality of updated alignment parameters. The one or more first 2D images may include first 2D images and / or one or more subsequent first 2D images representing a medical device within an anatomical region. The first 2D images and / or subsequent first 2D images may be part of a 2D image sequence, such as a live 2D image sequence.

[0089] Figure 9A shows a 2D image representing an anatomical region of a patient or other subject. The 2D image shown in Figure 9A is a digitally reconstructed simulation image generated from volumetric image data representing the anatomical region. In other words, the 2D image is a digitally reconstructed simulation image onto which planning data is projected. In Figure 9A, the planning data includes the locations of access routes 58 and anatomical structures 60, and in this embodiment, the locations of anatomical structures 60 include the aortic valve.

[0090] Figure 9B shows a 2D image representing an anatomical region. The 2D image shown in Figure 9B was acquired using the fluorescence fluoroscopy method described above. In Figure 9B, planning data, such as the location of the access route 58 and the anatomical structure 60, is projected onto the first 2D image. In this example, it can be seen that the medical device 42 is located at the access route 58 and extends to the location of the anatomical structure 60. This will assist the user in performing the medical procedure. The processing circuit 24 may optionally project the planning data onto one or more subsequent first 2D images in the 2D image sequence, either additionally or alternatively.

[0091] One embodiment provides a medical image processing device equipped with a processing circuit. The processing circuit receives a medical image volume (CT), a 2D medical image sequence (X-ray, etc.), and details of the intervention / procedure type (e.g., TAVI), acquires the tools / instruments in the 2D image(s), generates a "tool probability map" projected from the 3D CT volume using preoperative planning information, optimizes 2D / 3D alignment parameters to update the probability map to achieve the best alignment with the tools in the 2D image, and projects the planning information onto the 2D image(s) using the optimal alignment parameters.

[0092] Intervention objects in 2D images may be automatically detected. Tools may be detected by classical filtering operations (e.g., smoothing and vesselness filters). Tools may be detected by machine learning (e.g., CNN).

[0093] Intervention objects within a 2D image may be manually annotated (using point annotation, splines, etc.).

[0094] Access via the aorta may be obtained through interventional procedures (e.g., TAVI).

[0095] The aorta may be automatically segmented.

[0096] The aorta can be projected into 2D space without modification.

[0097] The projected image may be further processed (binarization, augmentation, and a combination of distance mapping) to generate a 2D possibility map.

[0098] The probability map encodes the mechanical characteristics of the probe used (minimum curvature of the path, racing line).

[0099] The aorta may be dilated before projecting it into 2D space.

[0100] Intervention access routes may be manually annotated. Intervention access routes may be automatically determined based on a standardized plan for interventions.

[0101] Interventional procedures may be accessed via an alternative route (either by obtaining automated segmentation of anatomical structures corresponding to the access path, or by manually obtaining the route).

[0102] The subsequent "live" fluorescence fluoroscopy alignment may be restricted to parameters close to the initialization parameters.

[0103] It would be preferable to use the terms "volumetric image data" and "volumetric imaging data" interchangeably. Volumetric image data may also be called 3D CT images.

[0104] While specific circuits are described herein, in alternative embodiments, one or more functions of these circuits may be provided by a single processing resource or other component, or a function provided by a single circuit may be provided by a combination of two or more processing resources or other components. A reference to a single circuit encompasses multiple components that provide the functionality of that circuit, regardless of whether such components are separated from each other. A reference to multiple circuits encompasses a single component that provides the functionality of those circuits.

[0105] While certain embodiments are described, these embodiments are presented for illustrative purposes only and are not intended to limit the scope of the invention. In practice, the novel methods and systems described herein can be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and modifications in the forms of methods and systems described herein may be made without departing from the spirit of the invention. The claims of the appended claims and equivalents thereof are intended to cover forms and modifications that fall within the scope of the invention. [Explanation of symbols]

[0106] 10 Medical Image Processing Equipment 12 Computing Devices 14 Scanners 16 Data Storage Unit 18 Display screens 20 Input devices 22 Medical imaging devices 22a Radiation source 22b,52 detectors 24 Processing Circuits 26 Rendering Circuit 28 Display circuit Stages 30, 31, 32, 33, 34, 35, 36, 38 40 bones 42 Medical devices 42a, 54a Simulation 44 Tiny 46. ​​Map 1 48 X-rays (light rays) 50 Virtual Light Sources 54 Aorta 55 splines 56 Outline 58 Access Routes 60 Anatomical Structures P0, P1, P2 points

Claims

1. Equipped with a processing circuit, The aforementioned processing circuit is Based on a 2D (two-dimensional) image acquired by a medical imaging device representing a medical device within an anatomical region of a patient or other subject, a first map is generated representing the estimated location of the medical device within the anatomical region. Based on the volumetric imaging data representing the anatomical region and a plurality of alignment parameters defining the pose of the medical imaging device, a second map is generated representing the expected location of the medical device within the anatomical region. The first map and the second map are compared with each other. Based on the comparison between the first map and the second map, the plurality of alignment parameters are updated. Medical image processing equipment.

2. The processing circuit initializes the alignment of at least one of the medical imaging device and / or the 2D image sequence having the volumetric imaging data based on a plurality of updated alignment parameters. The 2D image sequence includes at least one or more subsequent 2D images representing the 2D image and / or the medical device within the anatomical region. The medical image processing apparatus according to claim 1.

3. The predicted location of the medical device within the anatomical region is part of the planning data. The processing circuit generates the second map based on the planning data. The medical image processing apparatus according to claim 1.

4. The processing circuit projects the planning data onto one or more 2D images based on a plurality of updated alignment parameters. The one or more 2D images include at least one or more subsequent 2D images representing the 2D image and / or the medical device within the anatomical region. The medical image processing apparatus according to claim 3.

5. The processing circuit determines the planning data based on the medical procedure to be performed. The medical image processing apparatus according to claim 3.

6. The aforementioned planning data is defined by the user. The medical image processing apparatus according to claim 3.

7. The planning data includes at least one of the route of the medical device within the anatomical region and the location of the anatomical structure. The medical image processing apparatus according to claim 3.

8. The aforementioned medical procedure to be performed includes the procedure of transcatheter aortic valve implantation, and the aforementioned anatomical region includes the aorta of the patient or other subject. The medical image processing apparatus according to claim 5.

9. The processing circuit performs one or more repetitions of at least one of the following: generating the first map, generating the second map, comparing the first map and the second map, and updating the plurality of alignment parameters based on the comparison between the first map and the second map. The medical image processing apparatus according to claim 1.

10. In the first iteration, the plurality of alignment parameters include a plurality of initial alignment parameters, and in each subsequent iteration, the plurality of alignment parameters include a plurality of updated alignment parameters, and the plurality of updated alignment parameters include at least one alignment parameter that is updated relative to the corresponding previous alignment parameter. The medical image processing apparatus according to claim 9.

11. In each subsequent iteration, the processing circuit generates the second map based on a plurality of updated alignment parameters. The medical image processing apparatus according to claim 9.

12. The processing circuit detects the medical device in the 2D image. The medical image processing apparatus according to claim 1.

13. The processing circuit uses a filtering method or a machine learning method to detect the medical device in the 2D image. The medical image processing apparatus according to claim 12.

14. The processing circuit segments the anatomical region within the volumetric imaging data. The medical image processing apparatus according to claim 1.

15. The processing circuit projects the anatomical region after the segment onto the image plane to generate an image representing the anatomical region after the segment, and the anatomical region after the segment is projected onto the image plane based on the plurality of alignment parameters. The medical image processing apparatus according to claim 14.

16. The aforementioned processing circuit is Based on the generated image, a binary image representing the anatomical region after the segment is generated, To increase the size or dimension of the anatomical region after the aforementioned segment, Based on the generated image, a distance map is generated, Perform at least one of the following: The medical image processing apparatus according to claim 15.

17. The processing circuit generates the second map based on one or more properties of the medical device. The medical image processing apparatus according to claim 1.

18. The 2D image includes a fluorescence fluoroscopy image, and the field of view of the 2D image is reduced to focus on or zoom in on the anatomical region. The medical image processing apparatus according to claim 1.

19. The aforementioned anatomical region includes arteries, veins, or organs of the patient or other subject, and the organs include cylindrical or tubular shapes. The medical image processing apparatus according to claim 1.

20. Based on a 2D (two-dimensional) image acquired by a medical imaging device representing a medical device within an anatomical region of a patient or other subject, a first map is generated representing the estimated location of the medical device within the anatomical region. Based on volumetric imaging data representing the anatomical region and a plurality of alignment parameters defining the pose of the medical imaging device, a second map is generated representing the expected location of the medical device within the anatomical region. Comparing the first map and the second map with each other, Based on the comparison between the first map and the second map, the plurality of alignment parameters are updated. A medical image processing method including [a specific term].

Images