Medical image processing apparatus, medical image processing method, and program

Abstract

To improve the efficiency of medical treatment by doctors. [Solution] The medical image processing apparatus according to the embodiment comprises an acquisition unit, an extraction unit, an estimation unit, and a display control unit. The acquisition unit acquires a first medical image and a second medical image different from the first medical image. The extraction unit extracts the shape of the structure of interest at a first time point in time when the first medical image was collected. The estimation unit estimates the shape of the structure of interest at a second time point in time, which is different from the first time point, based on the shape of the structure of interest at the first time point. The display control unit superimposes the shape of the structure of interest at the second time point on the corresponding position in the second medical image.

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to a medical image processing apparatus, a medical image processing method, and a program.

Background Art

[0002] Conventionally, techniques for aligning and displaying a medical image collected before treatment and a medical image collected after treatment, and techniques for aligning and displaying a medical image collected before treatment and a medical image collected during treatment are known. Further, a treatment simulation technique for estimating the state of a target organ after treatment using a medical image of the target organ collected before treatment is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve the treatment efficiency of doctors. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. The problems corresponding to the respective effects of each configuration shown in the embodiments described later can also be positioned as other problems.

Means for Solving the Problems

[0005] The medical image processing apparatus according to the embodiment comprises an acquisition unit, an extraction unit, an estimation unit, and a display control unit. The acquisition unit acquires a first medical image and a second medical image different from the first medical image. The extraction unit extracts the shape of the structure of interest at a first time point in time when the first medical image was collected. The estimation unit estimates the shape of the structure of interest at a second time point different from the first time point, based on the shape of the structure of interest at the first time point. The display control unit superimposes the shape of the structure of interest at the second time point onto the corresponding position in the second medical image. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows an example of the configuration of a medical image processing apparatus according to the first embodiment. [Figure 2] Figure 2 is a flowchart showing the processing procedure performed by the medical image processing apparatus according to the first embodiment. [Figure 3] Figure 3 is a diagram illustrating an example of extracting a structure of interest according to the first embodiment. [Figure 4] Figure 4 is a diagram illustrating an example of a treatment simulation according to the first embodiment. [Figure 5] Figure 5 is a diagram illustrating an example of the process for determining the superposition position according to the first embodiment. [Figure 6] Figure 6 shows an example of superimposed display according to the first embodiment. [Figure 7] Figure 7 shows an example of superimposed display according to Modification 1. [Figure 8] Figure 8 shows an example of superimposed display according to Modification 2. [Figure 9] Figure 9 shows an example of superimposed display according to Modification 3. [Figure 10] Figure 10 is a flowchart showing the processing procedure performed by the medical image processing apparatus according to the second embodiment. [Modes for carrying out the invention]

[0007] The embodiments of the medical image processing apparatus, medical image processing method, and program will be described in detail below with reference to the drawings. However, the medical image processing apparatus, medical image processing method, and program according to this application are not limited to the embodiments shown below.

[0008] (First embodiment) Figure 1 shows an example configuration of a medical image processing device according to the first embodiment. For example, as shown in Figure 1, the medical image processing device 3 according to this embodiment is connected to the medical image diagnostic device 1 and the medical image storage device 2 via a network. Note that various other devices and systems may be connected to the network shown in Figure 1.

[0009] Medical imaging device 1 captures images of a subject and generates medical images. Then, medical imaging device 1 transmits the generated medical images to various devices on the network. For example, medical imaging device 1 may be an X-ray diagnostic device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, an ultrasound diagnostic device, a SPECT (Single Photon Emission Computed Tomography) device, a PET (Positron Emission Computed Tomography) device, etc.

[0010] The medical image storage device 2 stores various medical images related to the subject. Specifically, the medical image storage device 2 receives medical images from the medical image diagnostic device 1 via a network and stores these medical images in its internal memory circuit. For example, the medical image storage device 2 can be implemented using computer equipment such as a server or workstation. Alternatively, for example, the medical image storage device 2 can be implemented using a PACS (Picture Archiving and Communication System) and store medical images in a format compliant with DICOM (Digital Imaging and Communications in Medicine).

[0011] The medical image processing device 3 performs various image processing operations on medical images collected from a subject. Specifically, the medical image processing device 3 receives medical images from the medical image diagnostic device 1 or the medical image storage device 2 via a network and performs various information processing operations using those medical images. For example, the medical image processing device 3 is implemented using computer equipment such as a server or workstation.

[0012] For example, the medical image processing device 3 includes a communication interface 31, an input interface 32, a display 33, a storage circuit 34, and a processing circuit 35.

[0013] The communication interface 31 controls the transmission and communication of various data between the medical image processing device 3 and other devices connected via the network. Specifically, the communication interface 31 is connected to the processing circuit 35 and transmits data received from other devices to the processing circuit 35, or transmits data received from the processing circuit 35 to other devices. For example, the communication interface 31 can be implemented by a network card, network adapter, NIC (Network Interface Controller), etc.

[0014] The input interface 32 receives input operations of various instructions and various information from the user. Specifically, the input interface 32 is connected to the processing circuit 35, converts the input operations received from the user into electrical signals, and transmits them to the processing circuit 35. For example, the input interface 32 is realized by a trackball, a switch button, a mouse, a keyboard, a touch pad that performs an input operation by touching an operation surface, a touch screen in which a display screen and a touch pad are integrated, a non-contact input interface using an optical sensor, a voice input interface, and the like. In this specification, the input interface 32 is not limited to only those equipped with physical operation components such as a mouse and a keyboard. For example, a processing circuit for electrical signals that receives an electrical signal corresponding to an input operation from an external input device provided separately from the apparatus and transmits this electrical signal to the control circuit is also included in the example of the input interface 32.

[0015] The display 33 displays various information and various data. Specifically, the display 33 is connected to the processing circuit 35 and displays various information and various data received from the processing circuit 35. For example, the display 33 is realized by a liquid crystal display, a CRT (Cathode Ray Tube) display, a touch panel, or the like.

[0016] The memory circuit 34 stores various data and various programs. Specifically, the memory circuit 34 is connected to the processing circuit 35, stores the data received from the processing circuit 35, or reads out the stored data and transmits it to the processing circuit 35. For example, the memory circuit 34 is realized by a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, an optical disk, or the like.

[0017] The processing circuit 35 controls the entire medical image processing apparatus 3. For example, the processing circuit 35 performs various processes according to an input operation received from a user via the input interface 32. For example, the processing circuit 35 receives data transmitted by another device via the communication interface 31 and stores the received data in the storage circuit 34. Also, for example, the processing circuit 35 transmits the data received from the storage circuit 34 to the communication interface 31 to transmit the data to another device. Also, for example, the processing circuit 35 displays the data received from the storage circuit 34 on the display 33.

[0018] The configuration example of the medical image processing apparatus 3 according to the present embodiment has been described above. For example, the medical image processing apparatus 3 according to the present embodiment is installed in a medical facility such as a hospital or a clinic and supports treatment performed by a user such as a doctor. Specifically, the medical image processing apparatus 3 enables improvement of the doctor's treatment efficiency by superimposing and displaying the result of a simulation using a medical image collected before treatment on the medical image being collected during treatment. Hereinafter, the medical image processing apparatus 3 having such a configuration will be described in detail.

[0019] For example, as shown in FIG. 1, in the present embodiment, the processing circuit 35 of the medical image processing apparatus 3 executes a control function 351, an image acquisition function 352, an extraction function 353, a setting function 354, an estimation function 355, and a registration function 356. Here, the control function 351 is an example of a display control unit. Also, the image acquisition function 352 is an example of an acquisition unit. Also, the extraction function 353 is an example of an extraction unit. Also, the setting function 354 is an example of a setting unit. Also, the estimation function 355 is an example of an estimation unit. Also, the registration function 356 is an example of a registration unit.

[0020] The control function 351 controls the generation of various GUIs (Graphical User Interfaces) and various display information in response to operations via the input interface 32, and displays them on the display 33. For example, the control function 351 displays the results of processing performed by each function on the display 33. The control function 351 can also generate and display various display images based on medical images acquired by the image acquisition function 352.

[0021] The control function 351 controls the display 33 to display the results of processing using the medical image acquired by the image acquisition function 352. Specifically, the control function 351 overlays the morphology of the structure of interest at a second time point, estimated based on the morphology of the structure of interest at a first time point when the first medical image was collected, onto the corresponding position of the second medical image. The display processing by the control function 351 will be described in detail later.

[0022] The image acquisition function 352 acquires medical images of a subject from the medical image diagnostic device 1 or the medical image storage device 2 via the communication interface 31. Specifically, the image acquisition function 352 acquires a first medical image and a second medical image different from the first medical image. Here, the first medical image and the second medical image are medical images that include the structure of interest that is the target of treatment. The second medical image is a medical image of a different type from the first medical image; for example, the first medical image is a three-dimensional medical image, and the second medical image is a two-dimensional medical image. Also, for example, the first medical image is a still image, and the second medical image is a moving image. The image acquisition function 352 can also acquire multiple medical images obtained by taking multiple images in the time direction.

[0023] The image acquisition function 352 acquires medical images such as CT images, ultrasound images, MRI images, X-ray images, angio images, PET images, and SPECT images as described above. The processing circuit 35 receives medical images of the subject from the medical image diagnostic device 1 or the medical image storage device 2 by executing the image acquisition function 352, and stores the received medical images in the storage circuit 34.

[0024] The extraction function 353 extracts the morphology of the structure of interest at the first time point in time when the first medical image acquired by the image acquisition function 352 was collected. The processing performed by the extraction function 353 will be described in detail later.

[0025] The setting function 354 sets the treatment conditions for the structure of interest. The processing performed by the setting function 354 will be described in detail later.

[0026] The estimation function 355 estimates the morphology of the structure of interest at a second time point, which differs from the morphology at the first time point, based on the morphology of the structure of interest at the first time point. Specifically, the estimation function 355 estimates the morphology of the structure of interest at the second time point based on the treatment conditions for the structure of interest. The processing performed by the estimation function 355 will be described in detail later.

[0027] The alignment function 356 performs alignment between the first medical image and the second medical image. The processing performed by the alignment function 356 will be described in detail later.

[0028] The processing circuit 35 described above is implemented, for example, by a processor. In this case, each of the processing functions described above is stored in the memory circuit 34 in the form of a program that can be executed by a computer. The processing circuit 35 then reads and executes each program stored in the memory circuit 34, thereby realizing the function corresponding to each program. In other words, the processing circuit 35, with each program read, has the processing functions shown in Figure 1.

[0029] Next, the processing procedure by the medical image processing device 3 will be explained using Figure 2, and then the details of each process will be described. Figure 2 is a flowchart showing the processing procedure performed by the medical image processing device 3 according to the first embodiment.

[0030] For example, as shown in Figure 2, in this embodiment, the image acquisition function 352 acquires a pre-treatment medical image of the subject from the medical image diagnostic device 1 or the medical image storage device 2 (step S101). For example, the image acquisition function 352 acquires a medical image (first medical image) that includes morphological information of the anatomical structure (structure of interest) of the biological organ to be treated, in response to a medical image acquisition operation via the input interface 32. This process is realized, for example, by the processing circuit 35 calling and executing a program corresponding to the image acquisition function 352 from the storage circuit 34.

[0031] Next, the extraction function 353 extracts the structures of interest contained in the acquired medical image (first medical image) (step S102). This process is realized, for example, by the processing circuit 35 calling and executing a program corresponding to the extraction function 353 from the storage circuit 34.

[0032] Next, the setting function 354 sets treatment conditions for the structure of interest extracted in step S102 (step S103). This process is realized, for example, by the processing circuit 35 calling and executing a program corresponding to the setting function 354 from the memory circuit 34.

[0033] Next, the estimation function 355 estimates the post-treatment structure of the target structure extracted in step S102, assuming that it is treated according to the treatment conditions set in step S103 (step S104). This process is realized, for example, by the processing circuit 35 calling and executing a program corresponding to the estimation function 355 from the memory circuit 34.

[0034] Next, the image acquisition function 352 acquires medical images (medical images during treatment) that are being collected while treatment is being performed on the structure of interest (step S105). This process is realized, for example, by the processing circuit 35 calling and executing a program corresponding to the image acquisition function 352 from the storage circuit 34.

[0035] Next, the alignment function 356 performs alignment between the pre-treatment medical image acquired in step S101 and the medical image acquired in step S105 (step S106). This process is realized, for example, by the processing circuit 35 calling and executing a program corresponding to the alignment function 356 from the storage circuit 34.

[0036] Next, the control function 351 determines the superposition position for the post-treatment structure of interest estimated in step S104 when superimposing it onto the medical image during treatment, based on the alignment results performed in step S106 (step S107). The control function 351 superimposes and displays the post-treatment structure of interest at the determined superposition position in the medical image during treatment (step S108). This process is realized, for example, by the processing circuit 35 calling and executing a program corresponding to the control function 351 from the memory circuit 34.

[0037] In Figure 2, a medical image is obtained before treatment in step S101 and a medical image is obtained during treatment in step S105. However, the medical images before and after treatment may be obtained simultaneously, or the medical images before treatment may be obtained after the medical images after treatment.

[0038] The details of each process performed by the medical image processing device 3 will be described below. In the following description, the focus will be on the heart valves (mitral valve, aortic valve), but the target is not limited to these. Any living organ in which treatment is performed on part or all of the structure may be the target of this embodiment. For example, the left atrial appendage or the fossa ovale may be the focus of attention. Furthermore, the following description will explain, as an example, the acquisition of a CT image (3D medical image) collected by an X-ray CT device as the first medical image, and a fluoroscopic image collected by an X-ray diagnostic device as the second medical image.

[0039] (Processing of acquiring medical images before treatment) As explained in step S101 of Figure 2, the image acquisition function 352 acquires a medical image (first medical image) taken before treatment in response to a medical image acquisition operation via the input interface 32. For example, the image acquisition function 352 acquires a CT image of the heart valve in three dimensions before treatment for the purpose of diagnosis or treatment planning.

[0040] The process of acquiring pre-treatment medical images in step S101 may be initiated by user instructions via the input interface 32, as described above, but it may also be initiated automatically. In the latter case, for example, the image acquisition function 352 monitors the medical image storage device 2 and automatically acquires medical images whenever new medical images are stored.

[0041] Here, the image acquisition function 352 may determine newly stored medical images based on pre-set acquisition conditions and execute the acquisition process if the medical image satisfies the acquisition conditions. For example, acquisition conditions that can determine the state of a medical image are stored in the memory circuit 34, and the image acquisition function 352 determines newly stored medical images based on the acquisition conditions stored in the memory circuit 34.

[0042] For example, the memory circuit 34 stores acquisition conditions such as "acquire a medical image captured using an imaging protocol targeting the heart," "acquire an enlarged and reconstructed medical image," or a combination thereof. The image acquisition function 352 acquires a medical image that satisfies the above acquisition conditions. In addition, the image acquisition function 352 can also automatically acquire a medical image that satisfies the above acquisition conditions when acquiring a medical image during treatment in step S105.

[0043] (Extraction process of structures of interest) As explained in step S102 of Figure 2, the extraction function 353 extracts the structure of interest from the pre-treatment medical image acquired by the image acquisition function 352. Specifically, the extraction function 353 extracts the coordinate information of all or some of the pixels that represent the heart valve in the three-dimensional CT image. Here, the extraction function 353 can extract the structure of interest using various methods. For example, the extraction function 353 can extract the coordinate information of pixels corresponding to a region specified on the CT image via the input interface 32 as the structure of interest (heart valve). That is, the extraction function 353 extracts the region manually specified by the user as the heart valve.

[0044] Furthermore, for example, the extraction function 353 can extract structures of interest based on anatomical structures depicted in CT images using known region extraction techniques. For example, the extraction function 353 extracts structures of interest in CT images using methods such as Otsu's binarization method based on CT values, region expansion method, snake method, graph cut method, and mean shift method.

[0045] Furthermore, for example, the extraction function 353 can also extract pixel coordinate information corresponding to the region of the structure of interest in a CT image using a trained model of the structure of interest (such as a mitral valve or aortic valve) built on pre-prepared training data using machine learning techniques (including deep learning). Note that the above-described extraction process of the structure of interest does not have to target the entire image. For example, a region related to the structure of interest and larger than the structure of interest but smaller than the entire image (for example, the cardiac region if the structure of interest is a heart valve) can be identified, and the above-described known region extraction techniques or machine learning techniques can be applied only to the identified region to extract the region indicating the area of ​​interest.

[0046] Alternatively, the multiple structures constituting the structure of interest (for example, the anterior and posterior leaflets in the mitral valve) may be extracted as separate regions. In that case, the above method can be performed separately for each region, or the structure of interest extracted as a single region can be separated based on the respective feature quantities of the multiple structures. Note that the region targeted by this function is not limited in size; a region may be treated as the coordinates of only one point (for example, one pixel).

[0047] Furthermore, the extraction function 353 can extract a structure of interest on a CT image not as the entire region corresponding to the structure, but as a predetermined number of point clouds and / or point clouds in a predetermined arrangement on that region. In addition, the extraction function 353 can extract the structure of interest as a grid-like mesh set based on the above point clouds. Figure 3 is a diagram illustrating an example of the extraction of a structure of interest according to the first embodiment. In Figure 3, an example of the extraction of the mitral valve as the structure of interest is schematically shown.

[0048] For example, as shown in Figure 3, the extraction function 353 acquires morphological information of the mitral valve from a pre-treatment CT image, specifically mesh information consisting of the shape of the anterior leaflet of the mitral valve using 3D coordinates of 19 × 9 grid points, and mesh information consisting of the shape of the posterior leaflet using 3D coordinates of 25 × 9 grid points. Here, each grid point in the mesh information is assigned an index, and by specifying the index, the 3D coordinates of a given grid point can be identified. The index of each grid point is a label that can identify each grid point, and can use numbers or symbols, for example.

[0049] For example, as shown in Figure 3, for mesh information where the anterior leaflet is represented by a grid point cloud of 19 columns and 9 rows and the posterior leaflet by a grid point cloud of 25 columns and 9 rows, an identifier (x,y) can be assigned to each grid point, with the boundary between the anterior and posterior leaflets, at one end in the row direction, being the origin, the row coordinate being "x", and the column coordinate being "y". In this case, identifier (8,0) indicates the anterior commissure, and identifier (8,18) indicates the posterior commissure. Furthermore, the outermost part of the anterior and posterior leaflets (the position where the x-coordinate is "0" in Figure 3) is called the valve annulus. Furthermore, the innermost part of the anterior and posterior leaflets (the position where the x-coordinate is "8" in Figure 3) is called the valve tip.

[0050] (Setting treatment conditions) As explained in step S103 of Figure 2, the setting function 354 sets the treatment conditions for the extracted structure of interest. Here, the treatment conditions include at least one piece of information from the following: treatment location, treatment method, and treatment device. For example, the treatment conditions may include the location and angle of treatment, and the type of treatment device (size, shape, etc.). For example, in the treatment of implanting a MitraClip® for the mitral valve, the treatment conditions are the type (size) of the treatment device and the implantation location of the treatment device in the structure of interest.

[0051] Here, the setting function 354 can set treatment conditions using various methods. For example, the setting function 354 can set conditions specified by the user via the input interface 32. In such cases, the control function 351 displays a GUI for inputting treatment conditions on the display 33. For example, the control function 351 displays a dropdown list that defines the types of treatment devices that can be set for each treatment. The setting function 354 sets the treatment device selected by the user in the dropdown list as the treatment condition.

[0052] Furthermore, the control function 351 displays the CT image acquired in step S101 or the structure of interest extracted in step S102. The setting function 354 sets the position specified by the user as the treatment position for the CT image or the structure of interest. Here, if the structure of interest is extracted as a grid point cloud in step S102, the setting function 354 sets the grid point selected by the user as the treatment position. The setting function 354 can also set treatment conditions by the method described in Japanese Patent Application Publication No. 2024-73089.

[0053] Furthermore, the setting function 354 can also automatically set treatment conditions based on the characteristics of the structure of interest extracted in step S102. For example, the setting function 354 can identify the shape of the mitral valve orifice and set the placement position of the treatment device relative to the position where the valve orifice is most open (for example, the position where the distance between the anterior and posterior leaflets is greatest). The setting function 354 can also set the type of treatment device based on the size of the valve orifice and annulus. In such cases, for example, the memory circuit 34 stores correspondence information that associates the recommended size of the treatment device with the size of the valve orifice and annulus. The setting function 354 measures the extracted size of the mitral valve orifice and annulus and determines the type of treatment device based on the measured size and the correspondence information.

[0054] (Estimation process for the target structure after treatment) As explained in step S104 of Figure 2, the estimation function 355 estimates the post-treatment structure of the structure of interest extracted from pre-treatment medical images, assuming treatment under set treatment conditions. That is, the estimation function 355 estimates the morphology of the structure of interest at a second time point based on the treatment conditions for the structure of interest. Here, the estimation of the post-treatment structure of interest uses known treatment simulation techniques. For example, the estimation function 355 can estimate the post-operative morphology of the structure of interest using the finite element method, finite difference method, era boundary method, etc. For example, the estimation function 355 can cite the methods described in Japanese Patent Publication No. 2022-73363 or in the document "Ooida, Junichi, et al. "An In Silico Model for Predicting the Efficacy of Edge-to-Edge Repair for Mitral Regurgitation." Journal of biomechanical engineering 146.2 (2024)."

[0055] The following describes an example of a treatment simulation in which MitraClip® is implanted in the mitral valve. Figure 4 is a diagram illustrating an example of a treatment simulation according to the first embodiment. Here, Figure 4 shows the case in which the mitral valve is extracted as a grid point cloud in the structure extraction process of step S102. For example, in step S103, as shown in Figure 4, the implantation position 50 of MitraClip® is set for the mitral valve 40 before treatment (time t1). The estimation function 355 estimates the morphology of the mitral valve 41 after treatment (time t2) by applying a mathematical model or physical model set based on the implantation position 50 of MitraClip® to the morphology of the mitral valve 40 (each grid point).

[0056] The method for estimating the structure of interest after treatment is not limited to the method described above. For example, the estimation function 355 may estimate the post-treatment shape of the structure of interest from a shape model constructed by training pre-prepared training data using machine learning techniques (e.g., machine learning techniques including deep learning). That is, the estimation function 355 estimates the post-treatment shape of the structure of interest by inputting the pre-treatment CT image into the shape model and outputting the post-treatment shape.

[0057] Furthermore, the estimation function 355 acquires the positional relationship between the structure of interest at the first time point and the structure of interest at the second time point. Specifically, the estimation function 355 acquires information on the positional relationship between each position on the structure of interest before treatment, which is extracted in step S102, and each position on the structure of interest after treatment, which is estimated in this step. In other words, the estimation function 355 acquires the positional relationship of each pixel representing the structure of interest before and after treatment.

[0058] For example, if the mitral valve is extracted as a grid point cloud and the change in the morphology of the mitral valve due to treatment is estimated, the estimation function 355 calculates and obtains, for each grid point, the direction of movement and the distance or vector of movement from the grid point before treatment to the corresponding grid point after treatment. The estimation function 355 may also obtain the set of information such as the direction of movement and the distance or vector of movement at each grid point as a deformation field. Furthermore, it may obtain positional relationship information for all regions of the structure of interest (for example, all pixels or all grid points constituting the structure of interest), or it may obtain positional relationship information for only some regions. For example, the estimation function 355 obtains positional relationship information between the pre-treatment position and the estimated post-treatment position of characteristic structures such as the annulus and commissure in the mitral valve region.

[0059] It should be noted that the treatment of the target structure is not limited to the examples above, and other treatments may also be performed. Here, information on treatment devices, including those already in clinical use and those currently in clinical research, is made public by the manufacturers and distributors of each treatment device. Therefore, the medical image processing device 3 records this publicly available information in the memory circuit 34 in advance, associating it with identification information such as the name of the treatment device, and the estimation function 355 can retrieve the corresponding information from the memory circuit 34 and use it for estimation. Alternatively, for example, the user may define information on an unknown treatment device they anticipate and record it in the memory circuit 34.

[0060] (Processing of acquiring medical images during treatment) As explained in step S105 of Figure 2, the image acquisition function 352 acquires medical images (second medical images) collected during treatment. Here, the second medical image is a medical image that includes the morphology of the structure of interest at a different time point than the first time point (when the medical image before treatment was collected) and the second time point (the time point after treatment by simulation). For example, the image acquisition function 352 acquires fluoroscopic images taken by an X-ray diagnostic device during treatment of the mitral valve.

[0061] The type of medical image used during treatment can be any type of image that depicts the morphological information of the anatomical structure of the target biological tissue during treatment. For example, in addition to the fluoroscopic images mentioned above, other medical images from other medical imaging devices such as transthoracic ultrasound images, transesophageal ultrasound images, intravascular ultrasound images (IVUS), intracardiac ultrasound images (ICE), and intravascular optical coherence tomography (OCT) images may be used, or a four-dimensional image may be obtained by acquiring multiple images of these in the time direction. Step S105 may be initiated by user instructions via the input interface 32, or it may be controlled to start automatically when imaging by a predetermined medical imaging device is started during treatment.

[0062] (Alignment process) As explained in step S106 of Figure 2, the alignment function 356 performs alignment between the pre-treatment medical image acquired in step S101 and the medical image acquired in step S105. For example, the alignment function 356 performs alignment between the pre-treatment CT image and the fluoroscopic image during treatment. Here, when performing alignment between a 3D image (CT image) and a 2D image (fluoroscopic image), the alignment function 356 first generates a virtual pre-treatment fluoroscopic image from the pre-treatment CT image. Methods for generating virtual fluoroscopic images can be found in, for example, the literature "Gopalakrishnan, Vivek, and Polina Golland. "Fast auto-differentiable digitally reconstructed radiographs for solving inverse problems in intraoperative imaging." Workshop on Clinical Image-Based Procedures. Cham: Springer Nature Switzerland, 2022." and "Unberath, Mathias, et al. "DeepDRR-a catalyst for machine learning in fluoroscopy-guided procedures." Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11. Springer International Publishing, 2018."

[0063] For example, the alignment function 356 sets a virtual X-ray source at any position in space of the CT image and virtually irradiates an arbitrary projection plane in space of the CT image with X-rays, thereby generating a virtual fluoroscopic image corresponding to the CT image on that projection plane. Since each pixel in the virtual fluoroscopic image is generated based on each pixel in the CT image that lies on the straight line connecting that pixel and the virtual X-ray source, it is possible to determine the positional relationship between each pixel in the CT image and each pixel that constitutes the virtual fluoroscopic image.

[0064] The alignment function 356 acquires multiple candidate positions that could be the positions of a virtual X-ray source, based on the actual position of the X-ray source when the actual fluoroscopic image (fluoroscopic image during treatment) was acquired. The alignment function 356 generates a virtual fluoroscopic image for each of the acquired candidate positions, assuming a virtual X-ray source is set at each position, and calculates the similarity between the generated virtual fluoroscopic image and the actual fluoroscopic image (fluoroscopic image during treatment). Then, the alignment function 356 identifies the virtual fluoroscopic image that is most similar to the actual fluoroscopic image (fluoroscopic image during treatment) based on the similarity calculation results. The similarity between images is calculated using known similarity calculation methods as appropriate.

[0065] Next, the alignment function 356 performs alignment between the identified virtual fluoroscopic image and the fluoroscopic image being treated. Known linear or nonlinear deformation alignment methods can be used for the alignment, such as the FFD (Free-Form Deformation) method or the LDDMM (Large Deformation Diffeomorphic Metric Mapping) method.

[0066] The alignment function 356 can identify the positional relationship of each pixel between images before and after alignment by using such a reversible deformation alignment method. In other words, the alignment function 356 can identify the positional relationship between the structure of interest on the CT image before treatment and the structure of interest on the fluoroscopic image during treatment. The positional relationship for each pixel may be obtained as a deformation field, which is a set of information such as the direction and distance of movement of each pixel in one image relative to the corresponding pixel in the other image, or a vector.

[0067] Generally, fluoroscopic images during treatment are used as moving images by continuously displaying X-ray images at each point in time during X-ray irradiation. In this step, the fluoroscopic images during treatment may be moving images, in which case the positional relationship between the pre-treatment CT image and all or more of the fluoroscopic images at different points in time may be identified. Specifically, this can be identified by repeatedly applying the above-described alignment process to each fluoroscopic image at each point in time, or after identifying the positional relationship between the CT image and one fluoroscopic image at time, the positional relationship with fluoroscopic images at different points in time may be identified using known image tracking techniques.

[0068] Furthermore, the alignment method performed by the alignment function 356 is not limited to the method described above. For example, the alignment function 356 can detect the same anatomical feature points from both the CT images or virtual fluoroscopic images that are the target of alignment and the fluoroscopic image being treated, and perform alignment based on the detected feature points. Examples of feature points to be detected include, in the case of the mitral valve, the commissures of the left and right sides, the junctions between the chordae tendineae and the valve lobes, and the tips of the papillary muscles. In the case of the aortic valve, examples include the commissures of each valve lobe, the hinge point, the Nadir point, and the ostia of the left and right coronary arteries.

[0069] The method for detecting feature points may involve using known feature point detection processes, or it may involve using a pre-trained model of the location information of feature points of the structure of interest, built on pre-prepared training data using machine learning techniques (e.g., machine learning techniques including deep learning), to detect feature points of the structure of interest in the image to be aligned. Furthermore, while multiple feature points are desirable to align the orientation of the structure of interest, a small number of feature points may suffice if the orientation of the structure of interest in the target image (which may also be the patient's posture) can be determined.

[0070] (Process for determining the superposition position) As explained in step S107 of Figure 2, the control function 351 determines the superposition position when superimposing the post-treatment target structure onto the medical image during treatment. Specifically, the control function 351 determines the superposition position of the morphology of the target structure at a second time point on the medical image during treatment (second medical image) based on the alignment result by the alignment function 356. Furthermore, the control function 351 identifies the superposition position of the morphology of the target structure at a second time point on the medical image during treatment (second medical image) based on the positional relationship identified by the estimation function 355 (the positional relationship between the target structure before treatment and the target structure after treatment).

[0071] In other words, the control function 351 determines the superposition position of the post-treatment target structure on the medical image during treatment, which was estimated in step S104, based on the positional relationship information between the pre-treatment target structure and the post-treatment target structure identified in step S104, and the positional relationship information between the pre-treatment medical image and the medical image during treatment identified in step S106.

[0072] Figure 5 is a diagram illustrating an example of the superposition position determination process according to the first embodiment. For example, as shown in Figure 5, the control function 351 determines the position on the fluoroscopic image where the post-treatment mitral valve 41 is superimposed, based on the positional relationship (correspondence) between the pre-treatment mitral valve 40 extracted from the pre-treatment CT image and the post-treatment mitral valve 41 estimated from the mitral valve 40, and the positional relationship (correspondence) between the pre-treatment CT image and the fluoroscopic image during treatment.

[0073] In other words, the control function 351 does not directly identify the positional relationship between the post-treatment structure of interest estimated in step S104 and the fluoroscopic image during treatment, but rather determines the superposition position by identifying the positional relationship between the estimated post-treatment structure of interest and the fluoroscopic image during treatment via the position of the structure of interest in the pre-treatment CT image. For example, the control function 351 may integrate the deformation field acquired in step S104 and the deformation field acquired in step S106 based on a predetermined logical formula (such as sum or product), and then determine the superposition position by identifying the positional relationship between the estimated post-treatment structure of interest and the fluoroscopic image during treatment based on the integrated deformation field.

[0074] Furthermore, the positional relationship between all estimated post-treatment target structures and the fluoroscopic image during treatment may be identified to determine the superposition position, or the positional relationship between only a portion of the estimated post-treatment target structures may be identified, and the superposition position of other regions may be determined based on the morphology of the estimated post-treatment target structures regardless of the image information of the fluoroscopic image during treatment. For example, the control function 351 identifies the positional relationship between the annular position of the estimated post-treatment target structure and the annular position on the fluoroscopic image based on the annular position on the pre-treatment CT image, and determines the superposition position of the annular position of the estimated post-treatment target structure on the fluoroscopic image. Furthermore, for the superposition position on the fluoroscopic image of regions other than the annular position of the post-treatment target structure, the control function 351 determines it based only on the superposition position of the annular position of the post-treatment target structure on the fluoroscopic image and the morphology of regions other than the annular position.

[0075] (Superimposed display processing) As explained in step S108 of Figure 2, the control function 351 superimposes the post-treatment structure of interest estimated in step S104 onto the medical image acquired during treatment in step S105, based on the superposition position determined in step S107. Figure 6 shows an example of superimposition display according to the first embodiment. For example, as shown in Figure 6, the control function 351 superimposes the post-treatment mitral valve 42 (a mitral valve whose post-treatment morphology has been estimated by treatment simulation) onto the superposition position in the fluoroscopic image during treatment, and displays this superimposed image on the display 33 or on a display (not shown) located in the operating room where the treatment is being performed.

[0076] Here, the control function 351 can use a known method as the superimposed display method. That is, the control function 351 displays both pixel information, such as the pixel values ​​of some or all of the positions of the post-treatment structure estimated in step S104, and pixel information, such as the pixel values ​​of some or all of the positions constituting the fluoroscopic image acquired in step S105, for the superimposed position identified in step S107 in the image display area of ​​the image display software provided in the form of an application or the like.

[0077] To simultaneously display multiple pixel information at the same location in the image display area (i.e., to superimpose them), this can be achieved by generating integrated pixel information and displaying that integrated pixel information. Methods for integrating pixel information include, for example, integrating pixel values ​​calculated from the sum or product of pixel values ​​in multiple pixel information, or setting a transparency for each pixel information, changing each pixel value according to that transparency, and calculating the integrated pixel value from the sum or product of the changed pixel values. Alternatively, the pixel information can be integrated by representing each pixel information in different display formats such as color, transparency, or saturation. Regarding display at locations other than the superimposed position in the image display area, the control function 351 displays based on only one of the pixel information (in this example, the fluoroscopic image during treatment). Furthermore, to make the fluoroscopic image easier to view, the control function 351 can also control the superimposed post-treatment structure of interest to display only its contour on the fluoroscopic image. Additionally, if the structure of interest consists of multiple structures, the control function 351 can superimpose each structure in a different display format.

[0078] (Variation 1) In the embodiment described above, the case in which only the morphology of the target structure after treatment, estimated in step S104, is superimposed was explained. However, the embodiment is not limited to this, and for example, information regarding the treatment conditions set in step S103 may also be superimposed. In such a case, the control function 351 identifies the superimposed position on the medical image during treatment of the information corresponding to the treatment conditions set in step S103, based on the result of the alignment performed in step S106 and the positional relationship before and after treatment in step S104.

[0079] For example, the control function 351 identifies the superposition position on the fluoroscopic image corresponding to the implantation position of the treatment device set in step S103, and superimposes the structure of a virtual treatment device corresponding to the type of device (size, shape, etc.) set in step S103 onto that superposition position. Here, when the control function 351 displays information corresponding to the morphology of the structure of interest and the treatment conditions for the structure of interest at the second time point in the corresponding position on the medical image during treatment (second medical image), it can display each in a different display format.

[0080] For example, the control function 351 can superimpose the structure of interest in the form of a grid point cluster (mesh) generated by setting multiple grid points in a part of the region corresponding to the structure of interest, and superimpose the treatment device in the form of a set of pixels (mask) representing the treatment device. Alternatively, the control function 351 can also be controlled to superimpose the treatment device and the structure of interest with different colors and transparency settings.

[0081] Figure 7 shows an example of superimposed display according to Modification 1. Here, Figure 7 shows an example of superimposed display when simulating the implantation of an artificial valve 61 for the aortic valve 60. For example, as shown in Figure 7, the control function 351 superimposes the post-treatment morphology of the aortic valve 60, which is the structure of interest, and the artificial valve 61 onto the fluoroscopic image during treatment. In this case, the control function 351 can display the aortic valve 60 and the artificial valve 61 in different display formats so that they can be easily distinguished.

[0082] Furthermore, information regarding treatment conditions may not be limited to the placement position of the treatment device as described above, but may also include information on suturing or excision of the structure of interest. In such cases, the control function 351 can display the suturing position or excision position of the structure of interest in a different display format than the area corresponding to the structure of interest.

[0083] (Modification 2) In the embodiment described above, the case in which only the morphology of the target structure after treatment, estimated in step S104, is superimposed, was explained. However, the embodiment is not limited thereto, and for example, in step S104, a structure different from the target structure may be further estimated, and the estimated structure may be further superimposed on the medical image during treatment. In such a case, the estimation function 355 estimates information about a second target structure different from the first target structure, which is the target structure, and the control function 351 superimposes the information about the second target structure onto the corresponding position in the second medical image.

[0084] When estimating the target structure after treatment, it is sometimes necessary to estimate the location, number, and size of structures related to, but different from, the target structure, and then estimate the target structure after treatment based on these structures. For example, by using information (location and size) of the chordae tendineae of the mitral valve, it is possible to estimate the structure of the mitral valve after treatment more accurately than without using information on the chordae tendineae. However, because the chordae tendineae of the mitral valve are very fine structures, it is difficult to extract them from medical images in step S102.

[0085] Therefore, the estimation function 355 estimates the length, thickness, connection position, and number of chordae tendineae for each subject using, for example, the method described in Japanese Patent Publication No. 2022-73363, and uses this estimated information as estimation parameters to estimate the structure of the mitral valve after treatment. The control function 351 identifies the superimposed position on the fluoroscopic image of structures (chordae tendineae) that are different from the estimated structure of interest, based on the position and size of the structures (chordae tendineae) that are different from the structure of interest, and superimposes and displays the structures (chordae tendineae) that are different from the structure of interest at that superimposed position. Here, as in the modified example 1, the control function 351 can display information regarding the morphology of the structure of interest at the second time point and the second structure of interest that is different from the first structure of interest, which is the structure of interest, at the corresponding positions on the medical image during treatment (second medical image), using different display formats.

[0086] Figure 8 shows an example of superimposed display according to Modification 2. Here, Figure 2 shows an example of superimposed display of the mitral valve 42 and chordae tendineae 43 after treatment simulation. For example, as shown in Figure 8, the control function 351 superimposes the morphology of the mitral valve 42, which is the structure of interest, and the chordae tendineae 43, which is a structure different from the mitral valve, onto the fluoroscopic image during treatment. In this case, the control function 351 can display the mitral valve 42 and chordae tendineae 43 in different display formats so that they can be easily distinguished.

[0087] (Variation 3) In the embodiment described above, the case in which only the morphology of the target structure after treatment, estimated in step S104, is superimposed was explained. However, the embodiment is not limited thereto, and for example, information other than the morphology of the target structure may be further estimated, and the estimated information may be further superimposed on the medical image during treatment. In such a case, the estimation function 355 further estimates estimated information other than the morphology for the target structure at the second time point, and the control function 351 superimposes the estimated information on the corresponding position in the second medical image.

[0088] Specifically, the estimation function 355 estimates estimation information that is at least one of the following at a second time point: fluid information surrounding the structure of interest (e.g., blood flow), stress related to the structure of interest (e.g., pressure, tension), potential at the structure of interest, and physical property information of the structure of interest (e.g., stiffness). For example, an electrical circuit model that simulates the circulatory dynamics of a living organism (e.g., the human body) is pre-constructed and stored in the memory circuit 34 based on a known Windkessel model or pulse wave propagation model. The estimation function 355 inputs the morphological information of the structure of interest after treatment into the electrical circuit model to obtain blood flow information indicating the blood flow state after treatment.

[0089] Here, by designing the electrical circuit model to be able to estimate the regurgitation flow rate in the mitral valve based on the valve orifice area of ​​the mitral valve, the estimation function 355 can estimate the regurgitation flow rate in the mitral valve after treatment by inputting the valve orifice area of ​​the mitral valve after treatment to the electrical circuit model.

[0090] Furthermore, the calculation of the blood flow state is not limited to the electrical circuit model described above. It may also be performed by solving a system of necessary equations such as the Naviers-Stokes equations and the continuity equation, Maxwell's equations, and the equation of state, and inputting various parameters into these equations to numerically obtain the target fluid information. The post-treatment state may be estimated based on the morphology of the target structure before treatment extracted in step S102, or based on the morphology of the target structure after treatment estimated in step S104.

[0091] As described above, the control function 351 superimposes estimated information, which differs from the morphology of the structure of interest estimated by the estimation function 355, onto the medical image during treatment. Here, similar to the modification 1, the control function 351 can display the morphology of the structure of interest at the second time point and the estimated information, which differs from the morphology of the structure of interest at the second time point, at corresponding positions on the medical image during treatment (second medical image), using different display formats for each.

[0092] Figure 9 shows an example of superimposed display according to Modification 3. Here, Figure 9 shows an example of superimposed display when simulating the blood flow state after implantation of an artificial valve 61 in the aortic valve 60. For example, as shown in Figure 9, the control function 351 superimposes the post-treatment morphology of the aortic valve 60, which is the structure of interest, the artificial valve 61, and the blood flow information 62 in the aortic valve 60 onto the fluoroscopic image during treatment. In this case, the control function 351 can display the aortic valve 60, the artificial valve 61, and the blood flow information 62 in different display formats so that they can be easily distinguished.

[0093] (Modification 4) In the embodiment described above, the case in which treatment conditions are set in step S103 and the morphology of the target structure after treatment is estimated in step S104 was explained. However, the embodiment is not limited thereto, and the morphology of the target structure at a different point in time before treatment (before treatment and at a different point in time from when the pre-treatment medical image was collected) may also be estimated.

[0094] In such cases, the estimation function 355 further estimates the morphology of the structure of interest at multiple time points different from the second time point, based on the morphology of the structure of interest at the first time point. Here, the multiple time points are time points in the cardiac phase or respiratory phase different from the first time point. Specifically, the multiple time points are time points in the cardiac phase different from the second time point in a cardiac cycle including the second time point, or time points in the respiratory phase different from the second time point in a respiratory cycle including the second time point.

[0095] Organs such as the heart and lungs operate based on heartbeats and respiratory pulsations. For example, heart valves control blood flow by repeatedly opening and closing in accordance with heartbeats. If the CT image acquired in step S101 is a three-dimensional image, the morphology of the target structure before treatment can only be identified as the morphology of the target structure at a single point in time when the three-dimensional image was acquired. Therefore, the estimation function 355 estimates the morphology of the target structure at a different point in time than the point in time corresponding to the three-dimensional CT image acquired in step S101, using known bio-simulation techniques.

[0096] The control function 351 superimposes the morphology of the structure of interest at multiple time points onto the corresponding positions of the second medical image. For example, the control function 351 identifies the superimposed position on the fluoroscopic image during treatment of the estimated morphology of the structure of interest at each time point and superimposes the morphology of the structure of interest at each time point onto the fluoroscopic image. Here, the control function 351 can display the behavior of the mitral valve on the fluoroscopic image by sequentially superimposing the estimated morphology of the structure of interest at each time point (for example, the morphology of the mitral valve at multiple time points from the open state to the closed state of the valve) onto the fluoroscopic image.

[0097] Furthermore, the morphology of the structure of interest at multiple time points as described above is not limited to the pre-treatment morphology, but may also be estimated for the post-treatment morphology. For example, the morphology of the mitral valve at multiple time points after treatment may be estimated, and the estimated morphology of the mitral valve at each time point may be superimposed sequentially on the fluoroscopic image. This makes it possible to superimpose the structure of interest, which operates due to heartbeats or respiratory pulsations, on the fluoroscopic image during treatment, even if the structure of interest obtained from the CT image acquired before treatment is only the structure at one time point.

[0098] As described above, according to the first embodiment, the image acquisition function 352 acquires a first medical image and a second medical image different from the first medical image. The extraction function 353 extracts the morphology of the structure of interest at a first time point in time when the first medical image was collected. The estimation function 355 estimates the morphology of the structure of interest at a second time point in time, different from the first time point, based on the morphology of the structure of interest at the first time point. The control function 351 superimposes the morphology of the structure of interest at the second time point onto the corresponding position in the second medical image. Therefore, the medical image processing device 3 according to the first embodiment can superimpose the morphology of the structure of interest estimated from the first medical image onto a second medical image different from the first medical image, providing more information to the physician during treatment performed while observing the second medical image, and improving the physician's treatment efficiency.

[0099] Furthermore, according to the first embodiment, the second medical image is a different type of medical image from the first medical image. Also, the first medical image is a three-dimensional medical image, and the second medical image is a two-dimensional medical image. Also, the first medical image is a still image, and the second medical image is a moving image. Also, the second medical image is a medical image that includes the form of the structure of interest at a time different from the first and second time points. Therefore, the medical image processing device 3 according to the first embodiment can accommodate various combinations of the first medical image and the second medical image, and makes it possible to improve the treatment efficiency of physicians in various treatments.

[0100] Furthermore, according to the first embodiment, the alignment function 356 performs alignment of the first medical image and the second medical image. Based on the alignment result, the control function 351 determines the superposition position of the shape of the structure of interest on the second medical image at a second time point. Therefore, the medical image processing device 3 according to the first embodiment can determine an appropriate superposition position on the second medical image.

[0101] Furthermore, according to the first embodiment, the estimation function 355 acquires the positional relationship between the structure of interest at a first time point and the structure of interest at a second time point. Based on the positional relationship, the control function 351 determines the superposition position of the shape of the structure of interest at the second time point on the second medical image. Therefore, the medical image processing device 3 according to the first embodiment can appropriately determine the superposition position of the structure of interest at the second time point on the second medical image.

[0102] Furthermore, according to Modification 1, the estimation function 355 estimates the morphology of the structure of interest at a second time point based on the treatment conditions for the structure of interest. The control function 351 superimposes information corresponding to the treatment conditions onto the corresponding position in the second medical image. Therefore, the medical image processing device 3 according to Modification 1 can also superimpose information regarding treatment conditions onto the second medical image observed during treatment, thereby further improving the efficiency of the physician's treatment. For example, the medical image processing device 3 according to Modification 1 can superimpose information on the estimated optimal treatment position onto the medical image observed during treatment. As a result, during surgery, the physician can efficiently perform treatment targeting the information on the optimal treatment position.

[0103] Furthermore, according to Modification 1, the treatment conditions include at least one piece of information from among the treatment location, treatment method, and treatment device. Therefore, the medical image processing device 3 according to Modification 1 can superimpose and display various pieces of information as treatment conditions, thereby further improving the efficiency of the physician's treatment.

[0104] Furthermore, according to Modification 2, the estimation function 355 estimates information about a second structure of interest that is different from the first structure of interest, which is the structure of interest. The control function 351 superimposes the information about the second structure of interest onto the corresponding position in the second medical image. Therefore, the medical image processing device 3 according to Modification 2 can superimpose structures different from the structure of interest onto the second medical image observed during treatment, thereby further improving the efficiency of the physician's treatment. For example, the medical image processing device 3 according to Modification 2 can superimpose estimated values ​​of chordae tendineae, which are difficult to obtain from images and affect the determination of the treatment position, onto the medical image observed during treatment. As a result, the physician can perform treatment more efficiently during surgery.

[0105] Furthermore, according to Modification 3, the estimation function 355 further estimates estimation information different from morphology for the structure of interest at the second time point. The control function 351 superimposes the estimation information onto the corresponding position of the second medical image. Therefore, the medical image processing device 3 according to Modification 3 can further superimpose estimation information regarding the structure of interest onto the second medical image observed during treatment, thereby further improving the efficiency of the physician's treatment. For example, the medical image processing device 3 according to Modification 3 can superimpose estimated values ​​onto the medical image observed during treatment when considering stress and other factors for optimal treatment conditions. As a result, the physician can more efficiently search for optimal conditions during surgery.

[0106] Furthermore, according to Modification 3, the estimation function 355 estimates estimation information based on the morphology of the structure of interest at the second time point. Therefore, the medical image processing device 3 according to Modification 3 can estimate appropriate estimation information.

[0107] Furthermore, according to Modification 3, the estimated information is at least one of the following: fluid information surrounding the structure of interest at the second time point, stress related to the structure of interest, potential in the structure of interest, and physical property information of the structure of interest. Therefore, the medical image processing device 3 according to Modification 3 can estimate various estimated information and display them superimposed.

[0108] Furthermore, according to the modified examples 1 to 3 described above, the control function 351 displays at least two of the following information at corresponding positions in the second medical image: the morphology of the structure of interest at the second time point, information corresponding to the treatment conditions for the structure of interest, information regarding a second structure of interest different from the first structure of interest, and estimated information different from the morphology of the structure of interest at the second time point, each in a different display format. Therefore, the medical image processing device 3 can perform superimposed display that allows for easy distinction of each estimated piece of information.

[0109] Furthermore, according to Modification 4, the estimation function 355 further estimates the morphology of the structure of interest at multiple time points different from the second time point, based on the morphology of the structure of interest at the first time point. The control function 351 superimposes the morphology of the structure of interest at multiple time points onto the corresponding positions of the second medical image. Therefore, the medical image processing device 3 according to Modification 4 can superimpose the morphology of the structure of interest at time points when medical images have not been acquired, providing physicians with more information and improving treatment efficiency. For example, the medical image processing device 3 according to Modification 4 can superimpose the estimated dynamics of one cardiac cycle even if there are time phases that have not been acquired before surgery. As a result, physicians can more easily grasp the behavior of the living body, which is inherently dynamic, and perform treatment more efficiently.

[0110] Furthermore, according to Modification 4, the multiple time points are different from the first time point in the cardiac phase or respiratory phase. Also, the multiple time points are different cardiac phases in the cardiac cycle including the second time point, or different respiratory phases in the respiratory cycle including the second time point. Therefore, the medical image processing device 3 according to Modification 4 can perform superimposed display showing the behavior of the structure of interest in each time phase for the cardiac cycle and respiratory cycle.

[0111] (Second embodiment) The first embodiment described the case where treatment conditions are set based on input by the user. The second embodiment describes the case where treatment conditions are set based on medical images during treatment. In this case, the medical image processing device 3 according to the second embodiment differs from the first embodiment in the processing content by the setting function 354. The following explanation will focus on this point.

[0112] The setting function 354 according to the second embodiment sets treatment conditions for the structure of interest based on the second medical image. Specifically, the setting function 354 sets treatment conditions for the structure of interest based on information about the medical device included in the medical image during treatment. The processing performed by the setting function 354 will be described in detail later.

[0113] The estimation function 355 estimates at least one of the following based on the treatment conditions: the morphology of the structure of interest at a second time point, estimation information other than morphology, and information about a second structure of interest that is different from the first structure of interest. The control function 351 superimposes the estimated results onto the corresponding positions in the second medical image.

[0114] Next, the processing procedure of the medical image processing apparatus 3 according to the second embodiment will be explained using Figure 10, and then the details of each process will be described. Figure 10 is a flowchart showing the processing procedure performed by the medical image processing apparatus 3 according to the second embodiment. Steps S201 to S203 in Figure 10 are the same processes as steps S101, S102, and S105 in Figure 2, and steps S205 to S208 are the same processes as steps S106, S104, S107, and S108, so a detailed explanation will be omitted.

[0115] For example, as shown in Figure 10, in this embodiment, the image acquisition function 352 acquires a medical image before treatment (step S201), the extraction function 353 extracts a structure of interest from the acquired medical image before treatment (step S202), and the image acquisition function 352 acquires a medical image during treatment (step S203).

[0116] Next, the setting function 354 sets the treatment conditions based on the medical image being treated (step S204). This process is achieved, for example, by the processing circuit 35 calling and executing a program corresponding to the setting function 354 from the storage circuit 34.

[0117] Next, the alignment function 356 performs alignment between the medical image before treatment and the medical image during treatment (step S205), and the estimation function 355 estimates the target structure after treatment according to the treatment conditions (step S206). After that, the control function 351 determines the superposition position of the estimated target structure (step S207) and displays the estimated target structure superimposed on the medical image during treatment (step S208).

[0118] Subsequently, the setting function 354 determines whether or not the treatment conditions have changed (step S209). If it is determined that the treatment conditions have changed (step S209, Yes), the process returns to step S205, and the estimation function 355 estimates the target structure after treatment according to the treatment conditions. On the other hand, if it is determined in step S209 that the treatment conditions have not changed (step S209, No), the setting function 354 determines whether or not the treatment has ended (step S209). If the treatment has ended (step S209, Yes), the medical image processing device 3 terminates processing. The setting function 354 continues the determination in step S209 until the treatment is completed (step S210, No).

[0119] The details of each process performed by the medical image processing device 3 are described below.

[0120] (Setting treatment conditions) As explained in step S204 of Figure 10, the setting function 354 sets treatment conditions based on the medical image during treatment. Specifically, the setting function 354 extracts the treatment device from the medical image during treatment and sets the position of the extracted treatment device as a treatment condition. For example, the setting function 354 extracts the treatment device from the medical image during treatment acquired in step S203 using existing segmentation technology and sets the position of the extracted treatment device as the treatment position. That is, the setting function 354 acquires pixel information of the region corresponding to the treatment device in the medical image during treatment and sets the position of the acquired pixels as the treatment position.

[0121] (Estimation process for the target structure after treatment) The estimation function 355 estimates the target structure after treatment according to the treatment conditions set in step S204, based on the alignment results of the pre-treatment medical image and the medical image during treatment performed in step S205. Specifically, the estimation function 355 identifies the treatment position in the pre-treatment medical image based on the alignment results, and estimates the morphology of the target structure after treatment, estimation information different from the morphology, and structures different from the target structure, assuming treatment at the identified treatment position. Here, the various estimated information are superimposed in a determined position in step S207 and superimposed and displayed on the medical image during treatment in step S208.

[0122] (Determination process regarding treatment conditions) As explained in step S209 of Figure 10, the setting function 354 determines whether the treatment conditions have changed after the structure of interest is superimposed on the medical image during treatment. For example, if fluoroscopic images are collected as medical images during treatment, the image acquisition function 352 sequentially acquires moving images (multiple frames) as medical images during treatment. The setting function 354 extracts the treatment device from each frame (each fluoroscopic image) and determines whether the treatment conditions (position of the treatment device) have changed. For example, the setting function 354 determines that the treatment conditions (position of the treatment device) have changed if the position of the treatment device extracted from each frame changes by more than a threshold.

[0123] For example, when a treatment device is placed while referring to a fluoroscopic image, the treatment device is moved to target the optimal placement position. In the medical image processing device 3 according to the second embodiment, a treatment simulation is performed at each position of the moved treatment device, and the results of the treatment simulation can be superimposed on the fluoroscopic image. Therefore, the physician can check the results of the treatment simulation at each position of the treatment device in real time while performing the procedure while referring to the fluoroscopic image, and can perform the treatment efficiently.

[0124] As described above, according to the second embodiment, the setting function 354 sets treatment conditions for the structure of interest based on the second medical image. The estimation function 355 estimates at least one of the following based on the treatment conditions: the morphology of the structure of interest at the second time point, estimation information other than morphology, and information about a second structure of interest that is different from the first structure of interest. The control function 351 overlays the estimated results onto the corresponding positions of the second medical image. Therefore, the medical image processing device 3 according to the second embodiment can overlay the estimated results, based on the treatment conditions of the medical image being treated, onto the medical image being treated, thereby improving the efficiency of the physician's treatment.

[0125] (Other embodiments)

[0126] Furthermore, the processing circuits described in each of the embodiments above may be composed of a combination of multiple independent processors, with each processor executing a program to realize each processing function. Also, each processing function of the processing circuit may be implemented by appropriately distributing or integrating it across one or more processing circuits. Additionally, each processing function of the processing circuit may be implemented by a mixture of hardware such as circuits and software. While this description has described an example where the programs corresponding to each processing function are stored in a single memory circuit 34, the embodiments are not limited to this. For example, the programs corresponding to each processing function may be stored in a distributed manner across multiple memory circuits, and the processing circuit may read and execute each program from each memory circuit.

[0127] In the embodiments described above, examples were given in which each part of this specification is implemented by the respective functions of the processing circuit, but the embodiments are not limited to these. For example, each part of this specification may be implemented not only by the respective functions described in the embodiments, but also by hardware alone, software alone, or a combination of hardware and software.

[0128] Furthermore, the term "processor" used in the above-described embodiment refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). Here, instead of storing the program in a memory circuit, the processor may be configured to directly incorporate the program into its circuitry. In this case, the processor realizes its function by reading and executing the program incorporated into the circuitry. Moreover, each processor in this embodiment is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor, and its function may be realized in this way.

[0129] Here, the medical image processing program executed by the processor is provided pre-installed in ROM (Read Only Memory) or memory circuits. Alternatively, this medical image processing program may be provided as a file in an installable or executable format on a computer-readable, non-transient storage medium such as a CD (Compact Disk)-ROM, FD (Flexible Disk), CD-R (Recordable), or DVD (Digital Versatile Disk). Furthermore, this medical image processing program may be stored on a computer connected to a network such as the Internet and provided or distributed by downloading it via the network. For example, this medical image processing program consists of modules containing the processing functions described above. In actual hardware, the CPU reads the medical image processing program from a storage medium such as ROM and executes it, loading each module onto the main memory and generating it in the main memory.

[0130] Furthermore, in the embodiments and modifications described above, each component of each illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific form of distribution or integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads and usage conditions. Moreover, each processing function performed by each device can be realized in whole or in any part by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0131] Furthermore, among the processes described in the embodiments and modifications described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0132] According to at least one embodiment described above, it is possible to improve the efficiency of treatment by physicians.

[0133] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0134] 3 Medical Image Processing Equipment 351 Control Functions 352 Image acquisition function 353 Extraction function 354 Settings Function 355 Estimation Function 356 Alignment function

Claims

1. An acquisition unit that acquires a first medical image and a second medical image different from the first medical image, An extraction unit for extracting the morphology of the structure of interest at a first time point in time when the first medical image was acquired, An estimation unit that estimates the form of the structure of interest at a second time point, which is different from the form of the structure of interest at the first time point, based on the form of the structure of interest at the first time point, A display control unit that superimposes the shape of the structure of interest at the second time point onto the corresponding position in the second medical image, A medical image processing device equipped with [a specific feature].

2. The medical image processing apparatus according to claim 1, wherein the second medical image is a medical image of a different type from the first medical image.

3. The medical image processing apparatus according to claim 2, wherein the first medical image is a three-dimensional medical image, and the second medical image is a two-dimensional medical image.

4. The medical image processing apparatus according to claim 2, wherein the first medical image is a still image and the second medical image is a moving image.

5. The medical image processing apparatus according to claim 1, wherein the second medical image is a medical image that includes the form of the structure of interest at a time different from the first time and the second time.

6. The system further includes an alignment unit that performs alignment of the first medical image and the second medical image, The medical image processing apparatus according to claim 1, wherein the display control unit determines the superposition position of the form of the structure of interest at the second time point on the second medical image based on the alignment result.

7. The estimation unit obtains the positional relationship between the structure of interest at the first time point and the structure of interest at the second time point, The medical image processing apparatus according to any one of claims 1 to 6, wherein the display control unit determines the superposition position of the form of the structure of interest at the second time point on the second medical image based on the positional relationship.

8. The estimation unit estimates the morphology of the structure of interest at the second time point based on the treatment conditions for the structure of interest, The medical image processing apparatus according to any one of claims 1 to 6, wherein the display control unit overlays and displays information corresponding to the treatment conditions at the corresponding position of the second medical image.

9. The medical image processing apparatus according to claim 8, wherein the treatment conditions include at least one piece of information from among treatment location, treatment method, and treatment device.

10. The estimation unit estimates information about a second structure of interest that is different from the first structure of interest, which is the structure of interest. The medical image processing apparatus according to any one of claims 1 to 6, wherein the display control unit superimposes information relating to the second structure of interest onto the corresponding position of the second medical image.

11. The estimation unit further estimates estimation information different from the above-mentioned form for the structure of interest at the second time point, The medical image processing apparatus according to any one of claims 1 to 6, wherein the display control unit displays the estimated information superimposed on the corresponding position of the second medical image.

12. The medical image processing apparatus according to claim 11, wherein the estimation unit estimates the estimation information based on the form of the structure of interest at the second time point.

13. The medical image processing apparatus according to claim 11, wherein the estimation information is at least one of the fluid information surrounding the structure of interest at the second time point, the stress relating to the structure of interest, the potential in the structure of interest, and the physical property information of the structure of interest.

14. The medical image processing apparatus according to claim 1, wherein the display control unit displays at least two of the following in different display formats when displaying at the corresponding positions of the second medical image: the form of the structure of interest at the second time point, information corresponding to treatment conditions for the structure of interest, information relating to a second structure of interest different from the first structure of interest which is the structure of interest, and estimated information different from the form of the structure of interest at the second time point.

15. The estimation unit further estimates the structural form of interest at multiple time points different from the second time point, based on the structural form of interest at the first time point. The medical image processing apparatus according to any one of claims 1 to 6, wherein the display control unit superimposes the form of the structure of interest at the multiple time points onto the corresponding position of the second medical image.

16. The medical image processing apparatus according to claim 15, wherein the plurality of time points are different from the first time point in the cardiac phase or respiratory phase.

17. The medical image processing apparatus according to claim 15, wherein the plurality of time points are time points in a cardiac phase different from the second time point in a cardiac cycle including the second time point, or time points in a respiratory phase different from the second time point in a respiratory cycle including the second time point.

18. The system further includes a setting unit that sets treatment conditions for the structure of interest based on the second medical image, The estimation unit estimates, based on the treatment conditions, at least one of the following: the morphology of the structure of interest at the second time point, estimation information other than the morphology, and information regarding a second structure of interest that is different from the first structure of interest which is the structure of interest. The medical image processing apparatus according to any one of claims 1 to 6, wherein the display control unit superimposes the estimated result onto the corresponding position of the second medical image.

19. A first medical image and a second medical image different from the first medical image are acquired. The morphology of the structure of interest at the first time point in time when the first medical image was collected is extracted. Based on the morphology of the structure of interest at the first time point, the morphology of the structure of interest at a second time point, which is different from the morphology of the first time point, The morphology of the structure of interest at the second time point is superimposed onto the corresponding position in the second medical image. A medical image processing method, including the following.

20. A first medical image and a second medical image different from the first medical image are acquired. The morphology of the structure of interest at the first time point in time when the first medical image was collected is extracted. Based on the morphology of the structure of interest at the first time point, the morphology of the structure of interest at a second time point, which is different from the morphology of the first time point, The morphology of the structure of interest at the second time point is superimposed onto the corresponding position in the second medical image. A program that instructs a computer to perform various processes.

Images