Program, information processing method, information processing device and model generation method

Abstract

This program makes a computer execute processing for acquiring a first medical image on which a hollow organ of a patient before treatment is imaged and device information associated with a treatment device used for treating the hollow organ, and generating, when the first medical image and device information are input, a second medical image after treatment by inputting the acquired first medical image and device information to a model having learnt to generate the second medical image.

Description

[Technical field]

[0001] The present invention relates to a program, an information processing method, an information processing device, and a model generating method. [Background technology]

[0002] Treatment is performed based on medical images that visualize the inside of the human body, such as ultrasound images, optical coherence tomography (OCT) images, X-ray images, etc. Accordingly, various techniques have been proposed for processing medical images so that image observers can conveniently observe the medical images.

[0003] For example, Patent Document 1 discloses an ultrasound diagnostic device that displays ultrasound images of a lesion in a subject, generates an image by superimposing an ultrasound image before treatment and an ultrasound image after treatment, and displays treated and untreated areas of the lesion in different colors. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-119071 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the invention of Patent Document 1 merely superimposes an image acquired before treatment and an image acquired after treatment, and does not generate an image after treatment.

[0006] In one aspect, an object of the present invention is to provide a program or the like that can obtain a medical image that predicts a post-treatment condition from a pre-treatment medical image. [Means for solving the problem]

[0007] One aspect of the program is:Treatment in any of multiple treatment phases A first medical image obtained by imaging the luminal organ of a previous patient and device information related to a treatment device used to treat the luminal organ are obtained, and the first medical image and the device information are a class label representing the treatment stage corresponding to the first medical image; If you enter Treatment of the treatment phase The acquired first medical image and device information are input to a trained model to generate a second medical image. and the class label The computer is caused to execute a process of generating the second medical image by inputting the above. Effect of the Invention

[0008] In one aspect, a medical image predicting a post-treatment condition can be obtained from a pre-treatment medical image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram showing an example of the configuration of a treatment support system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a server. [Diagram 3] FIG. 1 is an explanatory diagram showing an overview of a first embodiment. [Figure 4] FIG. 1 is an explanatory diagram of a generation model. [Diagram 5] FIG. 2 is an explanatory diagram showing an example of a display screen for a medical image. [Figure 6] FIG. 1 is an explanatory diagram regarding treatment stages of endovascular treatment. [Figure 7] 11 is a flowchart showing the procedure of a generation process of a generative model. [Figure 8] 13 is a flowchart showing a procedure for generating a second medical image. [Figure 9] FIG. 11 is an explanatory diagram of a generation model according to the second embodiment. [Figure 10] FIG. 11 is an explanatory diagram showing an example of a display screen for a medical image according to the second embodiment. [Figure 11] 13 is a flowchart showing the procedure of a generation process of a second medical image according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will now be described in detail with reference to the drawings showing embodiments thereof. (Embodiment 1) Fig. 1 is an explanatory diagram showing an example of the configuration of a treatment support system. In this embodiment, a treatment support system is described that generates a medical image (hereinafter, "second medical image") that predicts the state of blood vessels after treatment based on a medical image (hereinafter, "first medical image") obtained by imaging a patient's blood vessels before intravascular treatment and device information related to a treatment device used in the intravascular treatment. The treatment support system includes an information processing device 1 and an image diagnostic system 2. The information processing device 1 and the image diagnostic system 2 are communicatively connected to a network N such as a LAN (Local Area Network) or the Internet.

[0011] In this embodiment, intravascular treatment will be described as an example, however, the target hollow organ is not limited to blood vessels, and may be other hollow organs such as the bile duct, pancreatic duct, bronchi, and intestines.

[0012] The image diagnostic system 2 includes an intravascular imaging device 21, a fluoroscopic imaging device 22, and a display device 23. The intravascular imaging device 21 is a device for imaging a tomographic image of the inside of a patient's blood vessel, and is, for example, an IVUS (Intravascular Ultrasound) device that performs an ultrasonic examination using a catheter 211. The catheter 211 is a medical instrument inserted into the patient's blood vessel, and includes an imaging core that transmits ultrasonic waves and receives reflected waves from inside the blood vessel. The intravascular imaging device 21 generates an ultrasonic tomographic image based on the signal of the reflected wave received by the catheter 211, and displays it on the display device 23.

[0013] In this embodiment, the intravascular diagnostic imaging apparatus 21 generates an ultrasonic tomographic image, but may capture an optical coherence tomographic image by an optical technique such as OCT or OFDI (Optical Frequency Domain Imaging).

[0014] The fluoroscopic image capturing device 22 is a device unit for capturing a fluoroscopic image by looking through the inside of a patient's body, for example, an angiography device for performing angiography. The fluoroscopic image capturing device 22 includes an X-ray source 221 and an X-ray sensor 222, and captures an X-ray fluoroscopic image of the patient by the X-ray sensor 222 receiving X-rays irradiated from the X-ray source 221.

[0015] Although ultrasonic tomographic images, optical coherence tomographic images, and angiographic images have been given above as examples of medical images, the medical images may also be computed tomography (CT) images, magnetic resonance imaging (MRI) images, etc.

[0016] The information processing device 1 is an information processing device capable of various information processing and transmitting and receiving information, such as a server computer, a personal computer, etc. In this embodiment, the information processing device 1 is a server computer, and will be referred to as a server 1 below for simplicity. The server 1 may be a local server installed in the same facility (hospital, etc.) as the image diagnostic system 2, or a cloud server connected for communication via the Internet, etc. The server 1 functions as a generating device that generates a second medical image from a first medical image (ultrasonic tomographic image and X-ray fluoroscopic image) generated by the image diagnostic system 2, and outputs the generated second medical image to the image diagnostic system 2.

[0017] Specifically, as described below, the server 1 performs machine learning in advance to learn predetermined training data, and prepares a generative model 50 (see FIG. 3, etc.) that generates a second medical image using a first medical image and device information as input. The device information will be described later. The server 1 acquires the first medical image and device information from the image diagnostic system 2 and inputs them to the generative model 50, generates a second medical image, and displays it on the display device 23.

[0018] In this embodiment, the second medical image is generated in a server 1 separate from the image diagnostic system 2. However, the generative model 50 generated by the server 1 through machine learning may be installed in the image diagnostic system 2, making it possible to generate the second medical image in the image diagnostic system 2.

[0019] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit . The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc., and performs various information processing, control processing, etc. by reading and executing a program P stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside.

[0020] The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores the program P and other data necessary for the control unit 11 to execute processing. The auxiliary storage unit 14 also stores a generative model 50. The generative model 50 is a machine learning model that has learned training data as described above, and is a model that generates a second medical image using a first medical image and device information as input. The generative model 50 is expected to be used as a program module that constitutes artificial intelligence software.

[0021] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of a plurality of computers, or may be a virtual machine virtually constructed by software.

[0022] In the present embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit for accepting operation input, a display unit for displaying images, etc. The server 1 may also include a reading unit for reading a portable storage medium 1a such as a CD (Compact Disk), a DVD (Digital Versatile Disc), or a USB (Universal Serial Bus) memory, and may read and execute the program P from the portable storage medium 1a. Alternatively, the server 1 may read the program P from a semiconductor memory 1b.

[0023] Fig. 3 is an explanatory diagram showing an overview of the first embodiment. Fig. 3 conceptually illustrates how a second medical image is generated from a first medical image and text information using a generative model 50. The overview of this embodiment will be described with reference to Fig. 3.

[0024] The generation model 50 is a machine learning model that generates a second medical image predicting a state after treatment using a first medical image acquired by the image diagnostic system 2 before the treatment of the patient and device information related to the treatment device used in the treatment as input. The first medical image is an ultrasonic tomographic image and an X-ray fluoroscopic image acquired by the intravascular diagnostic imaging device 21 and the fluoroscopic imaging device 22, respectively, and is an image obtained by inspecting the patient's blood vessels before treatment.

[0025] The device information is information related to devices used in intravascular treatment, such as stents, balloons, etc. used in PCI (Percutaneous Coronary Intervention). For example, the device information includes the length, diameter, type, and placement position of a stent placed in a patient's blood vessel, the type of balloon, the diameter after expansion by the balloon (hereinafter referred to as "expansion diameter"), expansion pressure, expansion time, etc. Note that the treatment device is not limited to a stent and a balloon, and may be another device such as a rotablator.

[0026] The second medical image is an image predicting the state inside the blood vessel after treatment using the treatment device, and is an ultrasonic tomographic image and an X-ray fluoroscopic image after treatment. Note that the server 1 may generate only one of the ultrasonic tomographic image and the X-ray fluoroscopic image as the second medical image, or may generate three or more types of images.

[0027] The server 1 learns training data including a first medical image and device information for training, and a second medical image, and generates a generative model 50. The training data is, for example, data of a patient who has undergone endovascular treatment, and includes an ultrasonic tomographic image and an X-ray fluoroscopic image (first medical image) before the treatment, device information related to the treatment device used in the treatment of the patient, and an ultrasonic tomographic image and an X-ray fluoroscopic image after the treatment. The server 1 performs machine learning using these data as training data, and generates a generative model 50 in advance.

[0028] The training data is not limited to actual patient data, but may be virtual data padded by a data generation means such as a Generative Adversarial Network (GAN).

[0029] For example, the server 1 acquires a first medical image and device information from the image diagnostic system 2 during treatment of a patient, generates a second medical image after treatment, and displays it on the display device 23. Specifically, the server 1 sequentially acquires a plurality of ultrasonic tomographic images (frame images) imaged continuously along the longitudinal direction of a blood vessel from the intravascular imaging diagnostic device 21 during an ultrasonic examination using a catheter 211. The server 1 also acquires X-ray fluoroscopic images captured simultaneously during the ultrasonic examination from the fluoroscopic image capturing device 22. The server 1 sequentially inputs a plurality of consecutive ultrasonic tomographic images and X-ray fluoroscopic images at the time of acquisition of each ultrasonic tomographic image together with device information to the generative model 50, and sequentially generates ultrasonic tomographic images and X-ray fluoroscopic images predicting a state after treatment.

[0030] In the example of Fig. 3, an ultrasonic tomographic image and an X-ray fluoroscopic image of a blood vessel after a stent is placed and expanded by a balloon are generated as a second medical image after treatment. Note that in Fig. 3, the part corresponding to the stent is illustrated by hatching. The server 1 predicts the state of the blood vessel after treatment using the generative model 50 and presents it as a second medical image.

[0031] In this embodiment, the second medical image is generated during treatment and output to the image diagnostic system 2, but it goes without saying that the recorded first medical image may be input into the generative model 50 afterwards to generate the second medical image.

[0032] 4 is an explanatory diagram of the generative model 50. In this embodiment, a GAN is used as the generative model 50. A GAN includes a generator that generates output data from input data, and a discriminator that discriminates whether the data generated by the generator is true or false, and the generator and the discriminator compete with each other to learn, thereby constructing a network.

[0033] The generator in GAN accepts random noise (latent variables) as input and generates output data. The discriminator uses the true data provided for learning and the data provided by the generator to learn whether the data provided by the generator is true or false. In GAN, a network is constructed so that the loss function of the generator is ultimately minimized and the loss function of the discriminator is maximized.

[0034] The generator of the generative model 50 according to the present embodiment includes an encoder that converts input data into latent variables and a decoder that generates output data from the latent variables, and generates a second medical image from a first medical image and device information. For example, as shown in FIG. 4, the generator of the generative model 50 includes three encoders that receive three types of input data, namely, an ultrasonic tomographic image before treatment, an X-ray fluoroscopic image, and device information. The generator also includes two decoders that generate ultrasonic tomographic images and X-ray fluoroscopic images after treatment. The generative model 50 also includes two discriminators that discriminate the authenticity of the data output from each decoder. The generator extracts the feature amount of the input data in each encoder, inputs a latent variable combining the feature amount of each input data to each decoder, and generates an ultrasonic tomographic image and an X-ray fluoroscopic image that predict the state after treatment. The two discriminators each discriminate the authenticity of the ultrasonic tomographic image and the X-ray fluoroscopic image.

[0035] The server 1 learns using the first medical image and device information given for training, and the second medical image, and generates a generative model 50. For example, the server 1 first fixes the parameters (weights, etc.) of the generator, inputs the first medical image and device information for training to the generator, and generates a second medical image. The server 1 then gives the second medical image generated by the generator as false data and the second medical image for training to the discriminator as true data, and optimizes the parameters of the discriminator. Next, the server 1 fixes the parameters of the discriminator to optimal values, and performs training of the generator. The server 1 optimizes the parameters of the generator so that when the second medical image generated by the generator is input to the discriminator, the probability of true or false approaches 50%. In this way, the server 1 generates the generative model 50. When actually generating a second medical image from a first medical image, only the generator is used.

[0036] Note that the generative model 50 is not limited to GAN, and may be a neural network such as VAE (Variational Autoencoder), CNN (e.g., U-net), or a model based on other learning algorithms.

[0037] Moreover, the network structure of the generative model 50 shown in FIG. 4 is an example, and the present embodiment is not limited to this. For example, two images, an ultrasonic tomographic image and an X-ray fluoroscopic image, may be combined and regarded as a single image, and the two images may be simultaneously processed by one type of encoder and decoder. Also, instead of preparing an encoder for extracting the feature amount of the device information, the device information (text data) may be encoded by a means such as one-hot encoding, and directly mapped (combined) to the latent space. In this way, various modifications of the network structure of the generative model 50 are possible.

[0038] It is preferable that the server 1 uses, as the second medical image for training, a second medical image in which an object of interest to the user, such as a stent, is displayed in a different display mode from other image regions, as exemplified in Fig. 3. For example, the server 1 uses an image in which an image region corresponding to an object, such as a stent, is labeled (processed) by color coding, edge enhancement, or the like. This allows the server 1 to generate a second medical image in which the object of interest to the user can be identified.

[0039] Note that the object to be labeled is not limited to a treatment device such as a stent, but may be other objects such as a lesion (plaque, etc.) in a blood vessel or a lumen boundary.

[0040] Although stent information has been mentioned above as input data other than medical images, other information related to endovascular treatment may also be input to the generative model 50. For example, the server 1 inputs lesion information related to a lesion in a blood vessel to the generative model 50. The lesion information includes, for example, the type of lesion (plaque, calcified tissue, vascular stenosis, etc.), its location, and properties (hardness of the lesion, etc.). For example, the server 1 inputs the lesion information into an encoder corresponding to the stent information together with the stent information, and converts it into a latent variable. This allows the generative model 50 to generate a second medical image while also referring to the properties and state of the lesion.

[0041] Fig. 5 is an explanatory diagram showing an example of a display screen of a medical image. Fig. 5 shows an example of a screen displayed by the display device 23, which displays a first medical image and a second medical image. A processing flow executed during intravascular treatment will be described with reference to Fig. 5.

[0042] For example, the display device 23 displays a current ultrasonic tomographic image and an X-ray fluoroscopic image of a blood vessel corresponding to the first medical image as a current tomographic image 501 and a current fluoroscopic image 502. The display device 23 also displays designation fields 503, 503, 503... in the menu bar on the left side of the screen to receive designation input of device information. The display device 23 also displays a status field 504 in the menu bar to receive designation input of the current treatment stage.

[0043] The display device 23 accepts input of device information designation on the screen. Specifically, the display device 23 accepts input of designation of a site (position of a lesion) in a blood vessel to be treated by accepting an operation input of drawing a rectangular frame on a current fluoroscopic image 502. Furthermore, the display device 23 accepts input of designation of the length and diameter of a stent, the diameter of expansion by a balloon, etc., through each designation field 503.

[0044] In the example of FIG. 5, device information for each item is input individually in multiple specification fields 503, but it may also be possible to specify each item simultaneously by inputting, for example, the product names of a stent or balloon.

[0045] Furthermore, the display device 23 accepts input of a designation of a current treatment stage of the intravascular treatment via the status field 504. The server 1 generates a second medical image according to the treatment stage designated in the status field 504, and outputs the second medical image to the display device 23.

[0046] Fig. 6 is an explanatory diagram of the treatment stages of intravascular treatment. Fig. 6 conceptually illustrates a general treatment flow when performing PCI. Generally, PCI is performed in the following procedure.

[0047] First, a guide wire for guiding the catheter 211 is inserted into the patient's blood vessel. Next, the catheter 211 is inserted to obtain an ultrasonic tomographic image, and the state of the lesion is confirmed in combination with the X-ray fluoroscopic image. Thereafter, the blood vessel is expanded with a balloon before placement as necessary so that the stent can be placed. Note that the balloon expansion before stent placement may be omitted.

[0048] Next, a stent is placed in the blood vessel, and the lesion is expanded with a balloon. After the stent is placed, ultrasound images are taken again and the condition of the lesion after expansion is confirmed in conjunction with the X-ray fluoroscopy images. If the stent is not sufficiently expanded, the blood vessel is expanded again with a balloon. Note that balloon expansion after stent placement may be omitted. Finally, the condition of the lesion is confirmed with each image, and the series of procedures is completed.

[0049] For simplicity, in the following description, balloon expansion before stent placement is referred to as "pre-dilation," and balloon expansion after stent placement is referred to as "post-dilation."

[0050] In this embodiment, the server 1 generates a second medical image after treatment from a first medical image before treatment at each treatment stage using a generative model 50 in accordance with the above pre-dilation, stent dilation, and post-dilation.

[0051] For example, the server 1 configures the generation model 50 as a model that can input class labels of input data, such as ConditionalGAN. When inputting a first medical image into the generation model 50, the server 1 inputs a value indicating whether the first medical image is before pre-dilation treatment, before stent expansion, or before post-dilation treatment, as label data indicating the class of the first medical image. Then, the server 1 generates a second medical image after any of the treatments according to the input label data. This allows the user to gradually check the second medical image that predicts the state after the treatment to be performed.

[0052] 5 illustrates an example of a display screen during stent expansion. The server 1 generates a second medical image after stent expansion according to the treatment stage specified in the status field 504. That is, the server 1 inputs an ultrasonic tomographic image generated by the intravascular imaging diagnostic device 21 before stent expansion, an X-ray fluoroscopic image taken by the fluoroscopic image taking device 22, and device information specified by the user to the generative model 50, and generates an ultrasonic tomographic image and an X-ray fluoroscopic image predicting the state after stent expansion. In this case, for example, the server 1 generates an ultrasonic tomographic image and an X-ray fluoroscopic image as the second medical image, in which the image region corresponding to the stent is displayed in a different display mode from other regions.

[0053] The server 1 outputs the generated second medical image and displays it on the display device 23. The display device 23 displays an ultrasonic tomographic image and an X-ray fluoroscopic image corresponding to the second medical image as a predicted tomographic image 505 and a predicted fluoroscopic image 506.

[0054] In addition, although FIG. 5 illustrates two-dimensional ultrasonic tomographic images and X-ray fluoroscopic images as the second medical images, the server 1 may reconstruct the two-dimensional ultrasonic tomographic images and X-ray fluoroscopic images to generate three-dimensional blood vessel images and display them on the display device 23. As described in FIG. 3, the server 1 inputs a plurality of ultrasonic tomographic images (transverse images) and X-ray fluoroscopic images sequentially acquired from the intravascular imaging diagnostic device 21 into the generation model 50, and generates a plurality of ultrasonic tomographic images and X-ray fluoroscopic images predicting the state after treatment as the second medical images. The server 1 aligns each ultrasonic tomographic image and the X-ray fluoroscopic image according to the position of the catheter 211 detected by a means such as an X-ray opaque marker during the ultrasonic examination, and converts them into a three-dimensional image by a known method. As a result, the state inside the blood vessel after treatment can be expressed in three dimensions and presented to the user.

[0055] 5, one pattern of second medical images is displayed by accepting a designation input of one pattern of device information, but multiple patterns of device information may be accepted to display multiple second medical images corresponding to the multiple patterns. For example, the server 1 accepts a designation input of multiple stents as candidates for stents to be used in treatment, generates second medical images for the cases where each stent is used separately, and displays previews on the display device 23. This can improve the convenience for the user.

[0056] Fig. 7 is a flowchart showing the procedure of the generation process of the generative model 50. The process contents when generating the generative model 50 by machine learning will be described with reference to Fig. 7. The control unit 11 of the server 1 acquires training data, which is data on a patient who has undergone treatment, including a first medical image, which is a medical image before the treatment, device information on the therapeutic device used in the treatment, and a second medical image, which is a medical image after the treatment (step S11). Specifically, the control unit 11 acquires an ultrasonic tomographic image and an X-ray fluoroscopic image before the treatment as the first medical image for a patient who has undergone intravascular treatment. The control unit 11 also acquires information on the stent, balloon, etc. used in the intravascular treatment as device information. The control unit 11 also acquires an ultrasonic tomographic image and an X-ray fluoroscopic image after the intravascular treatment as the second medical image.

[0057] The control unit 11 generates a generative model 50 that generates a second medical image when the first medical image and device information are input based on the training data (step S12). Specifically, the control unit 11 generates a GAN that receives an ultrasonic tomographic image, an X-ray fluoroscopic image, and device information before treatment as input, and outputs an ultrasonic tomographic image and an X-ray fluoroscopic image after treatment. The control unit 11 ends the series of processes.

[0058] 8 is a flowchart showing the procedure of the generation process of the second medical image. The process contents when generating the second medical image will be described with reference to FIG. The control unit 11 of the server 1 acquires device information related to a treatment device used to treat a patient's tubular organ from the image diagnostic system 2 (step S31). The control unit 11 also acquires a first medical image from the image diagnostic system 2 (step S32). Specifically, as described above, the control unit 11 acquires an ultrasonic tomographic image obtained by examining the patient to be treated and an X-ray fluoroscopic image captured simultaneously during the ultrasonic examination.

[0059] The control unit 11 determines whether or not pre-dilation is required before stent placement in response to an operation input from the user (step S33). If it is determined that pre-dilation is not required (S33: NO), the control unit 11 proceeds to step S36. If it is determined that pre-dilation is required (S33: YES), the control unit 11 inputs the first medical image acquired in step S32 into the generative model 50 to generate a second medical image after pre-dilation, and outputs it to the display device 23 (step S34).

[0060] The control unit 11 determines whether or not pre-expansion is completed in response to an operational input from the user (step S35). If it is determined that pre-expansion is not completed (S35: NO), the control unit 11 waits for processing.

[0061] When it is determined that pre-expansion is completed (S35: YES), the control unit 11 generates a second medical image after the stent expansion and outputs it to the display device 23 (step S36).

[0062] The control unit 11 determines whether or not post-dilatation is necessary in response to an operation input from the user (step S37). If it is determined that post-dilatation is not necessary (S37: NO), the control unit 11 ends the series of processes. If it is determined that post-dilatation is necessary (S37: YES), the control unit 11 reacquires the first medical image (step S38). The control unit 11 inputs the acquired first medical image into the generative model 50 to generate a second medical image after post-dilatation, and outputs it to the display device 23 (step S39). The control unit 11 ends the series of processes.

[0063] As described above, according to the first embodiment, a second medical image obtained by predicting a post-treatment state from a first medical image taken before treatment can be generated and presented to a user.

[0064] Furthermore, according to the first embodiment, the second medical images after each treatment can be presented in accordance with a plurality of treatment stages.

[0065] Furthermore, according to the first embodiment, second medical images after pre-expansion, after stent expansion, and after post-expansion can be generated in accordance with intravascular treatment and presented to the user.

[0066] Furthermore, according to the first embodiment, by receiving an input specifying a target site on an X-ray fluoroscopic image, device information can be suitably specified.

[0067] In addition, according to this embodiment 1, it is also possible to provide an image that represents the post-treatment condition in three dimensions from multiple ultrasonic tomographic images (transverse images) and X-ray fluoroscopic images that are generated sequentially using the generative model 50.

[0068] Furthermore, according to the first embodiment, by using lesion information as an input to the generative model 50 in addition to device information, it is possible to improve the generation accuracy of the second medical image.

[0069] (Embodiment 2) In this embodiment, a form will be described in which only ultrasonic tomographic images are used as the first medical image and the second medical image. Note that the same reference numerals are used to designate the same contents as in the first embodiment, and the description thereof will be omitted.

[0070] Fig. 9 is an explanatory diagram of a generative model 50 according to the second embodiment. The generative model 50 according to the present embodiment does not include an encoder and a decoder for processing X-ray fluoroscopic images, but only includes two encoders for receiving inputs of pre-treatment ultrasound tomographic images and device information, a decoder for generating post-treatment ultrasound tomographic images, and a classifier for identifying the authenticity of the ultrasound tomographic images generated by the decoder. The server 1 uses ultrasound tomographic images before and after treatment of a patient who has already undergone intravascular treatment as training data for the first and second medical images for training, and generates the generative model 50 shown in Fig. 9.

[0071] 10 is an explanatory diagram showing an example of a display screen of a medical image according to embodiment 2. In this embodiment, the display device 23 displays a current tomographic image 501 corresponding to a first medical image, as in embodiment 1. In this embodiment, the display device 23 further displays a longitudinal tomographic image 521 reconstructed from a plurality of ultrasonic tomographic images (transverse tomographic images).

[0072] The longitudinal tomographic image 521 is a longitudinal sectional image obtained by reconstructing a plurality of transverse cross-sectional images successively imaged by the intravascular imaging diagnostic device 21 in a form along the longitudinal direction of the blood vessel. The server 1 reconstructs the longitudinal tomographic image 521 from a plurality of transverse cross-sectional images (frame images) successively imaged along the longitudinal direction of the blood vessel in accordance with the scanning of the catheter 211, and displays the image on the display device 23. In this embodiment, the display device 23 accepts input of drawing a rectangular frame on the longitudinal tomographic image 521, thereby accepting input of designation of a site in the blood vessel to be treated. In addition, the display device 23 accepts input of designation of other device information through a designation field 503, as in the first embodiment.

[0073] The server 1 inputs an ultrasonic tomographic image corresponding to the first medical image and device information to the generation model 50, and generates an ultrasonic tomographic image corresponding to the second medical image. Then, the server 1 displays the generated ultrasonic tomographic image as a predicted tomographic image 505. Of course, as the predicted tomographic image 505, not only a transverse tomographic image but also a longitudinal tomographic image may be displayed.

[0074] As in the first embodiment, a plurality of ultrasonic tomographic images sequentially generated by the generative model 50 may be reconstructed to generate a three-dimensional image, which may be displayed on the display device 23.

[0075] Other than the above, this embodiment is the same as the first embodiment, so a detailed description of the flowchart and other details will be omitted.

[0076] (Embodiment 3) In this embodiment, a form will be described in which the generative model 50 receives input of correction information for the second medical image generated, and performs re-learning based on the input correction information.

[0077] For example, the display device 23 accepts an input of correction information from a user for correcting the second medical image generated by the generative model 50 on the display screen exemplified in Fig. 5. Specifically, the display device 23 accepts a correction input for correcting the position, length, diameter (width), etc. of a stent (treatment device) displayed in color. For example, the display device 23 accepts a drawing input for newly drawing a boundary (edge) of an image region corresponding to the stent on the predicted tomographic image 505 or the predicted fluoroscopic image 506. The display device 23 transmits the second medical image in which the image region corresponding to the stent is drawn by the user to the server 1 as correction information, and causes re-learning to be performed.

[0078] In this embodiment, the input of correction information for the second medical image after stent expansion is accepted, but the input of correction information for the second medical image after pre-expansion or post-expansion may also be accepted.

[0079] The server 1 performs re-learning based on the correction information acquired from the display device 23, and updates the generative model 50. Specifically, the server 1 provides the generative model 50 with the first medical image and device information input when the second medical image was generated, and the second medical image in which the image region of the stent is drawn, as training data for re-learning, and updates the parameters of the generator and the classifier.

[0080] 11 is a flowchart showing the procedure of the generation process of the second medical image according to embodiment 3. After generating the second medical image after the stent placement and outputting it to the display device 23 (step S36), the server 1 executes the following process. The control unit 11 of the server 1 accepts input of correction information for the output second medical image (step S201). For example, the control unit 11 accepts correction input for correcting the position, length, diameter, etc. of a stent on the second medical image. The control unit 11 shifts the process to step S37.

[0081] If NO in step S37, or after executing the process of step S39, the control unit 11 updates the generative model 50 based on the correction information input in step S201 (step S202). Specifically, the control unit 11 uses the first medical image and device information input when the second medical image was generated in step S36 as input data for training, and uses the image obtained by correcting the second medical image generated in step S36 as output data for training, and performs re-learning, updating the parameters of the generator and the classifier. The control unit 11 ends the series of processes.

[0082] As described above, according to the third embodiment, the generation accuracy of the second medical image can be improved through operation of this system.

[0083] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0084] 1. Server (information processing device) 1a Removable storage media 1b Semiconductor memory 11 Control section 12 Main memory 13. Communications Department 14 Auxiliary storage N Network P Program 2. Diagnostic imaging system 21 Intravascular imaging device 211 Catheter 22 Fluoroscopic imaging device 221 X-ray source 222 X-ray sensor 23 Display device 501 Current fault image 502 Current perspective image 503 Specified field 504 Status column 505 Predicted tomography 506 Predictive Fluoroscopic Images 521 Longitudinal tomography

Claims

1. A method for obtaining a first medical image of a patient's tubular organ before any of a plurality of treatment steps and device information related to a treatment device used to treat the tubular organ, When the first medical image and device information and a class label representing the treatment stage corresponding to the first medical image are input, the acquired first medical image and device information and the class label are input to a model that has been trained to generate a second medical image after treatment in the treatment stage, and the second medical image is generated. A program that causes a computer to carry out processing.

2. The treatment is an intravascular treatment using a catheter, A first medical image is acquired before a treatment in any one of a treatment stage of balloon expansion before stent placement, expansion of the stent, and balloon expansion after the stent placement, and the second medical image is generated. The program according to claim 1.

3. Displaying the first medical image on a display unit; receiving a designation input for designating a region in the hollow organ to be treated and a plurality of patterns of the treatment device to be used for treating the region in the hollow organ for the first medical image; The device information indicating the specified region and a plurality of patterns of treatment devices is input to the model, thereby generating a plurality of the second medical images corresponding to the plurality of patterns.

3. The program according to claim 1 or 2.

4. the first medical image includes a fluoroscopic image of the inside of the patient's body; Displaying the fluoroscopic image on a display unit; receiving a designation input for designating a region in the hollow organ to be treated and the treatment device to be used for treating the region on the fluoroscopic image; Inputting the device information into the model, the device information indicating the designated site and treatment device. The program according to any one of claims 1 to 3.

5. The first medical image includes a cross-sectional image of the inside of the tubular organ, generating a longitudinal tomographic image from a plurality of transverse tomographic images successively imaged along the longitudinal direction of the tubular organ and displaying the longitudinal tomographic image on a display unit; receiving a designation input for designating a region in the hollow organ to be treated and the treatment device to be used for treating the region on the longitudinal tomographic image; Inputting the device information into the model, the device information indicating the designated site and treatment device. The program according to any one of claims 1 to 3.

6. inputting the plurality of transverse images into the model to generate a plurality of post-treatment transverse images; A three-dimensional second medical image is generated from the generated multiple cross-sectional images after treatment. The program according to claim 5.

7. Accepting input of lesion information indicating the characteristics of a lesion in the tubular organ; Inputting the lesion information into the model to generate the second medical image. The program according to any one of claims 1 to 6.

8. Accepting input of correction information for the generated second medical image; Update the model based on the correction information. The program according to any one of claims 1 to 7.

9. The first medical image, device information, and the class label are input to the model to generate the second medical image in which a predetermined object is displayed in a different display manner from other image regions. The program according to any one of claims 1 to 8.

10. A method for obtaining a first medical image of a patient's tubular organ before any of a plurality of treatment steps and device information related to a treatment device used to treat the tubular organ, When the first medical image and device information and a class label representing the treatment stage corresponding to the first medical image are input, the acquired first medical image and device information and the class label are input to a model that has been trained to generate a second medical image after treatment in the treatment stage, and the second medical image is generated. An information processing method in which processing is performed by a computer.

11. An acquisition unit that acquires a first medical image of a patient's tubular organ before any of a plurality of treatment steps and device information related to a treatment device used to treat the tubular organ. a generation unit that inputs the acquired first medical image and device information and the class label into a model that has been trained to generate a second medical image after treatment at the treatment stage when the first medical image, device information, and a class label representing the treatment stage corresponding to the first medical image are input, and generates the second medical image; An information processing device comprising:

12. Obtaining training data including a first medical image of a patient's luminal organ before each of a plurality of treatment steps, a second medical image of the patient's luminal organ after each of a plurality of treatment steps, and device information related to a therapeutic device used to treat the luminal organ; Based on the training data, a trained model is generated that generates the second medical image when the first medical image and device information and a class label representing the treatment stage corresponding to the first medical image are input. A model generation method in which processing is performed by a computer.

Images