Information processing device, information processing method, program, and trained model generation method

Abstract

The present invention provides, for example, an information processing device that assists in understanding an image which is acquired with an image acquisition catheter.　The information processing device comprises: an image acquisition unit that acquires a catheter image (518) which has been obtained with an image acquisition catheter inserted into a first cavity; and a first classification data output unit that inputs the acquired catheter image (518) into a first classification trained model (621), which upon receiving the input of the catheter image (518) outputs first classification data (521) in which a biological tissue region is classified differently from a non-biological tissue region including a first inner cavity region inside the first cavity and a second inner cavity region inside a second cavity where the image acquisition catheter is not inserted, and that outputs the first classification data (521), wherein the first classification trained model (621) is generated with use of first training data which clearly indicates at least the biological tissue region and the non-biological tissue region including the first inner cavity region and the second inner cavity region.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a program, and a method for generating a trained model. [Background technology]

[0002] BACKGROUND ART A catheter system is used in which an image acquisition catheter is inserted into a hollow organ such as a blood vessel to acquire an image (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 164071 Summary of the Invention [Problem to be solved by the invention]

[0004] In locations with complex structures, such as the intracardiac region, it may be difficult for a user to quickly interpret images acquired by an imaging catheter.

[0005] In one aspect, an object of the present invention is to provide an information processing device or the like that supports the understanding of images acquired by an image acquisition catheter. [Means for solving the problem]

[0006] The information processing device includes an image acquisition unit that acquires a catheter image obtained by an image acquisition catheter inserted into a first cavity, and a first classified data output unit that, when the catheter image is input, inputs the acquired catheter image into a first classification trained model that outputs first classified data in which a non-living tissue region including a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the image acquisition catheter is not inserted and a living tissue region are classified as different regions, and outputs the first classified data, and the first classification trained model is generated using first training data in which at least the non-living tissue region including the first lumen region and the second lumen region and the living tissue region are clearly indicated. the imaging catheter is a radial scanning type tomographic image acquiring catheter, the catheter image is an RT format image in which a plurality of scanning line data acquired from the imaging catheter is arranged in parallel in order of scanning angle, and the first classification data is a classification result of each pixel in the RT format image. . [Effects of the Invention]

[0007] In one aspect, it is possible to provide an information processing device or the like that supports the understanding of images acquired by an image acquisition catheter. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of a catheter system. [Figure 2] FIG. 2 is an explanatory diagram illustrating an overview of an image acquisition catheter. [Figure 3] FIG. 1 is an explanatory diagram illustrating the configuration of a catheter system. [Figure 4] FIG. 1 is an explanatory diagram illustrating an overview of the operation of the catheter system. [Figure 5A] FIG. 10 is an explanatory diagram schematically showing the operation of the imaging catheter. [Figure 5B] FIG. 2 is an explanatory diagram schematically showing a catheter image taken by an image acquisition catheter. [Figure 5C] FIG. 10 is an explanatory diagram illustrating classification data generated based on a catheter image. [Figure 6] FIG. 1 is an explanatory diagram illustrating the configuration of a medical instrument trained model. [Figure 7] FIG. 2 is an explanatory diagram illustrating the configuration of a classification model. [Figure 8] FIG. 10 is an explanatory diagram illustrating an outline of processing related to position information. [Figure 9] FIG. 10 is an explanatory diagram illustrating the record layout of a medical instrument position training data DB. [Figure 10] 10 is an example of a screen used to create a medical instrument position training data DB. [Figure 11] 10 is a flowchart illustrating the flow of processing of a program for creating a medical instrument position training data DB. [Figure 12] 10 is a flowchart illustrating the processing flow of a medical instrument trained model generation program. [Figure 13] 10 is a flowchart illustrating the flow of processing of a program for adding data to a medical instrument position training data DB. [Figure 14] FIG. 10 is an explanatory diagram illustrating visualization of a medical instrument. [Figure 15] FIG. 1 is an explanatory diagram illustrating the configuration of an angle-learned model. [Figure 16] FIG. 2 is an explanatory diagram illustrating a location information model. [Figure 17] 11 is a flowchart illustrating the flow of processing of a program according to the third embodiment. [Figure 18] FIG. 2 is an explanatory diagram illustrating the configuration of a classification model. [Figure 19] FIG. 10 is an explanatory diagram illustrating first training data. [Figure 20] 10 is a flowchart illustrating the flow of processing of a program according to a fourth embodiment. [Figure 21] FIG. 13 is an explanatory diagram illustrating the configuration of a classification model according to a fifth embodiment. [Figure 22] FIG. 10 is an explanatory diagram illustrating second training data. [Figure 23] 13 is a flowchart illustrating the flow of processing of a program according to the fifth embodiment. [Figure 24] 10 is a flowchart illustrating the processing flow of a classification and synthesis subroutine. [Figure 25] FIG. 10 is an explanatory diagram illustrating the configuration of a trained model with hints 631. [Figure 26] FIG. 10 is an explanatory diagram illustrating the record layout of a hinted model training data DB. [Figure 27] 13 is a flowchart illustrating the flow of processing of a program according to a sixth embodiment. [Figure 28] 10 is a flowchart illustrating a processing flow of a program according to a modified example. [Figure 29] FIG. 20 is an explanatory diagram illustrating the configuration of a classification model according to a seventh embodiment. [Figure 30] FIG. 10 is an explanatory diagram illustrating the configuration of a classification model according to a modified example. [Figure 31] FIG. 20 is an explanatory diagram illustrating an outline of the processing of the eighth embodiment. [Figure 32A] FIG. 10 is an explanatory diagram illustrating an outline of a process for correcting position information. [Figure 32B] FIG. 10 is an explanatory diagram illustrating an outline of a process for correcting position information. [Figure 32C] FIG. 10 is an explanatory diagram illustrating an outline of a process for correcting position information. [Figure 32D] FIG. 10 is an explanatory diagram illustrating an outline of a process for correcting position information. [Figure 33] 13 is a flowchart illustrating the flow of processing of a program according to the eighth embodiment. [Figure 34] 10 is a flowchart illustrating a process flow of a three-dimensional display subroutine. [Figure 35] FIG. 20 is an explanatory diagram illustrating padding processing according to the ninth embodiment. [Figure 36] FIG. 10 is an explanatory diagram illustrating a modified polar padding process. [Figure 37] FIG. 20 is an explanatory diagram illustrating the configuration of a catheter system according to a tenth embodiment. [Figure 38] FIG. 22 is a functional block diagram of an information processing device according to an eleventh embodiment. [Figure 39] FIG. 22 is an explanatory diagram illustrating the machine learning process of the twelfth embodiment. [Figure 40] FIG. 10 is an explanatory diagram illustrating a contradiction loss function. [Figure 41]FIG. 10 is an explanatory diagram illustrating a contradiction loss function. [Figure 42] FIG. 10 is an explanatory diagram illustrating a contradiction loss function. [Figure 43] 22 is a flowchart illustrating the processing flow of a program according to the twelfth embodiment. [Figure 44] 22 is a flowchart illustrating the processing flow of a program according to the thirteenth embodiment. [Figure 45] 23 is an example of a display screen according to a fourteenth embodiment. [Figure 46] 23 is an example of a display screen according to a fourteenth embodiment. [Figure 47] 14 is an example of a display screen according to Modification 14-1. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Embodiment 1] Figure 1 is an explanatory diagram illustrating an overview of a catheter system 10. The catheter system 10 of this embodiment is used in IVR (Interventional Radiology), which treats various organs while performing fluoroscopy using an imaging diagnostic device such as an X-ray fluoroscopy device. Medical instruments for treatment can be accurately operated by referring to images acquired by the catheter system 10 placed near the area to be treated.

[0010] The catheter system 10 includes an image acquisition catheter 40, an MDU (Motor Driving Unit) 33, and an information processing device 20. The image acquisition catheter 40 is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is an input device such as a keyboard, a mouse, a trackball, or a microphone. The display device 31 and the input device 32 may be stacked together to form a touch panel. The input device 32 and the information processing device 20 may be configured as an integrated unit.

[0011] 2 is an explanatory diagram illustrating an overview of the imaging catheter 40. The imaging catheter 40 has a probe portion 41 and a connector portion 45 disposed at the end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the imaging catheter 40 farther from the connector portion 45 will be referred to as the tip side.

[0012] A shaft 43 is inserted inside the probe section 41. A sensor 42 is connected to the tip side of the shaft 43. A guidewire lumen 46 is provided at the tip of the probe section 41. The user inserts a guidewire to a position beyond the target site, and then inserts the guidewire into the guidewire lumen 46 to guide the sensor 42 to the target site. A ring-shaped tip marker 44 is fixed near the tip of the probe section 41.

[0013] The sensor 42 is, for example, an ultrasonic transducer that transmits and receives ultrasonic waves, or a transmitter / receiver for OCT (Optical Coherence Tomography) that emits near-infrared light and receives reflected light. In the following description, the image acquisition catheter 40 will be described as an example of an IVUS (Intravascular Ultrasound) catheter used to capture ultrasonic cross-sectional images from inside the circulatory system.

[0014] 3 is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 includes an information processing device 20, an MDU 33, and an image acquisition catheter 40. The information processing device 20 includes a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

[0015] The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, etc. The control unit 21 is connected to each hardware unit that constitutes the information processing device 20 via a bus.

[0016] The main memory device 22 is a storage device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs currently being executed by the control unit 21.

[0017] The auxiliary storage device 23 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 23 stores the medical instrument trained model 611, the classification model 62, the program to be executed by the control unit 21, and various data required for executing the program. The communication unit 24 is an interface that performs communication between the information processing device 20 and a network.

[0018] Display unit 25 is an interface that connects display device 31 to the bus. Input unit 26 is an interface that connects input device 32 to the bus. Catheter control unit 271 controls MDU 33 and sensor 42, and generates images based on signals received from sensor 42, etc.

[0019] The MDU 33 rotates the sensor 42 and the shaft 43 inside the probe section 41. The catheter control section 271 generates one catheter image 51 (see FIG. 4) for each rotation of the sensor 42. The generated catheter image 51 is a cross-sectional image centered on the probe section 41 and approximately perpendicular to the probe section 41.

[0020] The MDU 33 can also move the sensor 42 back and forth while rotating the sensor 42 and shaft 43 inside the probe section 41. By rotating the sensor 42 while pulling or pushing it, the catheter control section 271 continuously generates multiple catheter images 51 that are approximately perpendicular to the probe section 41. The continuously generated catheter images 51 can be used to construct a three-dimensional image. Therefore, the image acquisition catheter 40 functions as a three-dimensional scanning catheter that sequentially acquires multiple catheter images 51 along the longitudinal direction.

[0021] The operation of advancing and retracting the sensor 42 includes both an operation of advancing and retracting the entire probe unit 41 and an operation of advancing and retracting the sensor 42 inside the probe unit 41. The advancing and retracting operation may be performed automatically at a predetermined speed by the MDU 33 or manually by the user.

[0022] The imaging catheter 40 is not limited to a mechanical scanning type that mechanically rotates and moves back and forth, but may be an electronic radial scanning type imaging catheter 40 that uses a sensor 42 in which multiple ultrasonic transducers are arranged in a ring shape.

[0023] Using the image acquisition catheter 40, a catheter image 51 can be captured that includes not only the biological tissues that make up the circulatory system, such as the heart wall and blood vessel walls, but also reflectors present inside the circulatory system, such as red blood cells, and organs present outside the circulatory system, such as the respiratory and digestive systems.

[0024] In this embodiment, an example will be described in which the imaging catheter 40 is used for atrial septal puncture. In atrial septal puncture, the imaging catheter 40 is inserted into the right atrium, and then the Brockenbrough needle is inserted into the fossa ovalis, a thin-walled portion of the atrial septum, under ultrasound guidance. The tip of the Brockenbrough needle reaches the inside of the left atrium.

[0025] When performing atrial septal puncture, the catheter image 51 depicts the Brockenbrough needle in addition to biological tissues that make up the circulatory system, such as the atrial septum, right atrium, left atrium, and aorta, and reflectors such as red blood cells contained in the blood flowing inside the circulatory system. A user such as a doctor can safely perform atrial septal puncture by confirming the positional relationship between the fossa ovalis and the tip of the Brockenbrough needle using the catheter image 51. The Brockenbrough needle is an example of a medical instrument of this embodiment.

[0026] The use of the catheter system 10 is not limited to atrial septal puncture. For example, the catheter system 10 can be used for procedures such as transcatheter myocardial ablation, transcatheter valve replacement, and stent placement in coronary arteries. The site to be treated using the catheter system 10 is not limited to the area around the heart. For example, the catheter system 10 can be used to treat various sites, such as the pancreatic duct, bile duct, and blood vessels in the lower limbs.

[0027] The function and configuration of catheter control unit 271 are the same as those of conventionally used ultrasound diagnostic devices, and therefore detailed description will be omitted. Note that control unit 21 may also realize the function of catheter control unit 271.

[0028] The information processing device 20 is connected to various imaging diagnostic devices 37 such as an X-ray angiography device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, or an ultrasound diagnostic device via an HIS (Hospital Information System) or the like.

[0029] The information processing device 20 of this embodiment is a dedicated ultrasound diagnostic device, or a personal computer, tablet, smartphone, or the like having the functionality of an ultrasound diagnostic device. In the following description, an example will be given in which the information processing device 20 is also used for training a trained model such as the medical instrument trained model 611 and for creating training data. A computer or server other than the information processing device 20 may be used for training the trained model and creating training data.

[0030] In the following explanation, we will mainly use an example in which the control unit 21 performs software processing. The processing explained using the flowcharts and the various trained models may each be implemented by dedicated hardware.

[0031] Fig. 4 is an explanatory diagram illustrating an outline of the operation of the catheter system 10. Fig. 4 illustrates an example in which a plurality of catheter images 51 are captured while the sensor 42 is pulled at a predetermined speed, and the images are displayed in real time.

[0032] The control unit 21 captures one catheter image 51 (step S501). The control unit 21 acquires position information of the medical instrument depicted in the catheter image 51 (step S502). In Fig. 4, the position of the medical instrument in the catheter image 51 is indicated by an "x" mark.

[0033] The control unit 21 associates the catheter image 51, the position of the catheter image 51 in the longitudinal direction of the image acquisition catheter 40, and the position information of the medical instrument, and records them in the auxiliary storage device 23 or a large-capacity storage device connected to the HIS (step S503).

[0034] The control unit 21 generates classification data 52 by classifying each part constituting the catheter image 51 by the depicted subject (step S504). In Fig. 4, the classification data 52 is shown by a schematic diagram in which the catheter image 51 is colored differently based on the classification result.

[0035] The control unit 21 determines whether the user has designated two-dimensional display or three-dimensional display (step S505). If it is determined that the user has designated two-dimensional display (2D in step S505), the control unit 21 displays the catheter image 51 and the classification data 52 on the display device 31 in two-dimensional display (step S506).

[0036] 4, "2D / 3D" is used as if it were a selection between "two-dimensional display" and "three-dimensional display." However, if the user selects "3D," the control unit 21 may display both "two-dimensional display" and "three-dimensional display."

[0037] If it is determined that the user has designated three-dimensional display (3D in step S505), the control unit 21 determines whether the position information of the medical instrument sequentially recorded in step S503 is normal (step S511). If it is determined that the position information is not normal (NO in step S511), the control unit 21 corrects the position information (step S512). Details of the processes performed in steps S511 and S512 will be described later.

[0038] If it is determined to be normal (YES in step S511), or after step S512 is completed, the control unit 21 performs a three-dimensional display illustrating the structure of the region under observation and the position of the medical instrument (step S513). As described above, the control unit 21 may display both the three-dimensional display and the two-dimensional display on a single screen.

[0039] After step S506 or step S513 is completed, control unit 21 determines whether acquisition of catheter image 51 is completed (step S507). For example, when an end instruction is received from the user, control unit 21 determines to end the process.

[0040] If it is determined not to end the process (NO in step S507), the control unit 21 returns to step S501. If it is determined to end the process (YES in step S507), the control unit 21 ends the process.

[0041] 4 illustrates a process flow for performing two-dimensional display (step S506) or three-dimensional display (step S513) in real time while capturing a series of catheter images 51. Controller 21 may also perform two-dimensional display or three-dimensional display in non-real time based on the data recorded in step S503.

[0042] Fig. 5A is an explanatory diagram schematically showing the operation of the image acquisition catheter 40. Fig. 5B is an explanatory diagram schematically showing a catheter image 51 captured by the image acquisition catheter 40. Fig. 5C is an explanatory diagram schematically explaining classification data 52 generated based on the catheter image 51. The RT (Radius-Theta) format and the XY format will be explained using Figs. 5A to 5C.

[0043] As described above, the sensor 42 transmits and receives ultrasonic waves while rotating inside the imaging catheter 40. The catheter control unit 271 acquires radial scanning line data centered on the imaging catheter 40, as schematically shown by eight arrows in FIG. 5A.

[0044] The catheter control unit 271 can generate the catheter image 51 shown in FIG. 5B in two formats based on the scan line data: an RT format catheter image 518 and an XY format catheter image 519. The RT format catheter image 518 is an image generated by arranging the respective scan line data in parallel with each other. The horizontal direction of the RT format catheter image 518 indicates the distance from the image acquisition catheter 40.

[0045] The vertical direction of the RT format catheter image 518 indicates the scanning angle. One RT format catheter image 518 is formed by arranging the scanning line data acquired by the sensor 42 rotating 360 degrees in parallel in the order of the scanning angle.

[0046] In FIG. 5B, the left side of the RT catheter image 518 shows a location closer to the imaging catheter 40, and the right side of the RT catheter image 518 shows a location farther from the imaging catheter 40.

[0047] The XY-format catheter image 519 is an image generated by radially arranging each scan line data and interpolating it. The XY-format catheter image 519 shows a tomographic image of the subject cut perpendicular to the image acquisition catheter 40 at the position of the sensor 42.

[0048] 5C schematically shows classification data 52, which is generated by classifying each part of catheter image 51 by the depicted subject. Classification data 52 can also be displayed in two formats: RT format classification data 528 and XY format classification data 529. The image conversion method between RT format and XY format is well known, so a description thereof will be omitted.

[0049] In Figure 5C, thick hatching sloping downward to the right indicates biological tissue regions that form cavities, such as the atrial wall and ventricular wall, into which the imaging catheter 40 is inserted. Thin hatching sloping downward to the left indicates the inside of a first cavity, which is a blood flow region into which the tip of the imaging catheter 40 is inserted. Thin hatching sloping downward to the right indicates the inside of a second cavity, which is a blood flow region other than the first cavity.

[0050] When performing atrial septal puncture from the right atrium to the left atrium, the first chamber is the right atrium, and the second chamber is the left atrium, right ventricle, left ventricle, aorta, coronary artery, etc. In the following description, the inside of the first chamber is referred to as the first lumen region, and the inside of the second chamber is referred to as the second lumen region.

[0051] Thick hatching sloping downward to the left indicates a non-lamber region of the non-biological tissue region that is neither the first lumen region nor the second lumen region. The non-lamber region includes extracardiac regions and regions outside cardiac structures. If the imaging range of the imaging catheter 40 is small and the distal wall of the left atrium cannot be adequately imaged, the interior of the left atrium is also included in the non-lamber region. Similarly, if the distal wall of the lumen of the left ventricle, pulmonary artery, pulmonary vein, aortic arch, etc. cannot be adequately imaged, the non-lamber region also includes the non-lamber region.

[0052] The black areas indicate medical device regions in which medical devices such as Brockenbrough needles are depicted. In the following description, the biological tissue region and the non-biological tissue region may be collectively referred to as a biological tissue-related region.

[0053] It should be noted that the medical instrument is not necessarily inserted into the same first cavity as the imaging catheter 40. Depending on the procedure, the medical instrument may be inserted into the second cavity.

[0054] The hatching and black areas shown in FIG. 5C are examples of ways in which each region can be distinguished. Each region is displayed on the display device 31 using, for example, a different color. The control unit 21 realizes the function of a first-mode output unit that outputs the first lumen region, the second lumen region, and the biological tissue region in a manner in which they can be distinguished from one another. The control unit 21 also realizes the function of a second-mode output unit that outputs the first lumen region, the second lumen region, the non-lumen region, and the biological tissue region in a manner in which they can be distinguished from one another.

[0055] For example, XY format display is suitable for use during interventional radiology procedures, such as when confirming the position of a Brockenbrough needle for atrial septal puncture. However, in XY display, information near the imaging catheter 40 is compressed, reducing the amount of data, and data that does not actually exist is added by interpolation at positions far from the imaging catheter 40. Therefore, when analyzing catheter images 51, using RT format images can provide more accurate results than using XY format images.

[0056] In the following description, the control unit 21 generates RT-format classification data 528 based on the RT-format catheter image 518. The control unit 21 converts the XY-format catheter image 519 to generate the RT-format catheter image 518, and converts the RT-format classification data 528 to generate the XY-format classification data 529.

[0057] The classification data 52 will be described below using a specific example. A "biological tissue region label" is recorded for pixels classified into the "biological tissue region," a "first lumen region label" for pixels classified into the "first lumen region," a "second lumen region label" for pixels classified into the "second lumen region," a "non-lumen region label" for pixels classified into the "non-lumen region," a "medical instrument region label" for pixels classified into the "medical instrument region," and a "non-biological tissue region label" for pixels classified into the "non-biological tissue region." Each label is represented, for example, by an integer.

[0058] The control unit 21 may generate the XY-format classification data 529 based on the XY-format catheter image 519. The control unit 21 may generate the RT-format classification data 528 based on the XY-format classification data 529.

[0059] FIG. 6 is an explanatory diagram illustrating the configuration of the medical instrument trained model 611. The medical instrument trained model 611 is a model that receives the catheter image 51 and outputs first position information related to the position where the medical instrument is depicted. The medical instrument trained model 611 implements step S502 described using FIG. 4. The output layer of the medical instrument trained model 611 functions as a first position information output unit that outputs the first position information.

[0060] In Fig. 6, the input of the medical instrument trained model 611 is an RT-format catheter image 518. The first position information is the probability that a medical instrument is depicted for each part on the RT-format catheter image 518. In Fig. 6, locations where there is a high probability that a medical instrument is depicted are indicated by heavy hatching, and locations where there is a low probability that a medical instrument is depicted are indicated by no hatching.

[0061] The medical instrument trained model 611 is generated by machine learning using, for example, a neural network structure such as a convolutional neural network (CNN). Examples of CNNs that can be used to generate the medical instrument trained model 611 include R-CNN (Region Based Convolutional Neural Network), YOLO (You Only Look Once), U-Net, and GAN (Generative Adversarial Network). The medical instrument trained model 611 may also be generated using a neural network structure other than a CNN.

[0062] The medical instrument trained model 611 may be a model that receives a plurality of catheter images 51 acquired in time series and outputs first position information for the latest catheter image 51. The medical instrument trained model 611 can be generated by combining a model that receives time series input, such as an RNN (Recurrent Neural Network), with the above-described neural network structure.

[0063] The RNN is, for example, a long short-term memory (LSTM). When using the LSTM, the medical instrument trained model 611 includes a memory unit that stores information about previously input catheter images 51. The medical instrument trained model 611 outputs first position information based on the information stored in the memory unit and the latest catheter image 51.

[0064] When using multiple catheter images 51 acquired in time series, the medical instrument trained model 611 may include a recursive input unit that inputs an output based on a previously input catheter image 51 along with the next catheter image 51. The medical instrument trained model 611 outputs first position information based on the latest catheter image 51 and the input from the recursive input unit. By using catheter images 51 acquired in time series, it is possible to realize a medical instrument trained model 611 that is less susceptible to the effects of image noise and the like and outputs first position information with high accuracy.

[0065] The medical instrument trained model 611 may use the position of a single pixel on the catheter image 51 that has been received as input to output a location where there is a high probability that a medical instrument is depicted. For example, the medical instrument trained model 611 may be a model that calculates the probability that a medical instrument is depicted for each site on the catheter image 51 as shown in FIG. 6, and then outputs the position of the pixel with the highest probability. The medical instrument trained model 611 may output the position of the center of gravity of an area where the probability that a medical instrument is depicted exceeds a predetermined threshold. The medical instrument trained model 611 may output an area where the probability that a medical instrument is depicted exceeds a predetermined threshold.

[0066] In addition, there are cases where multiple medical instruments are used simultaneously. When multiple medical instruments are depicted in the catheter image 51, it is desirable that the medical instrument trained model 611 is a model that outputs the first position information of each of the multiple medical instruments.

[0067] The medical instrument trained model 611 may be a model that outputs only the first position information of one medical instrument. The control unit 21 can acquire the first position information of a second medical instrument by inputting the RT-format catheter image 518, in which the periphery of the first position information output from the medical instrument trained model 611 is masked, into the medical instrument trained model 611. By repeating the same process, the control unit 21 can also acquire the first position information of a third or subsequent medical instrument.

[0068] 7 is an explanatory diagram illustrating the configuration of the classification model 62. The classification model 62 is a model that receives the catheter image 51 and outputs classification data 52 obtained by classifying each part that constitutes the catheter image 51 by the depicted subject. The classification model 62 implements step S504 described using FIG. 4.

[0069] A specific example will be given below. Classification model 62 classifies each pixel constituting input RT-format catheter image 518 into, for example, a "biological tissue region," a "first lumen region," a "second lumen region," a "non-lumen region," and a "medical instrument region," and outputs RT-format classification data 528 that associates the position of each pixel with a label indicating the classification result.

[0070] The classification model 62 may divide the catheter image 51 into regions of any size, such as three pixels vertically and three pixels horizontally, for a total of nine pixels, and output classification data 52 obtained by classifying each region. The classification model 62 is, for example, a trained model that performs semantic segmentation on the catheter image 51. A specific example of the classification model 62 will be described later.

[0071] Figure 8 is an explanatory diagram outlining the processing related to position information. A plurality of catheter images 51 are captured while the sensor 42 is moved in the longitudinal direction of the image-acquiring catheter 40. In Figure 8, the line drawing of an approximate truncated cone schematically shows a biological tissue region that is three-dimensionally constructed based on the plurality of catheter images 51. The interior of the approximate truncated cone represents the first lumen region.

[0072] The white and black circles indicate the positions of the medical instruments obtained from each catheter image 51. Of these, the black circles are determined to be false positives because they are located far away from the white circles. The shape of the medical instrument can be reproduced by thick lines smoothly connecting the white circles. The cross marks indicate complementary information that complements the position information of medical instruments that could not be detected.

[0073] Details of the processing described using Fig. 8 will be described in embodiment 8. The processing described using Fig. 8 realizes the processing of steps S511 and S512 described using Fig. 4.

[0074] For example, when a medical instrument is in contact with a biological tissue region, it is known that even when a user such as an experienced doctor or laboratory technician reads a single still catheter image 51, it may be difficult to identify where the medical instrument is depicted. However, when observing a moving catheter image 51, the user can relatively easily determine the position of the medical instrument. This is because the user reads the image while expecting that the medical instrument is located in the same position as in the previous frame.

[0075] 8, the medical instrument is reconstructed without any inconsistencies using the position information of the medical instrument obtained from each of the multiple catheter images 51. By performing such processing, it is possible to realize a catheter system 10 that can accurately determine the position of the medical instrument and display the shape of the medical instrument in a three-dimensional image, just as when a user observes a video.

[0076] According to this embodiment, it is possible to provide a catheter system 10 that supports the understanding of the catheter image 51 acquired using the image acquisition catheter 40 by the display in steps S506 and S513. By using the catheter system 10 of this embodiment, the user can accurately grasp the position of the medical instrument, and perform IVR safely.

[0077] [Embodiment 2] This embodiment relates to a method for generating a medical instrument trained model 611. Explanation of parts common to embodiment 1 will be omitted. Note that this embodiment will be described taking as an example a case where the medical instrument trained model 611 is generated using the information processing device 20 described using FIG. 3 .

[0078] The medical instrument trained model 611 may be created using a computer or the like separate from the information processing device 20. The medical instrument trained model 611 for which machine learning has been completed may be copied to the auxiliary storage device 23 via a network. The medical instrument trained model 611 trained on one piece of hardware can be used by multiple information processing devices 20.

[0079] 9 is an explanatory diagram illustrating the record layout of the medical instrument position training data DB (Database) 71. The medical instrument position training data DB 71 is a database that records catheter images 51 and position information of medical instruments in association with each other, and is used for training the medical instrument trained model 611 by machine learning.

[0080] The medical instrument position training data DB 71 has a catheter image field and a position information field. The catheter image field stores catheter images 51, such as an RT format catheter image 518. The catheter image field may store so-called sound ray data indicating an ultrasonic signal received by the sensor 42. The catheter image field may store scan line data generated based on the sound ray data.

[0081] The position information field records the position information of the medical instrument depicted in the catheter image 51. The position information is, for example, information indicating the position of a pixel marked by a labeler on the catheter image 51, as will be described later. The position information may also be information indicating a circular area centered near a point marked by the labeler on the catheter image 51. The size of the circle does not exceed the size of the medical instrument depicted in the catheter image 51. The circle is, for example, large enough to inscribe a square of 50 pixels or less in length and width.

[0082] Fig. 10 is an example of a screen used to create the medical instrument position training data DB 71. The screen of Fig. 10 displays a pair of catheter images 51, which are an RT format catheter image 518 and an XY format catheter image 519. The RT format catheter image 518 and the XY format catheter image 519 are images created based on the same sound ray data.

[0083] A control button area 782 is displayed below the catheter image 51. Above the control button area 782, there are arranged the frame number of the catheter image 51 currently being displayed, and a jump button that the user can use to input an arbitrary frame number to jump the display.

[0084] Below the frame numbers, etc., are arranged various buttons that the user can use to perform operations such as fast forward, rewind, and frame advance. These buttons are similar to those commonly used in various image playback devices, and therefore a description thereof will be omitted.

[0085] In this embodiment, the user is a person who creates training data by viewing pre-recorded catheter images 51 and labeling the positions of medical instruments. In the following description, the person who creates the training data will be referred to as a labeler. A labeler is a doctor or technician who is skilled in interpreting catheter images 51, or a person who has been trained to perform accurate labeling. Furthermore, in the following description, the work of the labeler adding marks or the like to catheter images 51 to label them may be referred to as marking.

[0086] The labeler observes the displayed catheter image 51 and determines the position where the medical instrument is depicted. Generally, the area where the medical instrument is depicted is very small compared to the overall area of ​​the catheter image 51. The labeler moves the cursor 781 to approximately the center of the area where the medical instrument is depicted and performs a marking operation such as a click operation. If the display device 31 is a touch panel, the labeler may perform a marking operation by tapping with a finger or a stylus pen. The labeler may also perform a marking operation using a so-called flick operation.

[0087] The labeler may mark either the RT format catheter image 518 or the XY format catheter image 519. The control unit 21 may display the mark at the corresponding position on the other catheter image 51.

[0088] The control unit 21 creates a new record in the medical instrument position training data DB 71 and records the catheter image 51 in association with the position marked by the labeler. The control unit 21 displays the next catheter image 51 on the display device 31. By repeating the above process multiple times, the medical instrument position training data DB 71 is created.

[0089] That is, the labeler can sequentially mark multiple catheter images 51 by simply performing a click operation or the like on the catheter image 51 without operating each button in the control button area 782. The labeler only needs to perform a single click operation or the like on one catheter image 51 depicting one medical instrument.

[0090] As mentioned above, multiple medical instruments may be depicted on the catheter image 51. The labeler can mark each medical instrument with a single click or other operation. The following explanation will be given using as an example a case where one medical instrument is depicted on one catheter image 51.

[0091] Fig. 11 is a flowchart illustrating the processing flow of a program for creating the medical instrument position training data DB 71. The description will be given taking as an example a case where the medical instrument position training data DB 71 is created using the information processing device 20. The program in Fig. 11 may be executed on hardware separate from the information processing device 20.

[0092] 11, a large number of catheter images 51 are recorded in the auxiliary storage device 23 or an external mass storage device. In the following explanation, an example will be given in which the catheter images 51 are recorded in the auxiliary storage device 23 in the form of video data including a plurality of RT format catheter images 518 captured in time series.

[0093] The control unit 21 acquires one frame of the RT format catheter image 518 from the auxiliary storage device 23 (step S671). The control unit 21 converts the RT format catheter image 518 to generate an XY format catheter image 519 (step S672). The control unit 21 displays the screen described with reference to FIG. 10 on the display device 31 (step S673).

[0094] The control unit 21 accepts an input operation of position information by the labeler via the input device 32 (step S674). Specifically, the input operation is a click operation or a tap operation on the RT format catheter image 518 or the XY format catheter image 519.

[0095] The control unit 21 displays a mark such as a small circle at the position where the input operation is accepted (step S675). The acceptance of the input operation for the image displayed on the display device 31 via the input device 32 and the display of the mark on the display device 31 are user interfaces that have been used conventionally, and therefore detailed explanations thereof will be omitted.

[0096] The control unit 21 determines whether the image for which the input operation was accepted in step S674 is the RT-format catheter image 518 (step S676). If it is determined that the image is the RT-format catheter image 518 (YES in step S676), the control unit 21 also displays a mark at the corresponding position on the XY-format catheter image 519 (step S677). If it is determined that the image is not the RT-format catheter image 518 (NO in step S676), the control unit 21 also displays a mark at the corresponding position on the RT-format catheter image 518 (step S678).

[0097] The control unit 21 creates a new record in the medical instrument position training data DB 71. The control unit 21 associates the catheter image 51 with the position information input by the labeler and records them in the medical instrument position training data DB 71 (step S679).

[0098] The catheter image 51 recorded in step S679 may be only the RT format catheter image 518 acquired in step S671, or may be both the RT format catheter image 518 and the XY format catheter image 519 generated in step S672. The catheter image 51 recorded in step S679 may be sound ray data for one rotation received by the sensor 42, or scan line data generated by signal processing the sound ray data.

[0099] The position information recorded in step S679 is, for example, information indicating the position of one pixel on the RT-format catheter image 518, corresponding to the position where the labeler performed a click operation or the like using the input device 32. The position information may also be information indicating the position where the labeler performed a click operation or the like and the surrounding area.

[0100] The control unit 21 determines whether or not to end the processing (step S680). For example, when the processing of the catheter image 51 recorded in the auxiliary storage device 23 is completed, the control unit 21 determines to end the processing. When it is determined to end the processing (YES in step S680), the control unit 21 ends the processing.

[0101] If it is determined not to end the processing (NO in step S680), the control unit 21 returns to step S671. In step S671, the control unit 21 acquires the next RT format catheter image 518 and executes the processing of step S672 and thereafter. That is, the control unit 21 automatically acquires and displays the next RT format catheter image 518 without waiting for an operation on a button displayed in the control button area 782.

[0102] By the loop from step S671 to step S680, the control unit 21 records training data based on a large number of RT format catheter images 518 recorded in the auxiliary storage device 23 in the medical instrument position training data DB 71.

[0103] Note that control unit 21 may display, for example, a "Save button" on the screen described using Fig. 10, and execute step S679 when selection of the "Save button" is accepted. Furthermore, control unit 21 may display, for example, an "AUTO button" on the screen described using Fig. 10, and automatically execute step S679 while accepting selection of the "AUTO button" without waiting for selection of the "Save button."

[0104] In the following explanation, we will use as an example a case where the catheter image 51 recorded in the medical instrument position training data DB71 in step S679 is an RT format catheter image 518, and the position information is the position of a single pixel on the RT format catheter image 518.

[0105] FIG. 12 is a flowchart illustrating the processing flow of a program for generating a medical equipment trained model 611. Prior to execution of the program in FIG. 12, an untrained model is prepared, for example, by combining a convolutional layer, a pooling layer, and a fully connected layer. As described above, the untrained model is, for example, a CNN model. Examples of CNNs that can be used to generate the medical equipment trained model 611 include R-CNN, YOLO, U-Net, and GAN. The medical equipment trained model 611 may also be generated using a neural network structure other than a CNN.

[0106] The control unit 21 acquires a training record to be used for training one epoch from the medical instrument position training data DB 71 (step S571). As described above, the training record recorded in the medical instrument position training data DB 71 is a combination of the RT-format catheter image 518 and coordinates indicating the position of the medical instrument depicted in the RT-format catheter image 518.

[0107] The control unit 21 adjusts the parameters of the model so that when the RT format catheter image 518 is input to the input layer of the model, the position of the pixel corresponding to the position information is output from the output layer (step S572). In obtaining the training record and adjusting the model parameters, the program may have a function of causing the control unit 21 to accept corrections by the user, present the basis for judgment, perform additional learning, etc. as appropriate.

[0108] The control unit 21 determines whether to end the process (step S573). For example, the control unit 21 determines to end the process when learning for a predetermined number of epochs has been completed. The control unit 21 may acquire test data from the medical instrument position training data DB 71, input the test data to the model under machine learning, and determine to end the process when an output of a predetermined accuracy is obtained.

[0109] If it is determined not to end the processing (NO in step S573), the control unit 21 returns to step S571. If it is determined to end the processing (YES in step S573), the control unit 21 records the parameters of the trained medical instrument position training data DB 71 in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 ends the processing. Through the above processing, a medical instrument trained model 611 that accepts the catheter image 51 and outputs the first position information is generated.

[0110] 12, a model that accepts time-series input, such as an RNN, may be prepared. The RNN is, for example, an LSTM. In step S572, the control unit 21 adjusts the parameters of the model so that, when a plurality of RT-format catheter images 518 captured in time series are input to the input layer of the model, the output layer outputs the position of a pixel corresponding to the position information associated with the last RT-format catheter image 518 in time series.

[0111] Fig. 13 is a flowchart illustrating the processing flow of a program for adding data to the medical instrument position training data DB 71. The program in Fig. 13 is a program for adding training data to the medical instrument position training data DB 71 after creating a medical instrument trained model 611. The added training data is used for additional training of the medical instrument trained model 611.

[0112] 13, a large number of catheter images 51 that have not yet been used to create the medical instrument position training data DB 71 are recorded in the auxiliary storage device 23 or an external mass storage device. The following explanation will be given taking as an example a case where the catheter images 51 are recorded in the auxiliary storage device 23 in the form of video data including a plurality of RT format catheter images 518 captured in time series.

[0113] The control unit 21 acquires one frame of the RT format catheter image 518 from the auxiliary storage device 23 (step S701). The control unit 21 inputs the RT format catheter image 518 into the medical instrument trained model 611 to acquire first position information (step S702).

[0114] The control unit 21 converts the RT-format catheter image 518 to generate an XY-format catheter image 519 (step S703). The control unit 21 displays the screen described with reference to Fig. 10 on the display device 31 with a mark indicating the first position information acquired in step S702 superimposed on each of the RT-format catheter image 518 and the XY-format catheter image 519 (step S704).

[0115] If the labeler determines that the position of the automatically displayed mark is inappropriate, he or she inputs the correct position of the medical instrument by performing a single click operation, i.e., inputs a correction instruction for the automatically displayed mark.

[0116] The control unit 21 determines whether an input operation by the labeler via the input device 32 has been received within a predetermined time (step S705). The predetermined time can desirably be set by the labeler as appropriate. The input operation is specifically a click operation or a tap operation on the RT format catheter image 518 or the XY format catheter image 519.

[0117] If it is determined that an input operation has been accepted (YES in step S705), the control unit 21 displays a mark such as a small circle at the position where the input operation has been accepted (step S706). It is desirable that the mark displayed in step S706 has a different color or shape from the mark indicating the position information acquired in step S702. Note that the control unit 21 may erase the mark indicating the position information acquired in step S702.

[0118] The control unit 21 determines whether the image for which the input operation was accepted in step S705 is the RT-format catheter image 518 (step S707). If it is determined that the image is the RT-format catheter image 518 (YES in step S707), the control unit 21 also displays a mark at the corresponding position on the XY-format catheter image 519 (step S708). If it is determined that the image is not the RT-format catheter image 518 (NO in step S707), the control unit 21 also displays a mark at the corresponding position on the RT-format catheter image 518 (step S709).

[0119] The control unit 21 creates a new record in the medical instrument position training data DB 71. The control unit 21 records the correction data that associates the catheter image 51 with the position information input by the labeler in the medical instrument position training data DB 71 (step S710).

[0120] If it is determined that the input operation has not been accepted (NO in step S705), the control unit 21 creates a new record in the medical instrument position training data DB 71. The control unit 21 records uncorrected data that associates the catheter image 51 with the first position information acquired in step S532 in the medical instrument position training data DB 71 (step S711).

[0121] After step S710 or step S711 is completed, control unit 21 determines whether to terminate the process (step S712). For example, when the processing of catheter image 51 recorded in auxiliary storage device 23 is completed, control unit 21 determines to terminate the process. When it is determined to terminate (YES in step S712), control unit 21 terminates the process.

[0122] If it is determined not to end the processing (NO in step S712), the control unit 21 returns to step S701. In step S701, the control unit 21 acquires the next RT format catheter image 518 and executes the processing in step S702 and thereafter. By the loop from step S701 to step S712, the control unit 21 adds training data based on the many RT format catheter images 518 recorded in the auxiliary storage device 23 to the medical instrument position training data DB 71.

[0123] 10, the control unit 21 may display, for example, an "OK button" for approving the output by the medical instrument trained model 611. When the control unit 21 receives the selection of the "OK button," the control unit 21 determines that the instruction "NO" has been received in step S705, and executes step S711.

[0124] According to this embodiment, the labeler can mark one medical instrument depicted in the catheter image 51 with only one operation, such as one click or one tap. The control unit 21 may also accept an operation to mark one medical instrument by a so-called double click or double tap. Compared to marking the boundary line of a medical instrument, this significantly reduces the labor required for marking, thereby reducing the burden on the labeler. According to this embodiment, a large amount of training data can be created in a short amount of time.

[0125] According to this embodiment, when a plurality of medical instruments are depicted on the catheter image 51, the labeler can mark each of the medical instruments with a single click operation or the like.

[0126] Note that control unit 21 may display, for example, an "OK button" on the screen described with reference to FIG. 10, and execute step S679 when it receives selection of the "OK button."

[0127] According to this embodiment, by superimposing the position information acquired using the medical instrument trained model 611 onto the catheter image 51, additional training data can be quickly created while reducing the burden on the labeler.

[0128] [Variation 2-1] The medical instrument position training data DB 71 may have a field for recording the type of medical instrument. In this case, the control unit 21 accepts input of the type of medical instrument, such as "Brockenbrough needle," "guide wire," or "balloon catheter," on the screen described with reference to FIG.

[0129] By performing machine learning using the medical instrument position training data DB71 created in this way, a medical instrument trained model 611 is generated that outputs the type of medical instrument in addition to the position of the medical instrument.

[0130] [Embodiment 3] This embodiment relates to a catheter system 10 that uses two trained models to acquire second position information regarding the position of a medical instrument from a catheter image 51. Explanation of parts common to the second embodiment will be omitted.

[0131] 14 is an explanatory diagram for explaining the depiction of medical instruments, in which the medical instruments depicted in the RT format catheter image 518 and the XY format catheter image 519 are highlighted.

[0132] Generally, medical instruments reflect ultrasound more strongly than biological tissue. Ultrasound emitted from the sensor 42 has difficulty reaching a distance farther than the medical instrument. Therefore, the medical instrument is depicted by a hyperechoic region indicating the side closer to the imaging catheter 40 and a hypoechoic region following behind it. The hypoechoic region following behind the medical instrument is referred to as an acoustic shadow. In Figure 14, the acoustic shadow is indicated by vertical hatching.

[0133] In the RT format catheter image 518, the acoustic shadow is depicted as a horizontal line. In the XY format catheter image 519, the acoustic shadow is depicted as a fan shape. In either case, a high-brightness region is depicted in a region closer to the image acquisition catheter 40 than the acoustic shadow. Note that the high-brightness region may be depicted in the form of a so-called multiple echo, which is regularly repeated along the scan line direction.

[0134] Based on the scan angle direction of the RT format catheter image 518, ie, the lateral direction in FIG. 14, the scan angle at which the medical device is visualized can be determined.

[0135] 15 is an explanatory diagram illustrating the configuration of the angle-trained model 612. The angle-trained model 612 is a model that receives the catheter image 51 and outputs scan angle information related to the scan angle at which the medical instrument is depicted.

[0136] 15 schematically shows an angle trained model 612 that receives an RT-format catheter image 518 and outputs scan angle information indicating the probability that a medical instrument is depicted at each scan angle, i.e., in the vertical direction of the RT-format catheter image 518. Note that because a medical instrument is depicted across multiple scan angles, the total probability of outputting scan angle information exceeds 100%. The angle trained model 612 may extract and output angles at which the medical instrument is highly likely to be depicted.

[0137] The angle trained model 612 is generated by machine learning. By extracting the scanning angle from the position information field of the medical instrument position training data DB 71 described with reference to Fig. 9, the angle trained model 612 can be generated using the training data.

[0138] An overview of the process for generating the angle-trained model 612 will be described using the flowchart in Figure 12. Prior to execution of the program in Figure 12, an untrained model such as a CNN that combines a convolutional layer, a pooling layer, and a fully connected layer is prepared. The program in Figure 12 adjusts each parameter of the prepared model and performs machine learning.

[0139] The control unit 21 acquires a training record to be used for training one epoch from the medical instrument position training data DB 71 (step S571). As described above, the training record recorded in the medical instrument position training data DB 71 is a combination of the RT-format catheter image 518 and coordinates indicating the position of the medical instrument depicted in the RT-format catheter image 518.

[0140] The control unit 21 adjusts the parameters of the model so that, when the RT format catheter image 518 is input to the input layer of the model, a scanning angle corresponding to the position information is output from the output layer (step S572). In acquiring the training records and adjusting the model parameters, the program may have a function of causing the control unit 21 to execute operations such as accepting corrections by the user, presenting the basis for the judgment, and performing additional learning.

[0141] The control unit 21 determines whether to end the process (step S573). For example, the control unit 21 determines to end the process when learning for a predetermined number of epochs has been completed. The control unit 21 may acquire test data from the medical instrument position training data DB 71, input the test data to the model under machine learning, and determine to end the process when an output of a predetermined accuracy is obtained.

[0142] If it is determined not to end the processing (NO in step S573), the control unit 21 returns to step S571. If it is determined to end the processing (YES in step S573), the control unit 21 records the learned parameters of the medical instrument position training data DB 71 in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 ends the processing. Through the above processing, an angle learned model 612 is generated that receives the catheter image 51 and outputs information related to the scanning angle.

[0143] 12, a model that accepts time-series input, such as an RNN, may be prepared. The RNN is, for example, an LSTM. In step S572, the control unit 21 adjusts the parameters of the model so that, when a plurality of RT-format catheter images 518 captured in time series are input to the input layer of the model, information on the scanning angle associated with the last RT-format catheter image 518 in time series is output from the output layer.

[0144] Instead of using the angle learned model 612, the control unit 21 may determine the scan angle at which the medical instrument is imaged by pattern matching.

[0145] 16 is an explanatory diagram illustrating the position information model 619. The position information model 619 is a model that receives the RT-format catheter image 518 and outputs second position information indicating the position of the depicted medical instrument. The position information model 619 includes a medical instrument trained model 611, an angle trained model 612, and a position information synthesis unit 615.

[0146] The same RT-format catheter image 518 is input to both the medical instrument trained model 611 and the angle trained model 612. First position information is output from the medical instrument trained model 611. As described using FIG. 6, the first position information is the probability that a medical instrument is depicted at each site on the RT-format catheter image 518. In the following description, the probability that a medical instrument is depicted at a position where the distance from the center of the image acquisition catheter 40 is r and the scanning angle is θ is represented as P1(r, θ).

[0147] Scanning angle information is output from the angle trained model 612. The scanning angle information is the probability that the medical instrument is visualized at each scanning angle. In the following description, the probability that the medical instrument is visualized in the direction of the scanning angle θ is represented as Pt(θ).

[0148] The first position information and the scanning angle information are combined in a position information combining unit 615 to generate second position information. Like the first position information, the second position information is the probability that a medical instrument is depicted at each site on the RT format catheter image 518. An input terminal of the position information combining unit 615 performs the functions of a first position information acquiring unit and a scanning angle information acquiring unit.

[0149] Since the medical instrument is depicted with a certain degree of spread in the RT format catheter image 518, the sum of P1 and the sum of Pt may both be greater than 1. The second position information P2(r, θ) at a position where the distance from the center of the image acquisition catheter 40 is r and the scanning angle is θ is calculated, for example, by equation (1-1). P2(r,θ)=P1(r,θ)+kPt(θ) ‥‥‥ (1-1) k is a coefficient relating to the weighting between the first position information and the scanning angle information.

[0150] The second position information P2(r, θ) may be calculated by the formula (1-2). P2(r,θ)=P1(r,θ)×Pt(θ) ‥‥‥ (1-2)

[0151] The second position information P2(r, θ) may be calculated by equation (1-3): Equation (1-3) is an equation for calculating the average value of the first position information and the scanning angle information. P2(r,θ)=(P1(r,θ)+Pt(θ)) / 2 ‥‥‥ (1-3)

[0152] Note that the second position information P2(r, θ) in equations (1-1) to (1-3) is not a probability but a numerical value that relatively indicates the likelihood that a medical instrument is visualized. Combining the first position information and the scanning angle information improves the accuracy of the scanning angle direction. Note that the second position information may be information regarding the position where the value of P2(r, θ) is the largest. The second position information may be determined by a function other than the equations exemplified in equations (1-1) to (1-3).

[0153] The second position information is an example of the position information of the medical instrument acquired in step S502 described using Fig. 4. The medical instrument trained model 611, the angle trained model 612, and the position information synthesis unit 615 work together to realize step S502 described using Fig. 4. The output terminal of the position information synthesis unit 615 functions as a second position information output unit that outputs second position information based on the first position information and scanning angle information.

[0154] 17 is a flowchart illustrating the flow of processing of the program according to embodiment 3. The flowchart explained using FIG. 17 shows details of the processing of step S502 explained using FIG.

[0155] The control unit 21 acquires one frame of the RT-format catheter image 518 (step S541). The control unit 21 inputs the RT-format catheter image 518 to the medical instrument trained model 611 to acquire first position information (step S542). The control unit 21 inputs the RT-format catheter image 518 to the angle trained model 612 to acquire scanning angle information (step S543).

[0156] The control unit 21 calculates the second position information based on, for example, equation (1-1) or equation (1-2) (step S544). Thereafter, the control unit 21 ends the process. Thereafter, the control unit 21 uses the second position information calculated in step S544 as the position information in step S502.

[0157] According to this embodiment, a catheter system 10 that accurately calculates the position information of a medical instrument depicted in a catheter image 51 can be provided.

[0158] [Embodiment 4] This embodiment relates to a specific example of the classification model 62 described using Fig. 7. Fig. 18 is an explanatory diagram illustrating the configuration of the classification model 62. The classification model 62 includes a first classification trained model 621 and a classification data conversion unit 629.

[0159] The first classification trained model 621 accepts the RT-format catheter image 518 and outputs first classified data 521 in which each part constituting the RT-format catheter image 518 is classified into a "biological tissue region," a "non-biological tissue region," and a "medical instrument region." The first classification trained model 621 further outputs the reliability of the classification result for each part, that is, the probability that the classification result is correct. The output layer of the first classification trained model 621 functions as a first classified data output unit that outputs the first classified data 521.

[0160] The diagram in the upper right of Fig. 18 shows a schematic representation of the first classified data 521 in RT format. The thick, downward-sloping hatching indicates biological tissue regions such as the atrial wall and ventricular wall. The black areas indicate medical device regions in which medical devices such as the Brockenbrough needle are depicted. The checkered hatching indicates non-biological tissue regions that are neither medical device regions nor biological tissue regions.

[0161] The first classified data 521 is converted into classified data 52 by the classified data conversion unit 629. The diagram at the bottom right of Figure 18 schematically shows RT-format classified data 528. Non-biological tissue regions are classified into three types: first lumen regions, second lumen regions, and non-lumen regions. As in Figure 5C, thin hatching sloping downward to the left indicates first lumen regions. Thin hatching sloping downward to the right indicates second lumen regions. Thick hatching sloping downward to the left indicates non-lumen regions.

[0162] An overview of the processing performed by the classification data conversion unit 629 will be explained. Of the non-biological tissue regions, the region in contact with the image acquisition catheter 40, i.e., the rightmost region in the first classification data 521, is classified as the first lumen region. Of the non-biological tissue regions, the region surrounded by biological tissue regions is classified as the second lumen region. Note that the classification of the second lumen region is desirably determined after connecting the upper and lower ends of the RT format catheter image 518 to form a cylindrical shape. Of the non-biological tissue regions, regions that are neither the first lumen region nor the second lumen region are classified as non-lumen regions.

[0163] 19 is an explanatory diagram illustrating the first training data. The first training data is used when generating the first classification trained model 621 by machine learning. In the following explanation, an example will be described in which the first training data is created using the information processing device 20 described using FIG. 3. The first training data may be created using a computer or the like other than the information processing device 20.

[0164] The control unit 21 displays two types of catheter images 51, an RT format catheter image 518 and an XY format catheter image 519, on the display device 31. The labeler observes the displayed catheter image 51 and marks four types of boundary data: "boundary line between the first lumen region and the biological tissue region," "boundary line between the second lumen region and the biological tissue region," "boundary line between the non-lumen region and the biological tissue region," and "outline of the medical instrument region."

[0165] The labeler may mark either the RT format catheter image 518 or the XY format catheter image 519. The control unit 21 displays a boundary line corresponding to the marking at a corresponding position on the other catheter image 51. As described above, the labeler can check both the RT format catheter image 518 and the XY format catheter image 519 and make appropriate markings.

[0166] The labeler inputs whether each region separated by the four types of boundary data that have been marked is a "living tissue region," a "non-living tissue region," or a "medical instrument region." Note that the control unit 21 may automatically determine the region, and the labeler may instruct corrections as necessary. Through the above processing, first classification data 521 is created, which clearly indicates whether each region of the catheter image 51 is classified as a "living tissue region," a "non-living tissue region," or a "medical instrument region."

[0167] The first classification data 521 will be described using a specific example. A "biological tissue region label" is recorded for pixels classified into a "biological tissue region," a "first lumen region label" for pixels classified into a "first lumen region," a "second lumen region label" for pixels classified into a "second lumen region," a "non-lumen region label" for pixels classified into a "non-lumen region," a "medical instrument region label" for pixels classified into a "medical instrument region," and a "non-biological tissue region label" for pixels classified into a "non-biological tissue region." Each label is represented, for example, by an integer. The first classification data 521 is an example of label data that associates pixel positions with labels.

[0168] Control unit 21 associates and records catheter image 51 and first classified data 521. By repeating the above process and recording many sets of data, a first training data DB is created. In the following explanation, an example of a first training data DB in which RT format catheter image 518 and RT format first classified data 521 are recorded in association with each other will be explained.

[0169] The control unit 21 may generate the XY-format classification data 529 based on the XY-format catheter image 519. The control unit 21 may generate the RT-format classification data 528 based on the XY-format classification data 529.

[0170] An overview of the process of generating the first classification trained model 621 will be described using the flowchart in Figure 12. Prior to execution of the program in Figure 12, an untrained model such as a U-Net structure that realizes semantic segmentation is prepared.

[0171] The U-Net structure includes multiple encoder layers followed by multiple decoder layers. Each encoder layer includes a pooling layer and a convolutional layer. Semantic segmentation assigns a label to each pixel that makes up the input image. The untrained model may be the Mask R-CNN model or any other model that achieves image segmentation.

[0172] The control unit 21 acquires training records to be used for training one epoch from the first training data DB (step S571). The control unit 21 adjusts the parameters of the model so that when an RT format catheter image 518 is input to the input layer of the model, first classified data 521 in RT format is output from the output layer (step S572). In acquiring the training records and adjusting the model parameters, the program may have a function of causing the control unit 21 to accept corrections from the user, present the basis for judgment, perform additional learning, etc., as appropriate.

[0173] The control unit 21 determines whether to end the process (step S573). For example, the control unit 21 determines to end the process when learning for a predetermined number of epochs has been completed. The control unit 21 may acquire test data from the first training data DB, input it to the model under machine learning, and determine to end the process when an output with a predetermined accuracy is obtained.

[0174] If it is determined not to end the processing (NO in step S573), control unit 21 returns to step S571. If it is determined to end the processing (YES in step S573), control unit 21 records the parameters of trained first classification trained model 621 in auxiliary storage device 23 (step S574). Thereafter, control unit 21 ends the processing. Through the above processing, first classification trained model 621 that accepts catheter image 51 and outputs first classification data 521 is generated.

[0175] A model that accepts time-series input may be prepared prior to execution of the program of Fig. 12. The model that accepts time-series input may include, for example, a memory unit that stores information about previously input RT-format catheter images 518. The model that accepts time-series input may include a recursive input unit that inputs the output for the previously input RT-format catheter image 518 together with the next RT-format catheter image 518.

[0176] By using catheter images 51 acquired in time series, it is possible to realize a first classification trained model 621 that is less susceptible to the influence of image noise and the like and outputs first classification data 521 with high accuracy.

[0177] The first classification trained model 621 may be created using a computer or the like separate from the information processing device 20. The first classification trained model 621 for which machine learning has been completed may be copied to the auxiliary storage device 23 via a network. The first classification trained model 621 trained on one piece of hardware can be used by multiple information processing devices 20.

[0178] 20 is a flowchart illustrating the flow of processing of the program according to embodiment 4. The flowchart explained using FIG. 20 shows details of the processing performed by the classification model 62 explained using FIG.

[0179] The control unit 21 acquires one frame of the RT-format catheter image 518 (step S551). The control unit 21 inputs the RT-format catheter image 518 to the first classification trained model 621 to acquire the first classified data 521 (step S552). The control unit 21 extracts one continuous non-biological tissue region from the first classified data 521 (step S553). Note that the processing after the extraction of the non-biological tissue region is preferably performed with the upper and lower ends of the RT-format catheter image 518 connected to form a cylindrical shape.

[0180] The control unit 21 determines whether the non-biological tissue region extracted in step S552 is on the side in contact with the image acquisition catheter 40, i.e., the portion in contact with the left end of the RT-format catheter image 518 (step S554). If it is determined that the non-biological tissue region is on the side in contact with the image acquisition catheter 40 (YES in step S554), the control unit 21 determines that the non-biological tissue region extracted in step S553 is the first lumen region (step S555).

[0181] If it is determined that the non-living tissue region is not in contact with the imaging catheter 40 (NO in step S554), the control unit 21 determines whether the non-living tissue region extracted in step S552 is surrounded by a living tissue region (step S556). If it is determined that the non-living tissue region is surrounded by a living tissue region (YES in step S556), the control unit 21 determines that the non-living tissue region extracted in step S553 is a second lumen region (step S557). Through steps S555 and S557, the control unit 21 realizes the function of a lumen region extraction unit.

[0182] If it is determined that the non-biological tissue region is not surrounded by a biological tissue region (NO in step S556), the control unit 21 determines that the non-biological tissue region extracted in step S553 is a non-lumen region (step S558).

[0183] After step S555, step S557, or step S558 is completed, control unit 21 determines whether or not processing of all non-biological tissue regions is completed (step S559). If it is determined that processing is not completed (NO in step S559), control unit 21 returns to step S553. If it is determined that processing is completed (YES in step S559), control unit 21 ends processing.

[0184] The control unit 21 realizes the function of the classification data conversion unit 629 by the processes from step S553 to step S559.

[0185] The first classification trained model 621 may be a model that classifies the XY-format catheter image 519 into a biological tissue region, a non-biological tissue region, and a medical instrument region. The first classification trained model 621 may be a model that classifies the RT-format catheter image 518 into a biological tissue region and a non-biological tissue region. In such a case, the labeler does not need to mark the medical instrument region.

[0186] According to this embodiment, it is possible to generate a first classification trained model 621 that classifies a catheter image 51 into a biological tissue region, a non-biological tissue region, and a medical instrument region. According to this embodiment, it is possible to provide a catheter system 10 that generates classification data 52 using the generated first classification trained model 621.

[0187] [Variation 4-1] The labeler may input whether each region separated by the four types of marked boundary data is a "biological tissue region," a "first lumen region," a "second lumen region," a "non-lumen region," or a "medical instrument region." By performing machine learning using the first training data DB created in this way, it is possible to generate a first classification trained model 621 that classifies catheter image 51 into a "biological tissue region," a "first lumen region," a "second lumen region," a "non-lumen region," and a "medical instrument region."

[0188] As a result of the above, a classification model 62 can be realized that classifies a catheter image 51 into a "biological tissue region," "first lumen region," "second lumen region," "non-lumen region," and "medical instrument region" without using a classification data conversion unit 629.

[0189] [Embodiment 5] This embodiment relates to a catheter system 10 that uses a composite classification model 626 that combines classification data 52 output from two classification trained models. Explanation of parts common to the fourth embodiment will be omitted.

[0190] 21 is an explanatory diagram illustrating the configuration of classification model 62 according to embodiment 5. Classification model 62 includes a combined classification model 626 and a classification data conversion unit 629. Combined classification model 626 includes a first classification trained model 621, a second classification trained model 622, and a classification data synthesis unit 628. First classification trained model 621 is the same as in embodiment 4, and therefore description thereof will be omitted.

[0191] The second classification trained model 622 is a model that accepts the RT-format catheter image 518 and outputs second classified data 522 in which each part constituting the RT-format catheter image 518 is classified into a "biological tissue region," a "non-biological tissue region," and a "medical instrument region." The second classification trained model 622 further outputs the reliability of the classification result for each part, that is, the probability that the classification result is correct. Details of the second classification trained model 622 will be described later.

[0192] The classified data synthesis unit 628 synthesizes the first classified data 521 and the second classified data 522 to generate synthesized classified data 526. That is, the input terminal of the classified data synthesis unit 628 realizes the functions of the first classified data acquisition unit and the second classified data acquisition unit, and the output terminal of the classified data synthesis unit 628 realizes the function of the synthesized classified data output unit.

[0193] The details of combined classification data 526 will be described later. Combined classification data 526 is converted into classification data 52 by classification data conversion unit 629. The processing performed by classification data conversion unit 629 is the same as in the fourth embodiment, and therefore description thereof will be omitted.

[0194] 22 is an explanatory diagram illustrating the second training data. The second training data is used when generating the second classification trained model 622 by machine learning. In the following explanation, an example will be described in which the second training data is created using the information processing device 20 described using FIG. 3. The second training data may be created using a computer or the like other than the information processing device 20.

[0195] The control unit 21 displays two types of catheter images 51, an RT format catheter image 518 and an XY format catheter image 519, on the display device 31. The labeler observes the displayed catheter image 51 and marks two types of boundary data, namely, the "boundary line between the first lumen region and the biological tissue region" and the "outline of the medical instrument region."

[0196] The labeler may mark either the RT format catheter image 518 or the XY format catheter image 519. The control unit 21 displays a boundary line corresponding to the marking at a corresponding position on the other catheter image 51. As described above, the labeler can check both the RT format catheter image 518 and the XY format catheter image 519 and make appropriate markings.

[0197] The labeler inputs whether each region separated by the two types of boundary data that have been marked is a "living tissue region," a "non-living tissue region," or a "medical instrument region." Alternatively, the control unit 21 may automatically determine the region, and the labeler may issue correction instructions as necessary. Through the above processing, second classification data 522 is created, which clearly indicates whether each part of the catheter image 51 is classified as a "living tissue region," a "non-living tissue region," or a "medical instrument region."

[0198] The second classification data 522 will be described below using a specific example. A "biological tissue region label" is recorded for pixels classified as a "biological tissue region," a "non-biological tissue region label" is recorded for pixels classified as a "non-biological tissue region," and a "medical instrument region label" is recorded for pixels classified as a "medical instrument region." Each label is represented by, for example, an integer. The second classification data 522 is an example of label data that associates pixel positions with labels.

[0199] Control unit 21 associates catheter image 51 with second classification data 522 and records them. By repeating the above process and recording many sets of data, a second training data DB is created. By performing a process similar to the machine learning described in the fourth embodiment using the second training data DB, second classification trained model 622 can be generated.

[0200] The second classification trained model 622 may be a model that classifies the XY-format catheter image 519 into a biological tissue region, a non-biological tissue region, and a medical instrument region. The second classification trained model 622 may be a model that classifies the RT-format catheter image 518 into a biological tissue region and a non-biological tissue region. In such a case, the labeler does not need to mark the medical instrument region.

[0201] The second classified data 522 can be created in a shorter time than the first classified data 521. The labeler who creates the second classified data 522 can be trained in a shorter time than the labeler who creates the first classified data 521. As a result, a larger amount of training data can be registered in the second training data DB than in the first training DB.

[0202] Because a large amount of training data can be used, it is possible to generate a second classification trained model 622 that can distinguish the boundary between the first lumen region and the biological tissue region and the outline of the medical instrument region with higher accuracy than the first classification trained model 621. However, since the second classification trained model 622 has not learned about non-biological tissue regions other than the first lumen region, it cannot distinguish them from biological tissue regions.

[0203] The processing performed by the classified data synthesis unit 628 will be described. The same RT-format catheter image 518 is input to both the first classification trained model 621 and the second classification trained model 622. The first classified data 521 is output from the medical instrument trained model 611. The second classified data 522 is output from the second classification trained model 622.

[0204] In the following explanation, an example will be given in which both the first classification trained model 621 and the second classification trained model 622 output a classified label and the reliability of the label for each pixel of the RT-format catheter image 518. Note that the first classification trained model 621 and the second classification trained model 622 may output a classified label and probability for each range of, for example, 3 pixels vertically and 3 pixels horizontally of the RT-format catheter image 518, a total of 9 pixels.

[0205] For a pixel that is at a distance r from the center of the image acquisition catheter 40 and at a scanning angle θ, the reliability that the first classification trained model 621 has that the pixel is a biological tissue region is indicated by Q1t(r,θ). Note that for pixels that the first classification trained model 621 has classified as a region other than a biological tissue region, Q1t(r,θ) = 0.

[0206] Similarly, for a pixel that is at a distance r from the center of the image acquisition catheter 40 and at a scanning angle θ, the reliability that the second classification trained model 622 has of the pixel being a biological tissue region is denoted by Q2t(r,θ). Note that for pixels that the second classification trained model 622 has classified as a region other than a biological tissue region, Q2t(r,θ) = 0.

[0207] The classification data synthesis unit 628 calculates the synthesis value Qt(r, θ) based on, for example, equation (5-1). Note that Qt(r, θ) is not the probability that the classification as a biological tissue region is correct, but is a numerical value that relatively indicates the degree of reliability that the region is a biological tissue region. Qt(r,θ)=Q1t(r,θ)×Q2t(r,θ) ‥‥‥(5-1) The classification data synthesis unit 628 classifies pixels with Qt(r, θ) of 0.5 or more into the biological tissue region.

[0208] Similarly, the reliability that the first classification trained model 621 is in the medical equipment domain is indicated by Q1c(r,θ), and the reliability that the second classification trained model 622 is in the medical equipment domain is indicated by Q2c(r,θ).

[0209] The classification data synthesis unit 628 calculates the synthesis value Qc(r, θ) based on, for example, equation (5-2). Note that Qc(r, θ) is not the probability that the classification as being in the medical equipment domain is correct, but is a numerical value that relatively indicates the reliability of the classification as being in the medical equipment domain. Qc(r,θ)=Q1c(r,θ)×Q2c(r,θ) ‥‥‥(5-2)

[0210] The classification data synthesis unit 628 classifies pixels with Qc(r, θ) of 0.5 or more into the medical instrument region. The classification data synthesis unit 628 classifies pixels that are not classified into either the medical instrument region or the biological tissue region into the non-biological tissue region. As a result, the classification data synthesis unit 628 generates synthesized classification data 526 by synthesizing the first classification data 521 and the second classification data 522. The synthesized classification data 526 is converted into RT-format classification data 528 by the classification data conversion unit 629.

[0211] Note that equations (5-1) and (5-2) are merely examples. The thresholds used by the classified data synthesis unit 628 to perform classification are also merely examples. The classified data synthesis unit 628 may be a trained model that receives the first classified data 521 and the second classified data 522 and outputs the synthesized classified data 526.

[0212] The first classification data 521 may be input to the classification data synthesis unit 628 after being classified into "biological tissue region," "first lumen region," "second lumen region," "non-lumen region," and "medical instrument region" by the classification data conversion unit 629 described in embodiment 4.

[0213] The first classification trained model 621 may be a model that classifies the catheter image 51 described in variant example 4-1 into a "biological tissue region," a "first lumen region," a "second lumen region," a "non-lumen region," and a "medical instrument region."

[0214] When classified data on non-living tissue regions already classified into "first lumen region," "second lumen region," and "non-lmen region" is input to the classified data synthesis unit 628, the classified data synthesis unit 628 can output synthesized classified data 526 already classified into "living tissue region," "first lumen region," "second lumen region," "non-lmen region," and "medical device region." In this case, there is no need to input the synthesized classified data 526 to the classified data conversion unit 629 to convert it into RT-format classified data 528.

[0215] 23 is a flowchart illustrating the flow of processing of the program according to embodiment 5. The flowchart explained using FIG. 23 shows details of the processing performed by the classification model 62 explained using FIG.

[0216] The control unit 21 acquires one frame of the RT format catheter image 518 (step S581). By step S581, the control unit 21 realizes the function of an image acquisition unit. The control unit 21 inputs the RT format catheter image 518 to the first classification trained model 621 to acquire the first classified data 521 (step S582). The control unit 21 inputs the RT format catheter image 518 to the second classification trained model 622 to acquire the second classified data 522 (step S583).

[0217] The control unit 21 starts a classification and synthesis subroutine (step S584). The classification and synthesis subroutine is a subroutine that combines the first classification data 521 and the second classification data 522 to generate combined classification data 526. The processing flow of the classification and synthesis subroutine will be described later.

[0218] Control unit 21 extracts one continuous non-biological tissue region from composite classification data 526 (step S585). Note that the processing after the extraction of the non-biological tissue region is preferably performed in a state where the upper and lower ends of RT format catheter image 518 are connected to form a cylindrical shape.

[0219] The control unit 21 determines whether or not the non-biological tissue region extracted in step S585 is on the side that contacts the imaging catheter 40 (step S554). The subsequent processing up to step S559 is the same as the processing flow of the program of the fourth embodiment described using Fig. 20, and therefore description thereof will be omitted.

[0220] The control unit 21 determines whether or not the processing of all non-biological tissue regions has been completed (step S559). If it is determined that the processing has not been completed (NO in step S559), the control unit 21 returns to step S585. If it is determined that the processing has been completed (YES in step S559), the control unit 21 ends the processing.

[0221] 24 is a flowchart illustrating the processing flow of the classification and synthesis subroutine. The classification and synthesis subroutine is a subroutine that combines first classified data 521 and second classified data 522 to generate combined classified data 526.

[0222] The control unit 21 selects a pixel to be processed (step S601). The control unit 21 acquires a reliability Q1t(r, θ) that the pixel being processed is a biological tissue region from the first classification data 521 (step S602). The control unit 21 acquires a reliability Q2t(r, θ) that the pixel being processed is a biological tissue region from the second classification data 522 (step S603).

[0223] The control unit 21 calculates the composite value Qt(r, θ) based on, for example, equation (5-1) (step S604). The control unit 21 determines whether the composite value Qt(r, θ) is equal to or greater than a predetermined threshold (step S605). The predetermined threshold is, for example, 0.5.

[0224] If it is determined that the difference is equal to or greater than the predetermined threshold (YES in step S605), the control unit 21 classifies the pixel being processed as a "biological tissue region" (step S606). If it is determined that the difference is less than the predetermined threshold (NO in step S605), the control unit 21 acquires a reliability Q1c(r, θ) that the pixel being processed is a medical instrument region from the first classification data 521 (step S611). The control unit 21 acquires a reliability Q2c(r, θ) that the pixel being processed is a medical instrument region from the second classification data 522 (step S612).

[0225] The control unit 21 calculates the composite value Qc(r, θ) based on, for example, equation (5-2) (step S613). The control unit 21 determines whether the composite value Qc(r, θ) is equal to or greater than a predetermined threshold (step S614). The predetermined threshold is, for example, 0.5.

[0226] If it is determined that the difference is equal to or greater than the predetermined threshold (YES in step S614), the control unit 21 classifies the pixel being processed into a "medical device region" (step S615). If it is determined that the difference is less than the predetermined threshold (NO in step S614), the control unit 21 classifies the pixel being processed into a "non-biological tissue region" (step S616).

[0227] After step S606, step S615, or step S616 is completed, the control unit 21 determines whether or not processing of all pixels is completed (step S607). If it is determined that processing is not completed (NO in step S607), the control unit 21 returns to step S601. If it is determined that processing is completed (YES in step S607), the control unit 21 terminates processing. The control unit 21 realizes the function of the classification data synthesis unit 628 by the classification synthesis subroutine.

[0228] According to this embodiment, a catheter system 10 can be provided that generates RT-format classified data 528 using composite classified data 526 obtained by combining classified data 52 output from two classification trained models. By combining and using a second classification trained model 622, which can relatively easily collect a large amount of training data and improve classification accuracy, with a first classification trained model 621, which requires time and effort to collect training data, a catheter system 10 can be provided that has a good balance between the cost of generating the trained model and classification accuracy.

[0229] [Embodiment 6] This embodiment relates to a catheter system 10 that uses position information of a medical instrument as a hint to classify each part that constitutes a catheter image 51. Explanation of parts common to the first embodiment will be omitted.

[0230] 25 is an explanatory diagram illustrating the configuration of the hinted trained model 631. The hinted trained model 631 is used in step S604 described using FIG. 4 instead of the classification model 62 described using FIG. 7.

[0231] The hinted trained model 631 is a model that receives the RT-format catheter image 518 and the position information of the medical instrument depicted in the RT-format catheter image 518, and outputs hinted classification data 561 that classifies each part that makes up the RT-format catheter image 518 into a "biological tissue region," a "non-biological tissue region," and a "medical instrument region." The first classification trained model 621 further outputs the reliability of the classification result for each part, that is, the probability that the classification result is correct.

[0232] 26 is an explanatory diagram illustrating the record layout of the hinted training data DB 72. The hinted training data DB 72 is a database that records, in association with each other, catheter images 51, position information of medical instruments depicted in the catheter images 51, and classification data 52 that classifies each part of the catheter images 51 by the depicted subjects.

[0233] The classification data 52 is data created by a labeler based on the procedure described using Fig. 19. A hinted trained model 631 can be generated by performing the same process as the machine learning described in the fourth embodiment using the hinted training data DB 72.

[0234] 27 is a flowchart illustrating the flow of processing of the program according to embodiment 6. The flowchart explained using FIG. 27 shows details of the processing performed in step S504 explained using FIG.

[0235] The control unit 21 acquires one frame of the RT-format catheter image 518 (step S621). The control unit 21 inputs the RT-format catheter image 518 to the medical instrument trained model 611 described with reference to FIG. 6, for example, to acquire position information of the medical instrument (step S622). The control unit 21 inputs the RT-format catheter image 518 and the position information to the trained model with hints 631 to acquire the classification data with hints 561 (step S623).

[0236] The control unit 21 extracts one continuous non-biological tissue region from the classification data with hints 561 (step S624). Note that the processing after the extraction of the non-biological tissue region is preferably performed in a state where the upper and lower ends of the RT format catheter image 518 are connected to form a cylindrical shape.

[0237] The control unit 21 determines whether the non-biological tissue region extracted in step S624 is on the side that contacts the imaging catheter 40 (step S554). The subsequent processing up to step S559 is the same as the processing flow of the program of the fourth embodiment described using Fig. 20, and therefore description thereof will be omitted.

[0238] The control unit 21 determines whether or not the processing of all non-biological tissue regions has been completed (step S559). If it is determined that the processing has not been completed (NO in step S559), the control unit 21 returns to step S624. If it is determined that the processing has been completed (YES in step S559), the control unit 21 ends the processing.

[0239] According to this embodiment, it is possible to provide a catheter system 10 that generates classification data 52 with high accuracy by inputting position information of a medical instrument as a hint.

[0240] [Variation 6-1] 28 is a flowchart illustrating the flow of processing of a program according to a modified example. The processing described with reference to FIG. 28 is executed instead of the processing described with reference to FIG.

[0241] The control unit 21 acquires one frame of the RT format catheter image 518 (step S621). The control unit 21 acquires the position information of the medical instrument (step S622). The control unit 21 determines whether or not the acquisition of the position information of the medical instrument has been successful (step S631). For example, when the reliability output from the medical instrument trained model 611 is higher than a threshold, the control unit 21 determines that the acquisition of the position information has been successful.

[0242] Note that "success" in step S631 means that the medical instrument is depicted in the RT-format catheter image 518 and that the control unit 21 was able to acquire position information of the medical instrument with a reliability higher than the threshold. Cases where the result is not "success" include, for example, a case where the medical instrument is not present in the imaging range of the RT-format catheter image 518, and a case where the medical instrument is in close contact with the surface of the biological tissue region and is not clearly depicted.

[0243] If it is determined that the acquisition of the position information has been successful (YES in step S631), the control unit 21 inputs the RT-format catheter image 518 and the position information into the hinted trained model 631 to acquire the hinted classified data 561 (step S623). If it is determined that the acquisition of the position information has not been successful (NO in step S631), the control unit 21 inputs the RT-format catheter image 518 into the unhinted trained model 632 to acquire the hinted unclassified data (step S632).

[0244] The unlearned hint model 632 is the classification model 62 described using, for example, Figure 7, Figure 18, or Figure 21. Similarly, the unclassified hint data is the classified data 52 output from the classification model 62.

[0245] After step S623 or step S632 is completed, the control unit 21 extracts one continuous non-biological tissue region from the classification data with hints 561 or the classification model 62 (step S624). The subsequent processing is the same as the processing flow described using Fig. 27, and therefore description thereof will be omitted.

[0246] The hinted classification data 561 is an example of the first data. The hinted trained model 631 is an example of a first trained model that outputs the first data when the catheter image 51 and the position information of the medical instrument are input. The output layer of the hinted trained model 631 is an example of a first data output unit that outputs the first data.

[0247] The unclassified hint data is an example of the second data. The untrained hint model 632 is an example of the second trained model and the second model that outputs the second data when the catheter image 51 is input. The output layer of the untrained hint model 632 is an example of the second data output unit.

[0248] According to this modification, if the acquisition of position information is not successful, the classification model 62 that does not require the input of position information is used. Therefore, it is possible to provide a catheter system 10 that prevents malfunctions caused by inputting incorrect hints to the hint-trained model 631.

[0249] [Embodiment 7] This embodiment relates to a catheter system 10 that generates synthetic data 536 by synthesizing the output of a trained model with hints 631 and the output of a trained model without hints 632. Explanation of parts common to the sixth embodiment will be omitted. The synthetic data 536 is data that is used in place of the classified data 52 that is the output of step S504 described using FIG. 4.

[0250] 29 is an explanatory diagram illustrating the configuration of the classification model 62 according to the seventh embodiment. The classification model 62 includes a position classification analysis unit 66 and a third synthesis unit 543. The position classification analysis unit 66 includes a position information acquisition unit 65, a trained model with hints 631, a trained model without hints 632, a first synthesis unit 541, and a second synthesis unit 542.

[0251] The position information acquisition unit 65 acquires position information indicating the position where the medical instrument is depicted, for example, from the medical instrument trained model 611 described using FIG. 6 or the position information model 619 described using FIG. 16. The trained model with hints 631 is the same as in the sixth embodiment, and therefore description thereof will be omitted. The trained model without hints 632 is the classification model 62 described using FIG. 7, FIG. 18, or FIG. 21, for example.

[0252] The operation of the first synthesis unit 541 will be described. The first synthesis unit 541 synthesizes the hint-included classified data 561 output from the hint-included trained model 631 and the hint-unclassified data output from the hint-unclassified trained model 632 to create classification information. The input terminal of the first synthesis unit 541 functions as a first data acquisition unit that acquires the hint-included classified data 561 and a second data acquisition unit that acquires the hint-unclassified data. The output terminal of the first synthesis unit 541 functions as a first synthesized data output unit that outputs first synthesized data obtained by synthesizing the hint-included classified data 561 and the hint-unclassified data.

[0253] When data is input in which the non-biological tissue region is not classified into the first lumen region, the second lumen region, and the non-lumen region, the first synthesis unit 541 performs the function of the classification data conversion unit 629 and classifies the non-biological tissue region.

[0254] For example, when the location information acquisition unit 65 has succeeded in acquiring location information, the first synthesis unit 541 synthesizes the two models by making the weight of the hinted trained model 631 greater than the weight of the unhinted trained model 632. The method of weighting and synthesizing images is well known, and therefore will not be described here.

[0255] The first combining unit 541 may combine the hint-containing classified data 561 and the hint-uncategorized data by determining weighting based on the reliability of the location information acquired by the location information acquiring unit 65.

[0256] The first combining unit 541 may combine the hint-included classified data 561 and the hint-unclassified data based on the reliability of each of the regions of the hint-included classified data 561 and the hint-unclassified data. Combining based on the reliability of the classified data 52 can be performed, for example, by the same process as that of the classified data combining unit 628 described in the fifth embodiment.

[0257] The first synthesis unit 541 treats the medical device region output from the hinted trained model 631 and the unhinted trained model 632 in the same way as an adjacent non-biological tissue region. For example, if a medical device region exists in the first lumen region, the first synthesis unit 541 treats the medical device region in the same way as the first lumen region. Similarly, if a medical device region exists in the second lumen region, the first synthesis unit 541 treats the medical device region in the same way as the second lumen region.

[0258] A trained model that does not output a medical instrument region may be used as either the trained model with hint 631 or the trained model without hint 632. Therefore, as shown in the center of Fig. 29 , the classification information output from the first synthesis unit 541 does not include information related to the medical instrument region.

[0259] The first synthesis unit 541 may function as a switch that switches between the hint-included classified data 561 and the hint-unclassified data based on whether the location information acquisition unit 65 has succeeded in acquiring location information. The first synthesis unit 541 may further function as the classified data conversion unit 629.

[0260] Specifically, if the location information acquisition unit 65 succeeds in acquiring location information, the first synthesis unit 541 outputs classification information based on the hinted classified data 561 output from the hinted trained model 631. If the location information acquisition unit 65 does not succeed in acquiring location information, the first synthesis unit 541 outputs classification information based on the hint unclassified data output from the hint untrained model 632.

[0261] The operation of the second synthesis unit 542 will be described. If the location information acquisition unit 65 succeeds in acquiring location information, the second synthesis unit 542 outputs the medical instrument region output from the hinted trained model 631. If the location information acquisition unit 65 does not succeed in acquiring location information, the second synthesis unit 542 outputs the medical instrument region included in the hint unclassified data.

[0262] It is desirable to use the second classification trained model 622 described using Figure 21 for the hint-free trained model 632. As mentioned above, a large amount of training data can be used to train the second classification trained model 622, so the medical instrument region can be extracted with high accuracy.

[0263] If the location information acquisition unit 65 does not succeed in acquiring location information, the second synthesis unit 542 may synthesize and output the medical instrument region included in the classification data with hints 561 and the medical instrument region included in the classification data without hints. The synthesis of the classification data with hints 561 and the classification data without hints can be performed, for example, by the same process as that of the classification data synthesis unit 628 described in the fifth embodiment.

[0264] The output terminal of the second synthesis unit 542 functions as a second synthesis data output unit that outputs second synthesis data obtained by synthesizing the medical instrument region of the classification data with hints 561 and the medical instrument region of the classification data without hints.

[0265] The operation of the third synthesis unit 543 will be described. The third synthesis unit 543 outputs synthesized data 536 in which the medical instrument region output from the second synthesis unit 542 is superimposed on the classification information output from the first synthesis unit 541. In Fig. 29, the superimposed medical instrument region is indicated by black dots.

[0266] Instead of the first synthesis unit 541, the third synthesis unit 543 may perform the function of the classification data conversion unit 629 that classifies the non-biological tissue region into a first lumen region, a second lumen region, and a non-lumen region.

[0267] Some or all of the multiple trained models that make up the position classification analysis unit 66 may be models that accept multiple catheter images 51 acquired in chronological order and output information for the most recent catheter image 51.

[0268] This embodiment provides a catheter system 10 that acquires position information of a medical instrument with high accuracy and outputs the information in combination with classification information. The control unit 21 may generate composite data 536 based on each of a plurality of catheter images 51 captured consecutively along the longitudinal direction of the image acquisition catheter 40, and then stack the composite data 536 to construct and display three-dimensional data of the biological tissue and the medical instrument.

[0269] [Variation 7-1] 30 is an explanatory diagram illustrating the configuration of a modified classification model 62. An X% hint trained model 639 is added to the position classification analysis unit 66. The X% hint trained model 639 is a model trained using the hinted training data DB 72 under the condition that position information is input for X percent of the training data and not input for (100-X) percent. In the following description, the data output from the X% hint trained model 639 will be referred to as X% hint classification data.

[0270] The X% hint trained model 639 is identical to the hinted trained model 631 when X is "100", and is identical to the non-hinted trained model 632 when X is "0". X is, for example, "50".

[0271] The first synthesis unit 541 outputs data obtained by synthesizing, based on a predetermined weighting, the classified data 52 acquired from the hinted trained model 631, the hint-free trained model 632, and the X% hint trained model 639. The weighting changes depending on whether the location information acquisition unit 65 has succeeded in acquiring location information.

[0272] For example, if the location information acquisition unit 65 succeeds in acquiring location information, the output of the hinted trained model 631 and the output of the X% hint trained model 639 are combined. If the location information acquisition unit 65 fails to acquire location information, the output of the unhinted trained model 632 and the output of the X% hint trained model 639 are combined. The weighting during combination may vary based on the reliability of the location information acquired by the location information acquisition unit 65.

[0273] The position classification analysis unit 66 may include multiple X% hint trained models 639. For example, an X% hint trained model 639 in which X is "20" and an X% hint trained model 639 in which X is "50" can be used in combination.

[0274] In clinical settings, there are cases where it is not possible to extract a medical instrument region from the catheter image 51. For example, this may occur when the medical instrument is not inserted into the first cavity or when the medical instrument is in close contact with the surface of biological tissue. This modification makes it possible to realize a classification model 62 that matches the actual situation in such clinical settings. As a result, it is possible to provide a catheter system 10 that can accurately detect and classify position information.

[0275] [Embodiment 8] This embodiment relates to three-dimensional display of a catheter image 51. Explanation of parts common to the seventh embodiment will be omitted. Fig. 31 is an explanatory diagram for explaining an outline of the processing of the eighth embodiment.

[0276] In this embodiment, a plurality of RT format catheter images 518 captured continuously along the longitudinal direction of the image acquisition catheter 40 are used. The control unit 21 inputs each of the plurality of RT format catheter images 518 to the position classification analysis unit 66 described in the seventh embodiment. The position classification analysis unit 66 outputs classification information and a medical instrument region corresponding to each RT format catheter image 518. The control unit 21 inputs the classification information and medical instrument information to a third synthesis unit 543 to synthesize synthesized data 536.

[0277] The control unit 21 creates biological three-dimensional data 551 indicating the three-dimensional structure of biological tissue based on the multiple pieces of composite data 536. The biological three-dimensional data 551 is, for example, voxel data in which values ​​indicating a biological tissue label, a first lumen region label, a second lumen region label, a non-lumen region label, etc. are recorded for each volumetric grid in three-dimensional space. The biological three-dimensional data 551 may also be polygon data composed of multiple polygons indicating the boundaries of each region. A method for creating three-dimensional data 55 based on multiple pieces of RT format data is well known, so a description thereof will be omitted.

[0278] The control unit 21 acquires position information indicating the position of the medical instrument depicted in each RT format catheter image 518 from the position information acquisition unit 65 included in the position classification analysis unit 66. The control unit 21 creates medical instrument three-dimensional data 552 indicating the three-dimensional shape of the medical instrument based on the multiple pieces of position information. The medical instrument three-dimensional data 552 will be described in detail later.

[0279] The control unit 21 generates three-dimensional data 55 by combining the living body three-dimensional data 551 and the medical instrument three-dimensional data 552. The three-dimensional data 55 is used for the "3D display" in step S513 described with reference to FIG. 4. When combining the three-dimensional data 55, the control unit 21 replaces the medical instrument region included in the combined data 536 with a blank region or a non-living body region, and then combines the medical instrument three-dimensional data 552. The control unit 21 may generate the living body three-dimensional data 551 using classification information output from a first combining unit 541 included in the position classification analysis unit 66.

[0280] Figures 32A to 32D are explanatory diagrams outlining the process of correcting position information. Figures 32A to 32D are schematic diagrams showing, in chronological order, the state in which catheter images 51 are captured while the image acquisition catheter 40 is pulled to the right in the figure. The thick cylinder schematically shows the inner surface of the first cavity.

[0281] In Figure 32A, three catheter images 51 have been captured. The position information of the medical instruments extracted from each catheter image 51 is indicated by a white circle. Figure 32B shows the state after the fourth catheter image 51 has been captured. The position information of the medical instruments extracted from the fourth catheter image 51 is indicated by a black circle.

[0282] The medical device has been detected in a location that is clearly different from the three previously captured catheter images 51. Generally, medical devices used in IVR have a certain degree of rigidity and are unlikely to bend suddenly. Therefore, the position information indicated by the black circle is likely to be a false detection.

[0283] In Figure 32C, two more catheter images 51 have been taken. The white circles indicate the positional information of the medical device extracted from each catheter image 51. The five white circles are aligned almost in a line along the longitudinal direction of the image acquisition catheter 40, but the black circles are far apart, clearly indicating erroneous detection.

[0284] In Figure 32D, the position information interpolated based on the five white circles is indicated by X marks. By using the position information indicated by X marks instead of the position information indicated by the black circles, the shape of the medical device in the first cavity can be correctly displayed in the three-dimensional image.

[0285] If the position information acquisition unit 65 does not succeed in acquiring the position information, the control unit 21 may use, as the position information, the representative point of the medical instrument region acquired from the second synthesis unit 542 included in the position classification analysis unit 66. For example, the center of gravity of the medical instrument region can be used as the representative point.

[0286] Fig. 33 is a flowchart illustrating the flow of processing of a program according to embodiment 8. The program described using Fig. 33 is a program executed when it is determined that the user has designated three-dimensional display in step S505 described using Fig. 4 (3D in step S505).

[0287] The program in Fig. 33 can be executed while a plurality of catheter images 51 are being captured along the longitudinal direction of the image acquisition catheter 40. An example will be described in which, prior to execution of the program in Fig. 33, classification information and position information have been generated for each of the captured catheter images 51 and stored in the auxiliary storage device 23 or an external mass storage device.

[0288] The control unit 21 acquires position information corresponding to one catheter image 51 and records it in the main storage device 22 or the auxiliary storage device 23 (step S641). The control unit 21 processes the series of catheter images 51 in order, starting with the catheter image 51 that was stored first. In step S641, the control unit 21 may acquire and record position information from the first few catheter images 51 of the series of catheter images 51.

[0289] The control unit 21 acquires position information corresponding to the next catheter image 51 (step S642). In the following description, the position information being processed will be referred to as first position information. The control unit 21 extracts position information that is closest to the first position information from the position information acquired in step S641 and previous steps S641 (step S643). In the following description, the position information extracted in step S643 will be referred to as second position information.

[0290] In step S642, the distances between the positional information are compared with the multiple catheter images 51 projected onto a plane perpendicular to the imaging catheter 40. That is, when extracting the second positional information, the distance in the longitudinal direction of the imaging catheter 40 is not taken into consideration.

[0291] The control unit 21 determines whether the distance between the first position information and the second position information is equal to or less than a predetermined threshold value (step S644). The threshold value is, for example, 3 millimeters. If it is determined that the distance is equal to or less than the threshold value (YES in step S644), the control unit 21 records the second position information in the main storage device 22 or the auxiliary storage device 23 (step S645).

[0292] If it is determined that the threshold value is exceeded (NO in step S644), or after step S645 is completed, control unit 21 determines whether or not processing of the recorded position information is completed (step S646). If it is determined that processing is not completed (NO in step S646), control unit 21 returns to step S642.

[0293] 32 are examples of location information that is determined to exceed the threshold value in step S644. The control unit 21 ignores such location information without recording it in step S645. The control unit 21 performs the function of an exclusion unit that excludes location information that does not satisfy a predetermined condition by performing the process when the determination in step S644 is NO. Note that the control unit 21 may record location information that is determined to exceed the threshold value in step S644 by attaching a flag indicating an "error."

[0294] If it is determined that the process has ended (YES in step S646), control unit 21 determines whether or not the location information can be complemented based on the location information recorded in steps S641 and S645 (step S647).If it is determined that the location information can be complemented (YES in step S647), control unit 21 complements the location information (step S648).

[0295] In step S648, the control unit 21 complements position information that replaces the position information determined to exceed the threshold in step S644, for example. The control unit 21 may complement position information between catheter images 51. The complementation can be performed using any method, such as linear interpolation, spline interpolation, Lagrange interpolation, or Newtonian interpolation. In step S648, the control unit 21 realizes the function of a complementing unit that adds complementary information to the position information.

[0296] If it is determined that the position information cannot be complemented (NO in step S647), or after step S648 is completed, the control unit 21 starts a three-dimensional display subroutine (step S649). The three-dimensional display subroutine is a subroutine that performs three-dimensional display based on a series of catheter images 51. The processing flow of the three-dimensional display subroutine will be described later.

[0297] The control unit 21 determines whether to end the process (step S650). For example, when a new pullback operation is performed by the MDU 33, that is, when capturing a catheter image 51 to be used for generating a three-dimensional image is started, the control unit 21 determines to end the process.

[0298] If it is determined not to end the process (NO in step S650), control unit 21 returns to step S642. If it is determined to end the process (YES in step S650), control unit 21 ends the process.

[0299] 33, control unit 21 generates and records classification information and position information based on newly captured catheter image 51. That is, when it is determined that the process has ended in step S646, steps S647 and thereafter are executed, but new position information and classification information may have been generated while steps S647 to S650 are being executed.

[0300] 34 is a flowchart illustrating the processing flow of the three-dimensional display subroutine. The three-dimensional display subroutine is a subroutine that performs three-dimensional display based on a series of catheter images 51. The three-dimensional display subroutine enables control unit 21 to realize the function of a three-dimensional output unit.

[0301] The control unit 21 acquires composite data 536 corresponding to the series of catheter images 51 (step S661). The control unit 21 creates biological three-dimensional data 551 indicating the three-dimensional structure of biological tissue based on the series of composite data 536 (step S662).

[0302] As described above, when synthesizing the three-dimensional data 55, the control unit 21 replaces the medical instrument region included in the synthesized data 536 with a blank region or a non-biological region, and then synthesizes the medical instrument three-dimensional data 552. The control unit 21 may generate the biological three-dimensional data 551 using classification information output from the first synthesis unit 541 included in the position classification analysis unit 66. The control unit 21 may generate the biological three-dimensional data 551 based on the first classified data 521 described using FIG. 18 . That is, the control unit 21 can generate the biological three-dimensional data 551 directly based on a plurality of first classified data 521.

[0303] The control unit 21 may generate the biometric three-dimensional data 551 indirectly based on the plurality of first classified data 521. "Indirectly based" means generating the biometric three-dimensional data 551 based on a plurality of synthesized data 536 generated using the plurality of first classified data 521, as described using FIG. 31, for example. The control unit 21 may generate the biometric three-dimensional data 551 based on a plurality of data different from the synthesized data 536 generated using the first plurality of classified data 521.

[0304] The control unit 21 assigns thickness information to the curve defined by the series of positional information recorded in steps S641 and S645 of the program described with reference to FIG. 33 and the complementary information complemented in step S648 (step S663). The thickness information is preferably the thickness of a medical instrument commonly used in IVR procedures. The control unit 21 may receive information about a medical instrument currently in use and assign thickness information corresponding to that medical instrument. By assigning the thickness information, the three-dimensional shape of the medical instrument is reproduced.

[0305] The control unit 21 combines the three-dimensional shape of the medical instrument generated in step S662 with the three-dimensional biological data 551 generated in step S662 (step S664). The control unit 21 displays the combined three-dimensional data 55 on the display device 31 (step S665).

[0306] The control unit 21 receives instructions from the user for rotating, changing the cross section, enlarging, reducing, etc., the three-dimensionally displayed image, and changes the display. Since receiving instructions for three-dimensionally displayed images and changing the display are conventionally performed, a description thereof will be omitted. The control unit 21 then ends the processing.

[0307] According to this embodiment, it is possible to provide a catheter system 10 that can eliminate the influence of erroneous detection of position information and display a medical instrument with a correct shape. A user can easily grasp the positional relationship between, for example, the Brockenbrough needle and the fossa ovalis and perform an IVR procedure.

[0308] Instead of performing the processes from step S643 to step S645, abnormal location information that is significantly different from other location information may be removed by clustering a plurality of location information.

[0309] [Variation 8-1] This modification relates to a catheter system 10 that, when no medical instrument is erroneously detected, performs three-dimensional display based on a medical instrument region detected from a catheter image 51. Explanation of parts common to the eighth embodiment will be omitted.

[0310] 34, the control unit 21 determines the thickness of the medical instrument based on the medical instrument region output from, for example, the trained model with hints 631 or the trained model without hints 632. However, for a catheter image 51 whose position information is determined to be incorrect, the control unit 21 complements the thickness information based on the medical instrument regions of the catheter images 51 before and after it.

[0311] According to this modification, it is possible to provide a catheter system 10 that can appropriately display, in a three-dimensional image, a medical instrument whose diameter changes midway, such as a medical instrument with a needle protruding from a sheath.

[0312] [Embodiment 9] This embodiment relates to padding processing suitable for a trained model that processes an RT format catheter image 518 acquired using a radial scanning type imaging catheter 40. Explanation of parts common to the first embodiment will be omitted.

[0313] Padding is the process of adding data around input data before performing convolution processing. In convolution processing immediately after the input layer that accepts image input, the input data is the input image. In convolution processing other than immediately after the input layer, the input data is the feature map extracted in the previous stage. In trained models that process image data, so-called zero padding, in which "0" data is added around the input data input to the convolution layer, is commonly performed.

[0314] FIG. 35 is an explanatory diagram illustrating padding processing in the ninth embodiment. The right side of FIG. 35 is a schematic diagram of input data input to a convolutional layer. The convolutional layer is, for example, an example of the first convolutional layer included in the medical instrument trained model 611 and the second convolutional layer included in the angle trained model 612. The convolutional layer may be a convolutional layer included in any trained model used to process catheter images 51 captured using a radial scanning imaging catheter 40.

[0315] The input data is in RT format, with the horizontal direction corresponding to the distance from the sensor 42 and the vertical direction corresponding to the scanning angle. Enlarged schematic diagrams of the upper right and lower left corners of the input data are shown in the center of Figure 35. Each frame corresponds to a pixel, and the numerical value within the frame corresponds to the pixel value.

[0316] The right side of Figure 35 is a schematic diagram of data after padding according to this embodiment. The numbers in italics indicate data added by padding. Data "0" is added to the left and right ends of the input data. Data indicated by "A" at the bottom of the input data before padding is copied to the top of the input data. Data indicated by "B" at the top of the input data before padding is copied to the bottom of the input data.

[0317] That is, at the right end of Fig. 35, the same data as on the side with a larger scan angle is added to the outside of the side with a smaller scan angle, and the same data as on the side with a smaller scan angle is added to the outside of the side with a larger scan angle. In the following explanation, the padding process explained using Fig. 35 will be referred to as polar padding process.

[0318] In the case of a radial scanning type image acquisition catheter 40, the upper and lower ends of the RT format catheter image 518 are substantially the same. For example, a single medical instrument or a lesion may be separated into the upper and lower parts of the RT format catheter image 518. Polar padding processing utilizes this characteristic.

[0319] According to this embodiment, it is possible to generate a trained model that fully reflects the information about the top and bottom of an RT format image.

[0320] Polar padding may be performed on all convolutional layers included in the trained model, or on some of the convolutional layers.

[0321] 35 shows an example of padding in which one piece of data is added to each side of the input data, but the padding may also be performed by adding multiple pieces of data. The number of pieces of data added in the polar padding process is selected depending on the size and stride of the filter used in the convolution process.

[0322] [Variation 9-1] 36 is an explanatory diagram illustrating polar padding processing of a modified example. The polar padding processing of this modified example is effective for the convolution layer at the stage where the RT format catheter image 518 is first processed.

[0323] The upper part of Fig. 36 schematically shows a state in which radial scanning is performed while the sensor 42 is pulled rightward. An RT format catheter image 518, shown schematically in the lower left of Fig. 36, is generated based on the scan line data acquired while the sensor 42 makes one rotation. The RT format catheter image 518 is formed from top to bottom as the sensor 42 rotates.

[0324] The bottom right of Figure 36 schematically shows the state after padding has been performed on the RT format catheter image 518. Data for the end of the RT format catheter image 518 from one rotation before, indicated by hatching slanting downward to the left, is added to the top of the RT format catheter image 518. Data for the start of the RT format catheter image 518 from one rotation after, indicated by hatching slanting downward to the right, is added to the bottom of the RT format catheter image 518. Data "0" is added to the left and right of the RT format catheter image 518.

[0325] According to this modified example, padding processing is performed based on actual scan line data, making it possible to generate a trained model that more accurately reflects the information above and below the RT format image.

[0326] [Embodiment 10] Figure 37 is an explanatory diagram illustrating the configuration of a catheter system 10 according to a tenth embodiment. This embodiment relates to a configuration in which the catheter system 10 according to the present embodiment is realized by combining and operating a catheter control device 27, an MDU 33, an image acquisition catheter 40, a general-purpose computer 90, and a program 97. Explanations of parts common to the first embodiment will be omitted.

[0327] Catheter control device 27 is an ultrasonic diagnostic device for IVUS that controls MDU 33, controls sensor 42, and generates transverse and longitudinal images based on signals received from sensor 42. The function and configuration of catheter control device 27 are similar to those of conventional ultrasonic diagnostic devices, and therefore a description thereof will be omitted.

[0328] The catheter system 10 of this embodiment includes a computer 90. The computer 90 is equipped with a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, a smartphone, or a server computer.

[0329] The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read the program 97 stored in a semiconductor memory 98, such as a flash memory, implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown) and store it in the auxiliary storage device 23.

[0330] The program 97 is installed as a control program for the computer 90, and is loaded into and executed by the main storage device 22. This causes the computer 90 to function as the information processing device 20 described above.

[0331] The computer 90 may be a general-purpose personal computer, a tablet, a smartphone, a mainframe computer, a virtual machine running on a mainframe computer, a cloud computing system, or a quantum computer. The computer 90 may also be a plurality of personal computers performing distributed processing.

[0332] [Embodiment 11] 38 is a functional block diagram of an information processing device 20 according to embodiment 11. The information processing device 20 includes an image acquiring unit 81 and a first classified data output unit 82. The image acquiring unit 81 acquires a catheter image 51 obtained by an image acquisition catheter 40 inserted into the first cavity.

[0333] When a catheter image 51 is input, the first classified data output unit 82 inputs the acquired catheter image 51 to a first classification trained model 621 that outputs first classified data 521 in which a non-living tissue region including a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of the second cavity into which the image acquisition catheter 40 is not inserted and a living tissue region are classified as different regions, and outputs the first classified data 521. The first classification trained model 621 is generated using first training data that clearly indicates at least a non-living tissue region including the first lumen region and the second lumen region, and a living tissue region.

[0334] [Embodiment 12] This embodiment relates to a method for generating a classification model 62 that performs machine learning using an inconsistency loss function that is set to be large when there is an inconsistency between adjacent regions. Explanation of parts common to the first embodiment will be omitted.

[0335] Fig. 39 is an explanatory diagram illustrating the machine learning process of embodiment 12. In this embodiment, a classification model 62 is generated that receives an RT-format catheter image 518, as explained using Fig. 7, and outputs RT-format classification data 528 that classifies each part constituting the RT-format catheter image 518 by the depicted subject.

[0336] In this embodiment, a third training data DB is used, which records a large number of sets of third training data 733 that associates RT-format catheter images 518 with RT-format classification data 528 classified by a labeler. In the following description, the RT-format classification data 528 recorded in the third training data 733 may be referred to as correct classification data.

[0337] In the RT type classification data 528, thin hatching downward to the left indicates a first lumen region. Thin hatching downward to the right indicates a second lumen region. Thick hatching downward to the right occupies a biological tissue region. Thick hatching downward to the left indicates a non-lumen region. Solid black indicates a medical device region.

[0338] The control unit 21 inputs the RT-format catheter image 518 to the classification model 62 under training and obtains output classification data 523. The output classification data 523 is an example of output label data in this embodiment. In part B of the output classification data 523, the first lumen region indicated by thin hatching slanting downward to the left and the second lumen region indicated by thin hatching slanting downward to the right are adjacent to each other.

[0339] However, as explained using Fig. 18, the second lumen region is a region of the non-biological tissue region that is surrounded by the biological tissue region. Therefore, the state in which the first lumen region and the second lumen region are in contact with each other contradicts the definition of the second lumen region.

[0340] The control unit 21 calculates a composite loss function 643 by combining a differential loss function 641 indicating the difference between the RT-format classification data 528 recorded in the third training data 733 and the output classification data 523, and a contradiction loss function 642 indicating the contradiction with the definitions of the respective regions. The control unit 21 adjusts the parameters of the classification model 62 being trained by backpropagation so as to reduce the composite loss function 643.

[0341] For example, the control unit 21 quantifies the difference between each pixel constituting the RT-format classification data 528 and the corresponding pixel in the output classification data 523. The control unit 21 calculates the mean square error (MSE) or cross entropy (CE) of the quantified difference. In this way, the control unit 21 calculates an arbitrary differential loss function 641 that has traditionally been used in supervised machine learning.

[0342] 40 to 42 are explanatory diagrams illustrating the contradiction loss function 642. Fig. 40 is a schematic diagram of nine pixels extracted from the output classification data 523. Although not shown, each pixel has a recorded label indicating whether it has been classified into the first lumen region, the second lumen region, the biological tissue region, the non-lumen region, or the medical device region.

[0343] P1 indicates the penalty determined by the degree of conflict between the reference pixel shown in the center and the adjacent pixel on the right. P2 indicates the penalty between the reference pixel shown in the center and the adjacent pixel on the lower right. P3 indicates the penalty between the reference pixel shown in the center and the adjacent pixel on the lower side. A penalty of "0" means there is no conflict. A larger penalty value means a larger conflict.

[0344] Figure 41 shows a penalty conversion table in tabular form, which shows the penalty determined based on the relationship between the label recorded on the reference pixel and the label recorded on the adjacent pixel. As mentioned above, if the reference pixel is in the first lumen region and the adjacent pixel is in the second lumen region, this contradicts the definition of the second lumen region, so the penalty is set to 3 points. If the reference pixel is in the first lumen region and the adjacent pixel is in a non-biological tissue region, this contradicts the definition of the first lumen region, so the penalty is set to 1 point. There is no contradiction between the pixel adjacent to the first lumen region being the first lumen region, a biological tissue region, or a medical device region, so the penalty is set to 0 points.

[0345] If the reference pixel is in a biological tissue region, there is no contradiction regardless of the region of the adjacent pixel, so the penalty is set to 0 points in both cases. If the reference pixel is in a medical device region and the adjacent pixel is in a non-biological region, the penalty is set to 3 points. If the adjacent pixel is in a region other than a non-biological region, the penalty is set to 0 points.

[0346] If the reference pixel is in the second lumen region, the penalty is set to 3 points if the neighboring pixel is in the first lumen region or non-living tissue, and the penalty is set to 0 points if the neighboring pixel is in the living tissue region, medical device region, or second lumen region.

[0347] When the reference pixel is in a non-biological tissue region, if the adjacent pixel is in the first lumen region, the penalty is set to 1 point. When the adjacent pixel is in the medical device region or the second lumen region, the penalty is set to 3 points. When the adjacent pixel is in a biological tissue region or a non-biological tissue region, the penalty is set to 0 points. Note that the penalty conversion table shown in Figure 41 is an example and is not limited to this.

[0348] Returning to FIG. 40, the explanation continues. The penalty for the reference pixel is determined based on P1, P2, and P3. In the following explanation, an example will be given in which the penalty for the reference pixel is the sum of P1, P2, and P3. Note that the penalty for the reference pixel may be any representative value such as the arithmetic mean, geometric mean, harmonic mean, median, or maximum value of P1, P2, and P3.

[0349] Figure 42 is a schematic diagram of 25 pixels extracted from the output classification data 523. The label recorded in each pixel is indicated by the type of hatching, as in Figure 39. The penalty calculated for each pixel is indicated by a number.

[0350] For example, the pixel in the top left corner is classified as a biological tissue region. Since there is no contradiction regardless of the region of the neighboring pixels, P1, P2, and P3 are all 0 points, and the penalty for the top left pixel, which is the sum of these, is 0 points.

[0351] The center pixel is classified as the first lumen region. The neighboring pixel to the right is classified as a non-lung region, so P1 is 1 point. The neighboring pixel to the lower right is classified as a living tissue region, so P2 is 0 point. The neighboring pixel below is classified as the second lumen region, so P3 is 3 point. Therefore, the penalty for the center pixel is 4 points, which is the sum of P1, P2, and P3.

[0352] The pixel in the fourth row from the top, second column from the left is also classified as the first lumen region. The adjacent pixel on the right and the adjacent pixel on the lower right are classified as the second lumen region, so P1 and P2 are 3 points. The adjacent pixel below is classified as the biological tissue region, so P3 is 0 point. Therefore, the penalty for the pixel in the fourth row from the top, second column from the left is 6 points, which is the sum of P1, P2, and P3.

[0353] Similarly, the control unit 21 calculates the penalty for each pixel that constitutes the output classification data 523. The control unit 21 calculates the inconsistency loss function 642. The inconsistency loss function 642 is a representative value of the calculated penalty for each pixel, such as the mean square value, arithmetic mean value, median, or mode of the penalty.

[0354] The control unit 21 calculates the combined loss function 643 based on, for example, equation (12-1).

[0355]

number

[0356] Fig. 43 is a flowchart illustrating the flow of processing of the program according to embodiment 12. Prior to execution of the program in Fig. 43, an untrained classification model 62, such as a U-Net structure that realizes semantic segmentation, is prepared.

[0357] The control unit 21 initializes parameters of the classification model 62 (step S801). The control unit 21 acquires a set of third training data 733 from the third training data DB (step S802). As described above, the third training data 733 acquired in step S802 includes the RT-format catheter image 518 and the RT-format classification data 528, which is the correct classification data.

[0358] The control unit 21 inputs the RT format catheter image 518 to the classification model 62 and obtains the output classification data 523 (step S803). The control unit 21 calculates the differential loss function 641 based on the output classification data 523 and the correct classification data (step S804). The control unit 21 calculates the contradiction loss function 642 based on the output classification data 523 and the penalty conversion table (step S805).

[0359] The control unit 21 calculates the composite loss function 643 based on the formula (12-1) (step S806). The control unit 21 adjusts the parameters of the classification model 62 using, for example, backpropagation or the like (step S807).

[0360] The control unit 21 determines whether to end the parameter adjustment (step S808). For example, the control unit 21 determines to end the process when learning has been performed a predetermined number of times. The control unit 21 may acquire test data from the third training data DB, input it to the classification model 62 undergoing machine learning, and determine to end the process when an output of a predetermined accuracy is obtained.

[0361] If it is determined not to end the processing (NO in step S808), control unit 21 returns to step S802. If it is determined to end the processing (YES in step S808), control unit 21 records the parameters of trained classification model 62 in auxiliary storage device 23 (step S809). Thereafter, control unit 21 ends the processing. Through the above processing, classification model 62 that accepts catheter image 51 and outputs RT-format classification data 528 is generated.

[0362] According to this embodiment, machine learning is performed so that there is no contradiction between adjacent regions, thereby making it possible to generate a highly accurate classification model 62.

[0363] 40, a case has been described in which penalties are used for three adjacent pixels to the right, lower right, and below the reference pixel, but this is not limiting. For example, penalties for eight adjacent pixels around the reference pixel may be used. Penalties for four adjacent pixels above, below, left, and right of the reference pixel, or four adjacent pixels to the lower right, lower left, upper left, and upper right may be used. Penalties for pixels two or more pixels away from the reference pixel may also be used.

[0364] [Embodiment 13] This embodiment relates to a method for selecting a classification model 62 with high accuracy from a plurality of classification models 62 generated by machine learning using a contradiction loss function 642. Explanation of parts common to the first embodiment will be omitted.

[0365] In machine learning, models with different parameters are generated depending on conditions such as the initial values ​​of the parameters, the combination of training data used for learning, and the order in which the training data is used. Depending on the learning process, a model with advanced local optimization may be generated, or a model with advanced global optimization may be generated. Prior to this embodiment, a plurality of classification models 62 were generated and recorded in the auxiliary storage device 23 by the method described in the fourth or twelfth embodiment.

[0366] 44 is a flowchart illustrating the processing flow of the program according to the thirteenth embodiment. The control unit 21 acquires a test record from the third training data DB (step S811). The test record is third training data 733 that has not been used for machine learning, and includes the RT-format catheter image 518 and the RT-format classification data 528, which is the correct classification data, as described above.

[0367] The control unit 21 acquires one classification model 62 recorded in the auxiliary storage device 23 (step S812). The control unit 21 inputs the RT format catheter image 518 to the classification model 62 and acquires the output classification data 523 to be output (step S813).

[0368] The control unit 21 calculates the differential loss function 641 based on the output classification data 523 and the correct classification data (step S814). The control unit 21 calculates the contradiction loss function 642 based on the output classification data 523 and the penalty conversion table (step S815).

[0369] The control unit 21 calculates the composite loss function 643 based on equation (12-1) (step S816). The control unit 21 associates the calculated composite loss function 643 with the model acquired in step S812 and records it in the auxiliary storage device 23 (step S817). The control unit 21 determines whether or not processing of the classification model 62 recorded in 23 has ended (step S818). If it is determined that processing has not ended (NO in step S818), the control unit 21 returns to step S812.

[0370] If it is determined that the processing has ended (YES in step S818), the control unit 21 determines whether the processing of the test record has ended (step S819).If it is determined that the processing has not ended (NO in step S819), the control unit 21 returns to step S811.

[0371] If it is determined that the process is completed (YES in step S819), the control unit 21 calculates (step S820) a representative value of the composite loss function 643 recorded in step S817 for each classification model 62. The representative value may be, for example, an arithmetic mean value, a geometric mean value, a harmonic mean value, a median value, or a maximum value.

[0372] The control unit 21 selects a classification model 62 with high accuracy based on the representative value, that is, a classification model 62 with a small combined loss function 643 for the test data (step S821). The control unit 21 then ends the process.

[0373] According to this embodiment, it is possible to select a classification model 62 in which machine learning has progressed in a direction in which there is no contradiction between adjacent regions.

[0374] In step S821, the control unit 21 may select a classification model 62 that has a small representative value of the composite loss function 643 and a small standard deviation of the composite loss function 643. In this way, a classification model 62 with small variations in output results can be selected.

[0375] [Embodiment 14] In this embodiment, a description of the parts common to the eighth embodiment regarding the three-dimensional display of catheter image 51 will be omitted. Figs. 45 and 46 show examples of display screens in the fourteenth embodiment. The screen examples shown in Figs. 45 and 46 include a three-dimensional image field 76 and a display area selection field 77.

[0376] The display area selection field 77 is a pull-down menu. The user operates the display area selection field 77 to select an area to be displayed in the three-dimensional image field 76. The control unit 21 constructs a three-dimensional image of the area accepted via the display area selection field 77 and displays it in the three-dimensional image field 76. In this way, the control unit 21 realizes the function of a display area selection unit that accepts the user's selection of an area to be displayed.

[0377] The user can use a cursor or the like (not shown) to appropriately manipulate the orientation of the three-dimensional image, the position of the cross section, the direction of the virtual illumination light, and the like.

[0378] Figure 45 shows an example in which the user selects a biological tissue region. In the three-dimensional image field 76 in Figure 45, the biological tissue region is displayed with the front side of the screen removed. The user can observe the three-dimensional shape of the inner surface of the biological tissue region, i.e., the inner surface of the blood vessel into which the imaging catheter 40 has been inserted.

[0379] 45, the three-dimensional shape of the medical device area inside the blood vessel is also displayed. The user can observe the shape of the medical device used simultaneously with the image acquisition catheter 40 inside the blood vessel. The control unit 21 may receive a selection from the user as to whether or not to display the medical device area.

[0380] Fig. 46 shows an example in which the user selects the first lumen region. In the three-dimensional image field 76 in Fig. 46, the three-dimensional shape of the first lumen region and the three-dimensional shape of the medical instrument region are displayed. In Fig. 46, the three-dimensional shape of the medical instrument region is shown by a dashed line, but for example, the control unit 21 displays the three-dimensional image field 76 in such a manner that the first lumen region is semi-transparent, allowing the internal medical instrument region to be seen through.

[0381] The user can observe the medical device together with the external shape of the first lumen region, i.e., the three-dimensional shape of the entire blood vessel into which the imaging catheter 40 is inserted. This allows the user to grasp the relative position of the medical device with respect to the entire blood vessel. With this display, the control unit 21 can support, for example, a catheter ablation procedure for atrial fibrillation using an ablation catheter, which is one of the medical devices.

[0382] The controller 21 may accept a selection of the second lumen region or the non-biological tissue region. The controller 21 may accept a selection of multiple regions, such as the first lumen region and the second lumen region.

[0383] Note that "display" here refers to a display state that can be seen by the user. In Fig. 45 and Fig. 46, a display mode is exemplified in which the control unit 21 displays the area selected by the user and the medical instrument area, and does not display other areas. The control unit 21 may display the area selected by the user and the medical instrument area with low transmittance, and display other areas with high transmittance. The user may be able to set the transmittance of each area as appropriate.

[0384] [Variation 14-1] Fig. 47 is an example of a display screen of Modification Example 14-1. The example screen shown in Fig. 47 includes a first three-dimensional image field 761 and a second three-dimensional image field 762. The first three-dimensional image field 761 and the second three-dimensional image field 762 are arranged in different positions on the display screen.

[0385] Fig. 47 shows an example of a screen that the control unit 21 causes the display device 31 to display via the display unit 25 when the user instructs dual screen display while Fig. 45 or 46 is displayed. A three-dimensional image similar to the three-dimensional image field 76 in Fig. 45 is displayed in the first three-dimensional image field 761, and a three-dimensional image similar to the three-dimensional image field 76 in Fig. 46 is displayed in the second three-dimensional image field 762.

[0386] When an instruction to adjust the orientation, cross-sectional position, virtual illumination light direction, etc. of one of the three-dimensional images is received from the user, the control unit 21 changes both three-dimensional images in the same way. Because the display of the first three-dimensional image field 761 and the display of the second three-dimensional image field 762 are linked, the user can compare the two with a simple operation.

[0387] Note that the control unit 21 may accept an instruction not to link the display of the first three-dimensional image field 761 with the display of the second three-dimensional image field 762. For example, the user can rotate only the second three-dimensional image field 762 while leaving the first three-dimensional image field 761 in the state shown in Fig. 47 to compare the two.

[0388] The control unit 21 may display a display area selection field 77 near each of the first three-dimensional image field 761 and the second three-dimensional image field 762. The user can select an area to be displayed in each of the first three-dimensional image field 761 and the second three-dimensional image field 762.

[0389] For example, the user can rotate one of the first three-dimensional image fields 761 and 762 while selecting the first lumen region in both fields, allowing the user to compare the three-dimensional images of the first lumen region viewed from two different directions.

[0390] The first three-dimensional image field 761 and the second three-dimensional image field 762 may be arranged vertically on one screen. Three or more three-dimensional image fields 76 may be displayed on one screen. The first three-dimensional image field 761 and the second three-dimensional image field 762 may be displayed on two display devices 31 arranged so that the user can view them simultaneously.

[0391] (Appendix A1) an image acquisition unit that acquires a catheter image obtained by an image acquisition catheter inserted into the first cavity; a first classified data output unit that, when the catheter image is input, outputs first classified data in which a non-living tissue region including a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the image acquisition catheter is not inserted and a living tissue region are classified as different regions, and inputs the acquired catheter image to a first classification trained model, and outputs the first classified data; The first classification trained model is generated using first training data that specifies at least the non-living tissue region including the first lumen region and the second lumen region, and the living tissue region. Information processing device.

[0392] (Appendix A2) a lumen region extraction unit that extracts the first lumen region and the second lumen region from the non-biological tissue region in the first classified data; a first format output unit that converts the first classification data into a format that allows the first lumen region, the second lumen region, and the biological tissue region to be distinguished from one another, and outputs the converted data. 10. The information processing device according to claim 1,

[0393] (Appendix A3) extracting a non-luminal region that is neither the first lumen region nor the second lumen region from the non-biological tissue region in the first classified data; a second format output unit that converts the first classification data into a format that allows the first lumen region, the second lumen region, the non-lumen region, and the biological tissue region to be distinguished from one another, and outputs the converted data. An information processing device according to Appendix A1 or Appendix A2.

[0394] (Appendix A4) When the catheter image is input, the first classification trained model outputs the first classification data in which the biological tissue region, the first lumen region, the second lumen region, and the non-lumen region are classified as different regions. 10. The information processing device according to claim 9, wherein the information processing device is a

[0395] (Appendix A5) the imaging catheter is a radial scanning type tomographic imaging catheter, the catheter image is an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle; The first classification data is a classification result of each pixel in the RT format image. An information processing device according to any one of Appendix A1 to Appendix A4.

[0396] (Appendix A6) The first classification trained model is It includes multiple convolutional layers, At least one of the plurality of convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle. 10. The information processing device according to claim 9, wherein the information processing device is a

[0397] (Appendix A7) When a plurality of catheter images acquired in time series are input, the first classification trained model outputs the first classification data in which the latest catheter image among the plurality of catheter images is classified into the non-biological tissue region and the biological tissue region. An information processing device according to any one of Appendix A1 to Appendix A6.

[0398] (Appendix A8) The first classification trained model is a memory unit for storing information about the catheter image previously input; The first classification data is output based on the information stored in the memory unit and the latest catheter image among the plurality of catheter images. 10. The information processing device according to claim 7,

[0399] (Appendix A9) When the catheter image is input, the first classification trained model outputs the first classification data in which the biological tissue region, the non-biological tissue region, and a medical device region indicating a medical device inserted into the first cavity or the second cavity are classified as different regions. An information processing device according to any one of Appendix A1 to Appendix A8.

[0400] (Appendix A10) a second classified data acquisition unit that inputs the acquired catheter image into a second classification trained model that outputs second classified data in which the non-biological tissue region including the first lumen region and the biological tissue region are classified as different regions when the catheter image is input, and acquires the second classified data to be output; a combined classification data output unit that outputs combined classification data obtained by combining the first classification data with the second classification data, The second classification trained model is generated using second training data in which only the first lumen region is explicitly indicated among the non-biological tissue regions. An information processing device according to any one of Appendix A1 to Appendix A9.

[0401] (Appendix A11) When the catheter image is input, the second classification trained model outputs the second classification data in which the biological tissue region, the non-biological tissue region, and a medical device region indicating a medical device inserted into the first cavity or the second cavity are classified as different regions. The information processing device according to Appendix A10.

[0402] (Appendix A12) The first classification trained model further outputs a probability that each portion of the catheter image is the biological tissue region or the non-biological tissue region; The second classification trained model further outputs a probability that each portion of the catheter image is the biological tissue region or the non-biological tissue region; The composite classification data output unit outputs composite classification data obtained by combining the first classification data with the second classification data based on a result of calculating a probability that each part of the catheter image is the biological tissue region or the non-biological tissue region. The information processing device according to claim A10 or A11.

[0403] (Appendix A13) The imaging catheter is a three-dimensional scanning catheter that sequentially acquires a plurality of catheter images along the longitudinal direction of the imaging catheter. An information processing device according to any one of Appendix A1 to Appendix A12.

[0404] (Appendix A14) a three-dimensional output unit that outputs a three-dimensional image generated based on the plurality of first classification data generated from the plurality of acquired catheter images, 10. The information processing device according to claim 13.

[0405] (Appendix A15) A catheter image is obtained by an image acquisition catheter inserted into the first cavity, The acquired catheter image is input to a first classification trained model that is generated using first training data that clearly indicates a non-living tissue region including at least a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the image acquisition catheter is not inserted, and a living tissue region, and that outputs first classified data in which the non-living tissue region and the living tissue region are classified as different regions when the catheter image is input, and outputs the first classified data. An information processing method that causes a computer to execute a process.

[0406] (Appendix A16) A catheter image is obtained by an image acquisition catheter inserted into the first cavity, The acquired catheter image is input to a first classification trained model that is generated using first training data that clearly indicates a non-living tissue region including at least a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the image acquisition catheter is not inserted, and a living tissue region, and that outputs first classified data in which the non-living tissue region and the living tissue region are classified as different regions when the catheter image is input, and outputs the first classified data. A program that causes a computer to perform a process.

[0407] (Appendix A17) acquire multiple sets of training data in which catheter images obtained by an image acquisition catheter inserted into a first cavity and label data to which multiple labels have been assigned, for each portion of the catheter image, including a biological tissue region label indicating that the portion is a biological tissue region, a first lumen region indicating that the portion is the inside of the first cavity, a second lumen region indicating that the portion is the inside of a second cavity into which the image acquisition catheter is not inserted, and a non-lunment region that is neither the first lumen region nor the second lumen region, and Using the plurality of sets of training data, a trained model is generated in which, when the catheter image is input, the catheter image is input, the label data is output, and the trained model outputs the biological tissue region label and the non-biological tissue region label for each part of the catheter image. How to generate a trained model.

[0408] (Appendix A18) the non-biological tissue region labels of the plurality of sets of training data include a first lumen region label indicating the first lumen region, a second lumen region label indicating the second lumen region, and a non-lumen region label indicating the non-lumen region; Using the plurality of sets of training data, a trained model is generated that, when the catheter image is input, uses the catheter image as input and the label data as output, and outputs the biological tissue region label, the first lumen region label, the second lumen region label, and the non-lumen region label for each part of the catheter image. A method for generating a trained model as described in Appendix A17.

[0409] (Appendix A19) acquiring a plurality of sets of training data in which a catheter image obtained by an image acquisition catheter inserted into a first cavity and label data to which a plurality of labels are assigned, the labels including a biological tissue region label indicating a biological tissue region and a non-biological tissue region label including a first lumen region indicating the inside of the first cavity, the biological tissue region label being generated based on boundary data indicating the inner boundary of the first cavity in the catheter image, are associated and recorded; Using the plurality of sets of training data, a trained model is generated in which, when the catheter image is input, the catheter image is input, the label data is output, and the trained model outputs the biological tissue region label and the non-biological tissue region label for each part of the catheter image. How to generate a trained model.

[0410] (Appendix A20) the catheter image is an RT format image in which scanning line data for one rotation obtained by the radial scanning type imaging catheter is arranged in parallel in order of scanning angle, the trained model includes a plurality of convolutional layers; At least one of the convolution layers performs padding processing to add the same data as on the side with a larger scanning angle to the outside of the side with a smaller scanning angle, and to add the same data as on the side with a smaller scanning angle to the outside of the side with a larger scanning angle, and learns by doing so. A method for generating a trained model according to any one of Appendix A17 to Appendix A19.

[0411] (Appendix B1) an image acquisition unit that acquires a catheter image obtained by a radial scanning type image acquisition catheter; a first position information output unit that inputs the acquired catheter image into a medical instrument trained model that outputs first position information regarding the position of a medical instrument included in the catheter image when the catheter image is input, and outputs the first position information; An information processing device comprising:

[0412] (Appendix B2) The first position information output unit outputs the first position information using the position of one pixel included in the catheter image. 10. The information processing device according to claim 8, wherein the information processing device is a device for processing information according to claim 1.

[0413] (Appendix B3) the first position information output unit is a first position information acquisition unit that acquires the first position information in time series corresponding to each of the plurality of catheter images acquired in time series; an exclusion unit that excludes, from the time-series first location information, the first location information that does not satisfy a predetermined condition; a complementing unit that adds complement information that satisfies a predetermined condition to the first position information in time series; The information processing device according to Supplementary Note B1 or Supplementary Note B2, comprising:

[0414] (Appendix B4) When a plurality of catheter images acquired in time series are input, the medical instrument trained model outputs the first position information regarding the latest catheter image among the plurality of catheter images. An information processing device according to any one of Appendix B1 to Appendix B3.

[0415] (Appendix B5) The medical instrument trained model is a memory unit for storing information about the catheter image previously input; The first position information is output based on the information stored in the memory unit and the latest catheter image among the plurality of catheter images. 2. The information processing device according to claim 1, wherein the information processing device is a device for processing information according to claim 1.

[0416] (Appendix B6) The medical instrument trained model is Accepting input of the catheter image as an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle; a plurality of first convolution layers; At least one of the plurality of first convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle. An information processing device according to any one of Appendix B1 to Appendix B5.

[0417] (Appendix B7) a scanning angle information acquisition unit that inputs the acquired catheter image into an angle-trained model that outputs scanning angle information related to the position of a medical instrument included in the catheter image when the catheter image is input, and acquires the output scanning angle information; a second position information output unit that outputs second position information regarding a position of a medical instrument included in the catheter image based on the first position information output from the medical instrument trained model and the scanning angle information output from the angle trained model. An information processing device according to any one of Appendix B1 to Appendix B6.

[0418] (Appendix B8) The angle-trained model is Accepting input of the catheter image as an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle; a plurality of second convolutional layers; At least one of the plurality of second convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle. 10. The information processing device according to claim 7,

[0419] (Appendix B9) The medical instrument trained model is generated using a plurality of sets of training data in which the catheter images and the positions of the medical instruments included in the catheter images are recorded in association with each other. An information processing device according to any one of Appendix B1 to Appendix B8.

[0420] (Appendix B10) The training data is displaying the catheter image obtained by the image acquisition catheter; Accepting the position of the medical instrument included in the catheter image by a single click or tap on the catheter image; The catheter image is generated by a process of storing the image in association with the position of the medical instrument. 2. The information processing device according to claim 1, wherein the information processing device is an information processing device according to claim 1.

[0421] (Appendix B11) The training data is inputting the catheter image into the medical instrument trained model; The first position information output from the medical instrument trained model is superimposed on the input catheter image and displayed; if a correction instruction regarding the position of the medical instrument included in the catheter image is not received, storing uncorrected data associating the catheter image with the first position information as the training data; When a correction instruction regarding the position of the medical instrument included in the catheter image is received, correction data that associates the catheter image with information regarding the position of the medical instrument based on the correction instruction is generated by a process of storing the correction data as the training data. 2. The information processing device according to claim 1, wherein the information processing device is an information processing device according to claim 1.

[0422] (Appendix B12) acquiring a plurality of sets of training data in which catheter images acquired by an image acquisition catheter and first position information relating to the position of a medical instrument included in the catheter images are recorded in association with each other; A trained model is generated based on the plurality of sets of training data, which outputs first position information regarding the position of a medical instrument included in the catheter image when the catheter image is input. How to generate a trained model.

[0423] (Appendix B13) The first position information is information about the position of one pixel included in the catheter image. A method for generating a trained model as described in Appendix B12.

[0424] (Appendix B14) Displaying a catheter image including a lumen obtained by the image acquisition catheter; receiving, by a single click or tap on the catheter image, first position information relating to a position of the medical instrument inserted into the lumen, the first position information being included in the catheter image; storing training data in which the catheter image and the first position information are associated with each other; A method for generating training data that causes a computer to execute a process.

[0425] (Appendix B15) The first position information is information about the position of one pixel included in the catheter image. A training data generation method as described in Appendix B14.

[0426] (Appendix B16) When the first position information is received for the catheter image, Displaying different catheter images acquired consecutively in time series 16. The training data generation method according to claim 14 or 15.

[0427] (Appendix B17) the imaging catheter is a radial scanning type tomographic imaging catheter, The catheter image is displayed by arranging two images side by side: an RT format image in which a plurality of scanning line data acquired from the imaging catheter is arranged in parallel in order of scanning angle, and an XY format image in which data based on the scanning line data is arranged radially around the imaging catheter; The first position information is received from either the RT format image or the XY format image. A training data generation method according to any one of Appendix B14 to Appendix B16.

[0428] (Appendix B18) inputting a catheter image obtained by an image acquisition catheter into a medical instrument trained model that outputs first position information regarding the position of a medical instrument included in the catheter image when the catheter image is input; The first position information output from the medical instrument trained model is superimposed on the input catheter image and displayed; if a correction instruction regarding the position of the medical instrument included in the catheter image is not received, storing uncorrected data associating the catheter image with the first position information as training data; When a correction instruction regarding the position of the medical instrument included in the catheter image is received, correction data that associates the catheter image with the received information regarding the position of the medical instrument is stored as the training data. A method for generating training data that causes a computer to execute a process.

[0429] (Appendix B19) The uncorrected data and the corrected data are data relating to the position of one pixel included in the catheter image. A training data generation method as described in Appendix B18.

[0430] (Appendix B20) A plurality of the catheter images obtained in time series are input into the medical instrument trained model in order; The output positions are superimposed on the input catheter image and displayed in order. 10. The training data generation method according to claim 8, wherein the training data generation method is a method for generating training data according to claim 8.

[0431] (Appendix B21) The position of the medical device is accepted with a single click or tap. A training data generation method according to any one of Appendix B18 to Appendix B20.

[0432] (Appendix B22) the imaging catheter is a radial scanning type tomographic imaging catheter, The catheter image is displayed by arranging two images side by side: an RT format image in which a plurality of scanning line data acquired from the imaging catheter is arranged in parallel in order of scanning angle, and an XY format image in which data based on the scanning line data is arranged radially around the imaging catheter; The position of the medical instrument is received from either the RT format image or the XY format image. A training data generation method according to any one of Appendix B18 to Appendix B21.

[0433] (Appendix C1) an image acquisition unit for acquiring a catheter image including a lumen obtained by an image acquisition catheter; a position information acquiring unit that acquires position information regarding a position of a medical instrument inserted into the lumen, the position information being included in the catheter image; a first data output unit that inputs the acquired catheter image and the acquired position information into a first trained model that outputs first data in which, when the catheter image and the position information are input, each region of the catheter image is classified into at least three regions: a biological tissue region, a medical device region where the medical device is present, and a non-biological tissue region, and outputs the first data; An information processing device comprising:

[0434] (Appendix C2) The location information acquisition unit The acquired catheter image is input into a medical instrument trained model that outputs the position information included in the catheter image when the catheter image is input, and the position information is acquired from the medical instrument trained model. 10. The information processing device according to claim 9, wherein the information processing device is a device for processing information according to claim 1.

[0435] (Appendix C3) a second data acquisition unit that, when the catheter image is input without inputting the position information, outputs second data in which each region of the catheter image is classified into at least three regions: a biological tissue region, a medical device region where the medical device is present, and a non-biological tissue region, and inputs the acquired catheter image into a second model to acquire the second data; a combined data output unit that outputs combined data obtained by combining the first data and the second data; 10. The information processing device according to claim 9, wherein the information processing device is a

[0436] (Appendix C4) The composite data output unit a first combined data output unit that outputs first combined data obtained by combining data related to the biological tissue-related regions classified as the biological tissue region and the non-biological tissue region out of the first data and the second data; a second combined data output unit that outputs second combined data obtained by combining the data related to the medical device area from the first data and the second data. 10. The information processing device according to claim 9, wherein the information processing device is a device for processing information according to claim 1.

[0437] (Appendix C5) The second combined data output unit When the position information can be acquired from the medical instrument trained model, outputting the second combined data using data related to the medical instrument region included in the first data; When the position information cannot be acquired from the medical instrument trained model, the second synthetic data is output using data related to the medical instrument region included in the second data. 10. The information processing device according to claim 9, wherein the information processing device is a

[0438] (Appendix C6) The composite data output unit outputs the second composite data obtained by combining data related to the medical device region based on weighting according to the reliability of the first data and the reliability of the second data. 10. The information processing device according to claim 9, wherein the information processing device is a

[0439] (Appendix C7) The reliability is determined based on whether the position information can be acquired from the medical instrument trained model. 10. The information processing device according to claim 9, wherein the information processing device is a

[0440] (Appendix C8) The composite data output unit If the location information can be acquired from the medical instrument trained model, the reliability of the first data is set to be higher than the reliability of the second data; When the location information cannot be acquired from the medical instrument trained model, the reliability of the first data is set lower than the reliability of the second data. 10. The information processing device according to claim 9, wherein the information processing device is a

[0441] (Appendix C9) The imaging catheter is a three-dimensional scanning catheter that sequentially acquires a plurality of catheter images along the longitudinal direction of the imaging catheter. An information processing device according to any one of appendices C1 to C8.

[0442] (Appendix C10) Acquiring a catheter image including a lumen obtained by an image acquisition catheter; acquiring position information relating to a position of a medical instrument inserted into the lumen, the position information being included in the catheter image; The acquired catheter image and the acquired position information are input to a first trained model that outputs first data in which, when the catheter image and position information relating to the position of the medical instrument included in the catheter image are input, each region of the catheter image is classified into at least three regions: a biological tissue region, a medical instrument region where the medical instrument is present, and a non-biological tissue region, and the acquired catheter image and position information are output. An information processing method that causes a computer to execute a process.

[0443] (Appendix C11) Acquiring a catheter image including a lumen obtained by an image acquisition catheter; acquiring position information relating to a position of a medical instrument inserted into the lumen, the position information being included in the catheter image; The acquired catheter image and the acquired position information are input to a first trained model that outputs first data in which, when the catheter image and position information relating to the position of the medical instrument included in the catheter image are input, each region of the catheter image is classified into at least three regions: a biological tissue region, a medical instrument region where the medical instrument is present, and a non-biological tissue region, and the acquired catheter image and position information are output. A program that causes a computer to perform a process.

[0444] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0445] In the presently disclosed embodiment, the term "catheter image" refers to a two-dimensional image obtained by an imaging catheter. In particular, when the imaging catheter is an IVUS catheter, the term "catheter image" refers to an ultrasound tomographic image, which is a two-dimensional image.

[0446] Furthermore, "medical instruments" primarily refer to long medical instruments that are inserted into blood vessels, such as Brockenbrough needles and ablation catheters. [Explanation of symbols]

[0447] 10 Catheter System 20 Information processing equipment 21 Control section 22 Main storage 23 Auxiliary storage device 24 Communications Department 25 Display section 26 Input section 27 Catheter control device 271 Catheter control unit 29 Reading unit 31 Display device 32 Input Devices 33 MDU 37 Diagnostic imaging equipment 40 Imaging catheter 41 Probe section 42 sensors 43 Shaft 44 Tip Marker 45 Connector part 46 Guidewire lumen 51 Catheter image (2D image) 518 RT format catheter image (catheter image, 2D image) 519 XY catheter images (2D images) 52 Classified data (Hint unclassified data, secondary data) 521 First Classification Data (Label Data) 522 Second Classification Data (Label Data) 523 Output classification data (label data) 526 Synthetic Classification Data 528 RT format classification data 529 XY format classification data 536 Synthetic Data 541 1st synthesis section 542 2nd Synthesis Department 543 3rd Synthesis Department 55 Three-dimensional data 551 Biological 3D Data 552 Medical equipment 3D data 561 Classification data with hints (first data) 611 Medical Instrument Trained Models 612 Angle-trained model 615 Location information synthesis section 619 Location Information Model 62 Classification Model (Second Model) 621 First Classification Trained Model 622 Second Classification Trained Model 626 Synthetic Classification Model 628 Classification Data Synthesis Unit 629 Classification Data Conversion Unit 631 Hint-trained model (first trained model) 632 Hint-free trained model (second trained model) 639 X% Hint Trained Model 641 Differential Loss Function 642 Inconsistent Loss Function 643 Composite Loss Function 65 Location information acquisition unit 66 Location classification analysis section 71 Medical equipment position training data DB 72 Hint training data DB 733 Third training data 76 3D image column 761 1st 3D Image Column 762 2nd 3D Image Column 77 Display area selection field 781 cursor 782 Control Button Area 81 Image acquisition unit 82 First classification data output section 90 Computer 96 Portable recording media 97 Programs 98 Semiconductor Memory

Claims

1. an image acquisition unit that acquires a catheter image obtained by an image acquisition catheter inserted into the first cavity; a first classified data output unit that, when the catheter image is input, outputs first classified data in which a non-living tissue region including a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the image acquisition catheter is not inserted and a living tissue region are classified as different regions; and the first classification trained model is generated using first training data that explicitly indicates at least the non-biological tissue region including the first lumen region and the second lumen region, and the biological tissue region; the imaging catheter is a radial scanning type tomographic imaging catheter, the catheter image is an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle; The first classification data is a classification result of each pixel in the RT format image. Information processing device.

2. a lumen region extraction unit that extracts the first lumen region and the second lumen region from the non-biological tissue region in the first classified data; a first format output unit that converts the first classification data into a format that allows the first lumen region, the second lumen region, and the biological tissue region to be distinguished from one another and outputs the converted data. The information processing device according to claim 1 .

3. extracting a non-luminal region that is neither the first lumen region nor the second lumen region from the non-biological tissue region in the first classified data; a second format output unit that converts the first classification data into a format that allows the first lumen region, the second lumen region, the non-lumen region, and the biological tissue region to be distinguished from one another, and outputs the converted data.

3. The information processing device according to claim 1.

4. When the catheter image is input, the first classification trained model outputs the first classification data in which the biological tissue region, the first lumen region, the second lumen region, and the non-lumen region are classified as different regions. The information processing device according to claim 3 .

5. The first classification trained model is It includes multiple convolutional layers, At least one of the plurality of convolution layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle in the RT format image, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle.

5. The information processing device according to claim 1.

6. When a plurality of catheter images acquired in time series are input, the first classification trained model outputs the first classification data obtained by classifying the latest catheter image among the plurality of catheter images into the non-biological tissue region and the biological tissue region.

6. The information processing device according to claim 1.

7. The first classification trained model is a memory unit for storing information about the catheter image previously input; The first classification data is output based on the information stored in the memory unit and the latest catheter image among the plurality of catheter images. The information processing device according to claim 6 .

8. When the catheter image is input, the first classification trained model outputs the first classification data in which the biological tissue region, the non-biological tissue region, and a medical device region indicating a medical device inserted into the first cavity or the second cavity are classified as different regions.

8. The information processing device according to claim 1.

9. a second classified data acquisition unit that inputs the acquired catheter image into a second classification trained model that outputs second classified data in which the non-biological tissue region including the first lumen region and the biological tissue region are classified as different regions when the catheter image is input, and acquires the second classified data to be output; a combined classification data output unit that outputs combined classification data obtained by combining the first classification data with the second classification data, The second classification trained model is generated using second training data in which only the first lumen region is explicitly indicated among the non-biological tissue regions.

9. The information processing device according to claim 1.

10. When the catheter image is input, the second classification trained model outputs the second classification data in which the biological tissue region, the non-biological tissue region, and a medical device region indicating a medical device inserted into the first cavity or the second cavity are classified as different regions. The information processing device according to claim 9 .

11. The first classification trained model further outputs a probability that each portion of the catheter image is the biological tissue region or the non-biological tissue region; The second classification trained model further outputs a probability that each portion of the catheter image is the biological tissue region or the non-biological tissue region; The composite classification data output unit outputs composite classification data obtained by combining the first classification data with the second classification data based on a result of calculating a probability that each part of the catheter image is the biological tissue region or the non-biological tissue region.

11. The information processing device according to claim 9 or 10.

12. The imaging catheter is a three-dimensional scanning catheter that sequentially acquires a plurality of catheter images along the longitudinal direction of the imaging catheter.

12. The information processing device according to claim 1.

13. a three-dimensional output unit that outputs a three-dimensional image generated based on the plurality of first classification data generated from the plurality of acquired catheter images, respectively; The information processing device according to claim 12.

14. the image acquisition unit is configured to acquire a plurality of two-dimensional images obtained in time series using an image acquisition catheter; a three-dimensional output unit that outputs a three-dimensional image generated based on the plurality of first classification data generated from the plurality of acquired two-dimensional images; a display area selection unit that accepts a selection of a display target area to be displayed as the three-dimensional image from at least the living tissue area and the non-living tissue area; Furthermore, The three-dimensional output unit outputs the display target area received by the display area selection unit as the three-dimensional image.

12. The information processing device according to claim 1.

15. the image acquisition unit is configured to acquire a plurality of two-dimensional images obtained in time series using an image acquisition catheter; a three-dimensional output unit that outputs a three-dimensional image generated based on the plurality of first classification data generated from the plurality of acquired two-dimensional images to a display device; The three-dimensional output unit simultaneously outputs both the three-dimensional image corresponding to the biological tissue region and the three-dimensional image corresponding to the non-biological tissue region to different locations on the display device.

12. The information processing device according to claim 1.

16. a catheter image obtained by a radial scanning type tomographic image acquisition catheter inserted into the first cavity, in which a RT format image is obtained by arranging a plurality of scanning line data acquired from the tomographic image acquisition catheter in parallel in order of scanning angle; A non-living tissue region including at least a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the tomographic image acquisition catheter is not inserted, and a living tissue region are generated using first training data that are clearly indicated in an RT format image, and when the catheter image is input, the acquired catheter image is input to a first classification trained model that outputs first classification data that classifies the non-living tissue region and the living tissue region as different regions, and outputs the first classification data that is a classification result of each pixel in the RT format image. An information processing method that causes a computer to execute a process.

17. a catheter image obtained by a radial scanning type tomographic image acquisition catheter inserted into the first cavity, in which a RT format image is obtained by arranging a plurality of scanning line data acquired from the tomographic image acquisition catheter in parallel in order of scanning angle; A non-living tissue region including at least a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the tomographic image acquisition catheter is not inserted, and a living tissue region are generated using first training data that are clearly indicated in an RT-format image, and when the catheter image is input, the first classification trained model outputs first classification data that classifies the non-living tissue region and the living tissue region as different regions. The RT-format image that is the acquired catheter image is input to the first classification trained model, and the first classification data that is the classification result of each pixel in the RT-format image is output. A program that causes a computer to perform a process.

18. acquire a plurality of sets of training data in which a catheter image is an RT format image in which a plurality of scanning line data acquired from a radial scanning type tomographic image acquisition catheter inserted into a first cavity is arranged in parallel in order of scanning angle, and label data in which a plurality of labels are assigned to each portion of the catheter image that is the RT format image, the labels including a biological tissue region label indicating that the portion is a biological tissue region, a first lumen region indicating that the portion is the inside of the first cavity, a second lumen region indicating that the portion is the inside of a second cavity into which the tomographic image acquisition catheter is not inserted, and a non-lunment region that is neither the first lumen region nor the second lumen region; Using the plurality of sets of training data, a trained model is generated in which, when the catheter image that is the RT format image is input and the label data is output, the model outputs the biological tissue region label and the non-biological tissue region label for each part of the catheter image that is the RT format image. How to generate a trained model.

19. the non-biological tissue region labels of the plurality of sets of training data include a first lumen region label indicating the first lumen region, a second lumen region label indicating the second lumen region, and a non-lumen region label indicating the non-lumen region; Using the plurality of sets of training data, a trained model is generated in which, when the catheter image that is the RT format image is input and the label data is output, the trained model outputs the biological tissue region label, the first lumen region label, the second lumen region label, and the non-lumen region label for each part of the catheter image that is the RT format image. The method for generating a trained model according to claim 18.

20. Inputting the catheter image, which is the RT format image, into a trained model during training to obtain output label data; The parameters of the trained model are adjusted using a loss function in which the value at a location where the first lumen region label and the second lumen region label are adjacent in the output label data is larger than the value at other locations. The method for generating a trained model according to claim 19.

21. a catheter image, which is an RT format image in which a plurality of scanning line data acquired from a radial scanning type catheter for acquiring a tomographic image inserted into a first cavity is arranged in parallel in order of scanning angle; and label data, which is assigned a plurality of labels including a biological tissue region label indicating a biological tissue region and a non-biological tissue region label including a first lumen region indicating the inside of the first cavity, generated based on boundary line data indicating the inner boundary line of the first cavity in the catheter image, which is the RT format image, and which is recorded in association with each other; Using the plurality of sets of training data, a trained model is generated in which, when the catheter image that is the RT format image is input and the label data is output, the model outputs the biological tissue region label and the non-biological tissue region label for each part of the catheter image that is the RT format image. How to generate a trained model.

22. The trained model includes a plurality of convolutional layers, At least one of the convolution layers performs padding processing in the RT format image, adding the same data as on the side with a larger scanning angle to the outside of the side with a smaller scanning angle, and adding the same data as on the side with a smaller scanning angle to the outside of the side with a larger scanning angle, and learns. A method for generating a trained model according to any one of claims 19 to 21.

Images