Information Processing Apparatus, Information Processing Method, and Program

Abstract

Provided are an information processing device and the like that provide assistance in understanding of an image acquired by an image acquisition catheter. This information processing device is provided with: an image acquisition unit that acquires a catheter image (518) including an internal cavity and obtained by an image acquisition catheter; a positional information acquisition unit for acquiring positional information about the position of medical equipment inserted to the internal cavity included in the catheter image (518); and a first data output unit that inputs the acquired catheter image (518) and the acquired positional information to a first trained model (631) and that outputs first data (561), the first trained model being for outputting, upon input of the catheter image (518) and the positional information, the first data (561) in which regions in the catheter image (518) are classified into at least three regions, that is to say, a biological tissue region, a medical equipment region where the medical equipment is present, and a non-biological tissue region.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] A catheter system is used in which an image acquisition catheter is inserted into a lumen organ such as a blood vessel to acquire an image (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, in a location with a complex structure such as the intracardiac region, it may be difficult to quickly understand an image acquired by an image acquisition catheter.

[0005] On one side, an object is to provide an information processing apparatus or the like that supports understanding of an image acquired by an image acquisition catheter.

Means for Solving the Problems

[0006] The information processing apparatus includes an image acquisition unit that acquires a catheter image including a lumen obtained by an image acquisition catheter, a position information acquisition unit that acquires position information regarding the position of a medical instrument inserted into the lumen included in the catheter image, and when the catheter image and the position information are input, the first learning model that classifies each region of the catheter image into at least three of a living tissue region, a medical instrument region where the medical instrument exists, and a non-living tissue region outputs first data, and a first data output unit that inputs the acquired catheter image and the acquired position information and outputs the first data.

Advantages of the Invention

[0007] On one side, it is possible to provide an information processing device or the like that supports the understanding of an image acquired by an image acquisition catheter.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32A

Figure 32B

Figure 32C

Figure 32D

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Modes for Carrying Out the Invention

[0009] [Embodiment 1] FIG. 1 is an explanatory diagram for explaining the outline of the catheter system 10. The catheter system 10 of the present embodiment is used for IVR (Interventional Radiology) that performs treatment of various organs while performing fluoroscopy using an imaging diagnostic apparatus such as an X-ray fluoroscope. By referring to the image acquired by the catheter system 10 arranged near the treatment target site, the medical instrument for treatment can be accurately operated.

[0010] The catheter system 10 includes an image acquisition catheter 40, an MDU (Motor Driving Unit) 33, and an information processing device 20. The image acquisition catheter 40 is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is an input device such as a keyboard, a mouse, a trackball, or a microphone. The display device 31 and the input device 32 may be integrally stacked to form a touch panel. The input device 32 and the information processing device 20 may be integrally configured.

[0011] FIG. 2 is an explanatory diagram for explaining the outline of the image acquisition catheter 40. The image acquisition catheter 40 has a probe unit 41 and a connector unit 45 disposed at the end of the probe unit 41. The probe unit 41 is connected to the MDU 33 via the connector unit 45. In the following description, the side far from the connector unit 45 of the image acquisition catheter 40 is described as the distal end side.

[0012] A shaft 43 is inserted inside the probe unit 41. A sensor 42 is connected to the distal end side of the shaft 43. A guide wire lumen 46 is provided at the tip of the probe unit 41. After the user inserts a guide wire to a position beyond the target site, the user inserts the guide wire through the guide wire lumen 46 to guide the sensor 42 to the target site. An annular tip marker 44 is fixed near the tip of the probe unit 41.

[0013] The sensor 42 is, for example, an ultrasonic transducer that transmits and receives ultrasonic waves, or a transmission and reception unit for OCT (Optical Coherence Tomography) that irradiates near-infrared light and receives reflected light. In the following description, the case where the image acquisition catheter 40 is an IVUS (Intravascular Ultrasound) catheter used for taking an ultrasonic tomographic image from the inside of the circulatory system will be described as an example.

[0014] FIG. 3 is an explanatory diagram for explaining the configuration of the catheter system 10. As described above, the catheter system 10 includes an information processing device 20, an MDU 33, and an image acquisition catheter 40. The information processing device 20 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

[0015] The control unit 21 is an arithmetic control device that executes the program of the present embodiment. One or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), or multi-core CPUs, etc. are used for the control unit 21. The control unit 21 is connected to each hardware part constituting the information processing device 20 via a bus.

[0016] The main storage device 22 is a storage device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. Information necessary during the processing performed by the control unit 21 and the program being executed by the control unit 21 are temporarily stored in the main storage device 22.

[0017] The auxiliary storage device 23 is a storage device such as SRAM, flash memory, hard disk, or magnetic tape. The auxiliary storage device 23 stores a medical device learned model 611, a classification model 62, a program to be executed by the control unit 21, and various data necessary for the execution of the program. The communication unit 24 is an interface for performing communication between the information processing device 20 and a network.

[0018] The display unit 25 is an interface that connects a display device 31 and a bus. The input unit 26 is an interface that connects an input device 32 and a bus. The catheter control unit 271 performs control of the MDU 33, control of the sensor 42, and generation of an image based on the signal received from the sensor 42, etc.

[0019] MDU33 rotates the sensor 42 and the shaft 43 inside the probe unit 41. The catheter control unit 271 generates one catheter image 51 (see FIG. 4) for each rotation of the sensor 42. The generated catheter image 51 is a cross-sectional image centered on the probe unit 41 and substantially perpendicular to the probe unit 41.

[0020] MDU33 can further move the sensor 42 forward and backward while rotating the sensor 42 and the shaft 43 inside the probe unit 41. By operating the sensor 42 while pulling it or pushing it in, the catheter control unit 271 continuously generates a plurality of catheter images 51 substantially perpendicular to the probe unit 41. The continuously generated catheter images 51 can be used to construct a three-dimensional image. Therefore, the catheter 40 for image acquisition realizes the function of a three-dimensional scanning catheter that sequentially acquires a plurality of catheter images 51 along the longitudinal direction.

[0021] The forward and backward operation of the sensor 42 includes both an operation of moving the entire probe unit 41 forward and backward and an operation of moving the sensor 42 forward and backward inside the probe unit 41. The forward and backward operation may be automatically performed at a predetermined speed by the MDU33 or manually performed by the user.

[0022] Note that the catheter 40 for image acquisition is not limited to a mechanical scanning method that mechanically rotates and moves forward and backward. It may be an electronic radial scanning type catheter 40 for image acquisition that uses a sensor 42 in which a plurality of ultrasonic transducers are arranged in a ring shape.

[0023] Using the catheter 40 for image acquisition, it is possible to capture a catheter image 51 including reflectors existing inside the circulatory organs such as red blood cells, and organs existing outside the circulatory organs such as the respiratory organs and digestive organs, in addition to the biological tissues constituting the circulatory organs such as the heart wall and blood vessel wall.

[0024] In this embodiment, the case where the catheter 40 for image acquisition is used for atrial septum puncture will be described as an example. In atrial septum puncture, after inserting the catheter 40 for image acquisition into the right atrium, a Brockenbrough needle is punctured into the fossa ovalis, which is the thin-walled part of the atrial septum, under ultrasonic guidance. The tip of the Brockenbrough needle reaches the inside of the left atrium.

[0025] When performing atrial septum puncture, in the catheter image 51, in addition to the biological tissues constituting the circulatory organs such as the atrial septum, the right atrium, the left atrium, and the aorta, and the reflectors such as red blood cells contained in the blood flowing inside the circulatory organs, the Brockenbrough needle is depicted. A user such as a doctor can safely perform atrial septum puncture by using the catheter image 51 to confirm the positional relationship between the fossa ovalis and the tip of the Brockenbrough needle. The Brockenbrough needle is an example of the medical device of this embodiment.

[0026] Note that the use of the catheter system 10 is not limited to atrial septum puncture. For example, the catheter system 10 can be used for procedures such as transcatheter myocardial ablation, transcatheter valve replacement, and stent placement in the coronary arteries and the like. The site where treatment is performed using the catheter system 10 is not limited to the periphery of the heart. For example, the catheter system 10 can be used for the treatment of various sites such as the pancreatic duct, the bile duct, and the blood vessels of the lower extremities.

[0027] Since the functions and configurations of the catheter control unit 271 are the same as those of the conventionally used ultrasonic diagnostic apparatus, the detailed description thereof will be omitted. Note that the control unit 21 may realize the functions of the catheter control unit 271.

[0028] The information processing apparatus 20 is connected to various imaging diagnostic apparatuses 37 such as an X-ray angiography apparatus, an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, a PET (Positron Emission Tomography) apparatus, or an ultrasonic diagnostic apparatus via an HIS (Hospital Information System) or the like.

[0029] The information processing apparatus 20 according to the present embodiment is a dedicated ultrasonic diagnostic apparatus, or a personal computer, a tablet, or a smartphone having the function of an ultrasonic diagnostic apparatus. In the following description, the case where the information processing apparatus 20 is also used for learning of a learned model such as the medical instrument learned model 611 and creation of training data will be described as an example. For learning of a learned model and creation of training data, a computer or a server different from the information processing apparatus 20 may be used.

[0030] In the following description, the case where the control unit 21 mainly performs software processing will be described as an example. The processing described using a flowchart and various learned models may be implemented by dedicated hardware, respectively.

[0031] FIG. 4 is an explanatory diagram for explaining an outline of the operation of the catheter system 10. In FIG. 4, a case where a plurality of catheter images 51 are taken while pulling the sensor 42 at a predetermined speed and the images are displayed in real time will be described as an example.

[0032] The control unit 21 takes a single catheter image 51 (step S501). The control unit 21 acquires the position information of the medical instrument depicted in the catheter image 51 (step S502). In FIG. 4, the position of the medical instrument in the catheter image 51 is indicated by a "×" mark.

[0033] The control unit 21 associates the catheter image 51, the position of the catheter image 51 in the longitudinal direction of the image acquisition catheter 40, and the position information of the medical instrument, and records them in the auxiliary storage device 23 or a large-capacity storage device connected to the HIS (step S503).

[0034] The control unit 21 generates classification data 52 obtained by classifying each part constituting the catheter image 51 for each depicted subject (step S504). In FIG. 4, the classification data 52 is shown by a schematic diagram in which the catheter image 51 is shaded based on the classification result.

[0035] The control unit 21 determines whether the user has specified a two-dimensional display or a three-dimensional display (step S505). If it is determined that the user has specified a two-dimensional display (2D in step S505), the control unit 21 displays the catheter image 51 and the classification data 52 on the display device 31 by two-dimensional display (step S506).

[0036] In step S505 of FIG. 4, it is described as if it is a selection of either "two-dimensional display" or "three-dimensional display", such as "2D / 3D". However, when the user selects "3D", the control unit 21 may display both "two-dimensional display" and "three-dimensional display".

[0037] If it is determined that the user has specified a three-dimensional display (3D in step S505), the control unit 21 determines whether the position information of the medical device sequentially recorded in step S503 is normal (step S511). If it is determined that it is not normal (NO in step S511), the control unit 21 corrects the position information (step S512). Details of the processes performed in step S511 and step S512 will be described later.

[0038] If it is determined that it is normal (YES in step S511), or after the end of step S512, the control unit 21 performs a three-dimensional display showing the structure of the observed part and the position of the medical device (step S513). As described above, the control unit 21 may display both the three-dimensional display and the two-dimensional display on one screen.

[0039] After the end of step S506 or step S513, the control unit 21 determines whether the acquisition of the catheter image 51 has been completed (step S507). For example, when receiving an end instruction from the user, the control unit 21 determines to end the process.

[0040] If it is determined not to end the process (NO in step S507), the control unit 21 returns to step S501. If it is determined to end the process (YES in step S507), the control unit 21 ends the process.

[0041] In FIG. 4, the processing flow when performing two-dimensional display (step S506) or three-dimensional display (step S513) in real time during the shooting of a series of catheter images 51 was described. The control unit 21 may perform two-dimensional display or three-dimensional display non-real-time based on the data recorded in step S503.

[0042] FIG. 5A is an explanatory diagram schematically showing the operation of the catheter 40 for image acquisition. FIG. 5B is an explanatory diagram schematically showing the catheter image 51 taken by the catheter 40 for image acquisition. FIG. 5C is an explanatory diagram schematically explaining the classification data 52 generated based on the catheter image 51. Using FIGS. 5A to 5C, the RT (Radius-Theta) format and the XY format will be explained.

[0043] As described above, the sensor 42 transmits and receives ultrasonic waves while rotating inside the catheter 40 for image acquisition. The catheter control unit 271 acquires radial scan line data centered on the catheter 40 for image acquisition, as schematically shown by eight arrows in FIG. 5A.

[0044] The catheter control unit 271 can generate the catheter image 51 shown in FIG. 5B in two formats: the RT format catheter image 518 and the XY format catheter image 519, based on the scan line data. The RT format catheter image 518 is an image generated by arranging each scan line data parallel to each other. The horizontal direction of the RT format catheter image 518 indicates the distance from the catheter 40 for image acquisition.

[0045] The vertical direction of the RT format catheter image 518 indicates the scan angle. One RT format catheter image 518 is formed by arranging the scan line data acquired by the 360-degree rotation of the sensor 42 in parallel in the order of the scan angles.

[0046] In FIG. 5B, the left side of the RT format catheter image 518 indicates a location close to the image acquisition catheter 40, and the right side of the RT format catheter image 518 indicates a location far from the image acquisition catheter 40.

[0047] The XY format catheter image 519 is an image generated by arranging and interpolating each scan line data radially. The XY format catheter image 519 shows a tomographic image obtained by cutting the subject perpendicularly to the image acquisition catheter 40 at the position of the sensor 42.

[0048] FIG. 5C schematically shows classification data 52 classified for each depicted subject with respect to each part constituting the catheter image 51. The classification data 52 can also be displayed in two formats: RT format classification data 528 and XY format classification data 529. Since the image conversion method between the RT format and the XY format is known, the description thereof is omitted.

[0049] In FIG. 5C, thick downward-right hatching indicates a biological tissue region forming a cavity into which the image acquisition catheter 40 is inserted, such as an atrial wall and a ventricular wall. Thin downward-left hatching indicates the inside of the first cavity, which is a blood flow region where the tip portion of the image acquisition catheter 40 is inserted. Thin downward-right hatching indicates the inside of the second cavity, which is a blood flow region other than the first cavity.

[0050] When performing atrial septal puncture from the right atrium to the left atrium, the first cavity is the right atrium, and the second cavity is the left atrium, the right ventricle, the left ventricle, the aorta, the coronary artery, and the like. In the following description, the inside of the first cavity is referred to as the first inner cavity region, and the inside of the second cavity is referred to as the second inner cavity region.

[0051] Thick downward-left hatching indicates a non-lumen area that is neither in the first lumen area nor in the second lumen area among non-biological tissue areas. The non-lumen area includes an extracardiac area and an area outside the heart structure, etc. When the detectable range of the image acquisition catheter 40 is small and the distal wall of the left atrium cannot be sufficiently detected, the inside of the left atrium is also included in the non-lumen area. Similarly, lumens such as the left ventricle, pulmonary artery, pulmonary vein, and aortic arch are included in the non-lumen area when the distal wall cannot be sufficiently detected.

[0052] The black coloring indicates a medical device area where a medical device such as a Brockenbrough needle is detected. In the following description, the biological tissue area and the non-biological tissue area may be collectively referred to as a biological tissue-related area.

[0053] Note that the medical device is not necessarily inserted into the same first lumen as the image acquisition catheter 40. Depending on the procedure, the medical device may be inserted into the second lumen in some cases.

[0054] The hatching and black coloring shown in Fig. 5C are examples of modes that can distinguish the respective areas. Each area is displayed on the display device 31 using, for example, different colors. The control unit 21 realizes the function of a first mode output unit that outputs the first lumen area, the second lumen area, and the biological tissue area in a distinguishable mode. The control unit 21 also realizes the function of a second mode output unit that outputs the first lumen area, the second lumen area, the non-lumen area, and the biological tissue area in a distinguishable mode.

[0055] For example, during the IVR procedure, such as when confirming the position of a Brockenbrough needle for atrial septal puncture, XY format display is suitable. However, in XY display, information near the image acquisition catheter 40 is compressed and the data volume decreases, and data that does not originally exist is added by interpolation at positions far from the image acquisition catheter 40. Therefore, when analyzing the catheter image 51, using the RT format image can obtain more accurate results than using the XY format image.

[0056] In the following description, the control unit 21 generates RT format classification data 528 based on the RT format catheter image 518. The control unit 21 converts the XY format catheter image 519 to generate the RT format catheter image 518, and converts the RT format classification data 528 to generate the XY format classification data 529.

[0057] A specific example will be given to explain the classification data 52. For the pixels classified as "biological tissue region", a "biological tissue region label" is recorded; for the pixels classified as "first lumen region", a "first lumen region label" is recorded; for the pixels classified as "second lumen region", a "second lumen region label" is recorded; for the pixels classified as "non-lumen region", a "non-lumen region label" is recorded; for the pixels classified as "medical device region", a "medical device region label" is recorded; and for the pixels classified as "non-biological tissue region", a "non-biological tissue region label" is recorded. Each label is represented by an integer, for example.

[0058] Note that the control unit 21 may generate the XY format classification data 529 based on the XY format catheter image 519. The control unit 21 may also generate the RT format classification data 528 based on the XY format classification data 529.

[0059] FIG. 6 is an explanatory diagram for explaining the configuration of the learned medical device model 611. The learned medical device model 611 is a model that receives the catheter image 51 and outputs first position information regarding the position where the medical device is depicted. The learned medical device model 611 implements step S502 described with reference to FIG. 4. The output layer of the learned medical device model 611 functions as a first position information output unit that outputs the first position information.

[0060] In FIG. 6, the input to the learned medical device model 611 is the RT format catheter image 518. The first position information is the probability that the medical device is depicted for each part on the RT format catheter image 518. In FIG. 6, the places where the probability of the medical device being depicted is high are shown with dark hatching, and the places where the probability of the medical device being depicted is low are shown without hatching.

[0061] The trained medical device model 611 is generated by machine learning using, for example, the neural network structure of a CNN (Convolutional Neural Network). Examples of CNNs that can be used to generate the trained medical device model 611 include R-CNN (Region Based Convolutional Neural Network), YOLO (You only look once), U-Net, and GAN (Generative Adversarial Network). The trained medical device model 611 may be generated using a neural network structure other than a CNN.

[0062] The trained medical device model 611 may be a model that receives a plurality of catheter images 51 acquired in a time series and outputs first position information for the latest catheter image 51. A model that receives a time series input such as an RNN (Recurrent Neural Network) can be combined with the aforementioned neural network structure to generate the trained medical device model 611.

[0063] The RNN is, for example, an LSTM (Long short-term memory). When using an LSTM, the trained medical device model 611 includes a memory unit that holds information regarding the catheter images 51 input in the past. The trained medical device model 611 outputs the first position information based on the information held in the memory unit and the latest catheter image 51.

[0064] When using a plurality of catheter images 51 acquired in a time series, the trained medical device model 611 may include a recursive input unit that inputs an output based on the catheter images 51 input in the past together with the next catheter image 51. The trained medical device model 611 outputs the first position information based on the latest catheter image 51 and the input from the recursive input unit. By using the catheter images 51 acquired in a time series, it is possible to realize a trained medical device model 611 that is less affected by image noise and the like and outputs the first position information with high accuracy.

[0065] The learned medical device model 611 may output a location where there is a high probability that a medical device is depicted, using the position of one pixel on the catheter image 51 that has received the input. For example, the learned medical device model 611 may be a model that calculates the probability that a medical device is depicted for each part on the catheter image 51 as shown in FIG. 6, and then outputs the position of the pixel with the highest probability. The learned medical device model 611 may output the position of the centroid of a region where the probability that a medical device is depicted exceeds a predetermined threshold. The learned medical device model 611 may output a region where the probability that a medical device is depicted exceeds a predetermined threshold.

[0066] Note that multiple medical devices may be used simultaneously. When multiple medical devices are depicted in the catheter image 51, it is desirable for the learned medical device model 611 to be a model that outputs the first position information for each of the multiple medical devices.

[0067] The learned medical device model 611 may be a model that outputs only the first position information of one medical device. The control unit 21 can input the RT format catheter image 518 obtained by masking the periphery of the first position information output from the learned medical device model 611 to the learned medical device model 611, and acquire the first position information of the second medical device. By repeating the same process, the control unit 21 can also acquire the first position information of the third and subsequent medical devices.

[0068] FIG. 7 is an explanatory diagram for explaining the configuration of the classification model 62. The classification model 62 is a model that receives the catheter image 51 and outputs classification data 52 obtained by classifying each part constituting the catheter image 51 for each depicted subject. The classification model 62 implements step S504 described with reference to FIG. 4.

[0069] A specific example will be given for explanation. The classification model 62 classifies each pixel constituting the input RT format catheter image 518 into, for example, a "biological tissue region", a "first lumen region", a "second lumen region", a "non-lumen region", and a "medical device region", and outputs RT format classification data 528 that associates the position of the pixel with a label indicating the classification result.

[0070] The classification model 62 may divide the catheter image 51 into regions of an arbitrary size, such as a total of 9 pixels with 3 pixels in the vertical direction and 3 pixels in the horizontal direction, and output classification data 52 obtained by classifying each region. The classification model 62 is, for example, a learned model that performs semantic segmentation on the catheter image 51. Specific examples of the classification model 62 will be described later.

[0071] FIG. 8 is an explanatory diagram for explaining the outline of the processing related to the position information. While moving the sensor 42 in the longitudinal direction of the image acquisition catheter 40, a plurality of catheter images 51 are taken. In FIG. 8, the line drawing of a substantially frustum of a cone schematically shows the biological tissue region three-dimensionally constructed based on the plurality of catheter images 51. The inside of the substantially frustum of a cone means the first lumen region.

[0072] The white circles and black circles indicate the positions of the medical devices acquired from the respective catheter images 51. Among these, since the black circle is at a position far from the white circle, it is determined to be a false detection. The shape of the medical device can be reproduced by the thick line that smoothly connects the white circles. The cross marks indicate the complementary information that complements the position information of the medical devices that could not be detected.

[0073] Details of the processing described with reference to FIG. 8 will be described in Embodiment 8. By the processing described with reference to FIG. 8, the processing of steps S511 and S512 described with reference to FIG. 4 is realized.

[0074] For example, when a medical device is in contact with a living tissue area, it is known that even if a user such as a skilled doctor or medical technician reads a single catheter image 51 in a still image state, it may be difficult to identify where the medical device is depicted. However, when observing the catheter image 51 by a video, the user can relatively easily determine the position of the medical device. This is because the user reads the image while expecting the medical device to be present at a position similar to that in the previous frame.

[0075] In the process described with reference to FIG. 8, the medical device is reconstructed so as not to cause a contradiction by using the position information of the medical device respectively acquired from a plurality of catheter images 51. By performing such a process, it is possible to accurately determine the position of the medical device in the same manner as when the user observes a video, and to realize the catheter system 10 that displays the shape of the medical device in the three-dimensional image.

[0076] According to the present embodiment, by the displays in steps S506 and S513, it is possible to provide the catheter system 10 that supports the understanding of the catheter image 51 acquired by using the image acquisition catheter 40. By using the catheter system 10 of the present embodiment, the user can accurately grasp the position of the medical device and can perform IVR safely.

[0077] [Embodiment 2] The present embodiment relates to a method for generating a learned medical device model 611. For parts common to Embodiment 1, the description will be omitted. In the present embodiment, a case where the learned medical device model 611 is generated using the information processing apparatus 20 described with reference to FIG. 3 will be described as an example.

[0078] The learned medical device model 611 may be created using a computer or the like different from the information processing apparatus 20. The learned medical device model 611 for which machine learning has been completed may be copied to the auxiliary storage device 23 via a network. The learned medical device model 611 learned on one piece of hardware can be used in a plurality of information processing apparatuses 20.

[0079] FIG. 9 is an explanatory diagram for explaining the record layout of the medical instrument position training data DB (Database) 71. The medical instrument position training data DB 71 is a database that records the catheter image 51 in association with the position information of the medical instrument, and is used for training the learned medical instrument model 611 by machine learning.

[0080] The medical instrument position training data DB 71 has a catheter image field and a position information field. In the catheter image field, a catheter image 51 such as the RT format catheter image 518 is recorded. In the catheter image field, so-called sonogram data indicating the ultrasonic signal received by the sensor 42 may be recorded. In the catheter image field, scan line data generated based on the sonogram data may be recorded.

[0081] In the position information field, the position information of the medical instrument depicted in the catheter image 51 is recorded. The position information is, for example, information indicating the position of one pixel marked by the labeler in the catheter image 51 as described later. The position information may be information indicating a circular region centered on the vicinity of the point marked by the labeler in the catheter image 51. The circle has dimensions that do not exceed the size of the medical instrument depicted in the catheter image 51. The circle has a size inscribed in a square of, for example, 50 pixels or less in length and width.

[0082] FIG. 10 is an example of a screen used for creating the medical instrument position training data DB 71. On the screen of FIG. 10, a set of catheter images 51 including the RT format catheter image 518 and the XY format catheter image 519 is displayed. The RT format catheter image 518 and the XY format catheter image 519 are images created based on the same sonogram data.

[0083] Below the catheter image 51, a control button area 782 is displayed. Above the control button area 782, the frame number of the currently displayed catheter image 51 and a jump button used when the user inputs an arbitrary frame number to jump the display are arranged.

[0084] Below the frame number and the like, various buttons used when the user performs operations such as fast forward, rewind, and frame advance are arranged. Since these buttons are the same as those commonly used in various image playback devices and the like, the description thereof is omitted.

[0085] The user in the present embodiment is a person in charge of creating training data by viewing the pre-recorded catheter image 51 and attaching a label to the position of the medical device. In the following description, the person in charge of creating training data is referred to as a labeler. The labeler is a doctor, a medical technician who is proficient in reading the catheter image 51, or a person who has received training to be able to perform accurate labeling. Further, in the following description, the operation of the labeler attaching a mark or the like to the catheter image 51 for labeling may be described as marking.

[0086] The labeler observes the displayed catheter image 51 and determines the position where the medical device is depicted. Generally, the area where the medical device is depicted is very small with respect to the entire area of the catheter image 51. The labeler moves the cursor 781 to approximately the center of the area where the medical device is depicted and performs marking by a click operation or the like. When the display device 31 is a touch panel, the labeler may perform marking by a tap operation using a finger or a stylus pen or the like. The labeler may perform marking by a so-called flick operation.

[0087] Note that the labeler may perform marking on either the RT format catheter image 518 or the XY format catheter image 519 of the catheter image 51. The control unit 21 may display a mark at the corresponding position of the other catheter image 51.

[0088] The control unit 21 creates a new record in the medical device position training data DB 71 and records the catheter image 51 in association with the position marked by the labeler. The control unit 21 displays the next catheter image 51 on the display device 31. By repeating the above processing a number of times, the medical device position training data DB 71 is created.

[0089] That is, the labeler can sequentially mark a plurality of catheter images 51 simply by performing a click operation or the like on the catheter image 51 without operating each button in the control button area 782. The operation performed by the labeler on one catheter image 51 in which one medical device is depicted is only one click operation or the like.

[0090] As described above, a plurality of medical devices may be depicted in the catheter image 51. The labeler can mark each medical device with one click operation or the like. In the following description, the case where one medical device is depicted in one catheter image 51 will be described as an example.

[0091] FIG. 11 is a flowchart for explaining the processing flow of a program for creating the medical device position training data DB 71. The case of creating the medical device position training data DB 71 using the information processing apparatus 20 will be described as an example. The program in FIG. 11 may be executed by hardware separate from the information processing apparatus 20.

[0092] Prior to the execution of the program in FIG. 11, a large number of catheter images 51 are recorded in the auxiliary storage device 23 or an external large-capacity storage device. In the following description, the case where the catheter images 51 are recorded in the auxiliary storage device 23 in the form of moving image data including a plurality of RT format catheter images 518 taken in time series will be described as an example.

[0093] The control unit 21 acquires the RT format catheter image 518 of one frame from the auxiliary storage device 23 (step S671). The control unit 21 converts the RT format catheter image 518 to generate the XY format catheter image 519 (step S672). The control unit 21 displays on the display device 31 the screen described with reference to FIG. 10 (step S673).

[0094] The control unit 21 receives an input operation of position information by the labeler via the input device 32 (step S674). Specifically, the input operation is a click operation or a tap operation on the RT format catheter image 518 or the XY format catheter image 519.

[0095] The control unit 21 displays a mark such as a small circle at the position where the input operation was received (step S675). Since the reception of the input operation on the image displayed on the display device 31 via the input device 32 and the display of the mark on the display device 31 are conventional user interfaces, the details thereof will be omitted.

[0096] The control unit 21 determines whether the image for which the input operation was received in step S674 is the RT format catheter image 518 (step S676). If it is determined that the image is the RT format catheter image 518 (YES in step S676), the control unit 21 also displays a mark at the corresponding position of the XY format catheter image 519 (step S677). If it is determined that the image is not the RT format catheter image 518 (NO in step S676), the control unit 21 also displays a mark at the corresponding position of the RT format catheter image 518 (step S678).

[0097] The control unit 21 creates a new record in the medical device position training data DB71. The control unit 21 associates the catheter image 51 with the position information input by the labeler and records it in the medical device position training data DB71 (step S679).

[0098] The catheter image 51 recorded in step S679 may be only the RT format catheter image 518 acquired in step S671, or both the RT format catheter image 518 and the XY format catheter image 519 generated in step S672. The catheter image 51 recorded in step S679 may be the acoustic ray data for one rotation received by the sensor 42, or the scanning line data generated by signal processing of the acoustic ray data.

[0099] The position information recorded in step S679 is information indicating the position of one pixel on the RT format catheter image 518 corresponding to, for example, the position where the labeler performs a click operation or the like using the input device 32. The position information may be information indicating the position where the labeler performs a click operation or the like and the surrounding range.

[0100] The control unit 21 determines whether to end the process (step S680). For example, when the processing of the catheter image 51 recorded in the auxiliary storage device 23 is completed, the control unit 21 determines to end the process. When it is determined to end (YES in step S680), the control unit 21 ends the process.

[0101] When it is determined not to end the process (NO in step S680), the control unit 21 returns to step S671. The control unit 21 acquires the next RT format catheter image 518 in step S671 and executes the processes from step S672 and below. That is, the control unit 21 automatically acquires and displays the next RT format catheter image 518 without waiting for an operation on the button displayed in the control button area 782.

[0102] Through the loop from step S671 to step S680, the control unit 21 records training data based on a large number of RT format catheter images 518 recorded in the auxiliary storage device 23 in the medical device position training data DB71.

[0103] Note that the control unit 21 may display, for example, a "Save button" on the screen described with reference to FIG. 10, and execute step S679 when the selection of the "Save button" is received. Further, the control unit 21 may display, for example, an "AUTO button" on the screen described with reference to FIG. 10, and automatically execute step S679 without waiting for the selection of the "Save button" while the selection of the "AUTO button" is being received.

[0104] In the following description, an example will be described in which the catheter image 51 recorded in the medical device position training data DB 71 in step S679 is the RT format catheter image 518, and the position information is the position of one pixel on the RT format catheter image 518.

[0105] FIG. 12 is a flowchart for explaining the processing flow of the medical device learned model 611 generation program. Prior to the execution of the program in FIG. 12, for example, an unlearned model combining a convolutional layer, a pooling layer, and a fully connected layer is prepared. As described above, the unlearned model is, for example, a CNN model. Examples of CNNs that can be used to generate the medical device learned model 611 include R-CNN, YOLO, U-Net, and GAN. The medical device learned model 611 may be generated using a neural network structure other than CNN.

[0106] The control unit 21 acquires a training record to be used for one-epoch training from the medical device position training data DB 71 (step S571). As described above, the training record recorded in the medical device position training data DB 71 is a combination of the RT format catheter image 518 and the coordinates indicating the position of the medical device depicted in the RT format catheter image 518.

[0107] When the RT format catheter image 518 is input to the input layer of the model, the control unit 21 adjusts the parameters of the model so that the position of the pixel corresponding to the position information is output from the output layer (step S572). In acquiring the training records and adjusting the parameters of the model, the program may appropriately have a function of causing the control unit 21 to accept corrections by the user, present the basis for judgment, perform additional learning, and the like.

[0108] The control unit 21 determines whether to end the process (step S573). For example, when the control unit 21 finishes learning for a predetermined number of epochs, it determines to end the process. The control unit 21 may acquire test data from the medical instrument position training data DB 71, input it to the model being machine-learned, and determine to end the process when an output with a predetermined accuracy is obtained.

[0109] If it is determined not to end the process (NO in step S573), the control unit 21 returns to step S571. If it is determined to end the process (YES in step S573), the control unit 21 records the parameters of the learned medical instrument position training data DB 71 in the auxiliary storage device 23 (step S574). Then, the control unit 21 ends the process. Through the above process, a learned medical instrument model 611 that receives the catheter image 51 and outputs the first position information is generated.

[0110] Prior to the execution of the program in FIG. 12, a model that receives time-series inputs such as an RNN may be prepared. The RNN is, for example, an LSTM. In step S572, when a plurality of RT format catheter images 518 taken in time series are input to the input layer of the model, the control unit 21 adjusts the parameters of the model so that the position of the pixel corresponding to the position information associated with the last RT format catheter image 518 in time series is output from the output layer.

[0111] FIG. 13 is a flowchart for explaining the processing flow of a program for adding data to the medical device position training data DB 71. The program in FIG. 13 is a program for adding training data to the medical device position training data DB 71 after creating the medical device learned model 611. The added training data is used for additional learning of the medical device learned model 611.

[0112] Prior to the execution of the program in FIG. 13, a large number of catheter images 51 that have not yet been used for creating the medical device position training data DB 71 are recorded in the auxiliary storage device 23 or an external large-capacity storage device. In the following description, the case where the catheter images 51 are recorded in the auxiliary storage device 23 in the form of video data including a plurality of RT format catheter images 518 taken in time series will be described as an example.

[0113] The control unit 21 acquires one frame of the RT format catheter image 518 from the auxiliary storage device 23 (step S701). The control unit 21 inputs the RT format catheter image 518 to the medical device learned model 611 to obtain the first position information (step S702).

[0114] The control unit 21 converts the RT format catheter image 518 to generate an XY format catheter image 519 (step S703). The control unit 21 displays on the display device 31 the screen described with reference to FIG. 10 with the first position information obtained in step S702 superimposed on the RT format catheter image 518 and the XY format catheter image 519, respectively (step S704).

[0115] When the labeler determines that the position of the automatically displayed mark is inappropriate, the labeler performs a single click operation or the like to input the correct position of the medical device. That is, the labeler inputs a correction instruction for the automatically displayed mark.

[0116] The control unit 21 determines whether an input operation via the input device 32 by the labeler has been received within a predetermined time (step S705). It is desirable that the predetermined time can be appropriately set by the labeler. Specifically, the input operation is, for example, a click operation or a tap operation on the RT format catheter image 518 or the XY format catheter image 519.

[0117] When it is determined that an input operation has been received (YES in step S705), the control unit 21 displays a mark such as a small circle at the position where the input operation was received (step S706). It is desirable that the mark displayed in step S706 has a different color or a different shape from the mark indicating the position information acquired in step S702. Note that the control unit 21 may erase the mark indicating the position information acquired in step S702.

[0118] The control unit 21 determines whether the image for which an input operation was received in step S705 is the RT format catheter image 518 (step S707). When it is determined that it is the RT format catheter image 518 (YES in step S707), the control unit 21 also displays a mark at the corresponding position of the XY format catheter image 519 (step S708). When it is determined that it is not the RT format catheter image 518 (NO in step S707), the control unit 21 also displays a mark at the corresponding position of the RT format catheter image 518 (step S709).

[0119] The control unit 21 creates a new record in the medical device position training data DB71. The control unit 21 records the corrected data associating the catheter image 51 with the position information input by the labeler in the medical device position training data DB71 (step S710).

[0120] When it is determined that no input operation has been received (NO in step S705), the control unit 21 creates a new record in the medical device position training data DB71. The control unit 21 records the uncorrected data associating the catheter image 51 with the first position information acquired in step S532 in the medical device position training data DB71 (step S711).

[0121] After the end of step S710 or step S711, the control unit 21 determines whether to end the process (step S712). For example, when the process of the catheter image 51 recorded in the auxiliary storage device 23 is completed, the control unit 21 determines to end the process. When it is determined to end (YES in step S712), the control unit 21 ends the process.

[0122] When it is determined not to end the process (NO in step S712), the control unit 21 returns to step S701. The control unit 21 acquires the next RT format catheter image 518 in step S701 and executes the processes from step S702 and below. Through the loop from step S701 to step S712, the control unit 21 adds training data based on a number of RT format catheter images 518 recorded in the auxiliary storage device 23 to the medical device position training data DB71.

[0123] Note that the control unit 21 may display, for example, an "OK button" for approving the output by the medical device learned model 611 on the screen described with reference to FIG. 10. When the selection of the "OK button" is received, the control unit 21 determines that an instruction to the effect of "NO" has been received in step S705 and executes step S711.

[0124] According to the present embodiment, the labeler can mark one medical device drawn in the catheter image 51 with only one operation such as a single click operation or a single tap operation. The control unit 21 may receive an operation of marking one medical device by a so-called double click operation or double tap operation. Compared with the case of marking the boundary line of the medical device, the marking work can be significantly labor-saving, so the burden on the labeler can be reduced. According to the present embodiment, a large amount of training data can be created in a short time.

[0125] According to the present embodiment, when a plurality of medical devices are drawn in the catheter image 51, the labeler can mark each medical device with a single click operation or the like.

[0126] Note that the control unit 21 may display, for example, an "OK button" on the screen described with reference to FIG. 10, and execute step S679 when the selection of the "OK button" is received.

[0127] According to the present embodiment, by superimposing and displaying the position information acquired using the medical instrument learned model 611 on the catheter image 51, additional training data can be quickly created while reducing the burden on the labeler.

[0128] [Modification Example 2-1] The medical instrument position training data DB71 may have a field for recording the type of medical instrument. In this case, on the screen described with reference to FIG. 10, the control unit 21 accepts the input of the type of medical instrument, such as "broken blow needle", "guide wire", "balloon catheter", etc.

[0129] By performing machine learning using the medical instrument position training data DB71 created in this way, a medical instrument learned model 611 that outputs the type of medical instrument in addition to the position of the medical instrument is generated.

[0130] [Embodiment 3] The present embodiment relates to a catheter system 10 that uses two learned models to acquire second position information regarding the position of a medical instrument from a catheter image 51. Descriptions of parts common to Embodiment 2 are omitted.

[0131] FIG. 14 is an explanatory diagram for explaining the depiction of a medical instrument. In FIG. 14, the medical instruments depicted in the RT format catheter image 518 and the XY format catheter image 519 are emphasized and shown.

[0132] Generally, medical instruments strongly reflect ultrasonic waves compared to biological tissues. The ultrasonic waves emitted from the sensor 42 are less likely to reach areas farther away than the medical instrument. Therefore, the medical instrument is depicted by a high echo region indicating the side closer to the image acquisition catheter 40 and a low echo region following behind it. The low echo region following behind the medical instrument is described as an acoustic shadow. In FIG. 14, the portion of the acoustic shadow is indicated by vertical line hatching.

[0133] In the RT format catheter image 518, the acoustic shadow is depicted in a horizontal straight line shape. In the XY format catheter image 519, the acoustic shadow is depicted in a fan shape. In both cases, a high brightness region is depicted at a site closer to the image acquisition catheter 40 than the acoustic shadow. Note that the high brightness region may be depicted in a so-called multiple echo pattern that regularly repeats along the scanning line direction.

[0134] Based on the scanning angle direction of the RT format catheter image 518, that is, the horizontal direction feature in FIG. 14, the scanning angle at which the medical instrument is depicted can be determined.

[0135] FIG. 15 is an explanatory diagram for explaining the configuration of the angle-trained model 612. The angle-trained model 612 is a model that receives the catheter image 51 and outputs scanning angle information regarding the scanning angle at which the medical instrument is depicted.

[0136] In FIG. 15, a schematic illustration of the angle-trained model 612 that receives the RT format catheter image 518 and outputs scanning angle information indicating the probability of the medical instrument being depicted at each scanning angle, that is, in the vertical direction of the RT format catheter image 518, is shown. Note that since the medical instrument is depicted over a plurality of scanning angles, the total of the probabilities at which the scanning angle information is output exceeds 100%. The angle-trained model 612 may extract and output the angle at which the probability of the medical instrument being depicted is high.

[0137] The angle trained model 612 is generated by machine learning. Training data that can be used to generate the angle trained model 612 can be obtained by extracting the scanning angle from among the position information from the position information field of the medical instrument position training data DB71 described with reference to FIG. 9.

[0138] Using the flowchart of FIG. 12, an overview of the process for generating the angle trained model 612 will be described. Prior to the execution of the program of FIG. 12, an untrained model such as a CNN that combines, for example, a convolutional layer, a pooling layer, and a fully connected layer is prepared. By the program of FIG. 12, each parameter of the prepared model is adjusted and machine learning is performed.

[0139] The control unit 21 acquires a training record to be used for one epoch of training from the medical instrument position training data DB71 (step S571). As described above, the training record recorded in the medical instrument position training data DB71 is a combination of the RT format catheter image 518 and the coordinates indicating the position of the medical instrument depicted in the RT format catheter image 518.

[0140] The control unit 21 adjusts the parameters of the model so that when the RT format catheter image 518 is input to the input layer of the model, the scanning angle corresponding to the position information is output from the output layer (step S572). In acquiring the training record and adjusting the parameters of the model, the program may appropriately have a function of causing the control unit 21 to execute reception of correction by the user, presentation of the basis for judgment, additional learning, and the like.

[0141] The control unit 21 determines whether to end the process (step S573). For example, when the control unit 21 has completed learning for a predetermined number of epochs, it determines to end the process. The control unit 21 may acquire test data from the medical instrument position training data DB71, input it to the model during machine learning, and determine to end the process when an output with a predetermined accuracy is obtained.

[0142] When it is determined not to end the process (NO in step S573), the control unit 21 returns to step S571. When it is determined to end the process (YES in step S573), the control unit 21 records the parameters of the learned medical device position training data DB71 in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 ends the process. Through the above process, an angle learned model 612 that receives the catheter image 51 and outputs information regarding the scanning angle is generated.

[0143] Prior to the execution of the program in FIG. 12, a model that receives a time-series input such as an RNN may be prepared. The RNN is, for example, an LSTM. In step S572, when a plurality of RT format catheter images 518 captured in time series are input to the input layer of the model, the control unit 21 adjusts the parameters of the model so that information regarding the scanning angle associated with the last RT format catheter image 518 in time series is output from the output layer.

[0144] Instead of using the angle learned model 612, the control unit 21 may determine the scanning angle at which the medical device is depicted by pattern matching.

[0145] FIG. 16 is an explanatory diagram for explaining the position information model 619. The position information model 619 is a model that receives an RT format catheter image 518 and outputs second position information indicating the position of the depicted medical device. The position information model 619 includes a medical device learned model 611, an angle learned model 612, and a position information synthesis unit 615.

[0146] The same RT format catheter image 518 is input to both the medical device learned model 611 and the angle learned model 612. First position information is output from the medical device learned model 611. As described with reference to FIG. 6, the first position information is the probability that the medical device is depicted at each part on the RT format catheter image 518. In the following description, the probability that the medical device is depicted at a position where the distance from the center of the catheter 40 for image acquisition is r and the scanning angle is θ is represented by P1(r,θ).

[0147] Scanning angle information is output from the angle learning completed model 612. The scanning angle information is the probability that a medical device is depicted at each scanning angle. In the following description, the probability that a medical device is depicted in the direction of the scanning angle θ is denoted as Pt(θ).

[0148] The first position information and the scanning angle information are combined by the position information combining unit 615 to generate second position information. Similar to the first position information, the second position information is the probability that a medical device is depicted at each part on the RT format catheter image 518. The input end of the position information combining unit 615 functions as the first position information acquisition unit and the scanning angle information acquisition unit.

[0149] Since the medical device is depicted on the RT format catheter image 518 with a certain spread, the sum of P1 and the sum of Pt may both be greater than 1. The second position information P2(r,θ) at a position where the distance from the center of the catheter 40 for image acquisition is r and the scanning angle is θ is calculated, for example, by equation (1-1). P2(r,θ)=P1(r,θ)+kPt(θ) ‥‥‥ (1-1) k is a coefficient related to the weighting between the first position information and the scanning angle information.

[0150] The second position information P2(r,θ) may be calculated by equation (1-2). P2(r,θ)=P1(r,θ)×Pt(θ) ‥‥‥ (1-2)

[0151] The second position information P2(r,θ) may be calculated by equation (1-3). Equation (1-3) is an equation for calculating the average value of the first position information and the scanning angle information. P2(r,θ)=(P1(r,θ)+Pt(θ)) / 2 ‥‥‥ (1-3)

[0152] Note that the second position information P2(r,θ) in formulas (1-1) to (1-3) is not a probability but a numerical value that relatively indicates the magnitude of the possibility that a medical device is depicted. By synthesizing the first position information and the scanning angle information, the accuracy regarding the scanning angle direction is improved. Note that the second position information may be information regarding the position where the value of P2(r,θ) is the largest. The second position information may be determined by a function other than the formulas exemplified from formula (1-1) to formula (1-3).

[0153] The second position information is an example of the position information of the medical device obtained in step S502 described with reference to FIG. 4. The learned medical device model 611, the learned angle model 612, and the position information synthesis unit 615 cooperate to implement step S502 described with reference to FIG. 4. The output end of the position information synthesis unit 615 functions as a second position information output unit that outputs the second position information based on the first position information and the scanning angle information.

[0154] FIG. 17 is a flowchart for explaining the processing flow of the program according to Embodiment 3. The flowchart described with reference to FIG. 17 shows the details of the processing of step S502 described with reference to FIG. 4.

[0155] The control unit 21 acquires a one-frame RT format catheter image 518 (step S541). The control unit 21 inputs the RT format catheter image 518 to the learned medical device model 611 to acquire the first position information (step S542). The control unit 21 inputs the RT format catheter image 518 to the learned angle model 612 to acquire the scanning angle information (step S543).

[0156] The control unit 21 calculates the second position information based on, for example, formula (1-1) or formula (1-2) (step S544). Thereafter, the control unit 21 ends the process. Thereafter, the control unit 21 uses the second position information calculated in step S544 as the position information in step S502.

[0157] According to the present embodiment, a catheter system 10 can be provided that accurately calculates the position information of a medical instrument depicted in a catheter image 51.

[0158] [Embodiment 4] The present embodiment relates to a specific example of the classification model 62 described with reference to FIG. 7. FIG. 18 is an explanatory diagram for explaining the configuration of the classification model 62. The classification model 62 includes a first classified and learned model 621 and a classification data conversion unit 629.

[0159] The first classified and learned model 621 receives an RT format catheter image 518 and outputs first classification data 521 that classifies each part constituting the RT format catheter image 518 into a "biological tissue region", a "non-biological tissue region", and a "medical instrument region". The first classified and learned model 621 further outputs the reliability of the classification result for each part, that is, the probability that the classification result is correct. The output layer of the first classified and learned model 621 functions as a first classification data output unit that outputs the first classification data 521.

[0160] The upper right figure in FIG. 18 schematically shows the first classification data 521 in RT format. The thick downward hatching indicates a biological tissue region such as an atrial wall and a ventricular wall. The black area indicates a medical instrument region where a medical instrument such as a broken blow needle is depicted. The grid-like hatching indicates a non-biological tissue region that is neither a medical instrument region nor a biological tissue region.

[0161] The first classification data 521 is converted into classification data 52 by the classification data conversion unit 629. The lower right figure in FIG. 18 schematically shows the RT format classification data 528. The non-biological tissue region is classified into three types: a first lumen region, a second lumen region, and a non-lumen region. Similar to FIG. 5C, the thin downward hatching indicates the first lumen region. The thin upward hatching indicates the second lumen region. The thick downward hatching indicates the non-lumen region.

[0162] The outline of the process performed by the classification data conversion unit 629 will be described. Among the non-biological tissue regions, the region in contact with the catheter 40 for image acquisition, that is, the rightmost region in the first classification data 521, is classified as the first lumen region. Among the non-biological tissue regions, the region surrounded by the biological tissue region is classified as the second lumen region. Note that for the classification of the second lumen region, it is desirable to make the determination in a state where the upper end and the lower end of the RT format catheter image 518 are connected to form a cylindrical shape. Among the non-biological tissue regions, the regions that are neither the first lumen region nor the second lumen region are classified as non-lumen regions.

[0163] FIG. 19 is an explanatory diagram for explaining the first training data. The first training data is used when generating the first classification trained model 621 by machine learning. In the following description, the case of creating the first training data using the information processing apparatus 20 described with reference to FIG. 3 will be described as an example. The first training data may be created using a computer or the like separate from the information processing apparatus 20.

[0164] The control unit 21 displays the two types of catheter images 51, that is, the RT format catheter image 518 and the XY format catheter image 519, on the display device 31. The labeler observes the displayed catheter image 51 and marks four types of boundary line data: "the boundary line between the first lumen region and the biological tissue region", "the boundary line between the second lumen region and the biological tissue region", "the boundary line between the non-lumen region and the biological tissue region", and "the outer contour line of the medical device region".

[0165] Note that the labeler may mark either the RT format catheter image 518 or the XY format catheter image 51. The control unit 21 displays the boundary line corresponding to the marking at the corresponding position of the other catheter image 51. As described above, the labeler can confirm both the RT format catheter image 518 and the XY format catheter image 519 and perform appropriate marking.

[0166] The labeler inputs which of the respective regions separated by the four types of marked boundary line data are the "biological tissue region", the "non-biological tissue region", and the "medical device region". Note that the control unit 21 may automatically determine the regions, and the labeler may give a correction instruction as necessary. Through the above processing, first classification data 521 is created that specifies which of the regions of the catheter image 51 are classified into the "biological tissue region", the "non-biological tissue region", and the "medical device region".

[0167] A specific example will be given to explain the first classification data 521. A "biological tissue region label" is recorded for the pixels classified into the "biological tissue region", a "first lumen region label" is recorded for the pixels classified into the "first lumen region", a "second lumen region label" is recorded for the pixels classified into the "second lumen region", a "non-lumen region label" is recorded for the pixels classified into the "non-lumen region", a "medical device region label" is recorded for the pixels classified into the "medical device region", and a "non-biological tissue region label" is recorded for the pixels classified into the "non-biological tissue region". Each label is indicated by an integer, for example. The first classification data 521 is an example of label data that associates the position of the pixel with the label.

[0168] The control unit 21 records the catheter image 51 and the first classification data 521 in association with each other. By repeating the above processing to record a large number of sets of data, a first training data DB is created. In the following description, the first training data DB in which the RT format catheter image 518 and the RT format first classification data 521 are recorded in association with each other will be used as an example for explanation.

[0169] Note that the control unit 21 may generate XY format classification data 529 based on the XY format catheter image 519. The control unit 21 may also generate RT format classification data 528 based on the XY format classification data 529.

[0170] Using the flowchart of FIG. 12, the outline of the process for generating the first classification learned model 621 will be described. Prior to the execution of the program of FIG. 12, an unlearned model such as a U-Net structure that realizes semantic segmentation, for example, is prepared.

[0171] The U-Net structure includes a multi-layer encoder layer and a multi-layer decoder layer connected behind it. Each encoder layer includes a pooling layer and a convolutional layer. Through semantic segmentation, a label is assigned to each pixel constituting the input image. Note that the unlearned model may be a Mask R-CNN model or any other model that realizes segmentation of an image.

[0172] The control unit 21 acquires training records for one epoch of training from the first training data DB (step S571). The control unit 21 adjusts the parameters of the model so that the first classification data 521 in RT format is output from the output layer when the RT format catheter image 518 is input to the input layer of the model (step S572). In acquiring the training records and adjusting the parameters of the model, the program may appropriately have a function to cause the control unit 21 to execute acceptance of corrections by the user, presentation of the basis for judgment, additional learning, and the like.

[0173] The control unit 21 determines whether to end the process (step S573). For example, when the control unit 21 has completed learning for a predetermined number of epochs, it determines to end the process. The control unit 21 may also acquire test data from the first training data DB, input it to the model during machine learning, and determine to end the process when an output with a predetermined accuracy is obtained.

[0174] When it is determined not to end the process (NO in step S573), the control unit 21 returns to step S571. When it is determined to end the process (YES in step S573), the control unit 21 records the parameters of the learned first classification learned model 621 in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 ends the process. Through the above process, a learned first classification learned model 621 that receives the catheter image 51 and outputs the first classification data 521 is generated.

[0175] Prior to the execution of the program in FIG. 12, a model for receiving time-series inputs may be prepared. The model for receiving time-series inputs includes, for example, a memory unit that holds information regarding the RT format catheter image 518 input in the past. The model for receiving time-series inputs may include a recursive input unit that inputs the output for the RT format catheter image 518 input in the past together with the next RT format catheter image 518.

[0176] By using the catheter images 51 acquired in time series, it is possible to realize a learned first classification learned model 621 that is less affected by image noise and the like and outputs the first classification data 521 with high accuracy.

[0177] The learned first classification learned model 621 may be created using a computer or the like separate from the information processing apparatus 20. The learned first classification learned model 621 for which machine learning has been completed may be copied to the auxiliary storage device 23 via a network. The learned first classification learned model 621 learned on one piece of hardware can be used in a plurality of information processing apparatuses 20.

[0178] FIG. 20 is a flowchart for explaining the processing flow of the program of Embodiment 4. The flowchart described using FIG. 20 shows the details of the processing performed by the classification model 62 described using FIG. 7.

[0179] The control unit 21 acquires the RT format catheter image 518 of one frame (step S551). The control unit 21 inputs the RT format catheter image 518 into the first classification learned model 621 to obtain the first classification data 521 (step S552). The control unit 21 extracts one continuous non-biological tissue region from the first classification data 521 (step S553). It should be noted that the processing after the extraction of the non-biological tissue region is preferably performed in a state where the upper end and the lower end of the RT format catheter image 518 are connected to form a cylindrical shape.

[0180] The control unit 21 determines whether the non-biological tissue region extracted in step S552 is the side in contact with the image acquisition catheter 40, that is, the portion in contact with the left end of the RT format catheter image 518 (step S554). If it is determined that it is the side in contact with the image acquisition catheter 40 (YES in step S554), the control unit 21 determines that the non-biological tissue region extracted in step S553 is the first lumen region (step S555).

[0181] If it is determined that it is not the portion in contact with the image acquisition catheter 40 (NO in step S554), the control unit 21 determines whether the non-biological tissue region extracted in step S552 is surrounded by the biological tissue region (step S556). If it is determined that it is surrounded by the biological tissue region (YES in step S556), the control unit 21 determines that the non-biological tissue region extracted in step S553 is the second lumen region (step S557). By steps S555 and S557, the control unit 21 realizes the function of the lumen region extraction unit.

[0182] If it is determined that it is not surrounded by the biological tissue region (NO in step S556), the control unit 21 determines that the non-biological tissue region extracted in step S553 is the non-lumen region (step S558).

[0183] After the completion of step S555, step S557, or step S558, the control unit 21 determines whether the processing of all non-biological tissue regions has been completed (step S559). If it is determined that the processing has not been completed (NO in step S559), the control unit 21 returns to step S553. If it is determined that the processing has been completed (YES in step S559), the control unit 21 ends the processing.

[0184] The control unit 21 realizes the function of the classification data conversion unit 629 through the processing from step S553 to step S559.

[0185] Note that the first classification learned model 621 may be a model that classifies the XY format catheter image 519 into a biological tissue region, a non-biological tissue region, and a medical device region. The first classification learned model 621 may be a model that classifies the RT format catheter image 518 into a biological tissue region and a non-biological tissue region. In such a case, the labeler may not perform marking regarding the medical device region.

[0186] According to the present embodiment, a first classification learned model 621 that classifies the catheter image 51 into a biological tissue region, a non-biological tissue region, and a medical device region can be generated. According to the present embodiment, a catheter system 10 that generates classification data 52 using the generated first classification learned model 621 can be provided.

[0187] [Modification Example 4-1] The labeler may input which of the regions separated by the marked four types of boundary line data are "biological tissue region", "first lumen region", "second lumen region", "non-lumen region", and "medical device region". By performing machine learning using the first training data DB created in this way, a first classification learned model 621 that classifies the catheter image 51 into a "biological tissue region", "first lumen region", "second lumen region", "non-lumen region", and "medical device region" can be generated.

[0188] As described above, without using the classification data conversion unit 629, the classification model 62 that classifies the catheter image 51 into a "biological tissue region", a "first lumen region", a "second lumen region", a "non-lumen region", and a "medical instrument region" can be realized.

[0189] [Embodiment 5] This embodiment relates to a catheter system 10 that uses a composite classification model 626 that synthesizes classification data 52 output from two classification learned models respectively. For parts common to Embodiment 4, the description will be omitted.

[0190] FIG. 21 is an explanatory diagram for explaining the configuration of the classification model 62 of Embodiment 5. The classification model 62 includes a composite classification model 626 and a classification data conversion unit 629. The composite classification model 626 includes a first classification learned model 621, a second classification learned model 622, and a classification data synthesis unit 628. Since the first classification learned model 621 is the same as that in Embodiment 4, the description thereof will be omitted.

[0191] The second classification learned model 622 is a model that receives the RT format catheter image 518 and outputs second classification data 522 obtained by classifying each part constituting the RT format catheter image 518 into a "biological tissue region", a "non-biological tissue region", and a "medical instrument region". The second classification learned model 622 further outputs the reliability of the classification result for each part, that is, the probability that the classification result is correct. Details of the second classification learned model 622 will be described later.

[0192] The classification data synthesis unit 628 synthesizes the first classification data 521 and the second classification data 522 to generate composite classification data 526. That is, the input end of the classification data synthesis unit 628 realizes the functions of the first classification data acquisition unit and the second classification data acquisition unit. The output end of the classification data synthesis unit 628 realizes the function of the composite classification data output unit.

[0193] Details of the synthetic classification data 526 will be described later. The synthetic classification data 526 is converted into classification data 52 by the classification data conversion unit 629. Since the process performed by the classification data conversion unit 629 is the same as that in the fourth embodiment, the description thereof will be omitted.

[0194] FIG. 22 is an explanatory diagram for explaining the second training data. The second training data is used when generating the second classification trained model 622 by machine learning. In the following description, the case of creating the second training data using the information processing apparatus 20 described with reference to FIG. 3 will be described as an example. The second training data may be created using a computer or the like different from the information processing apparatus 20.

[0195] The control unit 21 displays two types of catheter images 51, namely the RT format catheter image 518 and the XY format catheter image 519, on the display device 31. The labeler observes the displayed catheter image 51 and marks two types of boundary line data, namely, the "boundary line between the first lumen region and the biological tissue region" and the "outer contour line of the medical instrument region".

[0196] Note that the labeler may mark either the RT format catheter image 518 or the XY format catheter image 51. The control unit 21 displays a boundary line corresponding to the marking at the corresponding position of the other catheter image 51. As described above, the labeler can confirm both the RT format catheter image 518 and the XY format catheter image 519 and perform appropriate marking.

[0197] The labeler inputs which of the "biological tissue region", "non-biological tissue region", and "medical instrument region" each region separated by the two marked boundary line data is. Note that the control unit 21 may automatically determine the region and the labeler may give a correction instruction as necessary. Through the above process, the second classification data 522 is created, which specifies which region of the "biological tissue region", "non-biological tissue region", and "medical instrument region" each part of the catheter image 51 is classified into.

[0198] For the second classification data 522, a specific example will be given for explanation. For the pixels classified as "biological tissue region", a "biological tissue region label" is recorded, for the pixels classified as "non-biological tissue region", a "non-biological tissue region label" is recorded, and for the pixels classified as "medical instrument region", a "medical instrument region label" is recorded respectively. Each label is indicated by an integer, for example. The second classification data 522 is an example of label data associating the positions of pixels with the labels.

[0199] The control unit 21 records by associating the catheter image 51 and the second classification data 522. By repeating the above process to record a large number of sets of data, the second training data DB is created. By performing the same process as the machine learning described in Embodiment 4 using the second training data DB, the second classification learned model 622 can be generated.

[0200] Note that the second classification learned model 622 may be a model that classifies the XY format catheter image 519 into a biological tissue region, a non-biological tissue region, and a medical instrument region. The second classification learned model 622 may be a model that classifies the RT format catheter image 518 into a biological tissue region and a non-biological tissue region. In such a case, the labeler does not have to perform marking regarding the medical instrument region.

[0201] The creation of the second classification data 522 can be performed in a shorter time compared to the creation of the first classification data 521. The training of the labeler for creating the second classification data 522 can be performed in a shorter time compared to the training of the labeler for creating the first classification data 521. As described above, a larger amount of training data can be registered in the second training data DB than in the first training DB.

[0202] Since a large amount of training data can be used, a second classification learned model 622 that can identify the boundary between the first lumen region and the biological tissue region and the outer shape of the medical instrument region with higher accuracy than the first classification learned model 621 can be generated. However, since the second classification learned model 622 has not learned about non-biological tissue regions other than the first lumen region, it cannot distinguish them from the biological tissue region.

[0203] The process performed by the classification data synthesis unit 628 will be described. The same RT format catheter image 518 is input to both the first classification trained model 621 and the second classification trained model 622. The first classification data 521 is output from the medical device trained model 611. The second classification data 522 is output from the second classification trained model 622.

[0204] In the following description, as an example, a case will be described in which both the first classification trained model 621 and the second classification trained model 622 output the classified label and the confidence of the label for each pixel of the RT format catheter image 518. Note that the first classification trained model 621 and the second classification trained model 622 may output the classified label and probability for each range such as a total of 9 pixels of 3 pixels in the vertical direction and 3 pixels in the horizontal direction of the RT format catheter image 518, for example.

[0205] For a pixel whose distance from the center of the image acquisition catheter 40 is r and whose scanning angle is θ, the confidence that the first classification trained model 621 is a biological tissue region is represented by Q1t(r,θ). Note that for a pixel classified by the first classification trained model 621 into a region other than the biological tissue region, Q1t(r,θ) = 0.

[0206] Similarly, for a pixel whose distance from the center of the image acquisition catheter 40 is r and whose scanning angle is θ, the confidence that the second classification trained model 622 is a biological tissue region is represented by Q2t(r,θ). Note that for a pixel classified by the second classification trained model 622 into a region other than the biological tissue region, Q2t(r,θ) = 0.

[0207] The classification data synthesis unit 628 calculates a synthesis value Qt(r,θ) based on, for example, equation (5-1). Note that Qt(r,θ) is not the probability that the classification as a biological tissue region is correct, but a numerical value that relatively indicates the magnitude of the confidence that it is a biological tissue region. Qt(r,θ)=Q1t(r,θ)×Q2t(r,θ) ‥‥‥(5-1) The classification data synthesis unit 628 classifies pixels where Qt(r, θ) is 0.5 or more as the living tissue region.

[0208] Similarly, the first pre-trained classification model 621 indicates the reliability that it is a medical instrument region by Q1c(r, θ), and the second pre-trained classification model 622 indicates the reliability that it is a medical instrument region by Q2c(r, θ).

[0209] The classification data synthesis unit 628 calculates a composite value Qc(r, θ) based on, for example, Equation (5-2). Note that Qc(r, θ) is not the probability that the classification that it is a medical instrument region is correct, but a numerical value that relatively indicates the magnitude of the reliability that it is a medical instrument region. Qc(r, θ)=Q1c(r, θ)×Q2c(r, θ) ‥‥‥(5-2)

[0210] The classification data synthesis unit 628 classifies pixels where Qc(r, θ) is 0.5 or more as the medical instrument region. The classification data synthesis unit 628 classifies pixels that have not been classified into either the medical instrument region or the living tissue region as the non-living tissue region. As described above, the classification data synthesis unit 628 generates the composite classification data 526 obtained by synthesizing the first classification data 521 and the second classification data 522. The composite classification data 526 is converted into the RT format classification data 528 by the classification data conversion unit 629.

[0211] Note that Equations (5-1) and (5-2) are examples. The threshold value when the classification data synthesis unit 628 performs classification is also an example. The classification data synthesis unit 628 may be a pre-trained model that receives the first classification data 521 and the second classification data 522 and outputs the composite classification data 526.

[0212] The first classification data 521 may be input to the classification data synthesis unit 628 after being classified into the "living tissue region", "first lumen region", "second lumen region", "non-lumen region", and "medical instrument region" by the classification data conversion unit 629 described in Embodiment 4.

[0213] The first classification-trained model 621 may be a model that classifies the catheter image 51 described in Modification 4-1 into a "biological tissue region", a "first lumen region", a "second lumen region", a "non-lumen region", and a "medical instrument region".

[0214] When data in which a non-biological tissue region has been classified into a "first lumen region", a "second lumen region", and a "non-lumen region" is input to the classification data synthesis unit 628, the classification data synthesis unit 628 can output combined classification data 526 that has been classified into a "biological tissue region", a "first lumen region", a "second lumen region", a "non-lumen region", and a "medical instrument region". In such a case, it is not necessary to input the combined classification data 526 to the classification data conversion unit 629 and convert it into RT format classification data 528.

[0215] FIG. 23 is a flowchart for explaining the processing flow of the program according to Embodiment 5. The flowchart described using FIG. 23 shows the details of the processing performed by the classification model 62 described using FIG. 7.

[0216] The control unit 21 acquires one-frame RT format catheter image 518 (step S581). By step S581, the control unit 21 realizes the function of the image acquisition unit. The control unit 21 inputs the RT format catheter image 518 to the first classification-trained model 621 and acquires first classification data 521 (step S582). The control unit 21 inputs the RT format catheter image 518 to the second classification-trained model 622 and acquires second classification data 522 (step S583).

[0217] The control unit 21 activates a classification synthesis subroutine (step S584). The classification synthesis subroutine is a subroutine that synthesizes the first classification data 521 and the second classification data 522 to generate combined classification data 526. The processing flow of the classification synthesis subroutine will be described later.

[0218] The control unit 21 extracts one continuous non-biological tissue region from the composite classification data 526 (step S585). Note that it is desirable that the processing after the extraction of the non-biological tissue region be performed in a state where the upper end and the lower end of the RT format catheter image 518 are connected to form a cylindrical shape.

[0219] The control unit 21 determines whether or not the non-biological tissue region extracted in step S585 is on the side in contact with the image acquisition catheter 40 (step S554). Thereafter, since the processing up to step S559 is the same as the processing flow of the program of Embodiment 4 described with reference to FIG. 20, the description thereof is omitted.

[0220] The control unit 21 determines whether or not the processing of all non-biological tissue regions has been completed (step S559). If it is determined that the processing has not been completed (NO in step S559), the control unit 21 returns to step S585. If it is determined that the processing has been completed (YES in step S559), the control unit 21 ends the processing.

[0221] FIG. 24 is a flowchart for explaining the processing flow of the classification synthesis subroutine. The classification synthesis subroutine is a subroutine that synthesizes the first classification data 521 and the second classification data 522 to generate the composite classification data 526.

[0222] The control unit 21 selects a pixel to be processed (step S601). The control unit 21 acquires the reliability Q1t(r,θ) indicating that the pixel being processed is a biological tissue region from the first classification data 521 (step S602). The control unit 21 acquires the reliability Q2t(r,θ) indicating that the pixel being processed is a biological tissue region from the second classification data 522 (step S603).

[0223] The control unit 21 calculates a composite value Qt(r,θ) based on, for example, equation (5-1) (step S604). The control unit 21 determines whether or not the composite value Qt(r,θ) is equal to or greater than a predetermined threshold value (step S605). The predetermined threshold value is, for example, 0.5.

[0224] When it is determined that the value is equal to or greater than a predetermined threshold (YES in step S605), the control unit 21 classifies the pixel being processed as a "biological tissue region" (step S606). When it is determined that the value is less than the predetermined threshold (NO in step S605), the control unit 21 acquires the reliability Q1c(r,θ) indicating that the pixel being processed is in the medical instrument region from the first classification data 521 (step S611). The control unit 21 acquires the reliability Q2c(r,θ) indicating that the pixel being processed is in the medical instrument region from the second classification data 522 (step S612).

[0225] The control unit 21 calculates a composite value Qc(r,θ) based on, for example, equation (5-2) (step S613). The control unit 21 determines whether or not the composite value Qc(r,θ) is equal to or greater than a predetermined threshold (step S614). The predetermined threshold is, for example, 0.5.

[0226] When it is determined that the value is equal to or greater than the predetermined threshold (YES in step S614), the control unit 21 classifies the pixel being processed as a "medical instrument region" (step S615). When it is determined that the value is less than the predetermined threshold (NO in step S614), the control unit 21 classifies the pixel being processed as a "non-biological tissue region" (step S616).

[0227] After the completion of step S606, step S615, or step S616, the control unit 21 determines whether or not the processing of all pixels has been completed (step S607). When it is determined that the processing has not been completed (NO in step S607), the control unit 21 returns to step S601. When it is determined that the processing has been completed (YES in step S607), the control unit 21 ends the processing. The control unit 21 realizes the function of the classification data synthesis unit 628 through a classification synthesis subroutine.

[0228] According to this embodiment, a catheter system 10 can be provided that generates RT format classification data 528 using combined classification data 526 obtained by combining classification data 52 output from two separately trained classification models. By using in combination a second pre-trained classification model 622 for which a relatively large amount of training data can be collected relatively easily to improve classification accuracy and a first pre-trained classification model 621 for which collection of training data is laborious, a catheter system 10 can be provided that achieves a good balance between the generation cost of the pre-trained model and the classification accuracy.

[0229] [Embodiment 6] This embodiment relates to a catheter system 10 that classifies each part constituting a catheter image 51 using the position information of a medical instrument as a hint. Descriptions of parts common to Embodiment 1 are omitted.

[0230] FIG. 25 is an explanatory diagram for explaining the configuration of a hint-using pre-trained model 631. The hint-using pre-trained model 631 is used in step S604 described with reference to FIG. 4 instead of the classification model 62 described with reference to FIG. 7.

[0231] The hint-using pre-trained model 631 is a model that receives an RT format catheter image 518 and the position information of a medical instrument depicted in the RT format catheter image 518, and outputs hint-using classification data 561 obtained by classifying each part constituting the RT format catheter image 518 into a "biological tissue region", a "non-biological tissue region", and a "medical instrument region". The first pre-trained classification model 621 further outputs, for each part, the reliability of the classification result, that is, the probability that the classification result is correct.

[0232] FIG. 26 is an explanatory diagram for explaining the record layout of a hint-using model training data DB72. The hint-using training data DB72 is a database that records by associating a catheter image 51, the position information of a medical instrument depicted in the catheter image 51, and classification data 52 obtained by classifying each part constituting the catheter image 51 for each depicted subject.

[0233] The classification data 52 is data created by the labeler based on the procedure described using, for example, FIG. 19. By performing the same processing as the machine learning described in Embodiment 4 using the hint-provided training data DB 72, a hint-provided learned model 631 can be generated.

[0234] FIG. 27 is a flowchart for explaining the processing flow of the program of Embodiment 6. The flowchart described using FIG. 27 shows the details of the processing performed in step S504 described using FIG. 4.

[0235] The control unit 21 acquires an RT format catheter image 518 of one frame (step S621). The control unit 21 inputs the RT format catheter image 518 into the medical device learned model 611 described using, for example, FIG. 6 to acquire the position information of the medical device (step S622). The control unit 21 inputs the RT format catheter image 518 and the position information into the hint-provided learned model 631 to acquire hint-provided classification data 561 (step S623).

[0236] The control unit 21 extracts one continuous non-biological tissue region from the hint-provided classification data 561 (step S624). It should be noted that the processing after the extraction of the non-biological tissue region is preferably performed in a state where the upper end and the lower end of the RT format catheter image 518 are connected to form a cylindrical shape.

[0237] The control unit 21 determines whether or not the non-biological tissue region extracted in step S624 is on the side in contact with the image acquisition catheter 40 (step S554). Thereafter, the processing up to step S559 is the same as the processing flow of the program of Embodiment 4 described using FIG. 20, and thus the description thereof is omitted.

[0238] The control unit 21 determines whether or not the processing of all non-biological tissue regions has been completed (step S559). If it is determined that the processing has not been completed (NO in step S559), the control unit 21 returns to step S624. If it is determined that the processing has been completed (YES in step S559), the control unit 21 ends the processing.

[0239] According to the present embodiment, by inputting the position information of the medical instrument as a hint, the catheter system 10 that can accurately generate the classification data 52 can be provided.

[0240] [Modification Example 6-1] FIG. 28 is a flowchart for explaining the flow of the processing of the program of the modification example. The processing described using FIG. 28 is executed instead of the processing described using FIG. 27.

[0241] The control unit 21 acquires one-frame RT format catheter image 518 (step S621). The control unit 21 acquires the position information of the medical instrument (step S622). The control unit 21 determines whether or not the acquisition of the position information of the medical instrument has been successful (step S631). For example, when the reliability output from the medical instrument learned model 611 is higher than the threshold value, the control unit 21 determines that the acquisition of the position information has been successful.

[0242] Note that "success" in step S631 means that the medical instrument is depicted in the RT format catheter image 518 and the control unit 21 can acquire the position information of the medical instrument with a reliability higher than the threshold value. The case of "not successful" includes, for example, the case where the medical instrument does not exist in the imaging range of the RT format catheter image 518 and the case where the medical instrument is in close contact with the surface of the biological tissue region and is not clearly depicted.

[0243] When it is determined that the acquisition of the position information has been successful (YES in step S631), the control unit 21 inputs the RT format catheter image 518 and the position information into the hint-available learned model 631 to obtain hint-available classification data 561 (step S623). When it is determined that the acquisition of the position information has not been successful (NO in step S631), the control unit 21 inputs the RT format catheter image 518 into the hint-unavailable unlearned model 632 to obtain hint-unavailable classification data (step S632).

[0244] The hint-unavailable unlearned model 632 is the classification model 62 described using, for example, FIG. 7, FIG. 18, or FIG. 21. Similarly, the hint-unavailable classification data is the classification data 52 output from the classification model 62.

[0245] After the end of step S623 or step S632, the control unit 21 extracts one continuous non-biological tissue region from the hint-available classification data 561 or the classification model 62 (step S624). Since the subsequent processing is the same as the processing flow described using FIG. 27, the description is omitted.

[0246] The hint-available classification data 561 is an example of the first data. The hint-available learned model 631 is an example of a first learned model that outputs the first data when the catheter image 51 and the position information of the medical instrument are input. The output layer of the hint-available learned model 631 is an example of a first data output unit that outputs the first data.

[0247] The hint-unavailable classification data is an example of the second data. The hint-unavailable unlearned model 632 is an example of a second learned model and a second model that outputs the second data when the catheter image 51 is input. The output layer of the hint-unavailable unlearned model 632 is an example of a second data output unit.

[0248] According to this modification, when the acquisition of the position information has not been successful, the classification model 62 that does not require the input of the position information is used. Therefore, it is possible to provide the catheter system 10 that prevents malfunction due to inputting incorrect hints into the hint-available learned model 631.

[0249] [Embodiment 7] This embodiment relates to a catheter system 10 that generates synthetic data 536 by synthesizing the output of the hint-trained model 631 and the output of the hint-untrained model 632. Descriptions of parts common to Embodiment 6 are omitted. Note that the synthetic data 536 is data used instead of the classification data 52 that is the output of step S504 described using FIG. 4.

[0250] FIG. 29 is an explanatory diagram for explaining the configuration of the classification model 62 of Embodiment 7. The classification model 62 includes a position classification analysis unit 66 and a third synthesis unit 543. The position classification analysis unit 66 includes a position information acquisition unit 65, a hint-trained model 631, a hint-untrained model 632, a first synthesis unit 541, and a second synthesis unit 542.

[0251] The position information acquisition unit 65 acquires position information indicating the position where the medical device is depicted from, for example, the medical device trained model 611 described using FIG. 6 or the position information model 619 described using FIG. 16. Since the hint-trained model 631 is the same as that in Embodiment 6, the description thereof is omitted. The hint-untrained model 632 is, for example, the classification model 62 described using FIG. 7, FIG. 18, or FIG. 21.

[0252] The operation of the first synthesis unit 541 will be described. The first synthesis unit 541 synthesizes the hint-classified data 561 output from the hint-trained model 631 and the hint-unclassified data output from the hint-untrained model 632 to create classification information. The input end of the first synthesis unit 541 functions as a first data acquisition unit that acquires the hint-classified data 561 and a second data acquisition unit that acquires the hint-unclassified data. The output end of the first synthesis unit 541 functions as a first synthesis data output unit that outputs the first synthesis data obtained by synthesizing the hint-classified data 561 and the hint-unclassified data.

[0253] When data in which the non-biological tissue area is not classified into the first lumen area, the second lumen area, and the non-lumen area is input, the first synthesizing unit 541 functions as the classification data conversion unit 629 to classify the non-biological tissue area.

[0254] For example, when the position information acquisition unit 65 succeeds in acquiring the position information, the first synthesizing unit 541 makes the weight of the learned model 631 with hint larger than the weight of the unlearned model 632 without hint, and synthesizes the two. Since the method of performing weighted synthesis of images is well-known, the description thereof is omitted.

[0255] The first synthesizing unit 541 may determine the weighting between the classified data 561 with hint and the unclassified data without hint based on the reliability of the position information acquired by the position information acquisition unit 65, and synthesize them.

[0256] The first synthesizing unit 541 may synthesize the classified data 561 with hint and the unclassified data without hint based on the reliability of each area of the classified data 561 with hint and the unclassified data without hint. The synthesis based on the reliability of the classification data 52 can be executed by the same processing as the classification data synthesizing unit 628 described in the fifth embodiment, for example.

[0257] Note that the first synthesizing unit 541 treats the medical instrument area output from the learned model 631 with hint and the unlearned model 632 without hint in the same manner as the adjacent non-biological tissue area. For example, when a medical instrument area exists in the first lumen area, the first synthesizing unit 541 treats the medical instrument area in the same manner as the first lumen area. Similarly, when a medical instrument area exists in the second lumen area, the first synthesizing unit 541 treats the medical instrument area in the same manner as the second lumen area.

[0258] A learned model that does not output a medical instrument area may be used for either the learned model 631 with hint or the unlearned model 632 without hint. Therefore, as shown in the central part of FIG. 29, the classification information output from the first synthesizing unit 541 does not include information regarding the medical instrument area.

[0259] The first synthesizing unit 541 may function as a switch that switches between hint-present classification data 561 and hint-absent classification data based on whether the position information acquisition unit 65 has successfully acquired position information. The first synthesizing unit 541 may further function as a classification data conversion unit 629.

[0260] Specifically, when the position information acquisition unit 65 has successfully acquired position information, the first synthesizing unit 541 outputs classification information based on the hint-present classification data 561 output from the hint-present learned model 631. When the position information acquisition unit 65 has not successfully acquired position information, the first synthesizing unit 541 outputs classification information based on the hint-absent classification data output from the hint-absent unlearned model 632.

[0261] The operation of the second synthesizing unit 542 will be described. When the position information acquisition unit 65 has successfully acquired position information, the second synthesizing unit 542 outputs the medical device area output from the hint-present learned model 631. When the position information acquisition unit 65 has not successfully acquired position information, the second synthesizing unit 542 outputs the medical device area included in the hint-absent classification data.

[0262] It is desirable to use the second classification learned model 622 described with reference to FIG. 21 for the hint-absent unlearned model 632. As described above, since a large number of training data can be used for the training of the second classification learned model 622, the medical device area can be accurately extracted.

[0263] When the position information acquisition unit 65 has not successfully acquired position information, the second synthesizing unit 542 may synthesize and output the medical device area included in the hint-present classification data 561 and the medical device area included in the hint-absent classification data. The synthesis of the hint-present classification data 561 and the hint-absent classification data can be performed by the same process as the classification data synthesis unit 628 described in Embodiment 5, for example.

[0264] The output terminal of the second synthesizing unit 542 functions as a second synthesized data output unit that outputs second synthesized data obtained by synthesizing the medical device area of the hint-present classification data 561 and the medical device area of the hint-absent classification data.

[0265] The operation of the third combining unit 543 will be described. The third combining unit 543 outputs combined data 536 in which the medical instrument area output from the second combining unit 542 is superimposed on the classification information output from the first combining unit 541. In FIG. 29, the superimposed medical instrument area is shown in black.

[0266] Instead of the first combining unit 541, the third combining unit 543 may function as a classification data conversion unit 629 that classifies the non-biological tissue area into a first lumen area, a second lumen area, and a non-lumen area.

[0267] Some or all of the plurality of learned models constituting the position classification analysis unit 66 may be models that receive a plurality of catheter images 51 acquired in time series and output information for the latest catheter image 51.

[0268] According to the present embodiment, a catheter system 10 can be provided that accurately acquires the position information of a medical instrument and outputs it in combination with classification information. After generating the combined data 536 based on each of the plurality of catheter images 51 continuously captured along the longitudinal direction of the image acquisition catheter 40, the control unit 21 may construct three-dimensional data of the biological tissue and the medical instrument by stacking the combined data 536 and display it.

[0269] [Modification Example 7-1] FIG. 30 is an explanatory diagram for explaining the configuration of the classification model 62 of the modification example. An X% hint learned model 639 is added to the position classification analysis unit 66. The X% hint learned model 639 is a model that is trained under the condition that position information is input for X percent of the training data and position information is not input for (100 - X) percent when learning is performed using the hint-containing training data DB 72. In the following description, the data output from the X% hint learned model 639 is referred to as X% hint classification data.

[0270] The X%-hint learned model 639 is identical to the hint-trained learned model 631 when X is "100", and is identical to the hint-untrained learned model 632 when X is "0". X is, for example, "50".

[0271] The first synthesis unit 541 outputs data obtained by synthesizing the classification data 52 respectively acquired from the hint-trained learned model 631, the hint-untrained learned model 632, and the X%-hint learned model 639 based on a predetermined weighting. The weighting changes depending on whether the position information acquisition unit 65 has successfully acquired the position information.

[0272] For example, when the position information acquisition unit 65 has successfully acquired the position information, the output of the hint-trained learned model 631 and the output of the X%-hint learned model 639 are synthesized. When the position information acquisition unit 65 has failed to acquire the position information, the output of the hint-untrained learned model 632 and the output of the X%-hint learned model 639 are synthesized. The weighting at the time of synthesis may change based on the reliability of the position information acquired by the position information acquisition unit 65.

[0273] The position classification analysis unit 66 may include a plurality of X%-hint learned models 639. For example, the X%-hint learned model 639 with X being "20" and the X%-hint learned model 639 with X being "50" can be combined and used.

[0274] In the clinical field, it may not be possible to extract the medical device area from the catheter image 51. For example, when the medical device is not inserted into the first cavity, and when the medical device is in close contact with the surface of the living tissue. According to this modification example, a classification model 62 that matches the actual situation in such a clinical field can be realized. Therefore, a catheter system 10 that can accurately detect and classify position information can be provided.

[0275] [Embodiment 8] This embodiment relates to the three-dimensional display of the catheter image 51. For parts common to Embodiment 7, the description will be omitted. FIG. 31 is an explanatory diagram for explaining the outline of the processing of Embodiment 8.

[0276] In this embodiment, a plurality of RT format catheter images 518 continuously captured along the longitudinal direction of the image acquisition catheter 40 are used. The control unit 21 inputs the plurality of RT format catheter images 518 into the position classification analysis unit 66 described in Embodiment 7 respectively. From the position classification analysis unit 66, classification information corresponding to each RT format catheter image 518 and a medical device region are output. The control unit 21 inputs the classification information and the medical device information into the third synthesis unit 543 to synthesize the synthesis data 536.

[0277] Based on the plurality of synthesis data 536, the control unit 21 creates a biological three-dimensional data 551 showing the three-dimensional structure of the biological tissue. The biological three-dimensional data 551 is, for example, voxel data in which values indicating a biological tissue label, a first lumen region label, a second lumen region label, a non-lumen region label, etc. are recorded for each volume lattice in a three-dimensional space. The biological three-dimensional data 551 may be polygon data composed of a plurality of polygons showing the boundaries of the respective regions. Since the method of creating the three-dimensional data 55 based on a plurality of RT format data is well-known, the description will be omitted.

[0278] The control unit 21 acquires position information indicating the position of the medical device depicted in each RT format catheter image 518 from the position information acquisition unit 65 included in the position classification analysis unit 66. Based on the plurality of position information, the control unit 21 creates medical device three-dimensional data 552 showing the three-dimensional shape of the medical device. Details of the medical device three-dimensional data 552 will be described later.

[0279] The control unit 21 synthesizes the biological three-dimensional data 551 and the medical instrument three-dimensional data 552 to generate the three-dimensional data 55. The three-dimensional data 55 is used for the "3D display" in step S513 described with reference to FIG. 4. When synthesizing the three-dimensional data 55, the control unit 21 replaces the medical instrument region included in the synthesized data 536 with a blank region or a non-biological region, and then synthesizes the medical instrument three-dimensional data 552. The control unit 21 may generate the biological three-dimensional data 551 using the classification information output from the first synthesis unit 541 included in the position classification analysis unit 66.

[0280] FIGS. 32A to 32D are explanatory diagrams for explaining the outline of the correction process of the position information. FIGS. 32A to 32D are schematic diagrams showing, in chronological order, a state in which the catheter image 51 is being taken while pulling the catheter for image acquisition 40 in the right direction in the figure. The thick cylinder schematically shows the inner surface of the first lumen.

[0281] In FIG. 32A, three catheter images 51 have been taken. The position information of the medical instrument extracted from each catheter image 51 is indicated by white circles. FIG. 32B shows a state in which the fourth catheter image 51 is being taken. The position information of the medical instrument extracted from the fourth catheter image 51 is indicated by black circles.

[0282] The medical instrument is detected at a location clearly different from the three previously taken catheter images 51. Generally, medical instruments used in IVR have a certain degree of rigidity and are unlikely to bend suddenly. Therefore, the position information indicated by the black circles is highly likely to be a false detection.

[0283] In FIG. 32C, two more catheter images 51 have been taken. The position information of the medical instrument extracted from each catheter image 51 is indicated by white circles. The five white circles are arranged almost in a line along the longitudinal direction of the catheter 40 for image acquisition, while the black circles are far apart, clearly indicating a false detection.

[0284] In FIG. 32D, the position information complemented based on the five white circles is indicated by a cross mark. By using the position information indicated by the cross mark instead of the position information indicated by the black circle, the shape of the medical instrument in the first cavity can be correctly displayed in the three-dimensional image.

[0285] In addition, when the position information acquisition unit 65 fails to acquire the position information, the control unit 21 may use the representative point of the medical instrument area acquired from the second composition unit 542 included in the position classification analysis unit 66 as the position information. For example, the center of gravity of the medical instrument area can be used as the representative point.

[0286] FIG. 33 is a flowchart for explaining the processing flow of the program according to Embodiment 8. The program described with reference to FIG. 33 is a program executed when it is determined in step S505 described with reference to FIG. 4 that the user has specified three-dimensional display (3D in step S505).

[0287] The program of FIG. 33 can be executed while a plurality of catheter images 51 are being taken along the longitudinal direction of the image acquisition catheter 40. Prior to the execution of the program of FIG. 33, classification information and position information have been generated for each of the captured catheter images 51 and are stored in the auxiliary storage device 23 or an external large-capacity storage device. This case will be described as an example.

[0288] The control unit 21 acquires the position information corresponding to one catheter image 51 and records it in the main storage device 22 or the auxiliary storage device 23 (step S641). Note that the control unit 21 processes the catheter images 51 in order from the catheter image 51 stored earlier among the series of catheter images 51. In step S641, the control unit 21 may acquire and record the position information from the first few catheter images 51 among the series of catheter images 51.

[0289] The control unit 21 acquires the position information corresponding to the following single catheter image 51 (step S642). In the following description, the position information during processing is referred to as the first position information. The control unit 21 extracts the position information that is closest to the first position information from among the position information acquired in step S641 and past steps S641 (step S643). In the following description, the position information extracted in step S643 is referred to as the second position information.

[0290] Note that in step S642, the distances between the position information are compared in a state where a plurality of catheter images 51 are projected onto a single plane orthogonal to the image acquisition catheter 40. That is, when extracting the second position information, the distance in the longitudinal direction of the image acquisition catheter 40 is not considered.

[0291] The control unit 21 determines whether the distance between the first position information and the second position information is equal to or less than a predetermined threshold value (step S644). The threshold value is, for example, 3 millimeters. If it is determined that the value is equal to or less than the threshold value (YES in step S644), the control unit 21 records the second position information in the main storage device 22 or the auxiliary storage device 23 (step S645).

[0292] If it is determined that the threshold value is exceeded (NO in step S644), or after the end of step S645, the control unit 21 determines whether the processing of the recorded position information has ended (step S646). If it is determined that the processing has not ended (NO in step S646), the control unit 21 returns to step S642.

[0293] The position information indicated by the black circles in FIG. 32 is an example of the position information determined to exceed the threshold value in step S644. The control unit 21 ignores such position information without recording it in step S645. The control unit 21 realizes the function of an exclusion unit that excludes position information that does not satisfy a predetermined condition by the processing when the determination in step S644 is NO. Note that the control unit 21 may record a flag indicating "error" for the position information determined to exceed the threshold value in step S644.

[0294] When it is determined that the process has ended (YES in step S646), the control unit 21 determines whether it is possible to complement the position information based on the position information recorded in steps S641 and S645 (step S647). When it is determined that it is possible (YES in step S647), the control unit 21 complements the position information (step S648).

[0295] In step S648, the control unit 21 complements, for example, the position information that replaces the position information determined to exceed the threshold value in step S644. The control unit 21 may complement the position information between the catheter images 51. The complementation can be performed using any method such as linear interpolation, spline interpolation, Lagrange interpolation, or Newton interpolation. The control unit 21 realizes the function of the complementation unit that adds the complementation information to the position information in step S648.

[0296] When it is determined that the position information cannot be complemented (NO in step S647), or after the end of step S648, the control unit 21 starts a subroutine for three-dimensional display (step S649). The subroutine for three-dimensional display is a subroutine that performs three-dimensional display based on a series of catheter images 51. The flow of the process of the subroutine for three-dimensional display will be described later.

[0297] The control unit 21 determines whether to end the process (step S650). For example, when a new pullback operation is started by the MDU 33, that is, when photographing of the catheter image 51 used for generating the three-dimensional image is started, the control unit 21 determines to end the process.

[0298] When it is determined that the process is not ended (NO in step S650), the control unit 21 returns to step S642. When it is determined that the process is ended (YES in step S650), the control unit 21 ends the process.

[0299] Note that while executing the program in FIG. 33, the control unit 21 generates and records classification information and position information based on the newly captured catheter image 51. That is, when it is determined that the process ends in step S646, the processes after step S647 are executed. However, new position information and classification information may be generated while executing steps S647 to S650.

[0300] FIG. 34 is a flowchart for explaining the processing flow of the three-dimensional display subroutine. The three-dimensional display subroutine is a subroutine for performing a three-dimensional display based on a series of catheter images 51. By the three-dimensional display subroutine, the control unit 21 realizes the function of the three-dimensional output unit.

[0301] The control unit 21 acquires composite data 536 corresponding to a series of catheter images 51 (step S661). The control unit 21 creates three-dimensional biological data 551 indicating the three-dimensional structure of the biological tissue based on the series of composite data 536 (step S662).

[0302] Note that when synthesizing the three-dimensional data 55 as described above, the control unit 21 replaces the medical instrument area included in the composite data 536 with a blank area or a non-biological area, and then synthesizes the three-dimensional medical instrument data 552. The control unit 21 may generate the three-dimensional biological data 551 using the classification information output from the first synthesis unit 541 included in the position classification analysis unit 66. The control unit 21 may generate the three-dimensional biological data 551 based on the first classification data 521 described with reference to FIG. 18. That is, the control unit 21 can generate the three-dimensional biological data 551 directly based on a plurality of first classification data 521.

[0303] The control unit 21 may generate the biological three-dimensional data 551 based indirectly on a plurality of first classification data 521. "Based indirectly" means, for example, as described with reference to FIG. 31, generating the biological three-dimensional data 551 based on a plurality of synthesized data 536 generated using the plurality of first classification data 521. The control unit 21 may generate the biological three-dimensional data 551 based on a plurality of data different from the synthesized data 536 generated using the first plurality of classification data 521.

[0304] The control unit 21 assigns thickness information to the curve determined by the series of position information recorded in steps S641 and S645 of the program described with reference to FIG. 33 and the complementary information complemented in step S648 (step S663). The thickness information is preferably the thickness of a medical instrument commonly used in IVR procedures. The control unit 21 may receive information regarding the medical instrument in use and assign thickness information corresponding to the medical instrument. By assigning the thickness information, the three-dimensional shape of the medical instrument is reproduced.

[0305] The control unit 21 synthesizes the three-dimensional shape of the medical instrument generated in step S662 with the biological three-dimensional data 551 generated in step S662 (step S664). The control unit 21 displays the synthesized three-dimensional data 55 on the display device 31 (step S665).

[0306] The control unit 21 receives instructions from the user such as rotation, cross-section change, enlargement, reduction, etc. for the three-dimensionally displayed image, and changes the display. Since the reception of instructions and the change of display for the three-dimensionally displayed image have been conventionally performed, the description thereof is omitted. The control unit 21 ends the process.

[0307] According to the present embodiment, it is possible to provide the catheter system 10 that removes the influence of erroneous detection of position information and displays a medical instrument of an appropriate shape. The user can easily grasp the positional relationship between, for example, a Brockenbrough needle and the fossa ovalis, and perform an IVR procedure.

[0308] Instead of performing the processes from step S643 to step S645, abnormal position information that is far from other position information may be removed by clustering a plurality of pieces of position information.

[0309] [Modification Example 8-1] This modification example relates to a catheter system 10 that performs three-dimensional display based on a medical instrument region detected from a catheter image 51 when the medical instrument has not been misdetected. Descriptions of parts common to Embodiment 8 are omitted.

[0310] In step S663 of the subroutine described with reference to FIG. 34, the control unit 21 determines the thickness of the medical instrument based on the medical instrument region output from, for example, the hint-trained model 631 or the hint-untrained model 632. However, for the catheter image 51 determined to have incorrect position information, the thickness information is complemented based on the medical instrument regions of the preceding and subsequent catheter images 51.

[0311] According to this modification example, for example, a catheter system 10 can be provided that appropriately displays a medical instrument whose thickness changes midway, such as a medical instrument with a needle protruding from a sheath, in a three-dimensional image.

[0312] [Embodiment 9] This embodiment relates to padding processing suitable for a learned model that processes an RT format catheter image 518 obtained using a radial scanning type image acquisition catheter 40. Descriptions of parts common to Embodiment 1 are omitted.

[0313] Padding processing is a process of adding data around the input data before performing convolution processing. In the convolution processing immediately after the input layer that receives image input, the input data is the input image. In convolution processing other than immediately after the input layer, the input data is the feature map extracted in the previous stage. In a learned model that processes image data, so-called zero-padding processing, which assigns "0" data around the input data input to the convolution layer, is generally performed.

[0314] FIG. 35 is an explanatory diagram for explaining the padding process of Embodiment 9. The right end of FIG. 35 is a schematic diagram of input data input to the convolutional layer. The convolutional layer is an example of the first convolutional layer included in, for example, the medical instrument learned model 611 and the second convolutional layer included in the angle learned model 612. The convolutional layer may be a convolutional layer included in any learned model used for processing the catheter image 51 captured using the radial scanning type image acquisition catheter 40.

[0315] The input data is in RT format, where the horizontal direction corresponds to the distance from the sensor 42 and the vertical direction corresponds to the scanning angle. Schematic diagrams of the upper right end and the lower left end of the input data are shown in the center of FIG. 35. Each frame corresponds to a pixel, and the numerical value within the frame corresponds to the pixel value.

[0316] The right end of FIG. 35 is a schematic diagram of the data after performing the padding process of the present embodiment. The numerical values shown in italics indicate the data added by the padding process. "0" data is added to the left and right ends of the input data. At the upper end of the input data, the data indicated by "A" at the lower end of the data before performing the padding process is copied. At the lower end of the input data, the data indicated by "B" at the upper end of the data before performing the padding process is copied.

[0317] That is, at the right end of FIG. 35, the same data as the side with the larger scanning angle is added to the outside of the side with the smaller scanning angle, and the same data as the side with the smaller scanning angle is added to the outside of the side with the larger scanning angle. In the following description, the padding process described with reference to FIG. 35 is referred to as polar padding process.

[0318] In the radial scanning type image acquisition catheter 40, the upper end and the lower end of the RT format catheter image 518 are substantially the same. For example, a single medical instrument or lesion may be separated above and below the RT format catheter image 518. The polar padding process is a process that utilizes such a feature.

[0319] According to this embodiment, a learned model that sufficiently reflects the upper and lower information of an RT format image can be generated.

[0320] Polar padding processing may be performed in all convolutional layers included in the learned model, or may be performed in some convolutional layers.

[0321] FIG. 35 shows an example of performing padding processing to add one piece of data to each of the four sides of the input data. However, the padding processing may be a process of adding a plurality of pieces of data. The number of data added by the polar padding processing is selected according to the size of the filter used for the convolution processing and the stride amount.

[0322] [Modification Example 9-1] FIG. 36 is an explanatory diagram for explaining the polar padding processing of the modification example. The polar padding processing of this modification example is effective for the convolutional layer at the stage of first processing the RT format catheter image 518.

[0323] The upper side of FIG. 36 schematically shows a state in which radial scanning is performed while pulling the sensor 42 to the right. Based on the scanning line data acquired while the sensor 42 makes one rotation, one RT format catheter image 518 schematically shown in the lower left of FIG. 36 is generated. The RT format catheter image 518 is formed from the upper side to the lower side as the sensor 42 rotates.

[0324] The lower right of FIG. 36 schematically shows a state in which padding processing is performed on the RT format catheter image 518. Data at the end portion of the RT format catheter image 518 before one rotation indicated by left-lower hatching is added to the upper side of the RT format catheter image 518. Data at the start portion of the RT format catheter image 518 after one rotation indicated by right-lower hatching is added to the lower side of the RT format catheter image 518. "0" data is added to the left and right of the RT format catheter image 518.

[0325] According to this modification example, since padding processing based on actual scan line data is performed, a learned model that more accurately reflects the upper and lower information of an RT format image can be generated.

[0326] [Embodiment 10] FIG. 37 is an explanatory diagram for explaining the configuration of the catheter system 10 according to Embodiment 10. This embodiment relates to a form of realizing the catheter system 10 of this embodiment by operating in combination with a catheter control device 27, an MDU 33, an image acquisition catheter 40, a general-purpose computer 90, and a program 97. Regarding the parts common to Embodiment 1, the description is omitted.

[0327] The catheter control device 27 is an ultrasonic diagnostic device for IVUS that performs control of the MDU 33, control of the sensor 42, and generation of cross-sectional images and longitudinal-sectional images based on signals received from the sensor 42. Since the functions and configuration of the catheter control device 27 are the same as those of conventionally used ultrasonic diagnostic devices, the description is omitted.

[0328] The catheter system 10 of this embodiment includes a computer 90. The computer 90 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, a smartphone, or a server computer.

[0329] The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. Further, the control unit 21 may read the program 97 stored in a semiconductor memory 98 such as a flash memory mounted in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown) and store it in the auxiliary storage device 23.

[0330] Program 97 is installed as a control program for computer 90, loaded into main memory device 22, and executed. As a result, computer 90 functions as information processing device 20 described above.

[0331] Computer 90 is a general-purpose personal computer, tablet, smartphone, mainframe computer, virtual machine operating on a mainframe computer, cloud computing system, or quantum computer. Computer 90 may be a plurality of personal computers or the like that perform distributed processing.

[0332] [Embodiment 11] FIG. 38 is a functional block diagram of information processing device 20 according to Embodiment 11. Information processing device 20 includes an image acquisition unit 81, a position information acquisition unit 84, and a first data output unit 85. Image acquisition unit 81 acquires a catheter image 51 including a lumen obtained by catheter 40 for image acquisition. Position information acquisition unit 84 acquires position information regarding the position of a medical instrument inserted into the lumen included in catheter image 51.

[0333] When catheter image 51 and the position information are input, first data output unit 85 inputs the acquired catheter image 51 and the acquired position information to first learned model 631 that outputs first data 561 obtained by classifying each region of catheter image 51 into at least three regions: a living tissue region, a medical instrument region where the medical instrument is present, and a non-living tissue region, and outputs first data 561.

[0334] (Appendix A1) An image acquisition unit that acquires a catheter image obtained by a catheter for image acquisition inserted into a first lumen; A first classification data output unit that inputs the catheter image and outputs first classification data obtained by classifying, as different regions, a non-living tissue region including a first lumen region inside the first lumen and a second lumen region inside a second lumen into which the catheter for image acquisition is not inserted and a living tissue region, by inputting the acquired catheter image to a first classification learned model that outputs the first classification data. The first classification trained model is generated using first training data in which at least the non-biological tissue region including the first lumen region and the second lumen region and the biological tissue region are specified. An information processing apparatus.

[0335] (Appendix A2) In the first classification data, a lumen region extraction unit that extracts the first lumen region and the second lumen region from the non-biological tissue region respectively, and a first mode output unit that changes and outputs the first classification data in a mode capable of distinguishing the first lumen region, the second lumen region, and the biological tissue region respectively. The information processing apparatus according to Appendix A1.

[0336] (Appendix A3) In the first classification data, a non-lumen region that is neither the first lumen region nor the second lumen region is extracted from the non-biological tissue region, and a second mode output unit that changes and outputs the first classification data in a mode capable of distinguishing the first lumen region, the second lumen region, the non-lumen region, and the biological tissue region respectively. The information processing apparatus according to Appendix A1 or Appendix A2.

[0337] (Appendix A4) When the catheter image is input, the first classification trained model outputs the first classification data in which the biological tissue region, the first lumen region, the second lumen region, and the non-lumen region are classified as different regions respectively. The information processing apparatus according to Appendix A3.

[0338] (Appendix A5) The catheter for image acquisition is a catheter for acquiring tomographic images of a radial scanning type, the catheter image is an RT format image in which a plurality of scan line data acquired from the catheter for image acquisition are arranged in parallel in the order of scan angles, and the first classification data is the classification result of each pixel in the RT format image. The information processing apparatus according to any one of Appendices A1 to A4.

[0339] (Appendix A6) The first classification learned model includes a plurality of convolutional layers, at least one of the plurality of convolutional layers is learned by performing padding processing of adding the same data as the side with a large scanning angle to the outside of the side with a small scanning angle and adding the same data as the side with a small scanning angle to the outside of the side with a large scanning angle. The information processing apparatus according to Appendix A5.

[0340] (Appendix A7) When the first classification learned model inputs a plurality of the catheter images acquired in time series, the first classification data obtained by classifying the non-biological tissue region and the biological tissue region with respect to the latest one of the plurality of catheter images is output. The information processing apparatus according to any one of Appendices A1 to A6.

[0341] (Appendix A8) The first classification learned model includes a memory unit that holds information regarding the catheter images input in the past, and outputs the first classification data based on the information held in the memory unit and the latest one of the plurality of catheter images. The information processing apparatus according to Appendix A7.

[0342] (Appendix A9) When the first classification learned model inputs the catheter image, the first classification data obtained by classifying the biological tissue region, the non-biological tissue region, and the medical instrument region indicating the medical instrument inserted into the first cavity or the second cavity as different regions is output. The information processing apparatus according to any one of Appendices A1 to A8.

[0343] (Appendix A10) When the catheter image is input, the acquired catheter image is input to a second classification trained model that outputs second classification data obtained by classifying the non-biological tissue region including the first lumen region and the biological tissue region as different regions, and the second classification data output is acquired by a second classification data acquisition unit. A composite classification data output unit that outputs composite classification data obtained by synthesizing the second classification data with the first classification data. The second classification trained model is generated using second training data in which only the first lumen region in the non-biological tissue region is specified. The information processing apparatus according to any one of Appendices A1 to A9.

[0344] (Appendix A11) When the catheter image is input, the second classification trained model outputs the second classification data obtained by classifying the biological tissue region, the non-biological tissue region, and the medical instrument region indicating the medical instrument inserted into the first lumen or the second lumen as different regions respectively. The information processing apparatus according to Appendix A10.

[0345] (Appendix A12) The first classification trained model further outputs the probability that each part of the catheter image is the biological tissue region or the probability that it is the non-biological tissue region. The second classification trained model further outputs the probability that each part of the catheter image is the biological tissue region or the probability that it is the non-biological tissue region. The composite classification data output unit outputs composite classification data obtained by synthesizing the second classification data with the first classification data based on the result of calculating the probability that each part of the catheter image is the biological tissue region or the probability that it is the non-biological tissue region. The information processing apparatus according to Appendix A10 or Appendix A11.

[0346] (Appendix A13) The catheter for image acquisition is a three-dimensional scanning catheter that sequentially acquires a plurality of the catheter images along the longitudinal direction of the catheter for image acquisition. The information processing apparatus according to any one of Appendices A1 to A12.

[0347] (Appendix A14) A three-dimensional output unit that outputs a three-dimensional image generated based on a plurality of the first classification data respectively generated from the acquired plurality of the catheter images. The information processing apparatus according to Appendix A13.

[0348] (Appendix A15) Acquire a catheter image obtained by a catheter for image acquisition inserted into the first cavity, At least a first inner cavity region that is inside the first cavity and a second inner cavity region that is inside the second cavity where the catheter for image acquisition is not inserted, a non-biological tissue region including these, and a biological tissue region are clearly shown. The first classification data that classifies the non-biological tissue region and the biological tissue region as different regions when the catheter image is input is output. Input the acquired catheter image into the first classification learned model to output the first classification data. An information processing method for causing a computer to execute processing.

[0349] (Appendix A16) Acquire a catheter image obtained by a catheter for image acquisition inserted into the first cavity, At least a first inner cavity region that is inside the first cavity and a second inner cavity region that is inside the second cavity where the catheter for image acquisition is not inserted, a non-biological tissue region including these, and a biological tissue region are clearly shown. The first classification data that classifies the non-biological tissue region and the biological tissue region as different regions when the catheter image is input is output. Input the acquired catheter image into the first classification learned model to output the first classification data. A program for causing a computer to execute processing.

[0350] (Appendix A17) Obtain a plurality of sets of training data, which are associated and recorded with the catheter image obtained by the image acquisition catheter inserted into the first cavity, and for each part of the catheter image, a biological tissue region label indicating that it is a biological tissue region, a first inner cavity region label indicating that it is inside the first cavity, a second inner cavity region label indicating that it is inside the second cavity where the image acquisition catheter is not inserted, and a non-biological tissue region label including a non-inner cavity region that is neither the first inner cavity region nor the second inner cavity region. Using the plurality of sets of training data, with the catheter image as the input and the label data as the output, generate a learned model that outputs the biological tissue region label and the non-biological tissue region label for each part of the catheter image when the catheter image is input. Method for generating a learned model.

[0351] (Appendix A18) The non-biological tissue region label of the plurality of sets of training data has a first inner cavity region label indicating the first inner cavity region, a second inner cavity region label indicating the second inner cavity region, and a non-inner cavity region label indicating the non-inner cavity region. Using the plurality of sets of training data, with the catheter image as the input and the label data as the output, generate a learned model that outputs the biological tissue region label, the first inner cavity region label, the second inner cavity region label, and the non-inner cavity region label for each part of the catheter image when the catheter image is input. Method for generating a learned model according to Appendix A17.

[0352] (Appendix A19) The catheter image obtained by the image acquisition catheter inserted into the first cavity, the biological tissue region label indicating that it is a biological tissue region generated based on the boundary line data indicating the inner boundary line of the first cavity in the catheter image, and the non-biological tissue region label including the first inner cavity region indicating that it is inside the first cavity are associated and recorded to obtain a plurality of sets of training data. Using the plurality of sets of training data, with the catheter image as the input and the label data as the output, when the catheter image is input, a learned model that outputs the biological tissue region label and the non-biological tissue region label for each part of the catheter image is generated. Method for generating a learned model.

[0353] (Appendix A20) The catheter image is obtained by the radial scanning type image acquisition catheter. It is an RT format image in which the scanning line data for one rotation is arranged in parallel in the order of the scanning angle. The learned model includes a plurality of convolutional layers. At least one of the convolutional layers performs padding processing of adding the same data as the side with a large scanning angle to the outside of the side with a small scanning angle and adding the same data as the side with a small scanning angle to the outside of the side with a large scanning angle for learning. The method for generating a learned model according to any one of Appendices A17 to A19.

[0354] (Appendix B1) An image acquisition unit that acquires a catheter image obtained by a radial scanning type image acquisition catheter. A first position information output unit that inputs the acquired catheter image to a medical device learned model that outputs first position information regarding the position of a medical device included in the catheter image when the catheter image is input, and outputs the first position information. An information processing apparatus comprising the same.

[0355] (Appendix B2) The first position information output unit outputs the first position information by using the position of one pixel included in the catheter image. The information processing apparatus according to appended note B1.

[0356] (Appended note B3) The first position information output unit includes a first position information acquisition unit that acquires the time-series first position information corresponding to each of a plurality of the catheter images obtained in time series, an exclusion unit that excludes the first position information that does not satisfy a predetermined condition from the time-series first position information, and a complementation unit that adds complementation information that satisfies a predetermined condition to the time-series first position information. The information processing apparatus according to appended note B1 or appended note B2, comprising the above.

[0357] (Appended note B4) When the learned medical device model inputs a plurality of the catheter images acquired in time series, it outputs the first position information regarding the latest catheter image among the plurality of catheter images. The information processing apparatus according to any one of appended note B1 to appended note B3.

[0358] (Appended note B5) The learned medical device model includes a memory unit that holds information regarding the catheter images input in the past, and outputs the first position information based on the information held in the memory unit and the latest catheter image among the plurality of catheter images. The information processing apparatus according to appended note B4.

[0359] (Appended note B6) The learned medical device model receives the input of the catheter image in an RT format image in which a plurality of scan line data acquired from the image acquisition catheter are arranged in parallel in the order of scan angles, includes a plurality of first convolutional layers, At least one of the plurality of first convolutional layers is learned by performing padding processing in which the same data as that on the side with a larger scanning angle is added to the outside on the side with a smaller scanning angle, and the same data as that on the side with a smaller scanning angle is added to the outside on the side with a larger scanning angle. The information processing apparatus according to any one of Appendices B1 to B5.

[0360] (Appendix B7) A scanning angle information acquisition unit that inputs the acquired catheter image into an angle learned model that outputs scanning angle information regarding the position of a medical instrument included in the catheter image when the catheter image is input, and acquires the output scanning angle information. A second position information output unit that outputs second position information regarding the position of a medical instrument included in the catheter image based on the first position information output from the medical instrument learned model and the scanning angle information output from the angle learned model. The information processing apparatus according to any one of Appendices B1 to B6.

[0361] (Appendix B8) The angle learned model receives an input of the catheter image in an RT format image in which a plurality of scan line data acquired from the image acquisition catheter are arranged in parallel in the order of scanning angles, includes a plurality of second convolutional layers, At least one of the plurality of second convolutional layers is learned by performing padding processing in which the same data as that on the side with a larger scanning angle is added to the outside on the side with a smaller scanning angle, and the same data as that on the side with a smaller scanning angle is added to the outside on the side with a larger scanning angle. The information processing apparatus according to Appendix B7.

[0362] (Appendix B9) The medical instrument learned model is generated using a plurality of sets of training data in which the catheter image and the position of the medical instrument included in the catheter image are associated and recorded. The information processing apparatus according to any one of Appendices B1 to B8.

[0363] (Appendix B10) The training data is displaying the catheter image obtained by the catheter for image acquisition, receiving the position of the medical instrument included in the catheter image by a single click operation or a single tap operation on the catheter image, and is generated by a process of associating and storing the catheter image and the position of the medical instrument. The information processing apparatus according to Appendix B9.

[0364] (Appendix B11) The training data is inputting the catheter image into the trained medical instrument model, displaying the first position information output from the trained medical instrument model by superimposing it on the input catheter image, when no correction instruction regarding the position of the medical instrument included in the catheter image is received, storing the uncorrected data associating the catheter image and the first position information as the training data, when a correction instruction regarding the position of the medical instrument included in the catheter image is received, it is generated by a process of storing the corrected data associating the catheter image and the information regarding the position of the medical instrument based on the correction instruction as the training data The information processing apparatus according to Appendix B9.

[0365] (Appendix B12) Obtaining a plurality of sets of training data in which a catheter image obtained by a catheter for image acquisition is associated with first position information regarding the position of a medical instrument included in the catheter image, generating a trained model that outputs first position information regarding the position of a medical instrument included in the catheter image when the catheter image is input, based on the plurality of sets of training data A method for generating a trained model.

[0366] (Appendix B13) The first position information is information regarding the position of one pixel included in the catheter image. The method for generating a learned model according to Supplementary Note B12.

[0367] (Supplementary Note B14) Display a catheter image including a lumen obtained by a catheter for image acquisition, Receive first position information regarding the position of a medical instrument inserted into the lumen included in the catheter image by a single click operation or a single tap operation on the catheter image, Store training data associating the catheter image and the first position information. A training data generation method for causing a computer to execute processing.

[0368] (Supplementary Note B15) The first position information is information regarding the position of one pixel included in the catheter image. The training data generation method according to Supplementary Note B14.

[0369] (Supplementary Note B16) When the first position information is received for the catheter image, Display another catheter image obtained continuously in time series. The training data generation method according to Supplementary Note B14 or Supplementary Note B15.

[0370] (Supplementary Note B17) The catheter for image acquisition is a radial scanning type tomographic image acquisition catheter, The display of the catheter image is to display two images side by side, namely an RT format image in which a plurality of scan line data obtained from the catheter for image acquisition are arranged in parallel in scanning angle order, and an XY format image in which data based on the scan line data are arranged radially around the catheter for image acquisition, The first position information is received from either the RT format image or the XY format image. The training data generation method according to any one of Supplementary Notes B14 to B16.

[0371] (Appendix B18) Input the catheter image obtained by the catheter for image acquisition into a trained medical device model that outputs first position information regarding the position of the medical device included in the catheter image, Superimpose and display the first position information output from the trained medical device model on the input catheter image, If a correction instruction regarding the position of the medical device included in the catheter image is not received, store the uncorrected data associating the catheter image and the first position information as training data, If a correction instruction regarding the position of the medical device included in the catheter image is received, store the corrected data associating the catheter image and the information regarding the received position of the medical device as the training data A training data generation method for causing a computer to execute processing.

[0372] (Appendix B19) The uncorrected data and the corrected data are data regarding the position of one pixel included in the catheter image The training data generation method described in Appendix B18.

[0373] (Appendix B20) Input a plurality of the catheter images obtained in a time series into the trained medical device model in order, Superimpose and display in order each output position on the input catheter image. The training data generation method described in Appendix B18 or Appendix B19.

[0374] (Appendix B21) The position of the medical device is received by a single click operation or a single tap operation. The training data generation method described in any one of Appendix B18 to Appendix B20.

[0375] (Appendix B22) The catheter for image acquisition is a radial scanning type catheter for tomographic image acquisition, The display of the catheter image displays two images side by side, namely an RT format image in which a plurality of scan line data acquired from the image acquisition catheter are arranged in parallel in the order of scan angles, and an XY format image in which data based on the scan line data are arranged radially around the image acquisition catheter, The position of the medical instrument is received from either the RT format image or the XY format image. The training data generation method according to any one of Appendices B18 to B21.

[0376] (Appendix C1) An image acquisition unit that acquires a catheter image including a lumen obtained by an image acquisition catheter, A position information acquisition unit that acquires position information regarding the position of a medical instrument inserted into the lumen included in the catheter image, A first data output unit that inputs the catheter image and the position information, and outputs first data obtained by classifying each region of the catheter image into at least three regions: a living tissue region, a medical instrument region where the medical instrument exists, and a non-living tissue region, to a first learned model that outputs the first data An information processing apparatus comprising the above.

[0377] (Appendix C2) The position information acquisition unit When the catheter image is input, the acquired catheter image is input to a learned medical instrument model that outputs the position information included in the catheter image, and the position information is acquired from the learned medical instrument model. The information processing apparatus according to Appendix C1.

[0378] (Appendix C3) When the catheter image is input without inputting the position information, the acquired catheter image is input to a second model that outputs second data obtained by classifying each region of the catheter image into at least three regions: a living tissue region, a medical device region where the medical device is present, and a non-living tissue region, and the second data is acquired by the second data acquisition unit and includes a composite data output unit that outputs composite data obtained by combining the first data and the second data. The information processing apparatus according to Supplementary Note C2.

[0379] (Supplementary Note C4) The composite data output unit includes a first composite data output unit that outputs first composite data obtained by combining data related to living tissue-related regions classified into the living tissue region and the non-living tissue region among the first data and the second data; and a second composite data output unit that outputs second composite data obtained by combining data related to the medical device region among the first data and the second data. The information processing apparatus according to Supplementary Note C3.

[0380] (Supplementary Note C5) The second composite data output unit when the position information can be acquired from the learned medical device model, outputs the second composite data using the data related to the medical device region included in the first data; and when the position information cannot be acquired from the learned medical device model, outputs the second composite data using the data related to the medical device region included in the second data. The information processing apparatus according to Supplementary Note C4.

[0381] (Supplementary Note C6) The composite data output unit outputs the second composite data obtained by combining the data related to the medical device region based on weighting according to the reliability of the first data and the reliability of the second data. The information processing apparatus according to Supplementary Note C4.

[0382] (Appendix C7) The reliability is determined based on whether the position information can be obtained from the learned medical device model. The information processing apparatus according to Appendix C6.

[0383] (Appendix C8) The composite data output unit When the position information can be obtained from the learned medical device model, sets the reliability of the first data higher than the reliability of the second data. When the position information cannot be obtained from the learned medical device model, sets the reliability of the first data lower than the reliability of the second data. The information processing apparatus according to Appendix C6.

[0384] (Appendix C9) The catheter for image acquisition is a three-dimensional scanning catheter that sequentially acquires a plurality of the catheter images along the longitudinal direction of the catheter for image acquisition. The information processing apparatus according to any one of Appendices C1 to C8.

[0385] (Appendix C10) Acquires a catheter image including a lumen obtained by a catheter for image acquisition, Acquires position information regarding the position of a medical device inserted into the lumen included in the catheter image, When the catheter image and the position information regarding the position of the medical device included in the catheter image are input, inputs the obtained catheter image and the obtained position information to a first learned model that outputs first data obtained by classifying each region of the catheter image into at least three regions: a living tissue region, a medical device region where the medical device is present, and a non-living tissue region, and outputs the first data. An information processing method for causing a computer to execute processing.

[0386] (Appendix C11) Acquires a catheter image including a lumen obtained by a catheter for image acquisition, Obtain position information regarding the position of a medical instrument inserted into the lumen included in the catheter image, When the catheter image and the position information regarding the position of the medical instrument included in the catheter image are input, input the obtained catheter image and the obtained position information into a first pre-trained model that outputs first data for classifying each region of the catheter image into at least three regions: a living tissue region, a medical instrument region where the medical instrument is present, and a non-living tissue region. A program for causing a computer to execute processing.

[0387] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.

Explanation of Reference Numerals

[0388] 10 Catheter system 20 Information processing device 21 Control unit 22 Main storage device 23 Auxiliary storage device 24 Communication unit 25 Display unit 26 Input unit 27 Catheter control device 271 Catheter control section 29 Reading unit 31 Display device 32 Input device 33 MDU 37 Image diagnostic device 40 Image acquisition catheter 41 Probe section 42 Sensor 43 Shaft 44 Tip marker 45 Connector part 46 Guide wire lumen 51 Catheter image 518 RT format catheter image (catheter image) 519 XY format catheter image 52 Classification data (hint unclassified data, second data) 521 First classification data (label data) 522 Second classification data (label data) 526 Composite classification data 528 RT format classification data 529 XY format classification data 536 Composite data 541 First synthesis part 542 Second synthesis part 543 Third synthesis part 55 Three-dimensional data 551 Biological three-dimensional data 552 Medical device three-dimensional data 561 Hint-classified data (first data) 611 Trained medical device model 612 Trained angle model 615 Position information synthesis part 619 Position information model 62 Classification model (second model) 621 First trained classification model 622 Second trained classification model 626 Composite classification model 628 Classification data synthesis part 629 Classification data conversion part 631 Trained hint-present model (first trained model) 632 Trained hint-absent model (second trained model) 639 X% hint-trained model 65 Position information acquisition part 66 Position classification analysis part 71 Medical device position training data DB 72 Trained hint-present data DB 781 Cursor 782 Control Button Area 81 Image Acquisition Unit 82 First Classification Data Output Unit 90 Computer 96 Portable Recording Medium 97 Program 98 Semiconductor Memory

Claims

1. An image acquisition unit that acquires a catheter image including a lumen obtained by a catheter for image acquisition; A position information acquisition unit that acquires position information regarding the position of a medical instrument inserted into the lumen included in the catheter image; A first data output unit that inputs the catheter image and the position information, and outputs first data obtained by classifying each region of the catheter image into at least three regions: a living tissue region, a medical instrument region where the medical instrument exists, and a non-living tissue region, to a first pre-trained model; An information processing apparatus comprising the above.

2. The position information acquisition unit inputs the acquired catheter image to a pre-trained medical instrument model that outputs the position information included in the catheter image when the catheter image is input, and acquires the position information from the pre-trained medical instrument model. The information processing apparatus according to Claim 1.

3. A second data acquisition unit that inputs the acquired catheter image to a second model that outputs second data obtained by classifying each region of the catheter image into at least three regions: a living tissue region, a medical instrument region where the medical instrument exists, and a non-living tissue region, when the catheter image is input without inputting the position information, and acquires the second data; A combined data output unit that outputs combined data obtained by combining the first data and the second data. The information processing apparatus according to Claim 2.

4. The combined data output unit includes a first combined data output unit that outputs first combined data obtained by combining data regarding a living tissue-related region classified into the living tissue region and the non-living tissue region among the first data and the second data; and a second combined data output unit that outputs second combined data obtained by combining data regarding the medical instrument region among the first data and the second data. The information processing apparatus according to Claim 3.

5. The second combined data output unit outputs the second combined data using the data regarding the medical instrument region included in the first data when the position information can be acquired from the pre-trained medical instrument model; and outputs the second combined data using the data regarding the medical instrument region included in the second data when the position information cannot be acquired from the pre-trained medical instrument model. The information processing apparatus according to claim 4.

6. The composite data output unit outputs the second composite data obtained by synthesizing data related to the medical instrument area based on weighting according to the reliability of the first data and the reliability of the second data. The information processing apparatus according to claim 4.

7. The reliability is determined based on whether the position information can be obtained from the learned medical instrument model. The information processing apparatus according to claim 6.

8. The composite data output unit when the position information can be obtained from the learned medical instrument model, sets the reliability of the first data higher than the reliability of the second data, when the position information cannot be obtained from the learned medical instrument model, sets the reliability of the first data lower than the reliability of the second data. The information processing apparatus according to claim 6.

9. The catheter for image acquisition is a three-dimensional scanning catheter that sequentially acquires a plurality of the catheter images along the longitudinal direction of the catheter for image acquisition. The information processing apparatus according to any one of claims 1 to 8.

10. Acquire a catheter image including a lumen obtained by a catheter for image acquisition, acquire position information regarding the position of a medical instrument inserted into the lumen included in the catheter image, When the catheter image and the position information regarding the position of the medical instrument included in the catheter image are input, the acquired catheter image and the acquired position information are input to a first learned model that outputs first data obtained by classifying each region of the catheter image into at least three regions: a living tissue region, a medical instrument region where the medical instrument exists, and a non-living tissue region, and the first data is output. An information processing method for causing a computer to execute the process.

11. Acquire a catheter image including a lumen obtained by a catheter for image acquisition, acquire position information regarding the position of a medical instrument inserted into the lumen included in the catheter image, When the catheter image and the position information regarding the position of the medical instrument included in the catheter image are input, the acquired catheter image and the acquired position information are input to a first learned model that outputs first data obtained by classifying each region of the catheter image into at least three regions: a living tissue region, a medical instrument region where the medical instrument exists, and a non-living tissue region, and the first data is output. A program for causing a computer to execute a process.

Images