Program, information processing method and information processing device

Abstract

To provide a program or the like capable of easily using a catheter system.SOLUTION: A program causes a computer to execute processing for acquiring a catheter image generated by using a catheter for image diagnosis inserted into a hollow organ, acquiring complement information 572 including an information lack part 473 of the acquired catheter image to complement it, correcting a region corresponding to the information lack part 473 and having reliability lower than the other regions in the acquired complement information 572 on the basis of the other regions, and displaying the catheter image and the corrected complement information 572.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a program, an information processing method, and an information processing system. [Background technology]

[0002] 2. Description of the Related Art A catheter system is used in which a diagnostic imaging catheter is inserted into a hollow organ such as a blood vessel to capture a tomographic image (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 164071 Summary of the Invention [Problem to be solved by the invention]

[0004] Guidewires, stents, or heavily calcified areas can cause information loss in tomographic images. Experts such as experienced doctors perform catheter treatments while filling in the missing information through guesswork. For this reason, extensive training is required to use the catheter system.

[0005] In one aspect, an object of the present invention is to provide a program or the like that enables easy use of a catheter system. [Means for solving the problem]

[0006] The program causes a computer to execute a process of acquiring a catheter image generated using an imaging diagnostic catheter inserted into a tubular organ, acquiring complementary information that complements the acquired catheter image including missing information parts, correcting areas of the acquired complementary information that correspond to the missing information parts and have a lower reliability than other areas based on the other areas, and displaying the catheter image and the corrected complementary information. Effect of the Invention

[0007] In one aspect, a program or the like can be provided that makes it easy to use the catheter system. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of a catheter system. [Diagram 2] FIG. 1 is an explanatory diagram for explaining an overview of a catheter for diagnostic imaging. [Diagram 3] FIG. 1 is an explanatory diagram illustrating a configuration of a catheter system. [Figure 4] 13 is an example of a screen displayed by the catheter system. [Diagram 5] 13 is an example of a screen displayed by the catheter system. [Figure 6] 13 is an example of a screen displayed by the catheter system. [Figure 7] 13 is an example of a screen displayed by the catheter system. [Figure 8] 11 is a flowchart illustrating a process flow of a program. [Figure 9] 13 is an example of a screen displayed by the catheter system of the second embodiment. [Figure 10] 10 is a flowchart illustrating the flow of processing of a program according to the second embodiment. [Figure 11] FIG. 2 is an explanatory diagram illustrating the configuration of a learning model. [Figure 12] 13 is an example of a screen displayed by the catheter system of the third embodiment. [Figure 13] 13 is an example of a screen displayed by the catheter system of the third embodiment. [Figure 14] 13 is a flowchart illustrating the flow of processing of a program according to a third embodiment. [Figure 15] FIG. 13 is an explanatory diagram illustrating the configuration of a learning model according to the fourth embodiment. [Figure 16] An explanatory diagram explaining how to use the learning model in embodiment 4. [Figure 17] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 18] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 19] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 20] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 21] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 22] 13 is a flowchart illustrating the flow of processing of a program according to a fourth embodiment. [Figure 23] FIG. 13 is an explanatory diagram for explaining a method of generating a screen to be displayed by the catheter system of the fifth embodiment. [Figure 24] 13 is a flowchart illustrating the flow of processing of a program according to a fifth embodiment. [Diagram 25] FIG. 23 is an explanatory diagram for explaining a method of generating a screen to be displayed by the catheter system of the sixth embodiment. [Figure 26] 20 is a flowchart illustrating the flow of processing of a program according to a sixth embodiment. [Figure 27] FIG. 2 is an explanatory diagram illustrating the record layout of a training data DB. [Figure 28] 20 is a flowchart illustrating the flow of processing of a program according to a seventh embodiment. [Figure 29] FIG. 1 is an explanatory diagram outlining a method for generating a learning model. [Diagram 30] 13 is a flowchart illustrating the flow of processing of a program according to an eighth embodiment. [Diagram 31] FIG. 23 is an explanatory diagram for explaining a method of generating an image displayed by the catheter system of the ninth embodiment. [Diagram 32] FIG. 23 is an explanatory diagram for explaining a method for generating a complementary line according to a ninth embodiment. [Diagram 33] FIG. 13 is an explanatory diagram for explaining a method for generating a complementary line in mode 2. [Diagram 34] 13 is a flowchart illustrating the flow of processing of a program according to a ninth embodiment. [Diagram 35] 13 is a flowchart illustrating a process flow of a subroutine for generating complementary lines. [Diagram 36] 13 is an example of a screen displayed by the catheter system of embodiment 9. [Figure 37] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Figure 38] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Figure 39] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Diagram 40] 13 is an example of a screen displayed by the catheter system of embodiment 9. [Diagram 41] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Diagram 42] FIG. 23 is a functional block diagram of an information processing system according to a tenth embodiment. [Diagram 43] FIG. 23 is an explanatory diagram illustrating the configuration of a catheter system according to an eleventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment 1] FIG. 1 is an explanatory diagram for explaining an overview of a catheter system 10. The catheter system 10 includes a catheter 40 for diagnostic imaging, an MDU (Motor Driving Unit) 33, and an information processing device 20. The catheter 40 for diagnostic imaging is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is an input device such as a keyboard, a mouse, a trackball, or a microphone. The display device 31 and the input device 32 may be stacked together to form a touch panel. The input device 32 and the information processing device 20 may be configured as one unit.

[0010] Fig. 2 is an explanatory diagram for explaining an overview of the diagnostic imaging catheter 40. Fig. 2 shows an example of a diagnostic imaging catheter 40 for IVUS (Intravascular Ultrasound) used when generating an ultrasonic tomographic image from inside a blood vessel, that is, for generating an ultrasonic tomographic image. The ultrasonic tomographic image is an example of a catheter image generated using the diagnostic imaging catheter 40. The IVUS catheter is an example of a catheter for generating a tomographic image.

[0011] The diagnostic imaging catheter 40 has a probe portion 41 and a connector portion 45 disposed at an end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the diagnostic imaging catheter 40 farther from the connector portion 45 will be referred to as the tip side.

[0012] A shaft 43 is inserted inside the probe portion 41. A sensor 42 is connected to the tip side of the shaft 43. The shaft 43 and the sensor 42 are rotatable and movable forward and backward inside the probe portion 41.

[0013] The sensor 42 is an ultrasonic transducer that transmits and receives ultrasonic waves. An annular tip marker 44 is fixed near the tip of the probe portion 41. The tip marker 44 is made of a material, such as metal, that is opaque to X-rays.

[0014] The diagnostic imaging catheter 40 may be a catheter for generating an optical tomographic image, such as for OCT (Optical Coherence Tomography) or OFDI (Optical Frequency Domain Imaging), which generates an optical tomographic image using near-infrared light. An optical tomographic image is an example of a catheter image generated using the diagnostic imaging catheter 40. The sensor 42 of these diagnostic imaging catheters 40 is a transceiver unit that irradiates near-infrared light and receives reflected light. A catheter for generating an optical tomographic image is an example of a catheter for generating a tomographic image.

[0015] The imaging diagnostic catheter 40 may have both an ultrasonic transducer and a transceiver unit for OCT or OFDI as the sensor 42. The imaging diagnostic catheter 40 may have a total of three sensors 42, including an ultrasonic transducer, a transceiver unit for OCT, and a transceiver unit for OFDI.

[0016] The diagnostic imaging catheter 40 is not limited to a mechanical scanning type that mechanically rotates and advances and retreats, but may be an electronic radial scanning type diagnostic imaging catheter 40 that uses a sensor 42 in which multiple ultrasonic transducers are arranged in a ring shape.

[0017] The diagnostic imaging catheter 40 may have a so-called linear scanning type sensor 42 in which a plurality of ultrasonic transducers are arranged in a row along the longitudinal direction. The diagnostic imaging catheter 40 may have a so-called two-dimensional array type sensor 42 in which a plurality of ultrasonic transducers are arranged in a matrix.

[0018] The tubular organ into which the diagnostic imaging catheter 40 is inserted and used is, for example, a blood vessel, a pancreatic duct, a bile duct, a bronchus, etc. In the following description, the diagnostic imaging catheter 40 will be described as an example of a mechanically scanned IVUS catheter shown in FIG.

[0019] Using the imaging diagnostic catheter 40, it is possible to generate tomographic images that include not only the walls of hollow organs such as blood vessel walls, but also reflectors present inside the hollow organs, such as red blood cells, and organs present outside the hollow organs, such as the epicardium and heart.

[0020] 3 is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 has an information processing device 20, an MDU 33, and an image diagnostic catheter 40. The information processing device 20 has a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

[0021] The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, etc. The control unit 21 is connected to each hardware unit constituting the information processing device 20 via a bus.

[0022] The main memory device 22 is a storage device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs being executed by the control unit 21.

[0023] The auxiliary storage device 23 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 23 stores programs to be executed by the control unit 21 and various data required for executing the programs. The communication unit 24 is an interface that performs communication between the information processing device 20 and a network.

[0024] The display unit 25 is an interface that connects the display device 31 to the bus. The input unit 26 is an interface that connects the input device 32 to the bus. The catheter control unit 271 controls the MDU 33 and the sensor 42, and generates an image based on a signal received from the sensor 42.

[0025] The MDU 33 rotates the sensor 42 and the shaft 43 inside the probe portion 41. The catheter control unit 271 generates one image for each rotation of the sensor 42. The generated image is a transverse slice image centered on the probe portion 41 and approximately perpendicular to the probe portion 41.

[0026] MDU 33 can also move sensor 42 and shaft 43 forward and backward while rotating them inside probe section 41. By a pullback operation in which sensor 42 is rotated while being pulled toward MDU 33 at a constant speed, catheter control section 271 successively generates multiple transverse slice images approximately perpendicular to probe section 41 at a predetermined interval.

[0027] The function and configuration of catheter control unit 271 are similar to those of conventional ultrasonic diagnostic devices, and detailed description thereof will be omitted. Note that control unit 21 may realize the function of catheter control unit 271.

[0028] The information processing device 20 is connected to various imaging diagnostic devices 37, such as an X-ray angiography device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, or an ultrasound diagnostic device, via a HIS (Hospital Information System) or the like.

[0029] The information processing device 20 of this embodiment is a dedicated ultrasound diagnostic device, or a personal computer, tablet, smartphone, or the like having the functions of an ultrasound diagnostic device.

[0030] 4 to 7 are examples of screens displayed by the catheter system 10. The screen shown in FIG. 4 includes a first image field 51, a stop button 581, a selection button 582, and a complement button 583. FIG. 4 shows a state in which the stop button 581 is not selected. The selection button 582 and the complement button 583 are set to a state in which they cannot be selected. A tomographic image obtained using the diagnostic imaging catheter 40 is displayed in real time in the first image field 51. In the following description, an image displayed in real time will be referred to as a real-time image.

[0031] In the real-time image shown in FIG. 4, a small crescent-shaped guidewire image 472 is displayed at the 5 o'clock direction relative to a circular catheter image 471 displayed in the center of the first image field 51. Since the ultrasonic waves emitted from the sensor 42 do not reach the back surface of the guidewire, which is a strong reflector, an approximately fan-shaped information missing portion 473 is formed outside the guidewire image 472. Such information missing portion 473 is called an acoustic shadow portion. Note that the information missing portion 473 is not limited to an acoustic shadow portion. Information missing portions are also formed by artifacts such as ringdown caused by vibration of the ultrasonic transducer, multiple echoes formed by a strong reflector such as the guidewire, and external noise.

[0032] For example, in a region where a stent is placed, information missing portions 473 are formed outside the image showing the wires that make up the mesh of the stent, which is a strong reflector (see FIG. 23). Similarly, information missing portions 473 are also formed outside the strongly calcified region and the highly attenuating plaque.

[0033] A doctor or other expert who is skilled in handling the diagnostic imaging catheter 40 can distinguish between information-missing areas 473 formed due to the influence of strong reflectors or high attenuation areas and areas that truly have low ultrasonic reflectivity, and can supplement the information missing due to the information-missing areas 473 by inference.

[0034] However, there may be variation in judgment between individuals regarding the parts that are supplemented by guesswork. For example, if there is variation in judgment between the doctor in charge of operating the diagnostic imaging catheter 40 and the medical staff such as nurses and technicians who support the doctor's procedure, the doctor's instructions may not be conveyed promptly, and in some cases, misunderstandings may occur.

[0035] 5 shows an example of a screen displayed when control unit 21 accepts the selection of stop button 581. Selection button 582 is set to a selectable state. Second image field 52 is displayed in the upper right corner of the screen. A real-time image is displayed in second image field 52. The image displayed at the time when selection of stop button 581 was accepted is displayed in a still state in first image field 51.

[0036] In the following description, a paused image refers to an image that is displayed when the selection of the stop button 581 is accepted. Note that the control unit 21 may accept the selection of the stop button 581 by voice recognition, or by operating a foot switch (not shown), or the like.

[0037] 6 shows an example of a screen displayed when the control unit 21 accepts selection of the selection button 582. A cursor 68 is displayed in the first image field 51. The user operates the cursor 68 via the input device 32 to place a designated point mark 571, for example, on the edge of a hollow organ.

[0038] 7 shows an example of a screen displayed when the control unit 21 accepts the selection of the complement button 583. The control unit 21 displays a circular, i.e., closed curved complement line 572 that complements the edge of the hollow organ based on the designated point marks 571 placed by the user. The complement line 572 is generated by connecting the designated point marks 571 placed by the user with a curve such as a spline curve. The complement line 572 is an example of complement information that complements the information missing portion 473.

[0039] Fig. 8 is a flowchart for explaining the flow of processing of the program. The program shown in Fig. 8 is a program that is started when the selection of the selection button 582 by the user is accepted.

[0040] The control unit 21 receives a position designation of the designated point mark 571 by the user (step S501). The control unit 21 determines whether or not the input of the designated point mark 571 has been completed (step S502). For example, the control unit 21 determines that the acceptance of the designated point mark 571 has been completed when the user instructs the end of the input. The control unit 21 may also determine that the input has been completed when a predetermined number of designated point marks 571 have been accepted.

[0041] If it is determined that the process is not completed (NO in step S502), the control unit 21 returns to step S501. If it is determined that the process is completed (YES in step S502), the control unit 21 generates a circular complementary line 572 that connects the designated point marks 571 accepted in step S501 (step S503).

[0042] The control unit 21 superimposes the generated complementary line 572 on the paused image, and displays the screen described with reference to Fig. 7 (step S504). The control unit 21 ends the process.

[0043] The control unit 21 may calculate parameters such as the area, the maximum diameter, and the minimum diameter based on the completed complement line 572, and display them on the screen described with reference to Fig. 7. The control unit 21 may accept a designation of the type of parameter to be calculated or a calculation formula.

[0044] According to the present embodiment, it is possible to provide a catheter system 10 that easily embodies and displays the complementary line 572 that the user imagines in his / her mind based on the tomographic image. Therefore, it is possible to provide a catheter system 10 that is easy for the user to use.

[0045] By displaying the completion state imagined by the user in his / her mind in a manner that can be recognized by other staff using the completion line 572, a catheter system 10 can be provided that facilitates communication between related medical staff.

[0046] According to the present embodiment, the user places the designated point mark 571 at a location where an image actually exists, so that the user is unlikely to be confused in his / her judgment. Therefore, the catheter system 10 can be provided that allows the user to quickly perform the task of creating the complementary line 572.

[0047] An example will be described in which a stent is placed in a blood vessel. A doctor inserts an image diagnostic catheter 40 into the blood vessel in which the stent is to be placed, and judges the state of stenosis of the blood vessel based on the tomographic image. When deciding which stent to use, the doctor or a medical staff member under the doctor's instructions operates the input device 32 to measure the area and inner diameter of the blood vessel lumen in the tomographic image. The doctor decides which stent to place based on the measurement results.

[0048] The control unit 21 can calculate the area and inner diameter of the blood vessel lumen based on the above-mentioned complement line 572. That is, it is possible to provide a catheter system 10 that calculates the area and inner diameter of the blood vessel lumen. The control unit 21 may display information about a recommended stent based on the calculated area and inner diameter. Note that since the calculation of the area and inner diameter of a portion surrounded by a closed curve has been performed conventionally, detailed explanation will be omitted.

[0049] After the stent is placed, the doctor inserts a diagnostic imaging catheter 40 into the same blood vessel and checks, based on the tomographic images, whether or not there is a gap between the blood vessel wall and the placed stent. If there is a gap between the blood vessel wall and the stent, the doctor will perform a procedure such as re-expanding the stent.

[0050] The tomographic image after the stent is placed contains many information missing parts 473 due to the wires of the stent. Therefore, the external elastic lamina is displayed as a broken line interrupted by the information missing parts 473. By displaying the above-mentioned complementary lines 572, other medical staff members, in addition to the doctor who set the position of the designated point mark 571, can easily understand the placement state of the stent.

[0051] A part or all of the program may be executed on a large-scale computer connected via a network, a virtual machine running on a large-scale computer, or a cloud computing system. A part or all of the program may be executed on multiple personal computers performing distributed processing.

[0052] Instead of real-time images, videos or still images stored in the auxiliary storage device 23 or the like may be used. It is possible to provide a catheter system 10 that can be used for recording medical records after the end of a case. In such a case, the information processing device 20 may be a general-purpose personal computer, tablet, smartphone, or the like that does not have a function for connecting the MDU 33 and the catheter 40 for image diagnosis.

[0053] [Variations] When the control unit 21 receives, for example, five or more designated point marks 571 in step S501, the control unit 21 generates (step S503) and displays (step S504) complementary lines 572. Thereafter, the control unit 21 returns to step S501 and further receives designation of the designated point marks 571.

[0054] The user places additional designated point marks 571 in a portion where the user actually wants to draw the complementary line 572, which is far from the displayed complementary line 572. In this manner, the user can create the desired complementary line 572 with a small number of designated point marks 571.

[0055] [Embodiment 2] This embodiment relates to a catheter system 10 that automatically displays a candidate point mark 573 at a location similar to a designated point mark 571 designated by the user. Description of parts common to the first embodiment will be omitted.

[0056] Fig. 9 is an example of a screen displayed by the catheter system 10 of embodiment 2. As described using Fig. 6, the user operates the cursor 68 via the input device 32 to specify the position of the specified point mark 571.

[0057] The control unit 21 detects an area similar to the periphery of the designated point mark 571 whose designation has been accepted. Specifically, the control unit 21 extracts a template area 574 of a predetermined number of pixels centered on the designated point mark 571. In Fig. 9, the template area 574 is indicated by a virtual line.

[0058] The control unit 21 uses a template matching technique to detect an area from the paused image that is similar to the template area 574. The template matching technique has been used for a long time, so a detailed explanation will be omitted. Note that it is preferable for the control unit 21 to exclude the vicinity of the location where the designated point mark 571 has already been placed from the target of template matching.

[0059] When a similar area is detected, the control unit 21 displays a candidate point mark 573 in the center of the area. The control unit 21 may simultaneously display multiple candidate point marks 573. The control unit 21 may display only the candidate point mark 573 with the highest similarity during template matching. The candidate point mark 573 is a mark with a different aspect from the designated point mark 571, and the user can easily distinguish between the two.

[0060] The user determines whether to approve the displayed candidate point mark 573 and operates the input device 32. For example, the user approves the candidate point mark 573 by clicking it, and rejects it by dragging and dropping it outside the first image field 51. The user may input an instruction to approve or reject by voice input.

[0061] The control unit 21 changes the candidate point mark 573 approved by the user to a specified point mark 571. The control unit 21 deletes the candidate point mark 573 rejected by the user. Note that the user may reserve judgment on the displayed candidate point mark 573 and specify the position of the next candidate point mark 573.

[0062] For example, when the user places a designated point mark 571 on the external elastic lamina, the control unit 21 extracts a point indicating the external elastic lamina by template matching and displays a candidate point mark 573. Similarly, when the user places a designated point mark 571 on the inner surface of the lumen wall, the control unit 21 extracts a point indicating the inner surface of the lumen wall and displays a candidate point mark 573.

[0063] 10 is a flowchart explaining the flow of processing of the program of the second embodiment. The control unit 21 accepts the position designation of the designated point mark 571 by the user (step S511). The control unit 21 determines whether or not the input of the designated point mark 571 has been completed (step S512). For example, the control unit 21 determines that the acceptance of the designated point mark 571 has been completed when the user instructs the end of the input. The control unit 21 may also determine that the input has been completed when a predetermined number of designated point marks 571 have been accepted.

[0064] If it is determined that the process has not ended (NO in step S512), the control unit 21 extracts a template region 574 from the paused image, centered on the position accepted in step S511 (step S513). The control unit 21 executes template matching processing to detect a region similar to the template region 574 from the paused image (step S514). Template matching has been performed conventionally, so detailed description will be omitted.

[0065] Control unit 21 determines whether or not the detection of a region similar to template region 574 has been successful (step S515). For example, control unit 21 determines that the detection has been successful when the similarity between template region 574 and the region detected in step S514 exceeds a predetermined threshold.

[0066] The similarity can be evaluated, for example, by the sum of squared differences (SSD), the sum of absolute differences (SAD), or normalized cross-correlation (NCC) of pixel values ​​of pixels at the same position in two regions. Note that the above methods are all examples. The method of evaluating the similarity is not limited to these.

[0067] If it is determined that the matching was not successful (NO in step S515), control unit 21 returns to step S511. If it is determined that the matching was successful (YES in step S515), control unit 21 displays candidate point mark 573 in the center of the area that matches template area 574 (step S516).

[0068] For example, control unit 21 displays candidate point mark 573 at the center of the area with the highest similarity. Control unit 21 may display candidate point mark 573 at the center of each of the multiple areas determined to satisfy the condition in step S515. Control unit 21 returns to step S511.

[0069] Although not shown in the flowchart, the user inputs approval or rejection of the displayed candidate point mark 573 at any timing by operating the cursor 68, by voice input, etc. The control unit 21 changes the approved candidate point mark 573 to the designated point mark 571, and deletes the rejected candidate point mark 573.

[0070] When it is determined that the process is completed (YES in step S512), the control unit 21 generates a circular complementary line 572 that connects the designated point marks 571 received in step S501 (step S517). The control unit 21 superimposes the generated complementary line 572 on the paused image, and displays the screen described using Fig. 7 (step S518). The control unit 21 ends the process.

[0071] According to the present embodiment, it is possible to provide a catheter system 10 that automatically displays a candidate point mark 573 at a position similar to the position where the user has specified the specified point mark 571. It is possible to provide a catheter system 10 that can reduce the burden on the user in inputting the specified point mark 571.

[0072] [Embodiment 3] This embodiment relates to a catheter system 10 that automatically generates a complementary line 572 in real time. Descriptions of parts common to the first embodiment will be omitted.

[0073] 11 is an explanatory diagram illustrating the configuration of the learning model 61. The learning model 61 is a model that receives an input image and outputs a prediction regarding an output image showing the complementary line 572. The learning model 61 is generated by machine learning. A method for generating the learning model 61 will be described later.

[0074] The learning model 61 is created for each of the complementary lines to be generated, such as a model that outputs a complementary line 572 corresponding to the external elastic lamina, a model that outputs a complementary line 572 corresponding to the inner surface of the lumen wall, a model that outputs a complementary line 572 corresponding to the periphery of a calcified portion, and a model that outputs a complementary line 572 corresponding to the periphery of a plaque. The learning model 61 may be created for each combination of the site where the diagnostic imaging catheter 40 is inserted, such as the coronary artery, the lower limb aorta, the bile duct, the pancreatic duct, and the bronchus, and the complementary line to be generated.

[0075] Each learning model 61 is stored in the auxiliary storage device 23. The learning model 61 may be stored in an external large-capacity storage device connected to the information processing device 20. The control unit 21 may acquire the learning model 61 stored in a server or the like via a network whenever necessary.

[0076] The learning model 61 may be a model that receives an input image and outputs an output image in which the complementary line 572 is superimposed on the input image. The input image input to the learning model 61 may be a longitudinal cross-sectional image. The learning model 61 may be a model that receives an input image and outputs a group of coordinates through which the complementary line 572 passes.

[0077] 12 and 13 are examples of screens displayed by the catheter system 10 of embodiment 3. Fig. 12 shows an example of a screen displayed by the control unit 21 on the display device 31 before creating a completion line 572. The screen shown in Fig. 12 includes a first image field 51, a target selection button 591, and a start button 584.

[0078] The first image field 51 displays a real-time image obtained using the diagnostic imaging catheter 40. The object selection button 591 is a pull-down menu button, and the user selects the area for which he or she wishes to display the completion line 572. In FIG. 12, "EEM", i.e., the external elastic lamina, is selected.

[0079] The user observes the real-time image displayed in the first image field 51 and determines the area for displaying the complementary line 572. The user operates the target selection button 591 to select the area for displaying the complementary line 572, and then selects the start button 584.

[0080] 13 shows an example of a screen that control unit 21 displays in first image field 51 after accepting selection of start button 584 by the user. The screen shown in FIG. 13 includes first image field 51, target selection button 591, and end button 585. A real-time image with complementary line 572 corresponding to "EEM" superimposed thereon is displayed in first image field 51. When wishing to end display of complementary line 572, the user selects end button 585.

[0081] 13 may display a button or the like for accepting a change in the display mode, such as the color and thickness, of the complementary line 572. The screen shown in Fig. 13 may display a button or the like for temporarily changing the complementary line 572 to a non-display state.

[0082] 14 is a flowchart for explaining the flow of processing of the program of the third embodiment. The control unit 21 acquires the complement target part set by the user using the target selection button 591 (step S521). The control unit 21 selects the learning model 61 corresponding to the complement target part acquired in step S521 (step S522). In the subsequent processing, the control unit 21 uses the learning model 61 selected in step S522.

[0083] The control unit 21 acquires a real-time image from the catheter control unit 271 (step S523). The control unit 21 inputs the acquired real-time image to the learning model 61 and acquires an output image showing the complementary line 572 (step S524). The control unit 21 displays the real-time image with the complementary line 572 superimposed in the first image field 51 (step S525).

[0084] The control unit 21 determines whether or not to end the process (step S526). For example, when the end button 585 is selected or when the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 determines that the process is to be ended.

[0085] When it is determined not to end the process (NO in step S526), ​​the control unit 21 returns to step S523. When it is determined to end the process (YES in step S526), ​​the control unit 21 ends the process.

[0086] When the frame rate of the real-time image is high and the process of step S524 cannot keep up, the control unit 21 may execute step S524, for example, once every two or three frames. The control unit 21 may determine the frame for performing the process of step S524 to be synchronized with the electrocardiogram.

[0087] According to this embodiment, it is possible to provide a catheter system 10 that automatically displays the complement line 572 on a real-time image. According to this embodiment, it is possible to provide a catheter system 10 that allows the user to specify the site indicated by the complement line 572.

[0088] [Embodiment 4] This embodiment relates to a catheter system 10 that displays an image in which an information missing portion 473 has been complemented. Description of parts common to the first embodiment will be omitted.

[0089] Fig. 15 is an explanatory diagram illustrating the configuration of a learning model 61 according to the fourth embodiment. The learning model shown in Fig. 15 is a model that, when an input image having an information missing portion 473 is input, outputs a prediction regarding an output image in which the information missing portion 473 has been complemented. The output image in Fig. 15 is an example of a complemented image generated so as to complement the information missing portion 473.

[0090] The learning model 61 includes a missing region extraction model 611, a complement model 612, a cutout unit 613, and a synthesis unit 614. The missing region extraction model 611 and the complement model 612 are neural network models having trainable parameters. The cutout unit 613 and the synthesis unit 614 are computing units that calculate the pixel values ​​of each pixel that constitutes an image.

[0091] The missing area extraction model 611 is a model that receives an input image including an information missing portion 473 and generates a missing area image 483 by extracting a portion indicating the information missing portion 473. The missing area extraction model 611 has a configuration of, for example, Mask-RCNN (Region Convolutional Neural Network), which is a type of object detection model.

[0092] 15, the information missing portion 473 in the missing area image 483 is indicated by hatching. For example, the missing area image 483 is an image of the same size as the input image, in which the pixel value of the pixel corresponding to the information missing portion 473 is "1" and the pixel value of the pixel of the other portion is "0".

[0093] The complementation model 612 is a model that receives an input image including an information missing portion 473 and generates an estimated image 486 without the information missing portion 473. The cutout unit 613 replaces pixel values ​​of pixels that do not correspond to the information missing portion 473, among the pixels of the estimated image 486, with pixel values ​​of the background color, thereby generating a cutout image 487. The complementation model 612 has a configuration in which, for example, multiple convolution layers are successively arranged.

[0094] The synthesis unit 614 synthesizes the input image and the cut image 487 to generate an output image. The output image is an image in which only the portion of the input image corresponding to the information missing portion 473 is replaced with an image generated by the complementation model 612. The cut image 487 is an example of complementation information that complements the information missing portion 473.

[0095] The learning model 61 has already been trained so as to output an output image that does not look strange to an expert when an input image having an information missing portion 473 is input. A method for generating the learning model 61 will be described later.

[0096] 16 is an explanatory diagram for explaining a method of using the learning model 61 according to the embodiment 4. The control unit 21 inputs an input image to the learning model 61, and acquires the cropped image 487 output from the cropping unit 613.

[0097] 17 to 21 are examples of screens displayed by the catheter system 10 of embodiment 4. The screen shown in Fig. 17 includes a first image field 51, a no complement button 580, a complement button 583, a color button 586, a border button 587, and a shaded button 588. Either the no complement button 580 or the complement button 583 is set to a state in which either one is selected at any one time.

[0098] Fig. 17 shows a state in which the No Complement button 580 is selected. The Complement button 583, the Color button 586, and the Frame button 587 are set to a state in which they cannot be selected. A real-time image including the information missing portion 473 is displayed in the first image field 51. In Fig. 17, the information missing portion 473 has not been complemented.

[0099] 18 shows an example of a screen displayed when control unit 21 accepts selection of complement button 583. No complement button 580 is deselected, and color button 586, border button 587, and shaded button 588 are set to a selectable state. Second image field 52 is displayed in the upper right corner of the screen.

[0100] The first image field 51 displays an image obtained by combining the real-time image with a cropped image 487 acquired from the learning model 61. The second image field 52 displays a real-time image. Based on the image in which the information-missing portion 473 displayed in the first image field 51 has been supplemented, the user can determine the cross-sectional shape, cross-sectional area, inner diameter, and the like of the hollow organ.

[0101] The control unit 21 can calculate the area, inner diameter, etc. of the blood vessel lumen based on the image without the information missing portion 473 displayed in the first image field 51. The control unit 21 may display information about a recommended stent based on the calculated area, inner diameter, etc.

[0102] If necessary, the user can observe the real-time image before complementation displayed in the second image field 52. By comparing the first image field 51 and the second image field 52, the user can confirm which part has been complemented by the learning model 61.

[0103] 19 shows an example of a screen displayed when the control unit 21 accepts the selection of the frame button 587. The control unit 21 combines the real-time image with the cropped image 487 with a frame added to the edge, and displays it in the first image field 51. The user can easily distinguish between the part complemented by the learning model 61 and the part actually acquired by the imaging diagnostic catheter 40.

[0104] 20 is an example of a screen displayed when the control unit 21 accepts the selection of the border button 587 and the shaded button 588. The control unit 21 combines the real-time image with a cropped image 487 with a border added to the edge and shaded, and displays it in the first image field 51. It is possible to provide a catheter system 10 in which even an inexperienced user will not mistake the portion complemented by the learning model 61 for the portion actually acquired by the diagnostic imaging catheter 40.

[0105] The user can select any combination of the coloring button 586, the border button 587, and the shading button 588. The coloring button 586, the border button 587, and the shading button 588 are examples of means for selecting the manner of marking on the cut-out image 487. The marking method is not limited to these. Any manner of marking can be set.

[0106] 21 is an example of a screen displayed when the control unit 21 accepts the selection of the coloring button 586. The control unit 21 combines the real-time image with a cut-out image 487 in which the hyperechoic portion is colored, and displays the result in the first image field 51. The user can easily distinguish between the portion complemented by the learning model 61 and the portion actually acquired by the imaging diagnostic catheter 40.

[0107] 22 is a flowchart explaining the flow of processing of the program of the fourth embodiment. The control unit 21 acquires a real-time image from the catheter control unit 271 (step S531). The control unit 21 determines whether or not the selection of the complement button 583 has been accepted (step S532). If it is determined that the selection has not been accepted (NO in step S532), the control unit 21 displays the real-time image in the first image field 51 (step S534).

[0108] If it is determined that the selection has been accepted (YES in step S532), the control unit 21 inputs the acquired real-time image into the learning model 61 to acquire the cropped image 487 (step S533). The control unit 21 determines whether or not the selection of marking for the cropped image 487 has been accepted via the coloring button 586, the border button 587, and the hatching button 588 (step S535).

[0109] If it is determined that the selection of the marking has not been accepted (NO in step S535), control unit 21 composites the real-time image with cropped image 487 (step S536). If it is determined that the selection of the marking has been accepted (YES in step S535), control unit 21 composites the real-time image with cropped image 487 with the designated marking (step S537).

[0110] After step S536 or step S537 is completed, the control unit 21 displays the combined image (step S538). After step S534 or step S538 is completed, the control unit 21 determines whether or not to end the process (step S539). For example, when the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 determines that the process is to be ended.

[0111] If it is determined that the process should not end (NO in step S539), the control unit 21 returns to step S531. If it is determined that the process should end (YES in step S539), the control unit 21 ends the process.

[0112] According to this embodiment, a catheter system 10 can be provided that maintains the portion actually acquired using the diagnostic imaging catheter 40, while complementing and displaying the missing information portion 473 in a natural manner.

[0113] According to this embodiment, it is possible to provide a catheter system 10 that allows the user to identify the completed area by various marking methods.

[0114] [Embodiment 5] This embodiment relates to a catheter system 10 that complements an information missing portion 473 in a transverse image by using a transverse image of another frame. Description of parts common to the first embodiment will be omitted.

[0115] 23 is an explanatory diagram for explaining a method for generating a screen displayed by the catheter system 10 of the embodiment 5. In this embodiment, an example will be described in which an image is acquired by inserting a catheter 40 for image diagnosis into a site where a stent has been placed.

[0116] The first and second tomographic images are transverse images taken at slightly different positions in the longitudinal direction of the probe portion 41. There is no significant difference in the structure of the luminal organ itself between the two transverse images. The stent wires cause multiple information missing areas 473 to radiate outward. Because the stent is mesh-like, when the slice plane of the transverse image is shifted in the longitudinal direction of the probe portion 41, the positions of the information missing areas 473 differ.

[0117] For example, the control unit 21 extracts the information missing portion 473 from the second tomographic image. The control unit 21 cuts out the portion corresponding to each of the information missing portions 473 from the first tomographic image, and combines the cut out portion with the second tomographic image. In this way, a combined image that does not include the information missing portion 473 is generated.

[0118] 24 is a flowchart explaining the flow of processing of the program of embodiment 5. Control unit 21 acquires a first real-time image from catheter control unit 271 and records it in main storage device 22 or auxiliary storage device 23 (step S551). The first real-time image is an example of a catheter image generated using catheter 40 for image diagnosis.

[0119] The control unit 21 extracts the information missing portion 473 in the first real-time image (step S552). The information missing portion 473 is extracted by template matching using, for example, a sector-shaped hypoechoic region as a template. A sector-shaped region in which the end portion on the central side is hyperechoic and the other portion is hypoechoic may be used as the template. The information missing portion 473 may be extracted by a learning model using an object detection algorithm such as Mask-RCNN.

[0120] The control unit 21 acquires the second real-time image from the catheter control unit 271 and records it in the main storage device 22 or the auxiliary storage device 23 (step S553). During execution of this program, the user slightly advances and retreats the diagnostic imaging catheter 40 near the site to be observed. This program may be executed during a pull-back operation using the MDU 33. When executed during a pull-back operation, the control unit 21 acquires the first real-time image, and then acquires the second real-time image one to several frames later. The second real-time image is an example of a second catheter image generated at a different time from the first catheter image.

[0121] The control unit 21 extracts the information missing portion 473 in the second real-time image (step S554). The control unit 21 determines whether the information missing portion 473 extracted in step S552 and the information missing portion 473 extracted in step S554 overlap (step S555). For example, the control unit 21 determines that the two information missing portions 473 overlap when the area of ​​the overlapping portion exceeds a predetermined area.

[0122] If it is determined that there is an overlap (YES in step S555), the control unit 21 returns to step S553. If it is determined that there is no overlap (NO in step S555), the control unit 21 combines the first real-time image and the second real-time image (step S556).

[0123] Specifically, for example, control unit 21 cuts out the portion corresponding to information missing portion 473 extracted in step S554 from the first real-time image and synthesizes it with the second real-time image. Control unit 21 may also cut out the portion corresponding to information missing portion 473 extracted in step S552 from the second real-time image and synthesize it with the first real-time image.

[0124] The control unit 21 displays the combined image in the first image field 51 of the screen described with reference to Figures 17 to 21, for example (step S557). The control unit 21 ends the process.

[0125] According to the present embodiment, the information missing portion 473 is complemented by using a transverse cross-sectional image of another frame, so that a catheter system 10 can be provided that is less susceptible to artifacts due to image synthesis.

[0126] [Embodiment 6] This embodiment relates to a catheter system 10 that complements an information missing portion 473 in an image by using an image of the same frame. Description of parts common to the first embodiment will be omitted.

[0127] 25 is an explanatory diagram for explaining a method of generating a screen displayed by the catheter system 10 of embodiment 6. In this embodiment, an image including an information missing portion 473 due to a guidewire will be described as an example.

[0128] For example, control unit 21 extracts information missing portion 473 from the original image. Control unit 21 copies attachment area 485 having a shape corresponding to information missing portion 473 from a portion of the original image where no information is missing, and composites it into information missing portion 473. In this manner, a composite image that does not include information missing portion 473 is generated.

[0129] 26 is a flowchart explaining the flow of processing of the program of embodiment 6. Control unit 21 acquires a real-time image from catheter control unit 271 (step S561). Control unit 21 extracts information missing portion 473 from the real-time image (step S562).

[0130] Control unit 21 extracts candidates for paste area 485 from the real-time image acquired in step S561 (step S563). Candidates for paste area 485 are extracted by, for example, cutting out a portion from the real-time image that corresponds to the shape obtained by rotating information missing portion 473 around the center of the image as an axis.

[0131] Control unit 21 synthesizes the real-time image acquired in step S561 with the candidate for paste area 485 extracted in step S563 (step S564). Control unit 21 extracts information missing portion 473 from the synthesized image synthesized in step S564 (step S565). The method of extracting information missing portion 473 in step S565 is the same as that in step S562. The control unit 21 determines whether or not the composite image includes the information missing portion 473 (step S566). Specifically, when the information missing portion 473 having a size exceeding a predetermined area is extracted, the control unit 21 determines that the composite image includes the information missing portion 473.

[0132] If it is determined that the image contains missing information portion 473 (YES in step S566), control unit 21 returns to step S563. If it is determined that the image does not contain missing information portion 473 (NO in step S566), control unit 21 displays the image synthesized in step S564 in, for example, first image field 51 of the screen described using Figures 17 to 21 (step S567). Control unit 21 ends the process.

[0133] According to this embodiment, the information missing portion 473 is complemented using the attachment area 485 acquired from the transverse image of the same frame, so that a catheter system 10 with little time lag associated with the complementation process can be provided.

[0134] [Embodiment 7] This embodiment relates to a method for generating the learning model 61 of the third embodiment described with reference to Fig. 11. Description of parts common to the third embodiment will be omitted.

[0135] 27 is an explanatory diagram illustrating the record layout of the training DB. The training data DB is a database in which a detection target, an input image, and an output image are recorded in association with each other, and is used for training the learning model 61 by machine learning. The training DB has a target field, an input image field, and an output image field.

[0136] The object field records the name of the object for which the complementary line 572 is to be created. The input image field records an image acquired using the diagnostic imaging catheter 40. The output image field records an image of the complementary line indicating the object recorded in the object field.

[0137] The training DB records a large number of combinations of the target name, an input image generated using the diagnostic imaging catheter 40, and an output image that has been confirmed to be correct by an expert, etc. The training DB is generated, for example, by a specialist, etc. who is familiar with the specifications of the diagnostic imaging catheter 40, based on case records in which the first or second embodiment is used.

[0138] 28 is a flowchart for explaining the flow of processing of the program according to the embodiment 7. The following will explain an example in which machine learning of the learning model 61 is performed using the information processing device 20.

[0139] 28 may be executed on hardware other than the information processing device 20, and the learning model 61 after the machine learning is completed may be copied to the auxiliary storage device 23 via a network. The learning model 61 trained on one piece of hardware may be used by multiple information processing devices 20.

[0140] Prior to the execution of the program in Fig. 28, an untrained model is prepared, for example, by combining a convolutional layer, a pooling layer, and a fully connected layer. The program in Fig. 28 adjusts each parameter of the prepared model and performs machine learning.

[0141] The control unit 21 acquires a training record to be used for training one epoch from the training DB (step S571). The control unit 21 adjusts the parameters of the model so that when an input image is input to the input layer of the model, an output image is output from the output layer (step S572).

[0142] The control unit 21 determines whether to end the process (step S573). For example, the control unit 21 determines to end the process when learning for a predetermined number of epochs is completed. The control unit 21 may obtain test data from the training DB, input the test data to the model under machine learning, and determine to end the process when an output with a predetermined accuracy is obtained.

[0143] If it is determined that the process should not be terminated (NO in step S573), control unit 21 returns to step S571. If it is determined that the process should be terminated (YES in step S573), control unit 21 records the parameters of the trained model in auxiliary storage device 23 (step S574). Thereafter, control unit 21 terminates the process. Through the above process, a trained model is generated.

[0144] According to this embodiment, the learning model 61 described in the third embodiment can be generated by machine learning.

[0145] [Embodiment 8] This embodiment relates to a method for generating the learning model 61 of the fourth embodiment described with reference to Fig. 15. Description of parts common to the fourth embodiment will be omitted.

[0146] FIG. 29 is an explanatory diagram for explaining an outline of a method for generating the learning model 61. As explained using FIG. 15, the learning model 61 is a model that receives an input image and generates an output image. The classifier 65 is a model that receives an output image output from the learning model 61 and determines whether the image is a true image or a false image. The classifier 65 has a configuration in which, for example, a convolution layer and a pooling layer are repeated, a fully connected layer, and a softmax layer are connected.

[0147] Generative Adversarial Networks (GAN) is used to train the classifier 65 and the learning model 61 and adjust the parameters, enabling the learning model 61 to generate natural output images.

[0148] 30 is a flowchart for explaining the flow of processing of the program according to the embodiment 8. The following will explain an example in which machine learning of the learning model 61 is performed using the information processing device 20.

[0149] 30 may be executed on hardware other than the information processing device 20, and the learning model 61 for which machine learning has been completed may be copied to the auxiliary storage device 23 via a network. The learning model 61 trained on one piece of hardware may be used by multiple information processing devices 20.

[0150] Prior to the execution of the program in FIG. 30, a model is prepared by combining an untrained learning model 61 having the configuration described using FIG. 15 with an untrained classifier 65.

[0151] The control unit 21 acquires a plurality of input images (step S581). The input images are recorded in the input image field of the training DB described with reference to Fig. 27, for example. The input images include an image that does not include the information missing portion 473.

[0152] The control unit 21 adjusts the parameters of the classifier 65 so as to output "false" for images input to the classifier 65 via the learning model 61, and "true" for images that do not include the information missing portion 473 and that are input to the classifier 65 without passing through the learning model 61 (step S582).

[0153] The control unit 21 adjusts the parameters of the learning model 61, specifically, the parameters of the missing area extraction model 611 and the complementation model 612, so that the classifier 65 outputs "true" and "false" with half the probability (step S583). The control unit 21 may repeat the process of steps S582 and S583 multiple times.

[0154] The control unit 21 records the parameters of the trained learning model 61 in the auxiliary storage device 23 (step S584). After that, the control unit 21 ends the process. Through the above process, the trained learning model 61 is generated.

[0155] According to this embodiment, the learning model 61 described in the fourth embodiment can be generated by machine learning.

[0156] [Embodiment 9] This embodiment relates to a catheter system 10 that distinguishes and displays a highly reliable portion from a less reliable portion of an interpolated line 572. Description of portions common to the third embodiment will be omitted.

[0157] 31 is an explanatory diagram for explaining a method for generating an image displayed by the catheter system 10 of embodiment 9. In this embodiment, an example will be described in which the display range of a tomographic image obtained using the catheter 40 for diagnostic imaging is small, that is, an input image displaying the vicinity of the catheter 40 for diagnostic imaging is used.

[0158] The input image displays a guidewire image 472 and a shadowed portion image 474. The shadowed portion image 474 is an image showing a strong reflector such as a strongly calcified portion, a stent, or another medical device used simultaneously with the diagnostic imaging catheter 40. An information missing portion 473 due to acoustic shadowing is formed outside the guidewire image 472 and outside the shadowed portion image 474. That is, the guidewire image 472 and the shadowed portion image 474 are examples of images showing a shadowed portion.

[0159] The control unit 21 generates two complementary lines 572, a first complementary line 561 and a second complementary line 562, based on the input image (step S101). In this embodiment, the first complementary line 561 indicates the inner surface of a blood vessel, and the second complementary line 562 indicates the external elastic lamina. The number of complementary lines 572 generated in step S101 may be one or three or more.

[0160] The control unit 21 can generate an image with complementary lines by combining the input image with the complementary lines 572 (step S102). Note that the image with complementary lines is shown for the convenience of explanation, and does not need to be generated in reality.

[0161] The control unit 21 extracts the guidewire image 472 and the shadow forming portion image 474 from the input image or the image with the complement line (step S103). The guidewire image 472 is an area with a brightness higher than a predetermined brightness, and is an area existing inside the first complement line 561. The control unit 21 may extract the guidewire image 472 by pattern matching based on a shape and dimensions specified in advance.

[0162] The shadow-forming portion image 474 is an area with a higher brightness than a predetermined brightness and is an area existing outside the first complementary line 561. The shadow-forming portion image 474 corresponds to, for example, a stent wire or a heavily calcified portion. The guidewire image 472 and the shadow-forming portion image 474 are examples of portions in which the cause of formation of an acoustic shadow portion is depicted.

[0163] The control unit 21 determines a low reliability region 55 based on the extracted guidewire image 472 and shadow formation portion image 474. The low reliability region 55 is a substantially fan-shaped region combining a region near the guidewire image 472 and a region outside the guidewire image 472, and a substantially fan-shaped region combining a region near the shadow formation portion image 474 and a region outside the shadow formation portion image 474.

[0164] Since the guidewire image 472 or the shadow-forming portion image 474 is present between the diagnostic imaging catheter 40 and the low-reliability region 55, there is a high possibility that an acoustic shadow is generated. Therefore, the reliability of the complement line 572 is low inside the low-reliability region 55. Note that the control unit 21 may include, in the low-reliability region 55, an area of ​​a predetermined width that is closer to the diagnostic imaging catheter than the guidewire image 472 and the shadow-forming portion image 474.

[0165] 32 is an explanatory diagram for explaining a method for generating the complementary line 572 according to the ninth embodiment. The learning model 61 according to the present embodiment is a model that receives an input image and outputs classification data. The classification data is data that associates each part constituting the input image with a label that classifies each subject depicted in that part. Each part is, for example, each pixel. The classification data can be used to generate a classification image in which the input image is colored differently for each subject depicted in the input image.

[0166] A concrete example will be given for explanation. The learning model 61 outputs classification data in which each pixel constituting the input image is classified into, for example, a first label, a second label, or a third label. An example of a classified image generated based on the classification data is shown. A first label region 541, a second label region 542, and a third label region 543 are arranged in a substantially concentric shape with a catheter image 471 at the center.

[0167] The first labeled region 541 indicates the lumen of the luminal organ into which the diagnostic imaging catheter 40 is inserted, i.e., the region of the blood vessel lumen through which blood flows. The second labeled region 542 indicates the lumen wall, i.e., the blood vessel wall. The third labeled region 543 indicates the outside of the lumen wall, i.e., the region outside the external elastic lamina which indicates the outer surface of the luminal organ. The third labeled region 543 includes, for example, muscle, nerve, fat, and other blood vessels adjacent to the blood vessel into which the diagnostic imaging catheter 40 is inserted.

[0168] The boundary line between the first label region 541 and the second label region 542 corresponds to the above-mentioned first complementary line 561. The boundary line between the second label region 542 and the third label region 543 corresponds to the above-mentioned second complementary line 562. As described above, the complementary line 572 created based on the input image may be referred to as a complementary line 572 of mode 1 in the following description.

[0169] 32, an input image displayed in a so-called XY format and a classification image in which classification data is displayed in the XY format are shown in schematic form. The learning model 61 may receive an input image in a so-called RT format formed by arranging scanning line data formed by transmitting and receiving ultrasonic waves in parallel in the order of scanning angles, and output classification data. The method of converting from the RT format to the XY format is well known, and therefore a description thereof will be omitted. Since there is no influence of the interpolation process, etc., which is performed when converting the input image from the RT format to the XY format, more appropriate classification data is generated.

[0170] The learning model 61 is, for example, a trained model that performs semantic segmentation on an input image. The trained model that performs semantic segmentation is generated by machine learning using training data that combines an input image generated using the imaging diagnostic catheter 40 and a classification image in which an expert colors the input image for each depicted subject.

[0171] The learning model 61 may be created for each site into which the diagnostic imaging catheter 40 is inserted, such as the coronary artery, the lower limb aorta, the bile duct, the pancreatic duct, and the bronchus. The learning model 61 may be created for each display range in which a tomographic image is generated using the diagnostic imaging catheter 40. The learning model 61 may be created for each patient attribute, such as the patient's age or sex.

[0172] The complementary line 572 of mode 1 may be generated using the learning model 61 of the third embodiment described with reference to Fig. 11. When the learning model 61 of the third embodiment is used, the first complementary line 561 indicating the inner surface of the blood vessel and the second complementary line 562 indicating the external elastic plate are generated using appropriate learning models 61. The complementary line 572 of mode 1 may be generated by any other method.

[0173] Fig. 33 is an explanatory diagram for explaining a method for generating a complementary line 572 in mode 2. As explained using Fig. 31, the explanation will start from the process after the generation of the complementary line 572 in mode 1 and the determination of the low-reliability region 55 are completed. In Fig. 33, the guidewire image 472 and the shadow-forming portion image 474 are omitted from illustration.

[0174] The control unit 21 deletes the portion of the complement line 572 of mode 1 that is included in the low reliability region 55 (step S111). The complement line 572 is broken in part. The control unit 21 generates a change line 565 that smoothly connects the broken portion of the complement line 572. In this way, the control unit 21 generates the complement line 572 of the second mode (step S112). In FIG. 33, the change lines 565 corresponding to the respective complement lines 572 are shown in bold.

[0175] Any interpolation method, such as spline interpolation, Lagrange interpolation, or linear interpolation, can be used to generate the modified line. The control unit 21 may accept a user's designation of the generation method. The second mode complementary line 572 is less susceptible to artifacts such as multiple echoes formed by a strong reflector such as a guide wire. The above-described second mode complementary line is a complementary line obtained by correcting an area of ​​the complementary line 572 that is less reliable than other areas based on an area that is not less reliable.

[0176] The control unit 21 determines that the reliability of the entire complement line 572 is low when a large portion of the complement line 572 is included in the low reliability region 55. For example, when the shadow-forming portion image 474 is a stent wire or when there is extensive and strong calcification, the range of the low reliability region 55 is wide and a large portion of the complement line 572 is included in the low reliability region 55.

[0177] 34 is a flowchart for explaining the flow of processing of the program of the ninth embodiment. The control unit 21 acquires a real-time image from the catheter control unit 271 (step S601). The control unit 21 determines whether or not an instruction to perform complementation has been received from the user (step S602). An example of a screen for receiving an instruction from the user will be described later.

[0178] If it is determined that the image has not been accepted (NO in step S602), the control unit 21 displays a real-time image in the first image field 51 (step S603). If it is determined that the image has been accepted (YES in step S602), the control unit 21 starts a complementary line generation subroutine (step S611). The complementary line generation subroutine is a subroutine that generates complementary line 572. The processing flow of the complementary line generation subroutine will be described later.

[0179] The control unit 21 extracts a high-luminance region from the real-time image (step S612). A high-luminance region is, for example, a region in which a greater number of pixels than a predetermined threshold are clustered together. The brightness threshold and the pixel count threshold may be appropriately set by the user. In step S612, the guidewire image 472 and the shadow forming portion image 474 are extracted.

[0180] The control unit 21 sets the range of the low reliability region 55 (step S613). In the XY format image, the low reliability region 55 is a substantially sector-shaped region outside the guidewire image 472 and the shadow formation portion image 474. The control unit 21 calculates the ratio of the length of the range included in the low reliability region 55 to the total length for one of the complementary lines 572 generated in step S611 (step S614).

[0181] The control unit 21 judges whether the proportion of the range included in the low reliability region 55 is greater than a predetermined threshold (step S615). If it is judged that the proportion is greater (YES in step S615), the control unit 21 temporarily records in the main storage device 22 or the auxiliary storage device 23 that the proportion of the range included in the low reliability region 55 for the complement line 572 being processed is large (step S616).

[0182] If it is determined that the number is not large (NO in step S615), the control unit 21 determines whether or not an instruction to display the complement lines 572 in mode 2 has been received from the user (step S617). An example of a screen for receiving an instruction from the user will be described later.

[0183] If it is determined that the input has been accepted (YES in step S617), control unit 21 generates complementary lines 572 of mode 2 described with reference to Fig. 33 (step S618). If it is determined that the input has not been accepted (NO in step S617), after the completion of step S618 or after the completion of step S616, control unit 21 determines whether or not the processing of all complementary lines 572 generated in step S611 has been completed (step S619).

[0184] If it is determined that the process has not ended (NO in step S619), the control unit 21 returns to step S614. If it is determined that the process has ended (YES in step S619), the control unit 21 displays the real-time image and the complementary line 572 in the first image field 51 (step S620).

[0185] After step S603 or step S620 is completed, the control unit 21 determines whether or not to end the process (step S621). For example, when the diagnostic imaging catheter 40 is detached from the MDU 33, the control unit 21 determines that the process is to be ended.

[0186] If it is determined that the process should not end (NO in step S621), the control unit 21 returns to step S601. If it is determined that the process should end (YES in step S621), the control unit 21 ends the process.

[0187] 35 is a flowchart explaining the process flow of the complementary line generation subroutine. The control unit 21 inputs the real-time image to the learning model 61 to acquire classification data (step S631). That is, in step S631, the control unit 21 acquires labels corresponding to the subjects depicted in each part constituting the real-time image. The control unit 21 extracts complementary lines corresponding to the boundaries between areas whose acquired labels are different from each other (step S632). The control unit 21 ends the process.

[0188] Fig. 36 is an example of a screen displayed by the catheter system 10 of embodiment 9. The control unit 21 displays the screen shown in Fig. 36 on the display device 31 in step S603 of the flowchart described using Fig. 34.

[0189] 36 includes a first image field 51, two object selection buttons 591, an intraluminal high-brightness area button 594, and an intramural high-brightness area button 595. Two mode selection buttons 592 and a non-display selection button 593 are displayed below each object selection button 591. A real-time image obtained using the diagnostic imaging catheter 40 is displayed in the first image field 51.

[0190] In Fig. 36, none of the buttons has been selected by the user. In step S602 of the flowchart described with reference to Fig. 34, control unit 21 determines that an instruction to perform complement has not been received from the user (NO in step S602), and executes step S603.

[0191] The target selection button 591 accepts the selection of the complementary line 572 to be extracted. In FIG. 36, the user can select "luminal inner surface" and "EEM". Note that "luminal inner surface" and "EEM" are examples. The control unit 21 appropriately displays the type of complementary line 572 corresponding to the luminal organ into which the diagnostic imaging catheter 40 is inserted. The control unit 21 may display the target selection button 591 for the type of complementary line 572 previously set by the user.

[0192] Mode selection button 592 accepts a selection regarding the mode of complementary line 572. When a selection of "mode 1" is accepted, control unit 21 generates complementary line 572 of mode 1 described using FIG. 31. When a selection of "mode 2" is accepted, control unit 21 generates complementary line 572 of mode 2 described using FIG. 33.

[0193] The non-display selection button 593 accepts a selection as to whether or not to display the complement line 572 for the portion included in the low-reliability region 55. The intraluminal high-brightness area button 594 accepts a selection as to whether or not to display information relating to a high-brightness area existing inside the lumen of a luminal organ. The intramural high-brightness area button 595 accepts a selection as to whether or not to display information relating to a high-brightness area existing inside the luminal wall, i.e., between the inner wall of a luminal organ and the external elastic lamina.

[0194] Figures 37 to 41 are examples of screens displayed by the catheter system 10 of embodiment 9. The control unit 21 displays the screens shown in Figures 37 to 41 on the display device 31 in step S620 of the flowchart described using Figure 34.

[0195] 37, selection of object selection buttons 591 corresponding to "Luminous inner surface" and "EEM" and mode selection button 592 for "Mode 1" are accepted. A legend field 596 is displayed to the right of the characters "Luminous inner surface" and "EEM."

[0196] In the first image field 51, an image in which a first complementary line 561 and a second complementary line 562 are superimposed on a real-time image obtained using the diagnostic imaging catheter 40 is displayed. Of the first complementary line 561 corresponding to the "inner surface of the lumen", the portion that does not overlap with the low-reliability region 55 described using Fig. 31 is displayed as a thick solid line, and the portion that overlaps with the low-reliability region 55 is displayed as a thin solid line. Of the second complementary line 562 corresponding to the "EEM", the portion that does not overlap with the low-reliability region 55 is displayed as a thick dashed line, and the portion that overlaps with the low-reliability region 55 is displayed as a thin dashed line.

[0197] 37, among the complementary lines 572, areas having a lower reliability than other areas are displayed in a manner different from areas having a higher reliability. Note that the complementary lines 572 and the areas having a lower reliability and areas having a higher reliability may be displayed in a manner distinguished from each other by color, brightness, or the like.

[0198] 37, the user can easily distinguish between a highly reliable portion and a less reliable portion of the complementary line 572 generated by the control unit 21. The user can appropriately select the complementary line 572 to be displayed in the first image field 51 by operating the target selection button 591.

[0199] 38, the selection of the non-display selection button 593 corresponding to “EEM” is accepted. In the first image field 51, of the second complementary line 562 displayed, the portion that overlaps with the low-reliability region 55 is in a deleted state.

[0200] Using the screen shown in FIG. 38, the user can visually check the unreliable parts of the second complementary line 562 and make an appropriate decision based on expert knowledge.

[0201] 39, the intraluminal high-brightness area button 594 and the intramural high-brightness area button 595 are selected. The first image field 51 displays the low-reliability area 55 generated by the guidewire image 472, which is an intraluminal high-brightness area, and the low-reliability area 55 generated by the low-reliability area 55, which is an intramural high-brightness area.

[0202] The control unit 21 may display the guidewire image 472 and the shadow formation portion image 474 in color. The user can confirm which portion the control unit 21 has determined to cause the low reliability region 55.

[0203] Figure 40 shows a real-time image of a stent-implanted area. A number of shadow-forming area images 474 formed by the stent wires are displayed in a circular arrangement. Therefore, low-reliability regions 55 corresponding to the respective shadow-forming area images 474 are generated.

[0204] FIG. 40 shows an example of a screen displayed by the control unit 21 when it is determined in step S615 described with reference to FIG. 34 that the range included in the low reliability region 55 for the second supplemental line 562 is larger than the threshold value.

[0205] The control unit 21 does not display the second complementary line 562 in the first image field 51. The control unit 21 displays a notification field 597 indicating that it is "not displayable" to the right of the characters "EEM" in place of the legend field 596. The user can understand that the second complementary line 562 generated by the control unit 21 is not displayed in the first image field 51 because its reliability is low.

[0206] 41 shows an example of a screen displayed by control unit 21 when the user, for example, clicks on notification field 597 to instruct display of unreliable second complementary line 562. Second complementary line 562 is displayed as a thin dashed line in its entirety, indicating that it is unreliable. A notification "EEM is unreliable" is displayed in the lower left corner of the screen, indicating that second complementary line 562 is unreliable.

[0207] According to this embodiment, it is possible to provide a catheter system 10 that distinguishes and displays the highly reliable portion of the complementary line 572 from the less reliable portion.

[0208] The learning model 61 may output labels corresponding to the guidewire image 472 and the shadow-forming portion image 474. In this case, step S612 described with reference to FIG. 34 is not necessary.

[0209] The learning model 61 may output a label corresponding to, for example, a highly attenuating plaque. In this case, the control unit 21 may set the region outside the highly attenuating plaque as the low-reliability region 55 in step S613 described with reference to FIG.

[0210] Instead of real-time images, videos or still images stored in the auxiliary storage device 23 or the like may be used. It is possible to provide a catheter system 10 that can be used for recording medical records after the end of a case. In such a case, the information processing device 20 may be a personal computer, tablet, smartphone, or the like that does not have a function for connecting the MDU 33 and the catheter 40 for image diagnosis.

[0211] [Embodiment 10] 42 is a functional block diagram of an information processing system 10 according to a tenth embodiment. The information processing system 10 includes an image acquiring unit 81, a complementary information acquiring unit 82, and a display unit 83. The image acquiring unit 81 acquires a catheter image generated using an imaging diagnostic catheter 40 inserted into a tubular organ. The complementary information acquiring unit 82 acquires complementary information that complements an information missing portion 473 of the catheter image acquired by the image acquiring unit 81. The display unit 83 displays the catheter image acquired by the image acquiring unit 81 and the complementary information acquired by the complementary information acquiring unit 82.

[0212] [Embodiment 11] Fig. 43 is an explanatory diagram for explaining the configuration of a catheter system 10 of an eleventh embodiment. This embodiment relates to a form in which the catheter system 10 of this embodiment is realized by combining and operating a catheter control device 27, an MDU 33, an image diagnostic catheter 40, a general-purpose computer 90, and a program 97. Explanations of parts common to the first embodiment will be omitted. The catheter system 10 is also an example of the information processing system of this embodiment.

[0213] Catheter control device 27 is an ultrasonic diagnostic device for IVUS that controls MDU 33, controls sensor 42, and generates transverse and longitudinal images based on signals received from sensor 42. The function and configuration of catheter control device 27 are similar to those of conventional ultrasonic diagnostic devices, and therefore a description thereof will be omitted.

[0214] The catheter system 10 of this embodiment includes a computer 90. The computer 90 includes a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, a smartphone, or a server computer.

[0215] The program 97 is recorded in a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read out the program 97 stored in a semiconductor memory 98 such as a flash memory implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown), and store it in the auxiliary storage device 23.

[0216] The program 97 is installed as a control program for the computer 90, and is loaded into and executed by the main storage device 22. This causes the computer 90 to function as the information processing device 20 described above.

[0217] The computer 90 is a general-purpose personal computer, a tablet, a smartphone, a mainframe computer, a virtual machine running on a mainframe computer, a cloud computing system, or a quantum computer. The computer 90 may be a plurality of personal computers performing distributed processing.

[0218] The technical features (constitutive elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims.

[0219] (Appendix 1) Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; Acquire complementary information for complementing an information missing portion of the acquired catheter image; Displaying the catheter image and the complementary information A program that causes a computer to carry out processing.

[0220] (Appendix 2) The acquired catheter image is input to a learning model that outputs complementary information when the catheter image is input, and complementary information output from the learning model is obtained. 2. The program described in Appendix 1.

[0221] (Appendix 3) The complementary information is a complementary image generated to complement the information-missing portion. 2. The program described in Appendix 2.

[0222] (Appendix 4) The missing information portion is complemented based on a second catheter image acquired at a time different from the catheter image. 2. The program described in Appendix 1.

[0223] (Appendix 5) The information missing portion is complemented based on a portion of the catheter image that is different from the information missing portion. 2. The program described in Appendix 1.

[0224] (Appendix 6) the imaging diagnostic catheter is a catheter for generating a tomographic image, the catheter image is a tomographic image generated using the tomographic image generating catheter, The information missing portion is a shaded portion. 6. A program according to any one of claims 1 to 5.

[0225] (Appendix 7) The catheter for generating a tomographic image is a catheter for generating an ultrasonic tomographic image. 6. The program described in Appendix 6.

[0226] (Appendix 8) The catheter for generating a tomographic image is a catheter for generating an optical tomographic image. 6. The program described in Appendix 6.

[0227] (Appendix 9) The catheter for generating a tomographic image is a catheter equipped with a sensor for generating an ultrasonic tomographic image and a sensor for generating an optical tomographic image. 6. The program described in Appendix 6.

[0228] (Appendix 10) extracting a plurality of points indicating an inner surface of a luminal wall constituting the luminal organ from the tomographic image; The shaded area is complemented by a curve connecting the extracted points. 10. The program according to any one of claims 6 to 9.

[0229] (Appendix 11) The acquired catheter image and a complemented image in which the missing information portion has been complemented are displayed side by side. 11. A program according to any one of claims 1 to 10.

[0230] (Appendix 12) Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; Acquire complementary information for complementing an information missing portion of the acquired catheter image; Displaying the catheter image and the complementary information An information processing method for causing a computer to execute processing.

[0231] (Appendix 13) an image acquisition unit that acquires a catheter image generated using a diagnostic imaging catheter inserted into a hollow organ; a complementary information acquisition unit that acquires complementary information for complementing an information missing portion of the acquired catheter image; a display unit that displays the catheter image and the complementary information; Information processing system.

[0232] (Appendix 14) Acquire training data in which a catheter image generated using a diagnostic imaging catheter inserted into a tubular organ and complementary information for complementing an information-missing portion of the catheter image are associated with each other and recorded; A learning model is generated that receives the catheter image as an input, outputs the complementary information, and outputs a prediction regarding the complementary information when an image generated using a catheter for image diagnosis is input. How to generate a learning model. [Explanation of symbols]

[0233] 10 Catheter system (information processing system) 20 Information processing device 21 Control section 22 Main storage 23 Auxiliary storage device 24 Communications Department 25 Display section 26 Input section 27 Catheter Control Device 271 Catheter control section 29 Reading unit 31 Display device 32 Input Devices 33 MDU 37 Diagnostic Imaging Devices 40 Diagnostic Imaging Catheter 41 Probe section 42 Sensors 43 Shaft 44 Tip Marker 45 Connector part 471 Catheter Image 472 Guidewire image 473 Information missing section 474 Shadow forming area image 483 Missing Area Images 485 Attachment Area 486 Estimated Images 487 Cropped image (supplementary information) 51 First Image 52 Second Image Column 541 First Label Area 542 Second Label Area 543 Third Label Area 55 Low confidence region 561 First Complementary Line 562 Second Complementary Line 565 Change Line 571 Designated point mark 572 Complementary Line (Complementary Information) 573 Candidate point mark 574 Template Area 580 No Completion Button 581 Stop button 582 Selection button 583 Completion button 584 Start button 585 Exit button 586 Colored Buttons 587 Border Button 588 Shaded Button 591 Target selection button 592 Mode selection button 593 Hidden Selection Button 594 Intraluminal High-Intensity Button 595 High-intensity button inside the wall 596 Legend 597 Notification column 61 Learning Model 611 Missing Region Extraction Model 612 Complementary Models 613 Cutout 614 Synthesis Section 65 Classifier 68 Cursor 90 Computer 96 Portable recording media 97 Program 98 Semiconductor Memory

Claims

1. Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; Acquire complementary information that complements the acquired catheter image including an information missing portion; correcting a region of the obtained complementary information that corresponds to the information missing portion and has a lower reliability than other regions based on the other regions; Displaying the catheter image and the corrected complementary information A program that causes a computer to carry out processing.

2. A region of the pre-correction complementary information that is less reliable than other regions is displayed in a manner different from the other regions. The program according to claim 1.

3. The acquired catheter image is input to a learning model that outputs complementary information when the catheter image is input, and complementary information output from the learning model is obtained. The program according to claim 1 or 2.

4. The complementary information is generated based on a second catheter image acquired at a time different from the catheter image. The program according to claim 1 or 2.

5. The complementary information is generated based on a portion of the catheter image that is different from the information-missing portion. The program according to claim 1 or 2.

6. the imaging diagnostic catheter is a catheter for generating a tomographic image, the catheter image is a tomographic image generated using the tomographic image generating catheter, The information missing portion is a shaded portion. The program according to any one of claims 1 to 5.

7. extracting a plurality of points corresponding to a luminal wall constituting the hollow organ from the tomographic image; The complementary information includes complementary lines connecting the extracted points. The program according to claim 6.

8. inputting the acquired catheter image into a learning model which outputs labels corresponding to each part constituting the catheter image when the catheter image is input, and acquiring labels output from the learning model; The complementary information includes complementary lines indicating boundaries between regions having different labels. The program according to claim 6.

9. the learning model outputs a first label indicating an inner cavity of a hollow organ into which the diagnostic imaging catheter is inserted, a second label indicating a luminal wall constituting the hollow organ, and a third label indicating an outside of the luminal wall; The complementary lines include a first complementary line indicating a boundary between an area corresponding to the first label and an area corresponding to the second label, and a second complementary line indicating a boundary between an area corresponding to the second label and an area corresponding to the third label. The program according to claim 8.

10. extracting a shadow forming portion from the tomographic image, the shadow forming portion corresponding to a cause of formation of the shadow portion; A portion of the complementary line where the shadow forming portion is interposed between the complementary line and the catheter for image diagnosis is displayed in a manner different from other portions. The program according to any one of claims 7 to 9.

11. extracting a shadow forming portion from the tomographic image, the shadow forming portion corresponding to a cause of formation of the shadow portion; A portion of the complementary line where the shadow forming portion is interposed between the complementary line and the catheter for diagnostic imaging is changed based on another portion of the complementary line. The program according to any one of claims 7 to 10.

12. extracting a shadow-forming portion from the tomographic image in which a cause of the formation of the shadow portion is depicted; A notification is output when a ratio of a portion of the complementary line where the shadow forming portion is interposed between the complementary line and the catheter for image diagnosis exceeds a threshold value. The program according to any one of claims 7 to 11.

13. The catheter for generating a tomographic image is a catheter for generating an ultrasonic tomographic image. The program according to any one of claims 6 to 12.

14. The catheter for generating a tomographic image is a catheter for generating an optical tomographic image. The program according to any one of claims 6 to 12.

15. The catheter for generating a tomographic image is a catheter equipped with a sensor for generating an ultrasonic tomographic image and a sensor for generating an optical tomographic image. The program according to any one of claims 6 to 12.

16. The acquired catheter image and a complemented image in which the missing information portion has been complemented are displayed side by side. A program according to any one of claims 1 to 15.

17. Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; Acquire complementary information that complements the acquired catheter image including an information missing portion; correcting a region of the obtained complementary information that corresponds to the information missing portion and has a lower reliability than other regions based on the other regions; Displaying the catheter image and the corrected complementary information An information processing method for causing a computer to execute processing.

18. an image acquisition unit that acquires a catheter image generated using a diagnostic imaging catheter inserted into a hollow organ; a complementary information acquisition unit that acquires complementary information that complements an information missing portion of the acquired catheter image; a correction unit that corrects a region of the acquired complementary information, the region corresponding to the information missing portion and having a lower reliability than other regions, based on the other regions; a display unit that displays the catheter image and the complementary information corrected by the correction unit. Information processing system.

Images