Program, information processing method and information processing device

Abstract

To provide a program or the like capable of easily using a catheter system.SOLUTION: A program causes a computer to execute processing for acquiring a catheter image generated by using a catheter for image diagnosis inserted into a hollow organ, acquiring complement information 572 including an information lack part 473 of the acquired catheter image to complement it, correcting a region corresponding to the information lack part 473 and having reliability lower than the other regions in the acquired complement information 572 on the basis of the other regions, and displaying the catheter image and the corrected complement information 572.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing method, and an information processing Device Regarding. [Background technology]

[0002] BACKGROUND ART A catheter system is used in which a diagnostic imaging catheter is inserted into a hollow organ such as a blood vessel to capture a tomographic image (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 164071 Summary of the Invention [Problem to be solved by the invention]

[0004] Guidewires, stents, or heavily calcified areas can cause missing information in tomographic images. Experienced physicians and other specialists perform catheter treatments while filling in the missing information through guesswork. This requires extensive training to use catheter systems.

[0005] In one aspect, an object is to provide a program or the like that makes it possible to easily use a catheter system. [Means for solving the problem]

[0006] The program acquires a catheter image generated using a diagnostic imaging catheter inserted into a hollow organ, acquires complementary information that complements information missing from the acquired catheter image, and 、 Less reliable than other areas determining a display mode of the complementary information based on the low-reliability region; The catheter image. The complementary information according to the display mode The computer is caused to execute a process of displaying the above. [Effects of the Invention]

[0007] In one aspect, a program or the like can be provided that makes it easy to use the catheter system. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of a catheter system. [Figure 2] FIG. 1 is an explanatory diagram illustrating an overview of a catheter for diagnostic imaging. [Figure 3] FIG. 1 is an explanatory diagram illustrating the configuration of a catheter system. [Figure 4] 10 is an example of a screen displayed by the catheter system. [Figure 5] 10 is an example of a screen displayed by the catheter system. [Figure 6] 10 is an example of a screen displayed by the catheter system. [Figure 7] 10 is an example of a screen displayed by the catheter system. [Figure 8] 10 is a flowchart illustrating the flow of processing of a program. [Figure 9] 10 is an example of a screen displayed by the catheter system of the second embodiment. [Figure 10] 10 is a flowchart illustrating the flow of processing of a program according to the second embodiment. [Figure 11] FIG. 2 is an explanatory diagram illustrating the configuration of a learning model. [Figure 12] 10 is an example of a screen displayed by the catheter system of the third embodiment. [Figure 13] 10 is an example of a screen displayed by the catheter system of the third embodiment. [Figure 14] 11 is a flowchart illustrating the flow of processing of a program according to the third embodiment. [Figure 15] FIG. 10 is an explanatory diagram illustrating the configuration of a learning model according to a fourth embodiment. [Figure 16] FIG. 10 is an explanatory diagram illustrating how to use the learning model of the fourth embodiment. [Figure 17] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 18] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 19] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 20] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 21] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 22] 10 is a flowchart illustrating the flow of processing of a program according to a fourth embodiment. [Figure 23] FIG. 10 is an explanatory diagram illustrating a method for generating a screen to be displayed by the catheter system of the fifth embodiment. [Figure 24] 13 is a flowchart illustrating the flow of processing of a program according to the fifth embodiment. [Figure 25] FIG. 20 is an explanatory diagram for explaining a method for generating a screen to be displayed by the catheter system of the sixth embodiment. [Figure 26] 13 is a flowchart illustrating the flow of processing of a program according to the sixth embodiment. [Figure 27] FIG. 10 is an explanatory diagram illustrating the record layout of a training data DB. [Figure 28] 13 is a flowchart illustrating the flow of processing of a program according to a seventh embodiment. [Figure 29] FIG. 1 is an explanatory diagram illustrating an overview of a method for generating a learning model. [Figure 30] 13 is a flowchart illustrating the flow of processing of a program according to the eighth embodiment. [Figure 31] FIG. 20 is an explanatory diagram illustrating a method for generating an image displayed by the catheter system of the ninth embodiment. [Figure 32] FIG. 20 is an explanatory diagram illustrating a method for generating a complementary line according to a ninth embodiment. [Figure 33] FIG. 10 is an explanatory diagram illustrating a method for generating a complementary line in mode 2. [Figure 34] 13 is a flowchart illustrating the flow of processing of a program according to a ninth embodiment. [Figure 35] 10 is a flowchart illustrating a process flow of a subroutine for generating complementary lines. [Figure 36] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Figure 37] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Figure 38] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Figure 39] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Figure 40] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Figure 41] 13 is an example of a screen displayed by the catheter system of the ninth embodiment. [Figure 42] FIG. 23 is a functional block diagram of an information processing system according to a tenth embodiment. [Figure 43] FIG. 20 is an explanatory diagram illustrating the configuration of a catheter system according to an eleventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Embodiment 1] FIG. 1 is an explanatory diagram illustrating an overview of a catheter system 10. The catheter system 10 includes a diagnostic imaging catheter 40, an MDU (Motor Driving Unit) 33, and an information processing device 20. The diagnostic imaging catheter 40 is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is an input device such as a keyboard, a mouse, a trackball, or a microphone. The display device 31 and the input device 32 may be stacked together to form a touch panel. The input device 32 and the information processing device 20 may be configured as an integrated unit.

[0010] Fig. 2 is an explanatory diagram illustrating an overview of an imaging diagnostic catheter 40. Fig. 2 shows an example of an imaging diagnostic catheter 40 for IVUS (Intravascular Ultrasound) used to generate an ultrasound tomographic image from inside a blood vessel, i.e., for generating an ultrasound tomographic image. An ultrasound tomographic image is an example of a catheter image generated using the imaging diagnostic catheter 40. An IVUS catheter is an example of a catheter for generating a tomographic image.

[0011] The diagnostic imaging catheter 40 has a probe portion 41 and a connector portion 45 disposed at the end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the diagnostic imaging catheter 40 farther from the connector portion 45 will be referred to as the tip side.

[0012] A shaft 43 is inserted inside the probe section 41. A sensor 42 is connected to the tip side of the shaft 43. The shaft 43 and the sensor 42 are rotatable and movable forward and backward inside the probe section 41.

[0013] The sensor 42 is an ultrasonic transducer that transmits and receives ultrasonic waves. An annular tip marker 44 is fixed near the tip of the probe part 41. The tip marker 44 is made of a material that is opaque to X-rays, such as metal.

[0014] The diagnostic imaging catheter 40 may be a catheter for generating optical tomographic images, such as for OCT (Optical Coherence Tomography) or OFDI (Optical Frequency Domain Imaging), which generate optical tomographic images using near-infrared light. Optical tomographic images are an example of catheter images generated using the diagnostic imaging catheter 40. The sensor 42 of these diagnostic imaging catheters 40 is a transceiver that emits near-infrared light and receives reflected light. A catheter for generating optical tomographic images is an example of a catheter for generating tomographic images.

[0015] The diagnostic imaging catheter 40 may have both an ultrasound transducer and an OCT or OFDI transceiver unit as the sensor 42. The diagnostic imaging catheter 40 may have a total of three sensors 42: an ultrasound transducer, an OCT transceiver unit, and an OFDI transceiver unit.

[0016] The diagnostic imaging catheter 40 is not limited to a mechanical scanning type that mechanically rotates and moves back and forth, but may be an electronic radial scanning type diagnostic imaging catheter 40 that uses a sensor 42 in which multiple ultrasonic transducers are arranged in a ring shape.

[0017] The diagnostic imaging catheter 40 may have a so-called linear scanning type sensor 42 in which a plurality of ultrasonic transducers are arranged in a row along the longitudinal direction. The diagnostic imaging catheter 40 may also have a so-called two-dimensional array type sensor 42 in which a plurality of ultrasonic transducers are arranged in a matrix.

[0018] The hollow organs into which the diagnostic imaging catheter 40 is inserted and used are, for example, blood vessels, pancreatic ducts, bile ducts, bronchi, etc. In the following explanation, the diagnostic imaging catheter 40 will be described as an example of a mechanical scanning type IVUS catheter shown in Figure 2.

[0019] Using the imaging diagnostic catheter 40, it is possible to generate tomographic images that include not only the walls of hollow organs such as blood vessel walls, but also reflectors present inside hollow organs, such as red blood cells, and organs present outside hollow organs, such as the epicardium and heart.

[0020] 3 is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 includes an information processing device 20, an MDU 33, and an imaging diagnostic catheter 40. The information processing device 20 includes a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

[0021] The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, etc. The control unit 21 is connected to each hardware unit that constitutes the information processing device 20 via a bus.

[0022] The main memory device 22 is a storage device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs currently being executed by the control unit 21.

[0023] The auxiliary storage device 23 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 23 stores programs to be executed by the control unit 21 and various data required for executing the programs. The communication unit 24 is an interface for communication between the information processing device 20 and a network.

[0024] Display unit 25 is an interface that connects display device 31 to the bus. Input unit 26 is an interface that connects input device 32 to the bus. Catheter control unit 271 controls MDU 33 and sensor 42, and generates images based on signals received from sensor 42, etc.

[0025] The MDU 33 rotates the sensor 42 and the shaft 43 inside the probe section 41. The catheter control section 271 generates one image for each rotation of the sensor 42. The generated image is a cross-sectional image centered on the probe section 41 and approximately perpendicular to the probe section 41.

[0026] The MDU 33 can also move the sensor 42 and shaft 43 forward and backward while rotating them inside the probe section 41. By performing a pullback operation that rotates the sensor 42 while pulling it toward the MDU 33 at a constant speed, the catheter control section 271 successively generates multiple transverse images approximately perpendicular to the probe section 41 at predetermined intervals.

[0027] The function and configuration of catheter control unit 271 are the same as those of conventionally used ultrasound diagnostic devices, and therefore detailed description will be omitted. Note that control unit 21 may also realize the function of catheter control unit 271.

[0028] The information processing device 20 is connected to various imaging diagnostic devices 37 such as an X-ray angiography device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, or an ultrasound diagnostic device via an HIS (Hospital Information System) or the like.

[0029] The information processing device 20 of this embodiment is a dedicated ultrasound diagnostic device, or a personal computer, tablet, smartphone, or the like having the functions of an ultrasound diagnostic device.

[0030] 4 to 7 are examples of screens displayed by the catheter system 10. The screen shown in FIG. 4 includes a first image field 51, a stop button 581, a select button 582, and an complement button 583. FIG. 4 shows a state in which the stop button 581 is not selected. The select button 582 and the complement button 583 are set to an unselectable state. A tomographic image obtained using the diagnostic imaging catheter 40 is displayed in real time in the first image field 51. In the following description, an image displayed in real time will be referred to as a real-time image.

[0031] In the real-time image shown in FIG. 4, a small crescent-shaped guidewire image 472 is displayed at the 5 o'clock direction relative to a circular catheter image 471 displayed in the center of the first image field 51. Because the ultrasonic waves emitted from the sensor 42 do not reach the back surface of the guidewire, which is a strong reflector, a roughly fan-shaped information missing portion 473 is formed outside the guidewire image 472. This information missing portion 473 is called an acoustic shadow portion. Note that the information missing portion 473 is not limited to an acoustic shadow portion. Information missing portions can also be formed by artifacts such as ringdown caused by vibration of the ultrasonic transducer, multiple echoes formed by strong reflectors such as the guidewire, and external noise.

[0032] For example, in a region where a stent is placed, information-missing areas 473 are formed outside the image showing the wires that make up the mesh of the stent, which is a strong reflector (see Figure 23). Similarly, information-missing areas 473 are also formed outside the strongly calcified region and the highly attenuating plaque.

[0033] A doctor or other expert skilled in handling the diagnostic imaging catheter 40 can distinguish between information-missing areas 473 formed by the influence of strong reflectors or high attenuation areas and areas where the ultrasound reflectivity is truly low, and can infer and supplement the information missing due to the information-missing areas 473.

[0034] However, when it comes to the parts that are filled in by guesswork, there may be variations in judgment depending on the person. For example, if there are variations in judgment between the doctor in charge of operating the diagnostic imaging catheter 40 and the medical staff such as nurses and technicians who assist the doctor in the procedure, it may be difficult for the doctor's instructions to be conveyed promptly, and in some cases misunderstandings may occur.

[0035] 5 shows an example of a screen displayed when control unit 21 accepts selection of stop button 581. Selection button 582 is set to a selectable state. Second image field 52 is displayed in the upper right corner of the screen. A real-time image is displayed in second image field 52. The image that was displayed at the time selection of stop button 581 was accepted is displayed in a still state in first image field 51.

[0036] In the following description, the image that is displayed at the time when the selection of the stop button 581 is accepted will be referred to as a paused image. Note that the control unit 21 may accept the selection of the stop button 581 by voice recognition or by operating a foot switch (not shown).

[0037] 6 shows an example of a screen displayed when the control unit 21 accepts selection of the selection button 582. A cursor 68 is displayed in the first image field 51. The user operates the cursor 68 via the input device 32 to place a designated point mark 571, for example, on the edge of a hollow organ.

[0038] 7 shows an example of a screen displayed when the control unit 21 accepts selection of the complement button 583. The control unit 21 displays a circular, i.e., closed curved, complement line 572 that complements the edge of the hollow organ based on the designated point marks 571 placed by the user. The complement line 572 is generated by connecting the designated point marks 571 placed by the user with a curve such as a spline curve. The complement line 572 is an example of complement information that complements the information missing portion 473.

[0039] 8 is a flowchart illustrating the flow of processing by the program. The program shown in FIG. 8 is a program that is started when the selection of the selection button 582 by the user is accepted.

[0040] The control unit 21 receives a position specification of the designated point mark 571 from the user (step S501). The control unit 21 determines whether or not the input of the designated point mark 571 has been completed (step S502). For example, the control unit 21 determines that the acceptance of the designated point mark 571 has been completed when the user instructs the end of input. The control unit 21 may also determine that the input has been completed when a predetermined number of designated point marks 571 have been accepted.

[0041] If it is determined that the process has not ended (NO in step S502), the control unit 21 returns to step S501. If it is determined that the process has ended (YES in step S502), the control unit 21 generates a circular complementary line 572 that connects the specified point marks 571 received in step S501 (step S503).

[0042] The control unit 21 superimposes the generated complementary line 572 on the paused image, and displays the screen described with reference to Fig. 7 (step S504). The control unit 21 then ends the process.

[0043] The control unit 21 may calculate parameters such as the area, maximum diameter, and minimum diameter based on the completed complementary line 572, and display them on the screen described with reference to Fig. 7. The control unit 21 may also accept a specification of the type of parameter to be calculated or a calculation formula.

[0044] According to this embodiment, it is possible to provide a catheter system 10 that can easily materialize and display the complementary line 572 that the user imagines in their mind based on the tomographic image. Therefore, it is possible to provide a catheter system 10 that is easy for the user to use.

[0045] By displaying the completion state that the user has imagined in their mind using the completion line 572 in a manner that can be recognized by other staff, a catheter system 10 can be provided that facilitates communication between related medical staff.

[0046] According to this embodiment, the user places the designated point mark 571 at a location where an image actually exists, which reduces the likelihood of confusion in judgment. Therefore, it is possible to provide a catheter system 10 that allows the user to quickly create the complementary line 572.

[0047] An example of a case where a stent is placed in a blood vessel will be described. A doctor inserts a diagnostic imaging catheter 40 into the blood vessel where the stent is to be placed and determines the state of stenosis of the blood vessel based on the tomographic image. To determine the stent to be used, the doctor or a medical staff member under the doctor's instructions operates the input device 32 to measure the area and inner diameter of the blood vessel lumen in the tomographic image. The doctor then determines the stent to be placed based on the measurement results.

[0048] The control unit 21 can calculate the area, inner diameter, etc. of the blood vessel lumen based on the interpolated line 572. That is, it is possible to provide a catheter system 10 that calculates the area, inner diameter, etc. of the blood vessel lumen. The control unit 21 may display information about a recommended stent based on the calculated area, inner diameter, etc. Note that calculation of the area and inner diameter of a portion surrounded by a closed curve has been performed conventionally, and therefore detailed explanation will be omitted.

[0049] After the stent is placed, the doctor inserts a diagnostic imaging catheter 40 into the same blood vessel and checks the cross-sectional images to see if there is any gap between the blood vessel wall and the placed stent. If there is any gap between the blood vessel wall and the stent, the doctor will perform a procedure such as re-expanding the stent.

[0050] The tomographic image after stent placement contains many information-missing areas 473 due to the stent wires. Therefore, the external elastic lamina is displayed as a broken line interrupted by the information-missing areas 473. By displaying the aforementioned complementary lines 572, not only the doctor who set the position of the designated point mark 571 but also other medical staff can easily understand the state of stent placement.

[0051] A part or all of the program may be executed on a supercomputer connected via a network, a virtual machine running on a supercomputer, or a cloud computing system. A part or all of the program may be executed on multiple personal computers performing distributed processing.

[0052] Instead of real-time images, video or still images stored in the auxiliary storage device 23 or the like may be used. A catheter system 10 that can be used for recording medical records after a case is completed can be provided. In this case, the information processing device 20 may be a general-purpose PC, tablet, smartphone, or the like that does not have the function to connect the MDU 33 and the diagnostic imaging catheter 40.

[0053] [Variations] In step S501, when control unit 21 receives, for example, five or more designated point marks 571, control unit 21 generates (step S503) and displays (step S504) complementary lines 572. Thereafter, control unit 21 returns to step S501 and receives further designation of designated point marks 571.

[0054] The user places additional designated point marks 571 at a location where the user actually wants to draw the complementary line 572, but the location is far from the displayed complementary line 572. In this way, the user can create the desired complementary line 572 using a small number of designated point marks 571.

[0055] [Embodiment 2] This embodiment relates to a catheter system 10 that automatically displays a candidate point mark 573 at a location similar to a designated point mark 571 designated by the user. Explanation of parts common to the first embodiment will be omitted.

[0056] 9 is an example of a screen displayed by the catheter system 10 of embodiment 2. As described using FIG. 6, the user operates the cursor 68 via the input device 32 to specify the position of the specified point mark 571.

[0057] The control unit 21 detects an area similar to the periphery of the designated point mark 571 whose designation has been accepted. Specifically, the control unit 21 extracts a template area 574 of a predetermined number of pixels centered on the designated point mark 571. In Fig. 9, the template area 574 is indicated by a virtual line.

[0058] The control unit 21 uses a template matching technique to detect an area from the paused image that is similar to the template area 574. The template matching technique has been used for a long time, so a detailed explanation will be omitted. It is desirable that the control unit 21 excludes the vicinity of a location where the designated point mark 571 has already been placed from the target of template matching.

[0059] When a similar area is detected, the control unit 21 displays a candidate point mark 573 in the center of the area. The control unit 21 may simultaneously display multiple candidate point marks 573. The control unit 21 may also display only the candidate point mark 573 that has the highest similarity during template matching. The candidate point mark 573 is a mark that has a different form from the designated point mark 571, allowing the user to easily distinguish between the two.

[0060] The user decides whether to approve the displayed candidate point mark 573 and operates the input device 32. For example, the user approves by clicking the candidate point mark 573, and rejects by dragging and dropping it outside the first image field 51. The user may also input an instruction to approve or reject by voice input.

[0061] The control unit 21 changes the candidate point mark 573 approved by the user to a specified point mark 571. The control unit 21 deletes the candidate point mark 573 rejected by the user. Note that the user may reserve judgment on the displayed candidate point mark 573 and specify the position of the next candidate point mark 573.

[0062] For example, when the user places a designated point mark 571 on the external elastic lamina, the control unit 21 extracts a point indicating the external elastic lamina by template matching and displays a candidate point mark 573. Similarly, when the user places a designated point mark 571 on the inner surface of the lumen wall, the control unit 21 extracts a point indicating the inner surface of the lumen wall and displays a candidate point mark 573.

[0063] 10 is a flowchart illustrating the flow of processing of the program of the second embodiment. The control unit 21 accepts the position designation of the designated point mark 571 by the user (step S511). The control unit 21 determines whether or not the input of the designated point mark 571 has ended (step S512). For example, the control unit 21 determines that the acceptance of the designated point mark 571 has ended when the user instructs the end of input. The control unit 21 may also determine that the input has ended when a predetermined number of designated point marks 571 have been accepted.

[0064] If it is determined that the process has not ended (NO in step S512), control unit 21 extracts template area 574 from the paused image, centered on the position accepted in step S511 (step S513). Control unit 21 executes template matching processing to detect an area similar to template area 574 from the paused image (step S514). Template matching has been performed conventionally, so detailed description will be omitted.

[0065] Control unit 21 determines whether or not the detection of a region similar to template region 574 has been successful (step S515). For example, control unit 21 determines that the detection has been successful when the similarity between template region 574 and the region detected in step S514 exceeds a predetermined threshold.

[0066] The similarity can be evaluated, for example, by the sum of squared differences (SSD), sum of absolute differences (SAD), or normalized cross-correlation (NCC) of pixel values ​​of pixels at the same position in two regions. Note that the above methods are merely examples. The similarity evaluation method is not limited to these.

[0067] If it is determined that the matching was not successful (NO in step S515), control unit 21 returns to step S511. If it is determined that the matching was successful (YES in step S515), control unit 21 displays candidate point mark 573 in the center of the area that matches template area 574 (step S516).

[0068] For example, the control unit 21 displays the candidate point mark 573 at the center of the area with the highest similarity. The control unit 21 may also display the candidate point mark 573 at the center of each of the multiple areas determined to satisfy the condition in step S515. The control unit 21 returns to step S511.

[0069] Although not shown in the flowchart, the user can accept or reject the displayed candidate point mark 573 at any time by operating the cursor 68, by voice input, etc. The control unit 21 changes the accepted candidate point mark 573 to the designated point mark 571, and deletes the rejected candidate point mark 573.

[0070] If it is determined that the process has ended (YES in step S512), the control unit 21 generates a circular complementary line 572 connecting the designated point marks 571 received in step S501 (step S517). The control unit 21 superimposes the generated complementary line 572 on the paused image, and displays the screen described with reference to FIG. 7 (step S518). The control unit 21 ends the process.

[0071] According to this embodiment, it is possible to provide a catheter system 10 that automatically displays a candidate point mark 573 at a position similar to the position where the user has specified the specified point mark 571. It is possible to provide a catheter system 10 that can reduce the burden on the user in inputting the specified point mark 571.

[0072] [Embodiment 3] This embodiment relates to a catheter system 10 that automatically generates a complementary line 572 in real time. Explanation of parts common to the first embodiment will be omitted.

[0073] 11 is an explanatory diagram illustrating the configuration of the learning model 61. The learning model 61 is a model that receives an input image and outputs a prediction regarding an output image showing the interpolated line 572. The learning model 61 is generated by machine learning. A method for generating the learning model 61 will be described later.

[0074] The learning model 61 is created for each complementary line to be generated, such as a model that outputs a complementary line 572 corresponding to the external elastic lamina, a model that outputs a complementary line 572 corresponding to the inner surface of the lumen wall, a model that outputs a complementary line 572 corresponding to the periphery of a calcified area, and a model that outputs a complementary line 572 corresponding to the periphery of a plaque. The learning model 61 may be created for each combination of the site into which the diagnostic imaging catheter 40 is inserted, such as the coronary artery, the aorta of the lower limb, the bile duct, the pancreatic duct, and the bronchi, and the complementary line to be generated.

[0075] Each learning model 61 is stored in the auxiliary storage device 23. The learning model 61 may be stored in an external large-capacity storage device connected to the information processing device 20. The control unit 21 may acquire the learning model 61 stored in a server or the like via a network whenever necessary.

[0076] The learning model 61 may be a model that receives an input image and outputs an output image in which the complementary line 572 is superimposed on the input image. The input image input to the learning model 61 may be a longitudinal cross-sectional image. The learning model 61 may be a model that receives an input image and outputs a group of coordinates through which the complementary line 572 passes.

[0077] 12 and 13 are examples of screens displayed by the catheter system 10 of embodiment 3. Fig. 12 shows an example of a screen displayed by the control unit 21 on the display device 31 before creating the interpolation line 572. The screen shown in Fig. 12 includes a first image field 51, a target selection button 591, and a start button 584.

[0078] The first image field 51 displays a real-time image obtained using the diagnostic imaging catheter 40. The target selection button 591 is a pull-down menu button, and the user selects the region for which he or she wishes to display the completion line 572. In FIG. 12, "EEM," or the external elastic lamina, is selected.

[0079] The user observes the real-time image displayed in the first image field 51 and determines the area where the complementary line 572 is to be displayed. The user operates the target selection button 591 to select the area where the complementary line 572 is to be displayed, and then selects the start button 584.

[0080] 13 shows an example of a screen that the control unit 21 displays in the first image field 51 after accepting the user's selection of the start button 584. The screen shown in FIG. 13 includes the first image field 51, a target selection button 591, and an end button 585. A real-time image with an interpolated line 572 corresponding to "EEM" superimposed thereon is displayed in the first image field 51. To end the display of the interpolated line 572, the user selects the end button 585.

[0081] 13 may display a button or the like for changing the display mode, such as the color and thickness, of the complementary line 572. The screen shown in Fig. 13 may display a button or the like for temporarily hiding the complementary line 572.

[0082] 14 is a flowchart illustrating the processing flow of the program of the third embodiment. The control unit 21 acquires the complement target part set by the user using the target selection button 591 (step S521). The control unit 21 selects a learning model 61 corresponding to the complement target part acquired in step S521 (step S522). In the subsequent processing, the control unit 21 uses the learning model 61 selected in step S522.

[0083] The control unit 21 acquires a real-time image from the catheter control unit 271 (step S523). The control unit 21 inputs the acquired real-time image into the learning model 61 and acquires an output image showing the complementary line 572 (step S524). The control unit 21 displays the real-time image with the complementary line 572 superimposed in the first image field 51 (step S525).

[0084] The control unit 21 determines whether or not to end the process (step S526). For example, when the end button 585 is selected or when the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 determines to end the process.

[0085] If it is determined not to end the process (NO in step S526), ​​the control unit 21 returns to step S523. If it is determined to end the process (YES in step S526), ​​the control unit 21 ends the process.

[0086] If the frame rate of the real-time image is high and the processing of step S524 cannot be completed in time, the control unit 21 may execute step S524, for example, once every two or three frames. The control unit 21 may determine the frame for performing the processing of step S524 to be synchronized with the electrocardiogram.

[0087] According to this embodiment, it is possible to provide a catheter system 10 that automatically displays the complementary line 572 on a real-time image. According to this embodiment, it is possible to provide a catheter system 10 that allows the user to specify the region indicated by the complementary line 572.

[0088] [Embodiment 4] This embodiment relates to a catheter system 10 that displays an image in which an information-missing portion 473 has been complemented. Explanation of parts common to the first embodiment will be omitted.

[0089] Fig. 15 is an explanatory diagram illustrating the configuration of a learning model 61 according to embodiment 4. The learning model shown in Fig. 15 is a model that, when an input image having an information missing portion 473 is input, outputs a prediction regarding an output image in which the information missing portion 473 has been complemented. The output image in Fig. 15 is an example of a complemented image generated so as to complement the information missing portion 473.

[0090] The learning model 61 includes a missing area extraction model 611, a complement model 612, a cutout unit 613, and a synthesis unit 614. The missing area extraction model 611 and the complement model 612 are neural network models with trainable parameters. The cutout unit 613 and the synthesis unit 614 are computing units that calculate the pixel values ​​of each pixel that constitutes an image.

[0091] The missing area extraction model 611 is a model that receives an input image including an information missing portion 473 and generates a missing area image 483 by extracting a portion that indicates the information missing portion 473. The missing area extraction model 611 has a configuration of, for example, Mask-RCNN (Region Convolutional Neural Network), which is a type of object detection model.

[0092] 15, the missing information portion 473 in the missing area image 483 is indicated by hatching. For example, the missing area image 483 is an image of the same size as the input image, in which the pixel value of the pixel corresponding to the missing information portion 473 is "1" and the pixel value of the pixels in the other portions is "0."

[0093] The interpolation model 612 is a model that receives an input image including an information missing portion 473 and generates an estimated image 486 without the information missing portion 473. The cropping unit 613 replaces the pixel values ​​of pixels in the estimated image 486 that do not correspond to the information missing portion 473 with pixel values ​​of the background color, thereby generating a cropped image 487. The interpolation model 612 has, for example, a configuration in which multiple convolution layers are connected.

[0094] The synthesis unit 614 synthesizes the input image and the cropped image 487 to generate an output image. The output image is an image in which only the portion of the input image corresponding to the information missing portion 473 is replaced with an image generated by the complement model 612. The cropped image 487 is an example of complementary information that complements the information missing portion 473.

[0095] The learning model 61 has been trained so that when an input image having an information missing portion 473 is input, it outputs an output image that does not look strange to an expert. A method for generating the learning model 61 will be described later.

[0096] 16 is an explanatory diagram illustrating a method of using the learning model 61 according to the embodiment 4. The control unit 21 inputs an input image to the learning model 61 and acquires the cropped image 487 output from the cropping unit 613.

[0097] 17 to 21 are examples of screens displayed by the catheter system 10 of embodiment 4. The screen shown in Fig. 17 includes a first image field 51, a no complement button 580, a complement button 583, a color button 586, a border button 587, and a shaded button 588. Either the no complement button 580 or the complement button 583 is always set to be selected.

[0098] 17 shows a state in which the No Completion button 580 is selected. The Completion button 583, Coloring button 586, and Frame button 587 are set to an unselectable state. A real-time image including the missing information portion 473 is displayed in the first image field 51. In FIG. 17, the missing information portion 473 has not been complemented.

[0099] 18 shows an example of a screen displayed when control unit 21 accepts selection of complement button 583. No complement button 580 is deselected, and color button 586, border button 587, and shaded button 588 are set to a selectable state. Second image field 52 is displayed in the upper right corner of the screen.

[0100] The first image field 51 displays an image obtained by combining a real-time image with a cropped image 487 acquired from the learning model 61. The second image field 52 displays a real-time image. Based on the image displayed in the first image field 51 with the missing information portion 473 complemented, the user can determine the cross-sectional shape, cross-sectional area, inner diameter, etc. of the hollow organ.

[0101] The control unit 21 can calculate the area, inner diameter, etc. of the blood vessel lumen based on the image without the information missing portion 473 displayed in the first image field 51. The control unit 21 may display information about a recommended stent based on the calculated area, inner diameter, etc.

[0102] If necessary, the user can observe the real-time image before completion displayed in the second image field 52. By comparing the first image field 51 and the second image field 52, the user can confirm which parts have been completed by the learning model 61.

[0103] 19 shows an example of a screen displayed when the control unit 21 receives selection of the border button 587. The control unit 21 combines the real-time image with a cropped image 487 with a border added to the edge, and displays the combined image in the first image field 51. The user can easily distinguish between the portion complemented by the learning model 61 and the portion actually acquired by the diagnostic imaging catheter 40.

[0104] 20 is an example of a screen displayed when the control unit 21 receives the selection of the border button 587 and the shaded button 588. The control unit 21 combines the real-time image with a cropped image 487 that has a border added to its edge and is shaded, and displays the combined image in the first image field 51. A catheter system 10 can be provided that prevents even an inexperienced user from mistaking the portion complemented by the learning model 61 for the portion actually acquired by the diagnostic imaging catheter 40.

[0105] The user can select any combination of the coloring button 586, the border button 587, and the shading button 588. The coloring button 586, the border button 587, and the shading button 588 are examples of means for selecting the manner of marking on the cutout image 487. The marking method is not limited to these. Any manner of marking can be set.

[0106] 21 is an example of a screen displayed when the control unit 21 accepts selection of the coloring button 586. The control unit 21 combines the real-time image with a cropped image 487 in which hyperechoic portions are colored, and displays the combined image in the first image field 51. The user can easily distinguish between the portion complemented by the learning model 61 and the portion actually acquired by the diagnostic imaging catheter 40.

[0107] 22 is a flowchart illustrating the processing flow of the program according to the fourth embodiment. The control unit 21 acquires a real-time image from the catheter control unit 271 (step S531). The control unit 21 determines whether or not selection of the complement button 583 has been accepted (step S532). If it is determined that selection has not been accepted (NO in step S532), the control unit 21 displays the real-time image in the first image field 51 (step S534).

[0108] If it is determined that the selection has been accepted (YES in step S532), the control unit 21 inputs the acquired real-time image into the learning model 61 to acquire the cropped image 487 (step S533). The control unit 21 determines whether or not the selection of marking for the cropped image 487 has been accepted via the coloring button 586, the border button 587, and the hatching button 588 (step S535).

[0109] If it is determined that the selection of marking has not been accepted (NO in step S535), control unit 21 composites the real-time image with cropped image 487 (step S536).If it is determined that the selection of marking has been accepted (YES in step S535), control unit 21 composites the real-time image with cropped image 487 with the designated marking (step S537).

[0110] After step S536 or step S537 is completed, the control unit 21 displays the synthesized image (step S538). After step S534 or step S538 is completed, the control unit 21 determines whether or not to end the process (step S539). For example, if the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 determines to end the process.

[0111] If it is determined not to end (NO in step S539), the control unit 21 returns to step S531. If it is determined to end (YES in step S539), the control unit 21 ends the process.

[0112] According to this embodiment, a catheter system 10 can be provided that maintains the parts actually acquired using the imaging diagnostic catheter 40, while complementing and displaying the missing information part 473 in a natural manner.

[0113] According to this embodiment, it is possible to provide a catheter system 10 that allows the user to identify the completed area using various marking methods.

[0114] [Embodiment 5] This embodiment relates to a catheter system 10 that complements an information-missing portion 473 in a transverse image by using a transverse image of another frame. Explanation of parts common to the first embodiment will be omitted.

[0115] 23 is an explanatory diagram illustrating a method for generating a screen displayed by the catheter system 10 according to embodiment 5. This embodiment will be described taking as an example a case where an imaging diagnostic catheter 40 is inserted into a site where a stent has already been placed and image data is acquired.

[0116] The first and second tomographic images are cross-sectional images acquired at slightly different positions in the longitudinal direction of the probe portion 41. There is no significant difference in the structure of the luminal organ itself between the two cross-sectional images. The stent wires cause multiple information-missing areas 473 to appear radially. Because the stent is mesh-like, if the slice plane of the cross-sectional image is shifted in the longitudinal direction of the probe portion 41, the positions of the information-missing areas 473 will differ.

[0117] For example, the control unit 21 extracts the information missing portions 473 from the second tomographic image. The control unit 21 cuts out the portions corresponding to the information missing portions 473 from the first tomographic image and combines them with the second tomographic image. In this way, a combined image that does not include the information missing portions 473 is generated.

[0118] 24 is a flowchart illustrating the processing flow of the program according to embodiment 5. The control unit 21 acquires a first real-time image from the catheter control unit 271 and records it in the main storage device 22 or the auxiliary storage device 23 (step S551). The first real-time image is an example of a catheter image generated using the diagnostic imaging catheter 40.

[0119] The control unit 21 extracts information-missing portion 473 from the first real-time image (step S552). The extraction of information-missing portion 473 is performed by template matching using, for example, a fan-shaped hypoechoic region as a template. A fan-shaped region in which the central end is hyperechoic and the remaining portion is hypoechoic may be used as the template. Information-missing portion 473 may be extracted by a learning model using an object detection algorithm such as Mask-RCNN.

[0120] The control unit 21 acquires the second real-time image from the catheter control unit 271 and records it in the main storage device 22 or the auxiliary storage device 23 (step S553). Note that while this program is being executed, the user slightly moves the diagnostic imaging catheter 40 forward and backward near the area to be observed. This program may be executed during a pullback operation using the MDU 33. When executed during a pullback operation, the control unit 21 acquires the first real-time image and then acquires the second real-time image one to several frames later. The second real-time image is an example of a second catheter image generated at a different time from the first catheter image.

[0121] The control unit 21 extracts the information missing portion 473 in the second real-time image (step S554). The control unit 21 determines whether the information missing portion 473 extracted in step S552 and the information missing portion 473 extracted in step S554 overlap (step S555). For example, the control unit 21 determines that the two information missing portions 473 overlap when the area of ​​the overlapping portion exceeds a predetermined area.

[0122] If it is determined that there is an overlap (YES in step S555), the control unit 21 returns to step S553. If it is determined that there is no overlap (NO in step S555), the control unit 21 combines the first real-time image and the second real-time image (step S556).

[0123] Specifically, for example, control unit 21 cuts out the portion corresponding to information missing portion 473 extracted in step S554 from the first real-time image and combines it with the second real-time image. Control unit 21 may also cut out the portion corresponding to information missing portion 473 extracted in step S552 from the second real-time image and combine it with the first real-time image.

[0124] The control unit 21 displays the combined image in the first image field 51 of the screen described with reference to Figures 17 to 21 (step S557), and the control unit 21 ends the process.

[0125] According to this embodiment, the information missing portion 473 is complemented using a transverse cross-sectional image of another frame, so that a catheter system 10 can be provided that is less likely to produce artifacts due to image synthesis.

[0126] [Embodiment 6] This embodiment relates to a catheter system 10 that complements an information missing portion 473 in an image by using an image of the same frame. Explanation of parts common to the first embodiment will be omitted.

[0127] 25 is an explanatory diagram illustrating a method for generating a screen displayed by the catheter system 10 according to embodiment 6. This embodiment will be described using an example of an image including an information missing portion 473 due to a guidewire.

[0128] For example, the control unit 21 extracts the information missing portion 473 from the original image. The control unit 21 copies a paste area 485 having a shape corresponding to the information missing portion 473 from a portion of the original image where no information is missing, and combines the copied paste area 485 with the information missing portion 473. In this way, a composite image that does not include the information missing portion 473 is generated.

[0129] 26 is a flowchart illustrating the flow of processing of the program according to embodiment 6. Control unit 21 acquires a real-time image from catheter control unit 271 (step S561). Control unit 21 extracts information missing portion 473 from the real-time image (step S562).

[0130] Control unit 21 extracts candidates for paste area 485 from the real-time image acquired in step S561 (step S563). Candidates for paste area 485 are extracted by, for example, cutting out a portion from the real-time image that corresponds to the shape obtained by rotating information missing portion 473 around the center of the image as an axis.

[0131] Control unit 21 combines the real-time image acquired in step S561 with the candidate for paste area 485 extracted in step S563 (step S564). Control unit 21 extracts information missing portion 473 from the combined image combined in step S564 (step S565). The method for extracting information missing portion 473 in step S565 is the same as that in step S562. The control unit 21 determines whether the composite image includes the information missing portion 473 (step S566). Specifically, when the information missing portion 473 having an area larger than a predetermined value is extracted, the control unit 21 determines that the composite image includes the information missing portion 473.

[0132] If it is determined that the image contains missing information portion 473 (YES in step S566), control unit 21 returns to step S563. If it is determined that the image does not contain missing information portion 473 (NO in step S566), control unit 21 displays the image synthesized in step S564 in first image field 51 of the screen described with reference to Figures 17 to 21 (step S567). Control unit 21 then ends the process.

[0133] According to this embodiment, the information missing portion 473 is complemented using the attachment region 485 acquired from the transverse image of the same frame, so that the catheter system 10 can be provided with a small time lag associated with the complementation process.

[0134] [Embodiment 7] This embodiment relates to a method for generating the learning model 61 of the third embodiment described with reference to Fig. 11. Explanation of parts common to the third embodiment will be omitted.

[0135] 27 is an explanatory diagram illustrating the record layout of the training DB. The training data DB is a database that records detection targets, input images, and output images in association with each other, and is used to train the learning model 61 using machine learning. The training DB has a target field, an input image field, and an output image field.

[0136] The object field records the name of the object for which the interpolated line 572 is to be created. The input image field records an image acquired using the diagnostic imaging catheter 40. The output image field records an image of the interpolated line indicating the object recorded in the object field.

[0137] The training DB contains a large number of records of combinations of the name of the subject, an input image generated using the diagnostic imaging catheter 40, and an output image that has been confirmed to be correct by an expert, etc. The training DB is generated, for example, by a specialist, etc. who is familiar with the specifications of the diagnostic imaging catheter 40, based on case records in which the first or second embodiment is used.

[0138] 28 is a flowchart illustrating the flow of processing of the program according to embodiment 7. An example will be described in which machine learning of the learning model 61 is performed using the information processing device 20.

[0139] 28 may be executed on hardware separate from the information processing device 20, and the learning model 61 after machine learning may be copied to the auxiliary storage device 23 via a network. The learning model 61 trained on one piece of hardware may be used by multiple information processing devices 20.

[0140] Before executing the program in Fig. 28, an untrained model is prepared, for example, by combining a convolutional layer, a pooling layer, and a fully connected layer. The program in Fig. 28 adjusts each parameter of the prepared model and performs machine learning.

[0141] The control unit 21 acquires a training record to be used for training one epoch from the training DB (step S571). The control unit 21 adjusts the parameters of the model so that when an input image is input to the input layer of the model, an output image is output from the output layer (step S572).

[0142] The control unit 21 determines whether to end the process (step S573). For example, the control unit 21 determines to end the process when learning for a predetermined number of epochs has been completed. The control unit 21 may acquire test data from the training DB, input it to the model under machine learning, and determine to end the process when an output with a predetermined accuracy is obtained.

[0143] If it is determined not to end the process (NO in step S573), control unit 21 returns to step S571. If it is determined to end the process (YES in step S573), control unit 21 records the parameters of the trained model in auxiliary storage device 23 (step S574). Thereafter, control unit 21 ends the process. Through the above process, a trained model is generated.

[0144] According to this embodiment, the learning model 61 described in the third embodiment can be generated by machine learning.

[0145] [Embodiment 8] This embodiment relates to a method for generating the learning model 61 of the fourth embodiment described with reference to Fig. 15. Explanation of parts common to the fourth embodiment will be omitted.

[0146] FIG. 29 is an explanatory diagram outlining a method for generating the learning model 61. As described using FIG. 15, the learning model 61 is a model that receives an input image and generates an output image. The classifier 65 is a model that receives an output image output from the learning model 61 and determines whether the image is a true image or a false image. The classifier 65 has a configuration in which, for example, a convolutional layer and a pooling layer are repeated, a fully connected layer, and a softmax layer are connected.

[0147] Generative Adversarial Networks (GAN) is used to train the classifier 65 and the learning model 61 and adjust their parameters, allowing the learning model 61 to generate natural output images.

[0148] 30 is a flowchart illustrating the flow of processing of the program according to embodiment 8. An example will be described in which machine learning of the learning model 61 is performed using the information processing device 20.

[0149] 30 may be executed on hardware separate from the information processing device 20, and the learning model 61 after machine learning may be copied to the auxiliary storage device 23 via a network. The learning model 61 trained on one piece of hardware may be used by multiple information processing devices 20.

[0150] Prior to the execution of the program in FIG. 30, a model is prepared that combines an untrained learning model 61 having the configuration described using FIG. 15 with an untrained classifier 65.

[0151] The control unit 21 acquires a plurality of input images (step S581). The input images are recorded in the input image field of the training DB described with reference to Fig. 27, for example. The input images include images that do not include the information missing portion 473.

[0152] The control unit 21 adjusts the parameters of the classifier 65 so that it outputs "false" for images input to the classifier 65 via the learning model 61, and "true" for images that do not include the information missing portion 473 and that are input to the classifier 65 without going through the learning model 61 (step S582).

[0153] The control unit 21 adjusts the parameters of the learning model 61, specifically the parameters of the missing area extraction model 611 and the complementation model 612, so that the classifier 65 outputs "true" and "false" with equal probability (step S583). The control unit 21 may repeat the processing of steps S582 and S583 multiple times.

[0154] The control unit 21 records the parameters of the trained learning model 61 in the auxiliary storage device 23 (step S584). Thereafter, the control unit 21 ends the processing. Through the above processing, the trained learning model 61 is generated.

[0155] According to this embodiment, the learning model 61 described in the fourth embodiment can be generated by machine learning.

[0156] [Embodiment 9] This embodiment relates to a catheter system 10 that distinguishes between highly reliable and less reliable portions of an interpolated line 572. Explanation of portions common to the third embodiment will be omitted.

[0157] 31 is an explanatory diagram illustrating a method for generating an image displayed by the catheter system 10 of embodiment 9. In this embodiment, an example will be described in which the display range of a tomographic image obtained using the diagnostic imaging catheter 40 is small, that is, an input image displaying the vicinity of the diagnostic imaging catheter 40 is used.

[0158] The input image displays a guidewire image 472 and a shadowed portion image 474. The shadowed portion image 474 is an image that shows a strong reflector, such as a strongly calcified area, a stent, or another medical device that is being used simultaneously with the diagnostic imaging catheter 40. Information-missing portions 473 due to acoustic shadows are formed outside the guidewire image 472 and outside the shadowed portion image 474. In other words, the guidewire image 472 and the shadowed portion image 474 are examples of images that show a shadowed portion.

[0159] The control unit 21 generates two complementary lines 572, a first complementary line 561 and a second complementary line 562, based on the input image (step S101). In this embodiment, the first complementary line 561 indicates the inner surface of a blood vessel, and the second complementary line 562 indicates the external elastic lamina. The number of complementary lines 572 generated in step S101 may be one or three or more.

[0160] The control unit 21 can generate an image with complementary lines by combining the input image with the complementary lines 572 (step S102). Note that the image with complementary lines is shown for the sake of convenience of explanation, and does not need to be generated in practice.

[0161] The control unit 21 extracts the guidewire image 472 and the shadow-forming portion image 474 from the input image or the image with the complementary lines (step S103). The guidewire image 472 is an area with a brightness higher than a predetermined brightness and is an area that exists inside the first complementary line 561. The control unit 21 may extract the guidewire image 472 by pattern matching based on a pre-specified shape and dimensions.

[0162] The shadowed portion image 474 is an area with a brightness higher than a predetermined value and located outside the first complementary line 561. The shadowed portion image 474 corresponds to, for example, a stent wire or a heavily calcified portion. The guidewire image 472 and the shadowed portion image 474 are examples of areas where the cause of the formation of an acoustic shadow is depicted.

[0163] The control unit 21 determines a low reliability region 55 based on the extracted guidewire image 472 and shadow formation portion image 474. The low reliability region 55 is a substantially fan-shaped region combining a region near the guidewire image 472 and a region outside the guidewire image 472, and a substantially fan-shaped region combining a region near the shadow formation portion image 474 and a region outside the shadow formation portion image 474.

[0164] Since the guidewire image 472 or the shadow-forming portion image 474 is present between the diagnostic imaging catheter 40 and the low-reliability region 55, there is a high possibility that an acoustic shadow is occurring. Therefore, the reliability of the interpolated line 572 is low within the low-reliability region 55. Note that the control unit 21 may include, in the low-reliability region 55, an area of ​​a predetermined width that is closer to the diagnostic imaging catheter than the guidewire image 472 and the shadow-forming portion image 474.

[0165] FIG. 32 is an explanatory diagram illustrating a method for generating a complementary line 572 according to the ninth embodiment. The learning model 61 according to the present embodiment is a model that receives an input image and outputs classification data. The classification data is data that associates each part constituting the input image with a label that classifies each object depicted in that part. Each part is, for example, each pixel. The classification data can be used to generate a classification image in which the input image is colored differently for each object depicted in the input image.

[0166] A specific example will be given. Learning model 61 outputs classification data in which each pixel constituting an input image is classified into, for example, a first label, a second label, or a third label. An example of a classified image generated based on the classification data is shown below. A first label region 541, a second label region 542, and a third label region 543 are arranged in approximately concentric circles with catheter image 471 at the center.

[0167] The first labeled region 541 indicates the lumen of the hollow organ into which the diagnostic imaging catheter 40 is inserted, i.e., the region of the blood vessel lumen through which blood flows. The second labeled region 542 indicates the lumen wall, i.e., the blood vessel wall. The third labeled region 543 indicates the outside of the lumen wall, i.e., the region outside the external elastic lamina which indicates the outer surface of the hollow organ. The third labeled region 543 includes, for example, muscle, nerve, fat, and other blood vessels adjacent to the blood vessel into which the diagnostic imaging catheter 40 is inserted.

[0168] The boundary line between the first label region 541 and the second label region 542 corresponds to the above-mentioned first complementary line 561. The boundary line between the second label region 542 and the third label region 543 corresponds to the above-mentioned second complementary line 562. As described above, the complementary line 572 created based on the input image may be referred to as the complementary line 572 of mode 1 in the following description.

[0169] 32 schematically shows an input image displayed in a so-called XY format and a classification image in which classification data is displayed in the XY format. The learning model 61 may accept an input image in a so-called RT format, which is formed by arranging scanning line data formed by transmitting and receiving ultrasonic waves by the sensor 42 in parallel in order of scanning angle, and output classification data. Since the method of converting from the RT format to the XY format is well known, a description thereof will be omitted. Since the input image is not affected by interpolation processing, etc., when converting from the RT format to the XY format, more appropriate classification data is generated.

[0170] The learning model 61 is a trained model that performs semantic segmentation on an input image, for example. The trained model that performs semantic segmentation is generated by machine learning using training data that combines an input image generated using the diagnostic imaging catheter 40 and classified images in which an expert colors the input image according to the subject depicted therein.

[0171] The learning model 61 may be created for each site into which the diagnostic imaging catheter 40 is inserted, such as the coronary artery, the lower limb aorta, the bile duct, the pancreatic duct, and the bronchi. The learning model 61 may be created for each display range in which a tomographic image is generated using the diagnostic imaging catheter 40. The learning model 61 may be created for each patient attribute, such as the patient's age or gender.

[0172] The complementary line 572 of mode 1 may be generated using the learning model 61 of embodiment 3 described with reference to Fig. 11. When the learning model 61 of embodiment 3 is used, the first complementary line 561 indicating the inner surface of the blood vessel and the second complementary line 562 indicating the external elastic plate are generated using appropriate learning models 61. The complementary line 572 of mode 1 may also be generated by any other method.

[0173] Fig. 33 is an explanatory diagram illustrating a method for generating a complementary line 572 in mode 2. As explained using Fig. 31, the explanation begins with the process after the generation of the complementary line 572 in mode 1 and the determination of the low-reliability region 55 are completed. In Fig. 33, the guidewire image 472 and the shadow-forming portion image 474 are not shown.

[0174] The control unit 21 deletes the portion of the complement line 572 of mode 1 that is included in the low-reliability region 55 (step S111). The complement line 572 is now partially broken. The control unit 21 generates a modified line 565 that smoothly connects the broken portion of the complement line 572. In this way, the control unit 21 generates the complement line 572 of the second mode (step S112). In FIG. 33, the modified line 565 corresponding to each complement line 572 is shown in bold.

[0175] Any interpolation method, such as spline interpolation, Lagrange interpolation, or linear interpolation, can be used to generate the modified line. The control unit 21 may accept a user's designation of the generation method. The second-mode interpolated line 572 is less susceptible to artifacts such as multiple echoes formed by a strong reflector such as a guidewire. The second-mode interpolated line described above is an interpolated line obtained by correcting areas of the interpolated line 572 that are less reliable than other areas based on areas that are not less reliable.

[0176] If a large portion of the complementary line 572 is included in the low-reliability region 55, the control unit 21 determines that the reliability of the entire complementary line 572 is low. For example, if the shadow-forming portion image 474 is a stent wire or if there is extensive, intense calcification, the low-reliability region 55 is wide and a large portion of the complementary line 572 is included in the low-reliability region 55.

[0177] 34 is a flowchart illustrating the processing flow of the program according to the ninth embodiment. The control unit 21 acquires a real-time image from the catheter control unit 271 (step S601). The control unit 21 determines whether or not an instruction to perform complementation has been received from the user (step S602). An example of a screen for receiving an instruction from the user will be described later.

[0178] If it is determined that the request has not been received (NO in step S602), the control unit 21 displays a real-time image in the first image field 51 (step S603). If it is determined that the request has been received (YES in step S602), the control unit 21 starts a complementary line generation subroutine (step S611). The complementary line generation subroutine is a subroutine that generates a complementary line 572. The processing flow of the complementary line generation subroutine will be described later.

[0179] The control unit 21 extracts a high-brightness region from the real-time image (step S612). A high-brightness region is, for example, a region where a larger number of pixels than a predetermined threshold are clustered together. The brightness threshold and the pixel count threshold may be set by the user as appropriate. In step S612, a guidewire image 472 and a shadow-forming portion image 474 are extracted.

[0180] The control unit 21 sets the range of the low reliability region 55 (step S613). In the XY format image, the low reliability region 55 is a substantially fan-shaped region outside the guidewire image 472 and the shadow formation portion image 474. The control unit 21 calculates the proportion of the length of the range included in the low reliability region 55 to the total length for one of the interpolated lines 572 generated in step S611 (step S614).

[0181] The control unit 21 determines whether the proportion of the range included in the low reliability region 55 is greater than a predetermined threshold (step S615). If it is determined that the proportion is greater (YES in step S615), the control unit 21 temporarily records in the main storage device 22 or the auxiliary storage device 23 that the proportion of the range included in the low reliability region 55 for the interpolated line 572 being processed is large (step S616).

[0182] If it is determined that the number is not large (NO in step S615), control unit 21 determines whether or not an instruction to display complementary lines 572 in mode 2 has been received from the user (step S617). An example of a screen for receiving an instruction from the user will be described later.

[0183] If it is determined that the request has been accepted (YES in step S617), control unit 21 generates complementary lines 572 of mode 2, which have been described using Fig. 33 (step S618). If it is determined that the request has not been accepted (NO in step S617), after step S618 or step S616, control unit 21 determines whether or not the processing of all complementary lines 572 generated in step S611 has been completed (step S619).

[0184] If it is determined that the process has not ended (NO in step S619), the control unit 21 returns to step S614. If it is determined that the process has ended (YES in step S619), the control unit 21 displays the real-time image and the complementary line 572 in the first image field 51 (step S620).

[0185] After step S603 or step S620 is completed, the control unit 21 determines whether or not to terminate the process (step S621). For example, when the diagnostic imaging catheter 40 is detached from the MDU 33, the control unit 21 determines to terminate the process.

[0186] If it is determined not to end (NO in step S621), the control unit 21 returns to step S601. If it is determined to end (YES in step S621), the control unit 21 ends the process.

[0187] 35 is a flowchart illustrating the processing flow of the complementary line generation subroutine. The control unit 21 inputs the real-time image to the learning model 61 to acquire classification data (step S631). That is, in step S631, the control unit 21 acquires labels corresponding to the subjects depicted in each part that constitutes the real-time image. The control unit 21 extracts complementary lines corresponding to the boundaries between areas whose acquired labels are different from each other (step S632). The control unit 21 ends the processing.

[0188] Fig. 36 is an example of a screen displayed by the catheter system 10 of embodiment 9. The control unit 21 displays the screen shown in Fig. 36 on the display device 31 in step S603 of the flowchart described using Fig. 34.

[0189] 36 includes a first image field 51, two object selection buttons 591, an intraluminal high-brightness area button 594, and an intramural high-brightness area button 595. Below each object selection button 591, two mode selection buttons 592 and a hide selection button 593 are displayed. A real-time image obtained using the diagnostic imaging catheter 40 is displayed in the first image field 51.

[0190] In Fig. 36, none of the buttons has been selected by the user. In step S602 of the flowchart described with reference to Fig. 34, control unit 21 determines that an instruction to perform completion has not been received from the user (NO in step S602), and executes step S603.

[0191] The target selection button 591 accepts the selection of the complementary line 572 to be extracted. In FIG. 36, the user can select "inner luminal surface" and "EEM." Note that "inner luminal surface" and "EEM" are examples. The control unit 21 appropriately displays the type of complementary line 572 corresponding to the luminal organ into which the diagnostic imaging catheter 40 is inserted. The control unit 21 may display the target selection button 591 for the type of complementary line 572 previously set by the user.

[0192] The mode selection button 592 accepts a selection regarding the mode of the complementary line 572. When a selection of "mode 1" is accepted, the control unit 21 generates the complementary line 572 of mode 1 described using FIG. 31. When a selection of "mode 2" is accepted, the control unit 21 generates the complementary line 572 of mode 2 described using FIG. 33.

[0193] The hide selection button 593 accepts a selection as to whether or not to display the complement line 572 for the portion included in the low-reliability region 55. The intraluminal high-brightness area button 594 accepts a selection as to whether or not to display information about high-brightness areas present in the lumen of the luminal organ. The intramural high-brightness area button 595 accepts a selection as to whether or not to display information about high-brightness areas present inside the luminal wall, i.e., between the inner wall of the luminal organ and the external elastic lamina.

[0194] 37 to 41 are examples of screens displayed by the catheter system 10 of embodiment 9. The control unit 21 displays the screens shown in Fig. 37 to 41 on the display device 31 in step S620 of the flowchart described using Fig. 34.

[0195] 37, selection of object selection button 591 corresponding to "Luminous Inner Surface" and "EEM" and mode selection button 592 for "Mode 1" are accepted. A legend field 596 is displayed to the right of the characters "Luminous Inner Surface" and "EEM."

[0196] The first image field 51 displays an image in which a first complementary line 561 and a second complementary line 562 are superimposed on a real-time image obtained using the diagnostic imaging catheter 40. Of the first complementary line 561 corresponding to the "inner surface of the lumen," the portion that does not overlap with the low-reliability region 55 described using FIG. 31 is displayed as a thick solid line, and the portion that overlaps with the low-reliability region 55 is displayed as a thin solid line. Of the second complementary line 562 corresponding to the "EEM," the portion that does not overlap with the low-reliability region 55 is displayed as a thick dashed line, and the portion that overlaps with the low-reliability region 55 is displayed as a thin dashed line.

[0197] 37, among the complementary lines 572, areas with lower reliability than other areas are displayed in a manner different from areas with higher reliability. Note that the complementary lines 572 may be displayed in a manner that distinguishes them from each other, and the areas with lower reliability from areas with higher reliability, using different colors or brightness, etc.

[0198] 37, the user can easily distinguish between highly reliable portions and less reliable portions of the complementary line 572 generated by the control unit 21. The user can appropriately select the complementary line 572 to be displayed in the first image field 51 by operating the target selection button 591.

[0199] 38, the selection of the hide selection button 593 corresponding to "EEM" is accepted. In the first image field 51, of the displayed second complementary line 562, the portion that overlaps with the low-reliability region 55 is erased.

[0200] The screen shown in FIG. 38 allows the user to visually check the unreliable parts of the second supplementary line 562 and make an appropriate decision based on specialized knowledge.

[0201] 39, the intraluminal high-brightness area button 594 and the intramural high-brightness area button 595 are selected. The first image field 51 displays the low-reliability area 55 generated by the guidewire image 472, which is the intraluminal high-brightness area, and the low-reliability area 55 generated by the low-reliability area 55, which is the intramural high-brightness area.

[0202] The control unit 21 may display the guidewire image 472 and the shadow-forming portion image 474 in color. The user can confirm which portion the control unit 21 has determined will cause the low-reliability region 55.

[0203] Figure 40 shows a real-time image of a stent-implanted area. Numerous shadow-forming area images 474 formed by the stent wires are visualized in a circular arrangement. Therefore, low-reliability regions 55 corresponding to each shadow-forming area image 474 are generated.

[0204] FIG. 40 shows an example of a screen displayed by the control unit 21 when it is determined in step S615 described with reference to FIG. 34 that the range included in the low reliability region 55 for the second supplemental line 562 is greater than the threshold value.

[0205] The control unit 21 does not display the second complementary line 562 in the first image field 51. The control unit 21 displays a notification field 597 indicating that the line is "not displayable" to the right of the characters "EEM" instead of the legend field 596. The user can understand that the second complementary line 562 generated by the control unit 21 is not displayed in the first image field 51 because its reliability is low.

[0206] 41 shows an example of a screen displayed by the control unit 21 when the user, for example, clicks on the notification field 597 to instruct the display of the unreliable second complementary line 562. The entire second complementary line 562 is displayed as a thin dashed line indicating its low reliability. A notification in the lower left corner of the screen saying "EEM has low reliability" indicates that the reliability of the second complementary line 562 is low.

[0207] According to this embodiment, it is possible to provide a catheter system 10 that displays highly reliable portions of the complementary line 572 and less reliable portions in a distinguishable manner.

[0208] The learning model 61 may output labels corresponding to the guidewire image 472 and the shadow-forming portion image 474. In this case, step S612 described with reference to FIG. 34 is not necessary.

[0209] The learning model 61 may output a label corresponding to, for example, a highly attenuating plaque. In this case, the control unit 21 may set the region outside the highly attenuating plaque as the low-reliability region 55 in step S613 described with reference to FIG.

[0210] Instead of real-time images, video or still images stored in the auxiliary storage device 23 or the like may be used. A catheter system 10 that can be used for recording medical records after a case is completed can be provided. In this case, the information processing device 20 may be a PC, tablet, smartphone, or the like that does not have the function to connect the MDU 33 and the diagnostic imaging catheter 40.

[0211] [Embodiment 10] 42 is a functional block diagram of an information processing system 10 according to a tenth embodiment. The information processing system 10 includes an image acquiring unit 81, a complementary information acquiring unit 82, and a display unit 83. The image acquiring unit 81 acquires a catheter image generated using an imaging diagnostic catheter 40 inserted into a hollow organ. The complementary information acquiring unit 82 acquires complementary information that complements information-missing portions 473 of the catheter image acquired by the image acquiring unit 81. The display unit 83 displays the catheter image acquired by the image acquiring unit 81 and the complementary information acquired by the complementary information acquiring unit 82.

[0212] [Embodiment 11] Figure 43 is an explanatory diagram illustrating the configuration of a catheter system 10 according to an eleventh embodiment. This embodiment relates to a configuration in which the catheter system 10 of this embodiment is realized by combining and operating a catheter control device 27, an MDU 33, an imaging diagnostic catheter 40, a general-purpose computer 90, and a program 97. Explanations of parts common to the first embodiment will be omitted. The catheter system 10 is also an example of the information processing system of this embodiment.

[0213] Catheter control device 27 is an ultrasonic diagnostic device for IVUS that controls MDU 33, controls sensor 42, and generates transverse and longitudinal images based on signals received from sensor 42. The function and configuration of catheter control device 27 are similar to those of conventional ultrasonic diagnostic devices, and therefore a description thereof will be omitted.

[0214] The catheter system 10 of this embodiment includes a computer 90. The computer 90 is equipped with a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, a smartphone, or a server computer.

[0215] The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read the program 97 stored in a semiconductor memory 98, such as a flash memory, implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown) and store it in the auxiliary storage device 23.

[0216] The program 97 is installed as a control program for the computer 90, and is loaded into and executed by the main storage device 22. This causes the computer 90 to function as the information processing device 20 described above.

[0217] The computer 90 may be a general-purpose personal computer, a tablet, a smartphone, a mainframe computer, a virtual machine running on a mainframe computer, a cloud computing system, or a quantum computer. The computer 90 may also be a plurality of personal computers performing distributed processing.

[0218] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0219] (Appendix 1) Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; acquiring complementary information for complementing missing information in the acquired catheter image; Displaying the catheter image and the complementary information A program that causes a computer to perform a process.

[0220] (Appendix 2) The acquired catheter image is input to a learning model that outputs complementary information when the catheter image is input, and the complementary information output from the learning model is obtained. The program described in Appendix 1.

[0221] (Appendix 3) The complementary information is a complementary image generated to complement the missing information portion. The program described in Appendix 2.

[0222] (Appendix 4) The missing information portion is complemented based on a second catheter image acquired at a time different from the catheter image. The program described in Appendix 1.

[0223] (Appendix 5) The information missing portion is complemented based on a portion of the catheter image that is different from the information missing portion. The program described in Appendix 1.

[0224] (Appendix 6) the diagnostic imaging catheter is a catheter for generating a tomographic image, the catheter image is a tomographic image generated using the tomographic image generating catheter, The information missing portion is a shaded portion. 10. A program according to any one of claims 1 to 5.

[0225] (Appendix 7) The catheter for generating a tomographic image is a catheter for generating an ultrasonic tomographic image. The program described in Appendix 6.

[0226] (Appendix 8) The catheter for generating a tomographic image is a catheter for generating an optical tomographic image. The program described in Appendix 6.

[0227] (Appendix 9) The catheter for generating a tomographic image is a catheter equipped with a sensor for generating an ultrasonic tomographic image and a sensor for generating an optical tomographic image. The program described in Appendix 6.

[0228] (Appendix 10) extracting a plurality of points representing the inner surface of a lumen wall constituting the hollow organ from the tomographic image; The shaded area is complemented by a curve connecting the extracted points. 10. A program according to any one of appendices 6 to 9.

[0229] (Appendix 11) The acquired catheter image and the interpolated image in which the missing information portion has been interpolated are displayed side by side. 11. A program according to any one of claims 1 to 10.

[0230] (Appendix 12) Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; acquiring complementary information for complementing missing information in the acquired catheter image; Displaying the catheter image and the complementary information An information processing method that causes a computer to execute a process.

[0231] (Appendix 13) an image acquisition unit that acquires a catheter image generated using a diagnostic imaging catheter inserted into a hollow organ; a complementary information acquisition unit that acquires complementary information that complements information missing portions of the acquired catheter image; a display unit that displays the catheter image and the complementary information; Information processing system.

[0232] (Appendix 14) acquiring training data in which catheter images generated using a diagnostic imaging catheter inserted into a hollow organ and complementary information for complementing missing information in the catheter images are associated with each other and recorded; A learning model is generated that receives the catheter image as an input, outputs the complementary information, and outputs a prediction regarding the complementary information when an image generated using a catheter for diagnostic imaging is input. How to generate a learning model. [Explanation of symbols]

[0233] 10 Catheter system (information processing system) 20 Information processing equipment 21 Control section 22 Main storage 23 Auxiliary storage device 24 Communications Department 25 Display section 26 Input section 27 Catheter control device 271 Catheter control unit 29 Reading unit 31 Display device 32 Input Devices 33 MDU 37 Diagnostic imaging equipment 40 Diagnostic imaging catheter 41 Probe section 42 sensors 43 Shaft 44 Tip Marker 45 Connector part 471 Catheter Image 472 Guidewire image 473 Missing information 474 Shadow forming area image 483 Missing Area Images 485 Attachment Area 486 Estimated Images 487 Cropped image (supplementary information) 51 First Image Column 52 Second Image Column 541 First Label Area 542 Second Label Area 543 Third Label Area 55 Low confidence region 561 First Complementary Line 562 Second Complementary Line 565 Change Line 571 Designated point mark 572 Complementary Line (Complementary Information) 573 Candidate point mark 574 Template Area 580 No Completion Button 581 Stop button 582 selection button 583 Completion Button 584 Start button 585 Exit button 586 Colored Buttons 587 Border Button 588 Shaded Button 591 Target selection button 592 Mode selection button 593 Hide selection button 594 Intraluminal High-Intensity Button 595 High-intensity button in the wall 596 Legend 597 Notification column 61 Learning Model 611 Missing Region Extraction Model 612 Complementary Model 613 Cutout 614 Synthesis Section 65 Classifier 68 Cursor 90 Computer 96 Portable recording media 97 Programs 98 Semiconductor Memory

Claims

1. Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; acquiring complementary information that complements the information missing portion of the acquired catheter image; determining a display mode of the complementary information based on a low-reliability area of ​​the acquired complementary information that has a lower reliability than other areas; Displaying the catheter image and the complementary information in the display mode A program that causes a computer to perform a process.

2. The low-reliability region includes a region outside the guidewire image and the shadow-forming portion image. The program according to claim 1.

3. The image of the shadow formation portion includes a region corresponding to at least one of a calcified portion, a stent wire, or a highly attenuating plaque. The program according to claim 2.

4. The complementary information is a line indicating the external elastic lamina of a blood vessel. The program according to claim 1.

5. The display mode is determined based on whether a proportion of the low-reliability region in the complementary information is greater than a threshold value. The program according to claim 1.

6. The display mode includes a mode in which the complementary information is not displayed. The program according to claim 1.

7. The complementary information is complementary information output from a learning model that outputs complementary information when a catheter image is input to the learning model and the acquired catheter image is input. The program according to any one of claims 1 to 6.

8. Inputting the acquired catheter image into a learning model that outputs labels corresponding to each part of the catheter image when the catheter image is input, and obtaining the labels output from the learning model; The complementary information includes a complementary line indicating a boundary between regions having different labels. The program according to any one of claims 1 to 6.

9. Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; acquiring complementary information that complements the information missing portion of the acquired catheter image; determining a display mode of the complementary information based on a low-reliability area of ​​the acquired complementary information that has a lower reliability than other areas; Displaying the catheter image and the complementary information in the display mode An information processing method in which processing is performed by a computer.

10. An information processing device having a control unit, The control unit Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ; acquiring complementary information that complements the information missing portion of the acquired catheter image; determining a display mode of the complementary information based on a low-reliability area of ​​the acquired complementary information that has a lower reliability than other areas; Displaying the catheter image and the complementary information in the display mode Information processing device.

Images