PROGRAM, INFORMATION PROCESSING METHOD AND INFORMATION PROCESSING APPARATUS

The program automatically rotates diagnostic imaging catheter images to a standard orientation, addressing the challenge of manual alignment, thereby improving usability and efficiency in medical imaging systems.

JP7577734B2Active Publication Date: 2024-11-05TERUMO KK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2022511730
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-03-30
Filing Date
2021-03-09
Publication Date
2024-11-05
Estimated Expiration
2041-03-09

AI Technical Summary

Technical Problem

The orientation of images obtained by diagnostic imaging catheters changes based on the bending state, requiring manual rotation by trained personnel, which is time-consuming and skill-dependent.

Method used

A program that automatically calculates and rotates images acquired by the catheter to a predetermined orientation by comparing them with standard images, ensuring correct interpretation without the need for manual adjustment.

Benefits of technology

Facilitates easy use of catheter systems by automatically aligning images to a standard orientation, reducing the need for skilled interpretation and enhancing efficiency in medical diagnostics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577734000001
    Figure 0007577734000001
  • Figure 0007577734000002
    Figure 0007577734000002
  • Figure 0007577734000003
    Figure 0007577734000003
Patent Text Reader

Abstract

Provided are a program and the like capable of facilitating use of a catheter system. The program causes a computer to execute processing for acquiring an image generated using a catheter for diagnostic imaging inserted into a hollow organ, and displaying the acquired image in the state where the image is turned so that the merkmal included in the image is oriented in a predetermined direction. The merkmal is extracted by image processing on the image. The acquired image is input to a learning model that, when an image is input thereto, outputs information relating to the merkmal included in the image, and the merkmal is extracted on the basis of the information output from the learning model.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] 2. Description of the Related Art A catheter system is used in which a diagnostic imaging catheter is inserted into a hollow organ such as a blood vessel, and an image for diagnostic imaging is displayed (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 164071 Summary of the Invention [Problem to be solved by the invention]

[0004] The orientation of an image obtained by a diagnostic imaging catheter changes depending on the bending state of the diagnostic imaging catheter, etc. In order for doctors and paramedical staff to quickly grasp the state of the luminal organ, an operation to manually rotate the image to a specified orientation may be performed. However, rotating an image to a specified orientation requires the ability to correctly interpret the depicted image, which requires a long period of training.

[0005] In one aspect, an object of the present invention is to provide a program or the like that enables easy use of a catheter system. [Means for solving the problem]

[0006] The program acquires images generated using an imaging catheter inserted into a hollow organ, Each time the image is acquired, Included in the acquired image A rotation angle until the merkmar reaches a predetermined orientation is calculated, and when the calculated rotation angle exceeds a predetermined threshold, the extracted The image is rotated so that the merkmar faces in the specified direction and displayed. Making a computer carry out a process . Effect of the Invention

[0007] In one aspect, a program or the like can be provided that makes it easy to use the catheter system. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of a catheter system. [Diagram 2] FIG. 1 is an explanatory diagram for explaining an overview of a catheter for diagnostic imaging. [Diagram 3] FIG. 1 is an explanatory diagram illustrating a configuration of a catheter system. [Figure 4] FIG. 2 is an explanatory diagram illustrating a record layout of a standard image DB. [Diagram 5] 13 is an example of a screen displayed by the catheter system. [Figure 6] 13 is an example of a screen displayed by the catheter system. [Figure 7] 13 is an example of a screen displayed by the catheter system. [Figure 8] 13 is a modified example of a screen displayed by the catheter system. [Figure 9] 11 is a flowchart illustrating a process flow of a program. [Figure 10] 10 is a flowchart illustrating a process flow of a rotation angle calculation subroutine. [Figure 11] 10 is a flowchart illustrating the flow of processing of a program according to the second embodiment. [Figure 12] FIG. 2 is an explanatory diagram illustrating the configuration of a learning model. [Figure 13] 13 is a flowchart illustrating a process flow of a rotation angle calculation subroutine according to the third embodiment. [Figure 14] FIG. 13 is an explanatory diagram illustrating the record layout of a Merkmal DB. [Figure 15] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 16]13 is a flowchart illustrating the flow of processing of a program according to a fourth embodiment. [Figure 17] 13 is an example of a screen displayed by the catheter system of the fifth embodiment. [Figure 18] 13 is an example of a screen displayed by the catheter system of the fifth embodiment. [Figure 19] 13 is a flowchart illustrating the flow of processing of a program according to a fifth embodiment. [Figure 20] FIG. 1 is an explanatory diagram illustrating the record layout of a training DB. [Figure 21] 20 is a flowchart illustrating the flow of processing of a program according to a sixth embodiment. [Figure 22] FIG. 13 is a functional block diagram of an information processing device according to a seventh embodiment. [Diagram 23] FIG. 13 is an explanatory diagram illustrating the configuration of a catheter system according to an eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment 1] FIG. 1 is an explanatory diagram for explaining an overview of a catheter system 10. The catheter system 10 includes a catheter 40 for diagnostic imaging, an MDU (Motor Driving Unit) 33, and an information processing device 20. The catheter 40 for diagnostic imaging is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is an input device such as a keyboard, a mouse, a trackball, or a microphone. The display device 31 and the input device 32 may be stacked together to form a touch panel. The input device 32 and the information processing device 20 may be configured as one unit.

[0010] Fig. 2 is an explanatory diagram for explaining an overview of the diagnostic imaging catheter 40. Fig. 2 shows an example of a diagnostic imaging catheter 40 for IVUS (Intravascular Ultrasound) used when generating an ultrasonic tomographic image from inside a blood vessel, that is, for generating an ultrasonic tomographic image.

[0011] The diagnostic imaging catheter 40 has a probe portion 41 and a connector portion 45 disposed at an end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the diagnostic imaging catheter 40 farther from the connector portion 45 will be referred to as the tip side.

[0012] A shaft 43 is inserted inside the probe portion 41. A sensor 42 is connected to the tip side of the shaft 43. The shaft 43 and the sensor 42 are rotatable and movable forward and backward inside the probe portion 41.

[0013] The sensor 42 is an ultrasonic transducer that transmits and receives ultrasonic waves. An annular tip marker 44 is fixed near the tip of the probe portion 41. The tip marker 44 is made of a material, such as metal, that is opaque to X-rays.

[0014] The diagnostic imaging catheter 40 may be a catheter for generating optical tomographic images, such as for OCT (Optical Coherence Tomography) or OFDI (Optical Frequency Domain Imaging), which generate optical tomographic images using near-infrared light. The sensor 42 of the diagnostic imaging catheter 40 is a transceiver that irradiates near-infrared light and receives reflected light.

[0015] The imaging diagnostic catheter 40 may have both an ultrasonic transducer and a transceiver unit for OCT or OFDI as the sensor 42. The imaging diagnostic catheter 40 may have a total of three sensors 42, including an ultrasonic transducer, a transceiver unit for OCT, and a transceiver unit for OFDI.

[0016] The diagnostic imaging catheter 40 is not limited to a mechanical scanning type that mechanically rotates and advances and retreats, but may be an electronic radial scanning type diagnostic imaging catheter 40 that uses a sensor 42 in which multiple ultrasonic transducers are arranged in a ring shape.

[0017] The diagnostic imaging catheter 40 may have a so-called linear scanning type sensor 42 in which a plurality of ultrasonic transducers are arranged in a row along the longitudinal direction. The diagnostic imaging catheter 40 may have a so-called two-dimensional array type sensor 42 in which a plurality of ultrasonic transducers are arranged in a matrix.

[0018] Using the imaging diagnostic catheter 40, it is possible to generate tomographic images that include not only luminal walls such as blood vessel walls, but also reflectors present inside luminal organs, such as red blood cells, and organs present outside luminal organs, such as the epicardium and heart.

[0019] The diagnostic imaging catheter 40 may be a vascular endoscope used for optically observing the inner wall of a hollow organ.

[0020] The tubular organ into which the diagnostic imaging catheter 40 is inserted and used is, for example, a blood vessel, a pancreatic duct, a bile duct, a bronchus, etc. In the following description, the diagnostic imaging catheter 40 will be described as an example of a mechanically scanned IVUS catheter shown in FIG.

[0021] 3 is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 has an information processing device 20, an MDU 33, and an image diagnostic catheter 40. The information processing device 20 has a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

[0022] The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, etc. The control unit 21 is connected to each hardware unit constituting the information processing device 20 via a bus.

[0023] The main memory device 22 is a storage device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs being executed by the control unit 21.

[0024] The auxiliary storage device 23 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 23 stores programs to be executed by the control unit 21, a standard image DB (Database) 65, and various data required for executing the programs. The standard image DB 65 may be stored in an external large-capacity storage device connected to the information processing device 20. The communication unit 24 is an interface that communicates between the information processing device 20 and a network.

[0025] Display unit 25 is an interface that connects display device 31 to the bus. Input unit 26 is an interface that connects input device 32 to the bus. Catheter control unit 271 controls MDU 33 and sensor 42, and generates images based on signals received from sensor 42.

[0026] The MDU 33 rotates the sensor 42 and the shaft 43 inside the probe portion 41. The catheter control unit 271 generates one image for each rotation of the sensor 42. The generated image is a transverse slice image centered on the probe portion 41 and approximately perpendicular to the probe portion 41.

[0027] MDU 33 can also move sensor 42 and shaft 43 forward and backward while rotating them inside probe section 41. By a pullback operation in which sensor 42 is rotated while being pulled toward MDU 33 at a constant speed, catheter control section 271 successively generates multiple transverse slice images approximately perpendicular to probe section 41 at a predetermined interval.

[0028] The function and configuration of catheter control unit 271 are similar to those of conventional ultrasonic diagnostic devices, and detailed description thereof will be omitted. Note that control unit 21 may realize the function of catheter control unit 271.

[0029] The information processing device 20 is connected to various imaging diagnostic devices 37, such as an X-ray angiography device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, or an ultrasound diagnostic device, via a HIS (Hospital Information System) or the like.

[0030] The information processing device 20 of this embodiment is a dedicated ultrasound diagnostic device, or a personal computer, tablet, smartphone, or the like having the functions of an ultrasound diagnostic device.

[0031] 4 is an explanatory diagram for explaining the record layout of the standard image DB 65. The standard image DB 65 is a database in which observation target regions and standard images are recorded in association with each other. The standard image DB 65 has a region field and a standard image field.

[0032] The site to be observed is recorded in the site field. "LAD" in the first line indicates the left anterior descending coronary artery. "LCX" and "RCA" similarly indicate the site of the coronary artery. The sites shown in FIG. 4 are examples and are not limited to these. The standard image DB 65 may also contain records corresponding to any site into which the imaging diagnostic catheter 40 may be inserted, such as blood vessels in the lower limbs, the bile duct, or a bronchial branch.

[0033] The standard image field records a standard image generated at the observation target site. The standard image is a typical image used by a doctor for diagnosis using the diagnostic imaging catheter 40, and is rotated in a direction that is familiar to the doctor. Generally, the standard image includes a landmark that the doctor uses as a guide for making a decision.

[0034] The landmark is, for example, a side branch structure of a hollow organ into which the diagnostic imaging catheter 40 is inserted, an organ such as the heart that is close to the hollow organ, or a hollow organ that runs parallel to the hollow organ. For example, when the diagnostic imaging catheter 40 is inserted into a coronary artery, doctors and technicians often use a side branch blood vessel branching off from the coronary artery as a landmark and rotate the displayed image so that the landmark is on the lower side, or use the epicardium and rotate the displayed image so that the landmark is on the upper side.

[0035] The standard image may be an image artificially generated using technology such as computer graphics. The standard image may be an image generated by synthesizing a plurality of case images. The standard image DB 65 has one record for one observation target site.

[0036] The standard image DB 65 may have a field for recording the attributes of the subject, such as the subject's age, sex, or underlying disease. The standard image DB 65 may have a field for recording the lesions included in the observation target site, such as the presence or absence of dissection or the presence or absence of calcification. In the standard image field, a standard image corresponding to each attribute or lesion is recorded.

[0037] 5 to 7 are examples of screens displayed by the catheter system 10. Fig. 5 shows an example of a screen displayed by the control unit 21 on the display device 31 when correspondence with a standard image has not been detected. The screen shown in Fig. 5 includes a first image field 51, a site selection button 591, a status notification field 571, and an automatic rotation button 581.

[0038] In the first image field 51, an image obtained using the diagnostic imaging catheter 40 is displayed in real time. In the following description, an image displayed in real time will be referred to as a real-time image. The site selection button 591 is a pull-down menu button, and the site to be observed that the user is observing is selected.

[0039] For example, the control unit 21 acquires the observation target site from an electronic medical record system. The control unit 21 may detect the position of the tip marker 44 from an image of the imaging diagnostic device 37 and determine the observation target site. The control unit 21 may acquire the observation target site determined by the imaging diagnostic device 37. The control unit 21 automatically sets the site selection button 591 to a state in which the observation target site is selected.

[0040] The control unit 21 may not automatically set the area selection button 591, but may wait for the user to operate the area selection button 591. Even if the control unit 21 automatically sets the area selection button 591, the user can change the area to be observed by operating the area selection button 591. The control unit 21 may accept the user's operation of the area selection button 591 by voice input.

[0041] The control unit 21 searches the standard image DB 65 using the observation target region set in the region selection button 591 as a key, and extracts a standard image. The control unit 21 rotates the real-time image and detects an angle at which the image becomes similar to the standard image.

[0042] Specifically, the control unit 21 rotates the real-time image by an arbitrary angle, for example, 3 degrees or 5 degrees, and then repeats the process of determining the similarity with the standard image. The control unit 21 may generate an image rotated by one or more sound ray data obtained from the sensor 42, and repeat the process of determining the similarity with the standard image.

[0043] The similarity can be evaluated, for example, by the sum of squared differences (SSD), the sum of absolute differences (SAD), or normalized cross-correlation (NCC) of pixel values ​​of pixels at the same position in two images. Note that the above methods are all examples. The method of evaluating the similarity is not limited to these.

[0044] After repeating the process until the real-time image rotates once, the control unit 21 extracts the angle with the highest similarity. The control unit 21 determines whether or not the similarity at that angle exceeds a predetermined threshold value.

[0045] If the similarity does not exceed the threshold, control unit 21 displays "not detected" in status notification field 571 as shown in Fig. 5, and sets auto-rotate button 581 to an unselectable state. If the similarity exceeds the threshold, control unit 21 sets auto-rotate button 581 to a selectable state as shown in Fig. 6. In Fig. 6, auto-rotate button 581 is set to the "OFF" state. Control unit 21 displays the real-time image in an unrotated state in first image field 51.

[0046] 7 shows an example of an image that the control unit 21 displays on the display device 31 when the user sets the automatic rotation button 581 to the "ON" state. The control unit 21 displays a real-time image rotated in a direction most similar to the standard image in the first image field 51. The user can easily grasp the condition of the hollow organ and its surroundings from the displayed real-time image.

[0047] Note that, when a similar orientation to the standard image is detected, the control unit 21 may rotate the real-time image and set the automatic rotation button 581 to the "ON" state. The control unit 21 may accept the operation of the automatic rotation button 581 by voice input.

[0048] Fig. 8 is a modified example of the screen displayed by the catheter system 10. In Fig. 8, the second image field 52 is displayed between the site selection button 591 and the automatic rotation button 581. The control unit 21 displays the rotated real-time image in the first image field 51 and the non-rotated real-time image in the second image field 52.

[0049] By displaying both the rotated real-time image and the non-rotated real-time image, it is possible to prevent the user from losing sight of the part of the image that he or she has focused on when rotating the image.

[0050] For example, the image described using FIG. 8 may be displayed on a display device 31 located in a place that is easy for the doctor who mainly operates the imaging diagnostic catheter 40 to observe, and the image described using FIG. 7 may be displayed on a display device 31 located in a place that is easy for other medical staff to observe.

[0051] 9 is a flowchart explaining the flow of the program processing. The control unit 21 acquires an observation target part (step S501). The control unit 21 searches the standard image DB 65 using the observation target part acquired in step S501 as a key, and acquires an extracted standard image (step S502).

[0052] The control unit 21 acquires a real-time image from the catheter control unit 271 (step S503). The control unit 21 displays the image described with reference to FIG. 5 (step S504). The control unit 21 starts a rotation angle calculation subroutine (step S505). The rotation angle calculation subroutine is a subroutine that calculates a rotation angle that increases the similarity between the standard image and the real-time image. The processing flow of the rotation angle calculation subroutine will be described later.

[0053] The control unit 21 determines whether the similarity between the real-time image and the standard image at the rotation angle calculated by the rotation angle calculation subroutine exceeds a predetermined threshold (step S506). If it is determined that the similarity does not exceed the predetermined threshold, that is, that the similarity is not sufficient (NO in step S506), the control unit 21 returns to step S503.

[0054] When it is determined that the threshold value has been exceeded (YES in step S506), the control unit 21 sets the rotation angle for rotating the real-time image when the automatic rotation button 581 is selected to the angle calculated in step S505 (step S507). The control unit 21 displays the images described using Figs. 6 to 8 on the display device 31 (step S508).

[0055] Thereafter, the control unit 21 updates in real time the image displayed in the first image field 51. The control unit 21 switches between rotating and not rotating the real-time image displayed in the first image field 51 based on the user's operation of the automatic rotation button 581. When an instruction to end is received from the user, or when the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 ends the process.

[0056] If the user operates the part selection button 591 to select a different part to be observed, the control unit 21 executes the program again after terminating the processing. In step S503 during re-execution, the control unit 21 displays the real-time image at the same rotation angle as immediately before the user operated the part selection button 591. When the user operates the part selection button 591, the image displayed in the first image column 51 is rotated, which can prevent confusion for the user who is observing the real-time image.

[0057] 10 is a flowchart illustrating the flow of processing in the rotation angle calculation subroutine. The rotation angle calculation subroutine is a subroutine for calculating a rotation angle that increases the similarity between the standard image and the real-time image.

[0058] The control unit 21 calculates the similarity between the standard image and the real-time image (step S511). As described above, the similarity can be evaluated using any method, such as the sum of squared differences, the sum of absolute differences, or normalized cross-correlation, between pixel values ​​of pixels at the same position in the two images. The control unit 21 may accept a user's selection of the method to be used.

[0059] The control unit 21 associates the rotation angle with the similarity and stores them in the auxiliary storage device 23 or the main storage device 22 (step S512). The initial value of the rotation angle is zero. The control unit 21 determines whether the real-time image has been rotated once (step S513). If it is determined that the image has not been rotated once (NO in step S513), the control unit 21 rotates the image being processed by a predetermined angle (step S514). The control unit 21 returns to step S511.

[0060] If it is determined that one rotation has been made (YES in step S513), control unit 21 calculates a rotation angle with high similarity based on the association between the rotation angle and the similarity recorded in step S512 (step S515). After that, control unit 21 ends the process.

[0061] Before calculating the similarity in step S511, the control unit 21 may perform image processing, such as edge emphasis or contrast correction, on the standard image and the real-time image to clearly extract landmarks.

[0062] The control unit 21 may calculate the similarity by using an image obtained by averaging images of multiple adjacent frames instead of the real-time image. Since reflectors such as red blood cells flowing inside the tubular organ are not depicted, the landmarks become clear. By using an image in which the landmarks are clearly extracted, the rotation angle at which the similarity between the two images becomes high can be calculated with high accuracy.

[0063] The control unit 21 may reduce the standard image and the real-time image before calculating the similarity in step S511. Alternatively, the standard image after reduction processing may be recorded in the standard image DB 65. By using the reduced image, the calculation process for calculating the similarity can be speeded up.

[0064] When the frame rate of the real-time image is high and the processing of step S505 cannot keep up, the control unit 21 may start step S505, for example, once every two or three frames. The control unit 21 may determine the frame for performing the processing of step S505 to be synchronized with the electrocardiogram.

[0065] A part or all of the program may be executed on a large-scale computer connected via a network, a virtual machine running on a large-scale computer, or a cloud computing system. A part or all of the program may be executed on multiple personal computers performing distributed processing.

[0066] According to this embodiment, it is possible to provide a catheter system 10 that automatically rotates a real-time image to a similar orientation to a standard image. For example, in a catheterization room where multiple medical staff such as doctors and nurses work together to diagnose and treat, each medical staff member can quickly grasp the condition of the hollow organ because the real-time image is displayed in an appropriate orientation. Therefore, catheterization treatment can be performed quickly and appropriately.

[0067] According to this embodiment, it is possible to provide a catheter system 10 that does not require skilled staff who can accurately interpret real-time images to manually perform rotation operations on real-time images.

[0068] Instead of real-time images, videos or still images stored in the auxiliary storage device 23 or the like may be used. It is possible to provide a catheter system 10 that can be used for recording medical records after the end of a case. In such a case, the information processing device 20 may be a personal computer, tablet, smartphone, or the like that does not have a function for connecting the MDU 33 and the catheter 40 for image diagnosis.

[0069] [Embodiment 2] This embodiment relates to a catheter system 10 that continuously calculates the rotation angle. Description of parts common to the first embodiment will be omitted.

[0070] Fig. 11 is a flowchart for explaining the flow of processing of the program according to the embodiment 2. The processing from step S501 to step S508 is common to the processing of the program according to the embodiment 1 explained using Fig. 9, and therefore the explanation will be omitted.

[0071] The control unit 21 acquires a real-time image from the catheter control unit 271 (step S521). The control unit 21 starts a subroutine for calculating a rotation angle (step S522). The subroutine for calculating a rotation angle is the same as the subroutine described with reference to FIG.

[0072] The control unit 21 determines whether the real-time image has been rotated (step S523). Specifically, the control unit 21 determines that the real-time image has been rotated when the difference between the rotation angle set in step S507 and the rotation angle calculated in step S522 is greater than a predetermined threshold. The predetermined threshold may be an angle for one sound ray data, or an angle that can be easily identified by the user visually, such as 30 degrees or 60 degrees. The control unit 21 may accept a threshold set by the user.

[0073] If it is determined that the image has rotated (YES in step S523), the control unit 21 updates the rotation angle set in step S507 to the rotation angle newly calculated in step S522 (step S524). If it is determined that the image has not rotated (NO in step S523) or after the completion of step S524, the control unit 21 determines whether or not to end the process (step S525).

[0074] For example, when an instruction to end the process is received from the user, when the diagnostic imaging catheter 40 is removed from the MDU 33, or when the user operates the site selection button 591 to select another site to be observed, the control unit 21 determines to end the process. When it is determined not to end the process (NO in step S525), the control unit 21 returns to step S508. When it is determined to end the process (YES in step S525), the control unit 21 ends the process.

[0075] According to this embodiment, it is possible to provide a catheter system 10 that automatically follows the rotation of a real-time image caused by the bending state of a tubular organ or the like and continues to correct the orientation of the real-time image.

[0076] [Embodiment 3] This embodiment relates to a catheter system 10 that uses a schema, which shows the structure of a hollow organ and its surrounding organs, as a standard image. In this embodiment, after converting a real-time image into an object arrangement image, the image is rotated in a direction that increases the similarity with the schema. Explanations of parts common to the first embodiment will be omitted.

[0077] 12 is an explanatory diagram for explaining the configuration of the learning model 61. The learning model 61 is a model that receives a real-time image and outputs an object arrangement image 482 in which the types of multiple objects included in the real-time image are associated with the ranges of each object and mapped. The learning model 61 is generated by machine learning.

[0078] In the object arrangement image 482 shown in FIG. 12, thin horizontal hatching indicates a cross section of the diagnostic imaging catheter 40, right-down hatching indicates the inside of a hollow organ, and thick left-down hatching indicates the epicardium.

[0079] The hatching in Fig. 12 shows that each object is painted in a different color. Painting each object in a different color is one example of a method for displaying each object in a distinctive manner. Each object may be displayed in any manner, such as by surrounding the outer edge of the object, so as to be distinguishable from other objects.

[0080] 12, a cross-sectional image is displayed in the portion other than the objects listed above in the object arrangement image 482. The control unit 21 may classify the entire surface of the object arrangement image 482 into some objects such as "hollow organ wall", "plaque", "calcification", "guide wire", etc., and display them in different colors.

[0081] The learning model 61 is, for example, a semantic segmentation model, and includes an input layer, a neural network, and an output layer. The neural network has a U-Net structure that realizes semantic segmentation. The U-Net structure is composed of a multi-layered encoder layer and a multi-layered decoder layer connected thereto. Semantic segmentation assigns a label indicating the type of object to each pixel that constitutes the input image.

[0082] By determining the display method for each pixel according to the label, the control unit 21 can generate an output image in which objects are mapped with different colors according to their types, as shown in the object arrangement image 482 in Fig. 4. In the standard image field of the standard image DB 65 of this embodiment, a schema of a standard image painted with the same colors as the object arrangement image is recorded.

[0083] 12 is an example of a model that outputs information about landmarks included in an image when the image is input. The learning model 61 may be a model that detects landmarks and the like from an image using Mask R-CNN (Regions with Convolutional Neural Networks).

[0084] Fig. 13 is a flowchart illustrating the flow of processing of a rotation angle calculation subroutine according to embodiment 3. The subroutine of Fig. 13 is a subroutine that is used instead of the rotation angle calculation subroutine described using Fig. 10.

[0085] The control unit 21 inputs the real-time image acquired from the catheter control unit 271 into the learning model 61, and acquires an object arrangement image 482 that associates the types of multiple objects contained in the real-time image with the range of each object (step S531).

[0086] The control unit 21 calculates the similarity between the standard image and the object arrangement image 482 (step S532). The control unit 21 associates the rotation angle and the similarity with each other and stores them in the auxiliary storage device 23 or the main storage device 22 (step S533). The initial value of the rotation angle is zero.

[0087] The control unit 21 determines whether the real-time image has rotated once (step S534). If it is determined that the real-time image has not rotated once (NO in step S534), the control unit 21 rotates the object arrangement image acquired in step S531 by a predetermined angle (step S535). The control unit 21 returns to step S532.

[0088] If it is determined that one rotation has been made (YES in step S534), the control unit 21 calculates a rotation angle with high similarity based on the relationship between the rotation angle and the similarity recorded in step S533 (step S536). After that, the control unit 21 ends the process.

[0089] According to the present embodiment, it is possible to provide a catheter system 10 that can use a schema for a standard image. For example, it is possible to provide a catheter system 10 that can be used even when an actual case image suitable for a standard image cannot be prepared, such as when a new type of diagnostic imaging catheter 40 is used or when the diagnostic imaging catheter 40 is used for a new purpose.

[0090] [Embodiment 4] This embodiment relates to a catheter system 10 that calculates a rotation angle based on a landmark designated by a user from a real-time image. Description of parts common to the third embodiment will be omitted.

[0091] 14 is an explanatory diagram illustrating the record layout of the Merkmal DB. The Merkmal DB is a DB that records the observation target site, the name of the Merkmal, and the Merkmal image indicating the position where the Merkmal is displayed in association with each other. The Merkmal DB has a site field, a Merkmal name field, and a Merkmal image field.

[0092] The part to be observed is recorded in the part field. The name of the landmark is recorded in the landmark name field. The image showing the range of the landmark is recorded in the landmark image field.

[0093] In the Merkmal DB, multiple Merkmal names may be recorded for one observation target site. One Merkmal name may be associated with multiple observation target sites. The Merkmal DB has one record for each combination of an observation target site and a Merkmal name.

[0094] A specific example will be given. In the first row of Fig. 14, a record is recorded in which the observation target site is "LAD" and the landmark name is "epicardium". In the landmark image field, a landmark image is recorded which illustrates a standard range in which the landmark "epicardium" is displayed while the diagnostic imaging catheter 40 is being inserted into the "LAD", that is, a standard position and standard shape in which the "epicardium" is displayed. In Fig. 14, the hatched portion is the standard range in which the landmark is displayed.

[0095] 14, a record is recorded in which the observation target site is "LAD" and the landmark name is "D1 (First Diagonal Branch)." In the landmark image field, a landmark image is recorded that illustrates a standard range for displaying the landmark "D1" while the diagnostic imaging catheter 40 is being inserted into the "LAD."

[0096] Fig. 15 is an example of a screen displayed by the catheter system 10 of embodiment 4. Fig. 15 is an example of an image displayed by the control unit 21 on the display device 31 when the user manually specifies a landmark. The screen shown in Fig. 15 includes a first image field 51, a site selection button 591, a landmark selection button 592, and an auto-rotation button 581.

[0097] A designated image for accepting a user's designation of a landmark is displayed in the first image field 51. The designated image is a real-time image obtained using the diagnostic imaging catheter 40, or a paused image obtained by pausing the update of the real-time image based on a user operation.

[0098] The site selection button 591 is a pull-down menu button, and an observation target site being observed by the user is selected. The landmark selection button 592 is a pull-down menu button, and a landmark name designated by the user is selected.

[0099] For example, the control unit 21 acquires the observation target site from an electronic medical record system. The control unit 21 may detect the position of the sensor 42 from an image being captured by the image diagnostic device 37 and determine the observation target site. The control unit 21 may acquire the observation target site determined by the image diagnostic device 37. The control unit 21 automatically sets the site selection button 591 to a state in which the observation target site is selected.

[0100] The control unit 21 may not automatically set the area selection button 591, but may wait for the user to operate the area selection button 591. Even if the control unit 21 automatically sets the area selection button 591, the user can change the area to be observed by operating the area selection button 591. The control unit 21 may accept the user's operation of the area selection button 591 by voice input.

[0101] The control unit 21 searches the landmark DB using the observation target site as a key, and extracts the landmark name associated with the site. The control unit 21 sets the extracted landmark name as an option when the pull-down menu of the landmark selection button 592 is opened. The control unit 21 accepts the operation of the landmark selection button 592 by the user.

[0102] The control unit 21 displays a cursor 68 in the first image field 51. The user operates the cursor 68 using an input device such as a mouse to input a designation frame 69 indicating the range of the landmark. The control unit 21 acquires the range of the landmark designated by the user.

[0103] For example, control unit 21 accepts the designation of the landmark range via a user interface similar to that of so-called paint software. Control unit 21 may display designation frame 69 of a shape suitable for designating the landmark range in first image field 51 based on the observation target region selected via region selection button 591 and region selection button 591 selected via landmark selection button 592, and accept movement and deformation operations by the user. After completing the designation of the landmark range, the user selects auto-rotate button 581.

[0104] The control unit 21 searches the landmark DB using the observation target area set in the area selection button 591 and the landmark name set in the landmark selection button 592 as keys, and extracts a landmark image. The control unit 21 detects an angle at which the range of the designation frame 69 specified by the user is similar to the range of the landmark included in the landmark image.

[0105] Specifically, the control unit 21 generates a designation frame image in which the area designated by the designation frame 69 is filled in in the same manner as the landmark image on a white background of the same dimensions as the designated acceptance image. The control unit 21 rotates the designation frame image by any angle, and then repeatedly performs a process of determining the similarity with the landmark image. After repeating the process until the designation frame image has rotated once, the control unit 21 extracts the angle with the highest similarity. The control unit 21 rotates the designated acceptance image together with the designation frame 69.

[0106] 16 is a flowchart explaining the flow of processing of the program of the embodiment 4. The control unit 21 acquires an observation target part (step S541). The control unit 21 searches the merkmal DB using the observation target part acquired in step S541 as a key, and extracts the merkmal name (step S542).

[0107] The control unit 21 sets the landmark name extracted in step S542 as an option for the landmark selection button 592. The control unit 21 acquires the landmark name selected by the user (step S543). The control unit 21 searches the landmark DB using the observation target site acquired in step S541 and the landmark name acquired in step S543 as keys, and extracts a landmark image (step S544).

[0108] The control unit 21 prompts the user to input a landmark. The user operates the cursor 68 to input a designation frame 69 indicating the range of the landmark depicted in the designation acceptance image. The control unit 21 acquires the designation frame 69 input by the user (step S545). The control unit 21 generates a designation frame image (step S546).

[0109] The control unit 21 starts a subroutine for calculating a rotation angle (step S547). The subroutine for calculating a rotation angle is the same as the subroutine described with reference to Fig. 10, and is a subroutine for calculating a rotation angle that increases the similarity between the standard image and the real-time image. In this embodiment, the control unit 21 inputs the landmark acquired in step S544 as the standard image instead of the standard image, and the designated frame image generated in step S546 as the real-time image, into the subroutine for calculating a rotation angle.

[0110] The control unit 21 sets the rotation angle for rotating the real-time image when the automatic rotation button 581 is selected to the angle calculated in step S547 (step S548). The control unit 21 rotates the designated accepted image being displayed in the first image column 51 (step S549).

[0111] When the user instructs to display a real-time image in the first image field 51, the control unit 21 updates in real time the image to be displayed in the first image field 51. The control unit 21 switches between rotating and not rotating the real-time image to be displayed in the first image field 51 based on the user's operation of the auto-rotate button 581. When an instruction to end is received from the user or when the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 ends the process.

[0112] The control unit 21 may change the position and shape of the designation frame 69 so that it follows the real-time image. The method of tracking a designated area or point on a moving image has been conventionally used, and therefore a description thereof will be omitted.

[0113] When the user operates the part selection button 591 to select a different part to be observed, the control unit 21 ends the processing and re-executes the program. When the control unit 21 receives an instruction to display a real-time image during re-execution, the control unit 21 displays the image at the same rotation angle as immediately before the user operated the part selection button 591. When the user operates the part selection button 591, the image displayed in the first image column 51 is rotated, which can prevent confusion for the user who is observing the real-time image.

[0114] According to the present embodiment, it is possible to provide a catheter system 10 that automatically rotates a real-time image to a similar orientation to a standard image based on a landmark specified by a user through image interpretation. The user can specify an appropriate landmark according to a planned treatment procedure.

[0115] [Variation 1] In the first modification, the control unit 21 uses the learning model 61 described with reference to Fig. 12 to assist in inputting the designation frame 69. The control unit 21 executes step S545 of the program described with reference to Fig. 16 by replacing it with the following process.

[0116] The control unit 21 inputs the designated accepted image into the learning model 61 to acquire an object arrangement image. The control unit 21 extracts the contour line of the object corresponding to the landmark name accepted via the landmark selection button 592. The control unit 21 uses the extracted contour line as a default designated frame 69 and displays it in the first image field 51. The control unit 21 accepts modifications to the designated frame 69 by the user.

[0117] According to this modification, it is possible to provide a catheter system 10 that allows the user to easily specify the specification frame 69 in a short period of time.

[0118] [Variation 2] In Modification 2, the rotation angle is automatically calculated without accepting any modification of the designation frame 69 by the user. The control unit 21 executes steps S545 and S546 of the program described with reference to FIG. 16 by replacing them with the following processes.

[0119] The control unit 21 inputs the designation acceptance image into the learning model 61 described with reference to Fig. 12 to acquire an object arrangement image. The control unit 21 extracts pixels corresponding to the object corresponding to the landmark name accepted via the landmark selection button 592. The control unit 21 generates a designation frame image by setting the pixels other than the extracted pixels to a background color such as white.

[0120] According to this modification, a catheter system 10 can be provided that does not require the user to specify the specification frame 69.

[0121] [Embodiment 5] This embodiment relates to a catheter system 10 that extracts a landmark during a pullback operation and rotates an image. Descriptions of parts common to the first embodiment will be omitted.

[0122] 17 and 18 are examples of screens displayed by the catheter system of embodiment 5. The screen shown in Fig. 17 includes a first image field 51, a second image field 52, a region selection button 591, a landmark selection button 592, and an automatic rotation button 581.

[0123] A real-time image is displayed in the first image field 51. A longitudinal tomographic image is displayed in the second image field 52. The right end of the longitudinal tomographic image corresponds to the real-time image, and as the sensor 42 moves toward the MDU 33 by the pull-back operation, the longitudinal tomographic image displayed in the second image field 52 extends to the right.

[0124] In Figures 17 and 18, the observation site is "SFA (Superficial Femoral Artery)" and the landmark is "DFA (Deep Femoral Artery)."

[0125] In the state shown in FIG. 17, the landmark has not been detected, and the auto-rotate button 581 is set to an unselectable state. FIG. 18 shows the state after the landmark has been detected. A detection mark 572 displayed on the edge of the second image field 52 indicates the position where the landmark was detected. After the landmark is detected and the rotation angle is calculated, the control unit 21 sets the auto-rotate button 581 to a selectable state. In FIG. 18, the auto-rotate button 581 has been selected, and rotated images are displayed in the first image field 51 and the second image field 52.

[0126] 19 is a flowchart for explaining the flow of processing of the program of the fifth embodiment. The control unit 21 acquires an observation target part (step S551). The control unit 21 searches the standard image DB 65 using the observation target part acquired in step S551 as a key, and acquires an extracted standard image (step S552). The control unit 21 accepts an instruction from the user to start a pullback operation (step S553).

[0127] The control unit 21 acquires a real-time image from the catheter control unit 271 (step S554). The control unit 21 displays the image described using FIG. 17 (step S555). The control unit 21 starts a subroutine for calculating a rotation angle (step S556). The subroutine for calculating a rotation angle is the same as the subroutine described using FIG. 10.

[0128] The control unit 21 judges whether the similarity between the real-time image and the standard image at the rotation angle calculated by the rotation angle calculation subroutine exceeds a predetermined threshold value (step S557). If it is judged that the similarity does not exceed the predetermined threshold value, that is, that the similarity is not sufficient (NO in step S557), the control unit 21 judges whether the pullback operation has ended (step S558).

[0129] If it is determined that the process has not ended (NO in step S558), control unit 21 returns to step S554. If it is determined that the process has ended (YES in step S558), control unit 21 ends the process.

[0130] If the predetermined threshold is exceeded, that is, if it is determined that there is sufficient similarity (YES in step S557), the control unit 21 sets the rotation angle for rotating the real-time image when the automatic rotation button 581 is selected to the angle calculated in step S557 (step S559). The control unit 21 displays the image described using Fig. 18 on the display device 31 (step S560).

[0131] Thereafter, control unit 21 updates in real time the images displayed in first image field 51 and second image field 52. Control unit 21 switches between rotating and not rotating the images displayed in first image field 51 and second image field 52 based on the user's operation of auto-rotate button 581. When the pullback operation is completed, control unit 21 ends the process.

[0132] Note that control unit 21 may rotate the images displayed in first image column 51 and second image column 52 when setting the rotation angle in step S560.

[0133] According to the present embodiment, it is possible to provide a catheter system 10 that detects landmarks during a pullback operation and rotates an image. Even if there are few locations where landmarks are depicted, it is possible to provide a catheter system 10 that automatically rotates an image in a predetermined direction.

[0134] [Embodiment 6] This embodiment relates to a program for generating a learning model 61. Explanation of parts common to the third embodiment will be omitted.

[0135] 20 is an explanatory diagram for explaining the record layout of the training DB. The training DB is a database that records inputs and correct labels in association with each other, and is used for training a model by machine learning. The training DB has an input data field and a color-coded data field.

[0136] The input data field records an input image acquired using the diagnostic imaging catheter 40. The color-coded data field records an image in which an expert such as a doctor has colored each object in a different color in the input image. That is, the color-coded data field records the object to which each pixel in the input image corresponds. The training DB records a large number of combinations of input images generated using the diagnostic imaging catheter 40 and images colored by an expert or the like.

[0137] 21 is a flowchart for explaining the flow of processing of the program according to the embodiment 6. An example will be explained in which machine learning of the learning model 61 is performed using the information processing device 20.

[0138] 21 may be executed on hardware other than the information processing device 20, and the learning model 61 after the machine learning process may be copied to the auxiliary storage device 23 via a network. The learning model 61 trained on one piece of hardware may be used on multiple information processing devices 20.

[0139] Prior to the execution of the program in Fig. 21, an untrained model such as a U-Net structure that realizes semantic segmentation is prepared. As described above, the U-Net structure is composed of a multi-layered encoder layer and a multi-layered decoder layer connected thereto. The program in Fig. 19 adjusts each parameter of the prepared model and performs machine learning.

[0140] The control unit 21 acquires a training record used for training one epoch from the training DB (step S571). The control unit 21 adjusts the parameters of the model so that a correct image label is output from the output layer when an input image is input to the input layer of the model (step S572). In acquiring the training record and adjusting the parameters of the model, the program may have a function of causing the control unit 21 to execute operations such as accepting corrections by the user, presenting the grounds for judgment, and re-learning, as appropriate.

[0141] The control unit 21 determines whether to end the process (step S573). For example, the control unit 21 determines to end the process when learning for a predetermined number of epochs is completed. The control unit 21 may obtain test data from the training DB, input the test data to the model under machine learning, and determine to end the process when an output with a predetermined accuracy is obtained.

[0142] If it is determined that the process should not be terminated (NO in step S573), control unit 21 returns to step S571. If it is determined that the process should be terminated (YES in step S573), control unit 21 records the parameters of the trained model in auxiliary storage device 23 (step S574). Thereafter, control unit 21 terminates the process. Through the above process, a trained model is generated.

[0143] According to this embodiment, the learning model 61 can be generated by machine learning.

[0144] [Embodiment 7] 22 is a functional block diagram of the information processing device 20 according to the seventh embodiment. The information processing device 20 includes an acquisition unit 81 and a display unit 82. The acquisition unit 81 acquires an image generated using a diagnostic imaging catheter inserted into a hollow organ. The display unit 82 displays the image acquired by the acquisition unit 81 in a rotated state such that the landmark included in the image is oriented in a predetermined direction.

[0145] [Embodiment 8] Fig. 23 is an explanatory diagram for explaining the configuration of a catheter system 10 of embodiment 8. This embodiment relates to a form in which the catheter system 10 of this embodiment is realized by combining and operating a catheter control device 27, an MDU 33, an image diagnostic catheter 40, a general-purpose computer 90, and a program 97. Explanations of parts common to embodiment 1 will be omitted.

[0146] Catheter control device 27 is an ultrasonic diagnostic device for IVUS that controls MDU 33, controls sensor 42, and generates transverse and longitudinal images based on signals received from sensor 42. The function and configuration of catheter control device 27 are similar to those of conventional ultrasonic diagnostic devices, and therefore a description thereof will be omitted.

[0147] The catheter system 10 of this embodiment includes a computer 90. The computer 90 includes a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, a smartphone, or a server computer.

[0148] The program 97 is recorded in a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read out the program 97 stored in a semiconductor memory 98 such as a flash memory implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown), and store it in the auxiliary storage device 23.

[0149] The program 97 is installed as a control program for the computer 90, and is loaded into and executed by the main storage device 22. This causes the computer 90 to function as the information processing device 20 described above.

[0150] The computer 90 is a general-purpose personal computer, a tablet, a smartphone, a mainframe computer, a virtual machine running on a mainframe computer, a cloud computing system, or a quantum computer. The computer 90 may be a plurality of personal computers performing distributed processing.

[0151] The technical features (constitutive elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0152] 10 Catheter System 20 Information processing device 21 Control section 22 Main storage 23 Auxiliary storage device 24 Communications Department 25 Display section 26 Input section 27 Catheter Control Device 271 Catheter control section 29 Reading unit 31 Display device 32 Input Devices 33 MDU 37 Diagnostic Imaging Devices 40 Diagnostic Imaging Catheter 41 Probe section 42 Sensors 43 Shaft 44 Tip Marker 45 Connector part 482 Object Arrangement 51 First Image 52 Second Image Column 571 Status notification column 572 Detection mark 581 Auto Rotation Button 591 Part selection button 592 Merkmal selection button 61 Learning Model 65 Standard Image DB 68 Cursor 69 Designated slots 81 Acquisition Department 82 Display section 90 Computer 96 Portable recording media 97 Program 98 Semiconductor Memory

Claims

1. Acquiring an image generated using an imaging catheter inserted into a hollow organ; Each time the image is acquired, Calculating a rotation angle until the landmark included in the acquired image is oriented in a predetermined direction; If the calculated rotation angle exceeds a predetermined threshold, the image is rotated so that the extracted landmark is oriented in a predetermined direction and displayed. A program that causes a computer to carry out processing.

2. The merkmal is extracted by performing image processing on the image. The program according to claim 1.

3. The acquired image is input to a learning model that outputs information about a landmark included in an image when the image is input, and a landmark is extracted based on the information output from the learning model. The program according to claim 1.

4. Accept input of the merkma through an input device The program according to claim 1.

5. The imaging diagnostic catheter is a catheter for generating ultrasonic tomographic images. The program according to any one of claims 1 to 4.

6. The imaging diagnostic catheter is a catheter for generating optical tomographic images. The program according to any one of claims 1 to 4.

7. The diagnostic imaging catheter is a catheter equipped with a sensor for generating an ultrasonic tomographic image and a sensor for generating an optical tomographic image. The program according to any one of claims 1 to 4.

8. The landmark includes the epicardium. The program according to any one of claims 5 to 7.

9. The merkmal includes a branch structure of the hollow organ. The program according to any one of claims 1 to 8.

10. Acquiring an image generated using an imaging catheter inserted into a hollow organ; Each time the image is acquired, Calculating a rotation angle until the landmark included in the acquired image is oriented in a predetermined direction; If the calculated rotation angle exceeds a predetermined threshold, the image is rotated so that the extracted landmark is oriented in a predetermined direction and displayed. An information processing method in which processing is performed by a computer.

11. an acquisition unit that acquires an image generated using an imaging diagnostic catheter inserted into a hollow organ; Each time the acquisition unit acquires the image, A calculation unit that calculates a rotation angle until a landmark included in the acquired image is oriented in a predetermined direction; a display unit that displays the image in a rotated state so that the extracted landmark is oriented in a predetermined direction when the rotation angle calculated by the calculation unit exceeds a predetermined threshold value; An information processing device comprising:

Citation Information

Patent Citations

  • Method and system for detecting and tracking deformable shape of candidate object in image

    JP2011000462A

  • Method for bringing probe, system, and image into registration (PROBE)

    JP2014180575A

  • Apparatus and method for the use of three-dimensional internal ultrasound

    JP2015534841A

  • Ultrasonography of Cardiac Performance by Single Degree of Freedom Heart Chamber Segmentation

    JP2018507738A

  • Semi-automated image segmentation system and method

    JP2019503833A