Program, information processing method, learning model generation method, learning model re-learning method, and information processing system

A program using machine learning models simplifies the use of image diagnostic catheters by identifying and mapping objects in tomographic images, allowing for easy interpretation and improved diagnostic capabilities.

JP7581331B2Active Publication Date: 2024-11-12TERUMO KK
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2022511733
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-03-30
Filing Date
2021-03-09
Publication Date
2024-11-12
Estimated Expiration
2041-03-09

AI Technical Summary

Technical Problem

Interpreting tomographic images taken using an image diagnostic catheter requires significant skill and long hours of training, making the catheter system difficult to use effectively.

Method used

A program that acquires tomographic images, identifies and maps the types and ranges of objects within the images using machine learning models, and provides user-friendly interfaces for easy interpretation and analysis.

Benefits of technology

Enables the catheter system to be used easily by users without extensive training, assisting in quick and accurate interpretation of tomographic images and facilitating informed diagnosis and treatment decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581331000006
    Figure 0007581331000006
  • Figure 0007581331000007
    Figure 0007581331000007
  • Figure 0007581331000008
    Figure 0007581331000008
Patent Text Reader

Abstract

Provided is a program or the like that enables a catheter system to be used easily. This program causes a computer to execute a process of, when a tomographic image (485) generated by using an image diagnosis catheter inserted to a lumen organ has been acquired and the tomographic image (485) has been inputted, inputting the acquired tomographic image (485) to a first model (61) for outputting types of a plurality of objects included in the tomographic image (485) and the respective ranges of the objects in an associated manner, thereby outputting the ranges and types of the objects outputted from the first model (61).
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a program, an information processing method, a method for generating a learning model, a method for relearning a learning model, and an information processing system. [Background technology]

[0002] 2. Description of the Related Art A catheter system is used in which a diagnostic imaging catheter is inserted into a hollow organ such as a blood vessel to capture a tomographic image (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 164071 Summary of the Invention [Problem to be solved by the invention]

[0004] Skill is required to interpret tomographic images taken using diagnostic imaging catheters, and therefore extensive training is required to use the catheter system.

[0005] In one aspect, an object of the present invention is to provide a program or the like that enables easy use of a catheter system. [Means for solving the problem]

[0006] The program acquires a tomographic image generated using a diagnostic imaging catheter inserted into a hollow organ, and when the tomographic image is input, inputs the acquired tomographic image into a first model which outputs the types of a plurality of objects included in the tomographic image in association with the ranges of each of the objects, and outputs the types and ranges of the objects output from the first model. the tomographic images are transverse cross-sectional images showing cross sections obtained by cutting the hollow organ and its periphery in a direction intersecting the longitudinal direction of the hollow organ, and the type and range of the object are mapped and stored for each of the multiple transverse cross-sectional images; and the type and range of the object are mapped and output on longitudinal cross-sectional images, which are generated based on the multiple transverse cross-sectional images and show cross sections obtained by cutting the hollow organ and its periphery in a direction parallel to the longitudinal direction of the hollow organ. The processing is executed by a computer. Effect of the Invention

[0007] In one aspect, a program or the like can be provided that makes it easy to use the catheter system. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of a catheter system. [Diagram 2] FIG. 1 is an explanatory diagram for explaining an overview of a catheter for diagnostic imaging. [Diagram 3] FIG. 1 is an explanatory diagram illustrating a configuration of a catheter system. [Figure 4] FIG. 2 is an explanatory diagram for explaining a first model. [Diagram 5] 13 is an example of a screen displayed by the catheter system. [Figure 6] 13 is an example of a screen displayed by the catheter system. [Figure 7] 13 is an example of a screen displayed by the catheter system. [Figure 8] 11 is a flowchart illustrating a process flow of a program. [Figure 9] FIG. 11 is an explanatory diagram illustrating the configuration of a catheter system according to a second embodiment. [Figure 10] 13 is an example of a screen displayed by the catheter system of the third embodiment. [Figure 11] 13 is an example of a screen displayed by the catheter system of the third embodiment. [Figure 12] 13 is a flowchart illustrating the flow of processing of a program according to a third embodiment. [Figure 13] FIG. 11 is an explanatory diagram illustrating a configuration of a second model. [Figure 14] 13 is an example of a screen displayed by the catheter system of the fourth embodiment. [Figure 15] 13 is a flowchart illustrating the flow of processing of a program according to a fourth embodiment. [Figure 16] 13 is an example of a screen displayed by the catheter system of the fifth embodiment. [Figure 17]13 is a flowchart illustrating the flow of processing of a program according to a fifth embodiment. [Figure 18] FIG. 2 is an explanatory diagram illustrating the record layout of a training data DB. [Figure 19] 20 is a flowchart illustrating the flow of processing of a program according to a sixth embodiment. [Figure 20] 20 is a flowchart illustrating the flow of processing of a program according to a sixth embodiment. [Figure 21] FIG. 11 is an explanatory diagram illustrating the record layout of a first revision DB. [Figure 22] 13 is an example of a screen displayed by the catheter system of the seventh embodiment. [Figure 23] FIG. 13 is an explanatory diagram illustrating the record layout of a second revision DB. [Figure 24] 13 is an example of a screen displayed by the catheter system of the seventh embodiment. [Diagram 25] FIG. 23 is a functional block diagram of a catheter system according to an eighth embodiment. [Figure 26] FIG. 13 is an explanatory diagram illustrating the configuration of a catheter system according to a ninth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment 1] FIG. 1 is an explanatory diagram for explaining an overview of a catheter system 10. The catheter system 10 includes a catheter 40 for diagnostic imaging, an MDU (Motor Driving Unit) 33, and an information processing device 20. The catheter 40 for diagnostic imaging is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is, for example, a keyboard, a mouse, a trackball, or a microphone. The display device 31 and the input device 32 may be stacked together to form a touch panel. The input device 32 and the information processing device 20 may be configured as an integrated unit.

[0010] 2 is an explanatory diagram for explaining an overview of the diagnostic imaging catheter 40. The diagnostic imaging catheter 40 has a probe portion 41 and a connector portion 45 disposed at an end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the diagnostic imaging catheter 40 farther from the connector portion 45 is referred to as the tip side.

[0011] A shaft 43 is inserted inside the probe portion 41. A sensor 42 is connected to the tip side of the shaft 43. A ring-shaped tip marker 44 is fixed near the tip of the probe portion 41.

[0012] The function of the MDU 33 allows the sensor 42 and shaft 43 to advance and retreat while rotating inside the probe portion 41. By a pull-back operation in which the sensor 42 is rotated while being pulled toward the MDU 33 side at a constant speed, a plurality of transverse slice images 485 (see FIG. 4) approximately perpendicular to the probe portion 41 and centered on the probe portion 41 are continuously captured at a predetermined interval.

[0013] The sensor 42 is, for example, an ultrasonic transducer that transmits and receives ultrasonic waves, or a transmitting / receiving unit for OCT (Optical Coherence Tomography) that irradiates near-infrared light and receives reflected light. The tubular organ into which the diagnostic imaging catheter 40 is inserted and used is, for example, a blood vessel, a pancreatic duct, a bile duct, or a bronchus.

[0014] 2 shows an example of an imaging diagnostic catheter 40 for IVUS (Intravascular Ultrasound) used when capturing an ultrasonic tomographic image from inside a blood vessel. In the following description, the imaging diagnostic catheter 40 will be described as an IVUS catheter.

[0015] The diagnostic imaging catheter 40 is not limited to a mechanical scanning type that mechanically rotates and advances and retreats. The diagnostic imaging catheter 40 may be an electronic radial scanning type that uses a sensor 42 in which multiple ultrasonic transducers are arranged in a ring shape.

[0016] The diagnostic imaging catheter 40 may have a so-called linear scanning type sensor 42 in which a plurality of ultrasonic transducers are arranged in a row along the longitudinal direction. The diagnostic imaging catheter 40 may have a so-called two-dimensional array type sensor 42 in which a plurality of ultrasonic transducers are arranged in a matrix.

[0017] The diagnostic imaging catheter 40 can capture cross-sectional images including not only luminal walls such as blood vessel walls, but also reflectors present inside the luminal cavity of a luminal organ, such as red blood cells, and organs present outside the luminal organ, such as the pericardium and heart.

[0018] 3 is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 has an information processing device 20, an MDU 33, and an image diagnostic catheter 40. The information processing device 20 has a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

[0019] The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, etc. The control unit 21 is connected to each hardware unit constituting the information processing device 20 via a bus.

[0020] The main memory device 22 is a storage device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs being executed by the control unit 21.

[0021] The auxiliary storage device 23 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 23 stores a program to be executed by the control unit 21, the first model 61, and various data required for executing the program. The communication unit 24 is an interface that communicates between the information processing device 20 and a network.

[0022] Display unit 25 is an interface that connects display device 31 to the bus. Input unit 26 is an interface that connects input device 32 to the bus. Catheter control unit 271 controls MDU 33, controls sensor 42, and generates transverse tomographic images 485 and longitudinal tomographic images based on signals received from sensor 42. The function and configuration of catheter control unit 271 are similar to those of conventionally used ultrasound diagnostic devices, and therefore a description thereof will be omitted. Note that control unit 21 may realize the function of catheter control unit 271.

[0023] The information processing device 20 is connected to various imaging diagnostic devices 37, such as an X-ray angiography device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, or an ultrasound diagnostic device, via a HIS (Hospital Information System) or the like.

[0024] The information processing device 20 of this embodiment is a dedicated ultrasound diagnostic device, or a personal computer, tablet, smartphone, or the like having the functions of an ultrasound diagnostic device.

[0025] 4 is an explanatory diagram for explaining the first model 61. The first model 61 is a model that receives a cross-sectional image 485 and outputs an object arrangement image 482 in which the types of a plurality of objects included in the cross-sectional image 485 are associated with the ranges of the respective objects and mapped. The first model 61 is generated by machine learning.

[0026] In the object arrangement image 482 shown in Figure 4, the vertical hatching indicates the "cross section of the diagnostic imaging catheter 40", the horizontal hatching indicates the "wall of the hollow organ", the right-sloping hatching indicates the "inside of the hollow organ", the left-sloping hatching indicates the "guide wire", and the thin grid-like hatching indicates "calcification".

[0027] The term "guidewire" includes the guidewire itself, multiple echoes produced by the guidewire, and the acoustic shadow produced by the guidewire. Similarly, the term "calcification" includes the calcified portion itself, multiple echoes produced by the calcified portion, and the acoustic shadow produced by the calcified portion.

[0028] The hatching in Fig. 4 shows that each object is painted in a different color. Painting each object in a different color is one example of a method for displaying each object in a distinctive manner. Each object may be displayed in any manner, such as by surrounding the outer edge of the object, so as to be distinguishable from other objects.

[0029] Here, the "cross section of the diagnostic imaging catheter 40", the "wall of a hollow organ", the "inside of a hollow organ", the "guidewire", and "calcification" are examples of objects included in the transverse image 485. For example, the "guidewire itself", the "multiple echoes generated by the guidewire", and the "acoustic shadow generated by the guidewire" may each be classified as different objects. Similarly, lesions such as "plaque" and "dissection" generated in the wall of a hollow organ may each be classified as different objects.

[0030] In the following description, a set of a plurality of transverse images 485 from which a longitudinal image can be generated is referred to as one set of transverse images 485. Similarly, acquiring a set of transverse images 485 from which a longitudinal image can be generated using the imaging diagnostic catheter 40 is referred to as one image acquisition. An example of the first model 61 that accepts input of a set of transverse images 485 and outputs an object arrangement image 482 corresponding to each of the transverse images 485 will be described below.

[0031] One set of transverse images 485 is acquired, for example, by one pull-back operation by the MDU 33. One set of transverse images 485 may be acquired while a user manually pushes and pulls the catheter for diagnostic imaging 40. Here, the pushing and pulling operation of the catheter for diagnostic imaging 40 includes both the operation of pushing and pulling the probe portion 41 and the operation of pushing and pulling the sensor 42 inside the probe portion 41.

[0032] For example, the user pulls back or pushes in the sensor 42 at a substantially constant speed. The transverse cross-sectional images 485 acquired during the period from when the user issues an instruction to start acquisition by voice input or the like to when the user issues an instruction to end acquisition constitute one set of transverse cross-sectional images 485.

[0033] A sensor or the like may be provided to detect the amount that the user pushes or pulls the sensor 42. Images acquired while the user pulls back or pushes the sensor 42 through a predetermined range constitute a set of cross-sectional images 485.

[0034] When the position of the sensor 42 can be detected, the user may push and pull the sensor 42 at any speed and in any direction. The transverse cross-sectional images 485 rearranged in order along the longitudinal direction of the probe portion 41 constitute one set of transverse cross-sectional images 485. When the intervals between the transverse cross-sectional images 485 are not constant, position information along the longitudinal direction of the probe portion 41 is associated with each of the transverse cross-sectional images 485 and recorded. In the following description, a case where the intervals between the transverse cross-sectional images 485 are constant will be described as an example.

[0035] As described above, the first model 61 may be a model that accepts one set of transverse cross-sectional images 485 obtained by a pull-back operation using the MDU 33, or may be a model that accepts an input of one set of transverse cross-sectional images 485 obtained by manually moving the sensor 42 forward and backward. The first model 61 may be a model that accepts an input of one transverse cross-sectional image 485. The first model 61 may be a model that accepts an input of half, one-third, or the like of the transverse cross-sectional images 485 obtained by one pull-back operation.

[0036] The first model 61 is, for example, a semantic segmentation model, and includes an input layer, a neural network, and an output layer. The neural network has, for example, a U-Net structure that realizes semantic segmentation. The U-Net structure is composed of a multi-layered encoder layer and a multi-layered decoder layer connected thereto. By semantic segmentation, a label indicating the type of object is assigned to each pixel that constitutes the input image.

[0037] The control unit 21 determines the display method for each pixel according to the label, thereby generating an output image in which objects are mapped in different colors or background patterns for each type, as shown in the object arrangement image 482 of FIG.

[0038] The first model 61 may be a model that realizes image segmentation, generated based on the Mask R-CNN (Regions with Convolutional Neural Networks) model or any other machine learning algorithm.

[0039] The first model 61 may be a model that realizes object detection such as R-CNN. When using a model that performs object detection without performing segmentation, the control unit 21 surrounds the part where the object is detected with a bounding box and displays text such as "calcification" near the bounding box.

[0040] By using the entire set of transverse cross-sectional images 485 as input data, information on adjacent transverse cross-sectional images 485 is reflected in the object arrangement image 482. Therefore, it is possible to realize a first model 61 that is less susceptible to the influence of noise in each transverse cross-sectional image 485 and accurately outputs the range of the object.

[0041] Fig. 5 is an example of a screen displayed by the catheter system 10. The screen shown in Fig. 5 has a transverse layer image field 51, a transverse object arrangement image field 515, and a vertical object arrangement image field 525. The transverse layer image field 51 displays the transverse layer image 485 described above. The transverse object arrangement image field 515 displays an object arrangement image 482 corresponding to the transverse layer image 485 displayed in the transverse layer image field 51.

[0042] An object arrangement image 482 corresponding to the longitudinal tomographic image is displayed in the longitudinal object arrangement image field 525. The object arrangement image 482 corresponding to the longitudinal tomographic image is formed based on the object arrangement image 482 corresponding to each of a set of transverse slice images 485, similar to the longitudinal tomographic image.

[0043] Specifically, from each object arrangement image 482, pixels at positions corresponding to the longitudinal tomographic images are extracted, and an object arrangement image corresponding to the longitudinal tomographic images is formed by performing interpolation processing or the like to reconstruct the images. This processing is similar to the method of forming longitudinal tomographic images from a set of transverse cross-sectional images 485, and therefore a detailed description thereof will be omitted.

[0044] Three-dimensional data may be generated based on a set of transverse slice images 485, and three-dimensional semantic segmentation may be performed to assign a label indicating the type of object to each voxel. From the three-dimensional object arrangement image generated by the three-dimensional semantic segmentation result, an object arrangement image 482 corresponding to the longitudinal slice images can be generated.

[0045] A transverse layer position marker 551 indicating the position of the transverse layer image 485 displayed in the transverse layer image field 51 and the transverse object arrangement image field 515 is displayed on the edge of the transverse layer image field 525. A longitudinal layer position marker 552 indicating the position of the longitudinal layer image displayed in the longitudinal layer image field 52 is displayed near the edge of the transverse layer image field 51 and the transverse object arrangement image field 515.

[0046] The user can appropriately change the cross section to be displayed by operating the input device 32 to change the positions of the transverse slice position marker 551 and the longitudinal slice position marker 552. The control unit 21 may also accept voice input from the user.

[0047] The user can check the state of the lesion in the longitudinal direction on the screen of Fig. 5. Note that Fig. 5 is an example of a screen display, and is not limited to this. For example, a three-dimensional display as a collection of cross sections may be performed.

[0048] Figure 6 is an example of a screen displayed by the catheter system 10. When the user instructs to display the probability that the determination of "calcification" is correct for each pixel constituting an object indicating "calcification", the control unit 21 displays the screen shown in Figure 6. An enlarged view of the portion indicating "calcification" is shown in the upper part of Figure 6.

[0049] The probability that the determination of each pixel is correct is output from the first model 61 described with reference to Fig. 4. The control unit 21 colors the pixels differently based on the probability that each pixel is classified as a "calcification" object.

[0050] For example, the control unit 21 can express the probability that the determination of "calcification" is correct by applying a gradation to the lightness or saturation while keeping the hue representing "calcification" common. The control unit 21 may display the entire horizontal object arrangement image column 515 in different colors based on the determination probability that indicates the correctness of the object determination for each pixel.

[0051] The user can recognize how reliable the determination of "calcification" is from the screen shown in Fig. 6. If the determination of "calcification" is output with a relatively low probability, it is desirable for the user to thoroughly observe the transverse cross-sectional image 485 displayed in the transverse cross-sectional image field 51 from a professional perspective.

[0052] Fig. 7 is an example of a screen displayed by the catheter system 10. When the user instructs to indicate the basis for the portion indicating "calcification", the control unit 21 displays the screen shown in Fig. 7. The transverse cross-sectional image field 51 displays the transverse cross-sectional image 485 on which is superimposed a basis marker 561 indicating the basis region that is the basis for the determination of "calcification". The basis marker 561 is an example of basis information regarding the basis of the object to be displayed in the transverse object arrangement image field 515.

[0053] The control unit 21 extracts the grounds region by a model visualization method such as Gradient-weighted Class Activation Mapping (Grad-CAM) or Grad-CAM++. The grounds region is a region that strongly influenced the output of pixels determined to be "calcification" in the multiple cross-sectional images 485 input to the learning model 65. The grounds marker 561 is displayed with finer hatching in areas that have a greater influence on the output.

[0054] Using the screen shown in FIG. 7, the user can judge from a professional standpoint whether the basis for the determination made by the control unit 21 is appropriate or not.

[0055] 8 is a flow chart for explaining the flow of the program processing. The control unit 21 acquires one set of transverse cross-sectional images 485 from the catheter control unit 271 (step S701). The control unit 21 inputs the acquired transverse cross-sectional images 485 to the first model 61, and acquires an object arrangement image 482 in which the types of objects included in the transverse cross-sectional image 485 are associated with the ranges of each object, and the probability that the object determination for each pixel is correct (step S702).

[0056] The control unit 21 generates a longitudinal section image of the object arrangement image 482 based on one set of the object arrangement image 482 (step S703). In the following description, the longitudinal section image of the object arrangement image 482 is referred to as a vertical object arrangement image. In response to the user's operation of the longitudinal section position marker 552, the control unit 21 records the generated vertical object arrangement image in the auxiliary storage device 23 so that the vertical object arrangement image based on the specified cross section can be quickly displayed.

[0057] The control unit 21 displays the screen described with reference to Fig. 5 on the display device 31 (step S704). The control unit 21 determines whether an instruction to display the determination probability of the object has been received from the user (step S705). The user can input an instruction to display the probability by, for example, double-clicking an object in the horizontal object arrangement image field 515. The control unit 21 may also receive a voice input from the user.

[0058] If it is determined that an instruction to display the probability has been received (YES in step S705), the control unit 21 displays the image described using FIG. 6 based on the probability that the object determination for each pixel obtained in step S702 is correct (step S706).

[0059] When it is determined that the instruction to display the probability has not been received (NO in step S705) or after the end of step S706, the control unit 21 determines whether or not an instruction to display the grounds has been received from the user (step S707). The user can input an instruction to display the grounds by, for example, swiping an object in the horizontal object arrangement image field 515. The control unit 21 may also receive a voice input from the user.

[0060] When it is determined that the instruction to display the grounds has been received (YES in step S707), the control unit 21 acquires an item for displaying the grounds (step S708). The control unit 21 extracts a grounds region related to the item acquired in step S708 by a model visualization method such as Grad-CAM or Grad-CAM++ (step S709).

[0061] The control unit 21 uses the screen described with reference to Fig. 7 to display the cross-sectional image 485 with the grounds marker 561 superimposed thereon in the cross-sectional image field 51 (step S710). If it is determined that an instruction to display the grounds has not been received (NO in step S707), or after step S710 is completed, the control unit 21 ends the process.

[0062] According to the present embodiment, it is possible to provide a catheter system 10 that can be easily used even by a user who is not sufficiently skilled in interpreting tomographic images.

[0063] According to the present embodiment, it is possible to provide a catheter system 10 that assists a user in reading images by using the object arrangement image 482. Since the user can quickly understand what is displayed where in the tomographic image, the user can focus on observing the important parts for diagnosis and treatment.

[0064] According to the present embodiment, it is possible to provide a catheter system 10 that displays the probability of object determination for each pixel. It is possible to provide a catheter system 10 that allows the user to know how reliable the determination by the first model 61 is.

[0065] According to the present embodiment, it is possible to provide a catheter system 10 that displays the basis for the determination of an object. It is possible to provide a catheter system 10 that allows the user to confirm the basis for the determination made by the first model 61.

[0066] [Embodiment 2] This embodiment relates to a catheter system 10 in which a catheter control device 27 and an information processing device 20 are separate entities. Descriptions of parts common to the first embodiment will be omitted.

[0067] 9 is an explanatory diagram illustrating the configuration of a catheter system 10 of embodiment 2. The catheter system 10 of this embodiment has an information processing device 20, a catheter control device 27, an MDU 33, and an image diagnostic catheter 40. The information processing device 20 has a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, and a bus.

[0068] Catheter control device 27 is an ultrasonic diagnostic device for IVUS that controls MDU 33, controls sensor 42, and generates transverse image 485 and longitudinal image based on a signal received from sensor 42. The function and configuration of catheter control device 27 are similar to those of conventional ultrasonic diagnostic devices, and therefore a description thereof will be omitted.

[0069] Catheter control device 27 and information processing device 20 may be directly connected via a cable or wireless communication, or may be connected via a network.

[0070] The information processing device 20 of this embodiment is a general-purpose personal computer, a tablet, a smartphone, a mainframe computer, a virtual machine running on a mainframe computer, a cloud computing system, or a quantum computer. The information processing device 20 may be a plurality of personal computers performing distributed processing.

[0071] [Embodiment 3] This embodiment relates to a catheter system 10 that displays quantitative information such as the length and area of ​​a portion depicted in a tomographic image. Description of portions common to the first embodiment will be omitted.

[0072] 10 to 12 are examples of screens displayed by the catheter system of embodiment 3. In the screen shown in Fig. 10, value labels 538 are displayed in the horizontal object arrangement image field 515 and the vertical object arrangement image field 525 of the screen using Fig. 5.

[0073] The control unit 21 calculates quantitative information such as length, area, and volume related to an object designated by a user, based on an object arrangement image 482 corresponding to one set of cross-sectional images 485. For example, when calculating an area, the control unit 21 multiplies the number of pixels constituting the object by the area per pixel. When calculating a volume, the control unit 21 further multiplies the calculated area by the thickness per slice. A method for calculating quantitative information from an arbitrary figure is well known, and detailed explanation will be omitted.

[0074] The explanation will be continued by taking as an example a case where the specified object is the "inside of a hollow organ" and a narrowed lesion is included in one set of transverse slice images 485. The control unit 21 determines the position where the diameter is the smallest, i.e., the position where the narrowing is greatest, based on the diameter of the "inside of a hollow organ" calculated from each object arrangement image 482. The control unit 21 determines both ends of the lesion based on a predetermined determination criterion.

[0075] In Figure 10, the value label 538 displayed in the lower left of the horizontal object arrangement image column 515 displays the area of ​​the “inside of the tubular organ” indicated by hatching slanting downward to the right, the minimum diameter, and the maximum diameter.

[0076] A total of four vertical lines indicating the positions of the "reference blood vessel," two "lesion ends," and the "minimum diameter portion" are displayed in the vertical object arrangement image field 525. The "reference blood vessel" refers to a portion without a lesion.

[0077] At the end of each vertical line, a corresponding value label 538 is displayed. The minimum and maximum diameters of the "inside of the hollow organ" at each position and the plaque burden are displayed in the value label 538. The "lesion length" is displayed between the two vertical lines indicating the two "lesion ends."

[0078] For example, when the user clicks near the vertical line indicating the position of the "minimum diameter part", the transverse layer position marker 551 moves to the position of the "minimum diameter part". The transverse layer image 485 of the "minimum diameter part" and the transverse object arrangement image field 515 are displayed in the transverse layer image field 51 and the transverse object arrangement image field 515. The user can check the state of the "minimum diameter part".

[0079] Similarly, when the user clicks near the vertical line indicating the position of the "reference vessel" or "lesion end", the transverse layer position marker 551 moves to the corresponding position. The user can easily check the state of the part that is the key point for judgment.

[0080] The user can perform diagnosis and treatment based on the tissue properties and values ​​displayed in the value label 538 and criteria established by an academic society or the like. For example, the user determines the necessity of angioplasty and the procedure based on the blood vessel diameter, the degree of stenosis, the lesion length, or the distribution state of calcification. The control unit 21 may display information on the related criteria together with the value label 538.

[0081] 11 shows a state in which the user has moved the transverse layer position marker 551 to the right side of the vertical object arrangement image field 525. In the transverse layer image field 51, a transverse layer image 485 different from that in FIG. 10 is displayed. In the horizontal object arrangement image field 515, an object arrangement image 482 corresponding to the transverse layer image field 51 is displayed.

[0082] In the horizontal object arrangement image field 515 in Fig. 11, the objects "cross section of diagnostic imaging catheter 40" indicated by vertical hatching, "inside of hollow organ" indicated by hatching slanting downward to the right, and "calcification" indicated by lattice-like hatching are displayed. In the portion where no objects are displayed, a transverse slice image 485 is displayed. The user can check the condition of the hollow organ existing around the "calcification" object.

[0083] The object showing "calcification" is displayed in the shape of a circular arc. The control unit 21 calculates the angle of the "calcification" based on the center of the "hollow organ wall" and displays it in the value label 538.

[0084] A specific example of a method for calculating an angle will be described. The control unit 21 calculates a circle having a shape that approximates the outer periphery of the "inside of the hollow organ", i.e., the inner surface of the wall of the hollow organ. The control unit 21 extracts vectors that extend from the center of the calculated circle to both ends of the object indicating "calcification". The control unit 21 calculates the angle between the two extracted vectors. Note that the method for calculating the angle is arbitrary and is not limited to this.

[0085] For a cylindrical object such as the "wall of a hollow organ" or the "inside of a hollow organ", the control unit 21 calculates the inner diameter, outer diameter, minimum inner diameter and maximum inner diameter, and displays them in the value label 538. For an arc-shaped object such as the "calcification" shown in Fig. 11, the control unit 21 calculates the angle and displays it in the value label 538. Note that the control unit 21 may also display the thickness, volume, etc., for the "calcification" object shown in Fig. 11.

[0086] The user may specify each time which item is to be displayed in the value label 538. The items to be displayed in the value label 538 may be determined by default according to the type, shape, etc. of the object.

[0087] Fig. 12 is a flowchart for explaining the flow of processing of the program of embodiment 3. The program of Fig. 12 is executed when the user instructs to display value label 538 after control unit 21 displays the screen explained using Fig. 5 in step S704 of the program explained using Fig. 8.

[0088] The user can instruct the display of the value label 538 by, for example, tapping an object displayed in the horizontal object arrangement image field 515. Note that the control unit 21 may also accept a voice input from the user.

[0089] Based on an instruction from the user, the control unit 21 acquires the designation of the object for which the value label 538 is to be displayed (step S721). The control unit 21 determines the item to be calculated based on the shape of the designated object, etc. (step S722).

[0090] The control unit 21 calculates the item determined in step S722 (step S723). The control unit 21 displays the value label 538 as described using Figures 10 and 11 (step S724). Thereafter, the control unit 21 ends the process.

[0091] [Embodiment 4] This embodiment relates to a catheter system 10 that uses a second model 62 that outputs information about the state of a hollow organ or the surroundings of the hollow organ when a tomographic image is input. Explanations of parts common to the first embodiment will be omitted.

[0092] 13 is an explanatory diagram for explaining the configuration of the second model 62. The second model 62 is a model that receives one set of transverse slice images 485 and outputs findings related to the state of a hollow organ or the state around the hollow organ, such as the necessity of treatment, the presence or absence of blood flow stasis, or the presence or absence of branching. Note that "necessity of treatment" may be the necessity of IVR (Interventional Radiology) that performs treatment inside the hollow organ, or the necessity of general treatment including medication and dietary therapy.

[0093] The findings output by the second model 62 are probabilities for predetermined options, such as "yes" or "no," for each of a plurality of items. Tables 1 to 5 show examples of items for which the second model 62 outputs probabilities. Each row in Tables 1 to 5 shows one item. The second model 62 outputs the probability of the option for each item. Tables 1 to 5 show examples of items included in the findings output by the second model 62.

[0094] Table 1 provides information regarding the need for treatment.

[0095] [Table 1]

[0096] Table 2 shows items related to blood flow information.

[0097] [Table 2]

[0098] Table 3 shows items related to qualitative shape information of hollow organs and their surroundings.

[0099] [Table 3]

[0100] Table 4 shows items related to property information indicating the properties of hollow organs and the surrounding area of ​​hollow organs.

[0101] [Table 4]

[0102] "In-stent stenosis" shown in Table 4 indicates the presence or absence of stenosis of a stent placed in a luminal organ several months to several years ago. When the transverse image 485 is taken immediately after the procedure of placing the stent, it indicates the presence or absence of stenosis of the placed stent. In other words, the transverse image 485 may be a tomographic image of an untreated luminal organ, a tomographic image of a luminal organ under follow-up observation after the procedure, or a tomographic image of a luminal organ taken immediately after the end of a series of procedures in the luminal organ.

[0103] Table 5 shows items related to device information that indicate the status of an indwelling device, such as a stent, placed in a luminal organ.

[0104] [Table 5]

[0105] Each item shown in Tables 1 to 5 is an example. The second model 62 may output probabilities for some of the items shown in Tables 1 to 5. The second model 62 may output probabilities for items other than the items shown in Tables 1 to 5.

[0106] The options for each item shown in Tables 1 to 5 are examples. For example, items shown in each table with two options, "Yes" and "No," may be shown with three or more options, such as "Large," "Small," and "No."

[0107] In the following description, the second model 62 that receives an input of one set of transverse images 485 obtained by one image acquisition and outputs findings on the state of the hollow organ or the state around the hollow organ will be described as an example. Note that the second model 62 may be a model that receives an input of one transverse image 485 and outputs findings on the state of the hollow organ or the state around the hollow organ.

[0108] As described above, one image may be acquired by one pullback operation by the MDU 33. The second model 62 may be a model that receives an input of a partial transverse image 485, such as half or one-third, of the transverse image 485 acquired by one pullback operation, and outputs findings regarding the state of the hollow organ or the state around the hollow organ.

[0109] The second model 62 includes an input layer, a neural network 629, a plurality of softmax layers 625, and an output layer. The neural network 629 is, for example, a convolutional neural network (CNN) having a set of a plurality of convolution layers and pooling layers, and a fully connected layer. One softmax layer 625 is provided for each row shown in Tables 1 to 5.

[0110] The input layer receives an image obtained by combining a set of cross-sectional images 485 in the scanning order. The image is output to the output layer via a neural network 629 and a softmax layer 625, which indicates the probability of each item shown in Tables 1 to 5.

[0111] For example, in Fig. 13, the probability that "need for treatment" is "no" is 95 percent, the probability that "blood flow stagnation" is "no" is 90 percent, and the probability that "branching" is "yes" is 90 percent. The second model 62 may be separate for each of Tables 1 to 5. The second model 62 may be separate for each item to be output.

[0112] A selection layer may be provided after the softmax layer 625 to select and output the option with the highest probability.

[0113] To the second model 62, data at a stage prior to the formation of the transverse cross-sectional image 485, such as sound ray data acquired by the catheter control unit 271 from the sensor 42, may be input.

[0114] Figure 14 is an example of a screen displayed by the catheter system 10 of embodiment 4. The screen shown in Figure 14 includes a transverse layer image field 51, a transverse object arrangement image field 515, and a findings field 53. The transverse layer image field 51 and the transverse object arrangement image field 515 are the same as those displayed by the catheter system 10 of embodiment 1 described using Figure 5, and therefore description thereof will be omitted.

[0115] The findings are displayed in the findings column 53. The control unit 21 selects, from among the findings output from the second model 62, a finding having a higher probability than a predetermined threshold, and displays it in the findings column 53.

[0116] The control unit 21 may select a finding related to the transverse cross-sectional image 485 being displayed in the transverse cross-sectional image field 51 and display it in the finding field 53. Although not shown in Fig. 14, the control unit 21 may display the vertical object arrangement image field 525 in the same manner as in Fig. 5.

[0117] Fig. 15 is a flowchart for explaining the flow of processing of the program of embodiment 4. Since the steps up to step S703 are the same as those of the program of embodiment 1 explained using Fig. 8, the explanation will be omitted.

[0118] The control unit 21 inputs the cross-sectional image 485 acquired in step S701 into the second model 62 to acquire findings (step S731). The control unit 21 displays the screen described using FIG. 14 on the display device 31 (step S732). The control unit 21 determines whether an instruction to display the determination probability of the object has been received from the user (step S705). The subsequent processing is the same as the program of the first embodiment described using FIG. 8, and therefore description thereof will be omitted.

[0119] According to this embodiment, a catheter system 10 that displays findings in addition to the object placement image 482 can be provided.

[0120] The control unit 21 may display a reason marker 561 indicating the reason for the finding displayed in the finding column 53. The control unit 21 can extract a reason region related to the finding output from the second model 62 by a model visualization method such as Grad-CAM or Grad-CAM++.

[0121] In addition to the transverse image 485, medical information acquired in real time, such as images captured using the diagnostic imaging device 37, blood pressure, heart rate, or oxygen saturation, may be input to the first model 61 and the second model 62. In addition to the transverse image 485, medical information acquired from an electronic medical record, such as medical history, height, weight, and images captured in the past using the diagnostic imaging device 37, may be input to the first model 61 and the second model 62.

[0122] In this case, the first model 61 receives the transverse image 485 and the medical information, and outputs an object arrangement image 482 in which the types of objects included in the transverse image 485 are associated with the ranges of the respective objects and mapped. Similarly, the second model 62 receives the transverse image 485 and the medical information, and outputs findings regarding the state of the hollow organ or the state around the hollow organ.

[0123] By including medical information other than the transverse image 485 in the input data of the first model 61, it is possible to provide a catheter system 10 that accurately classifies objects. By including medical information other than the transverse image 485 in the input data of the second model 62, it is possible to provide a catheter system 10 that outputs findings with high accuracy.

[0124] [Embodiment 5] This embodiment relates to a catheter system 10 that displays the position of a tomographic image captured using a diagnostic imaging catheter 40 superimposed on an image acquired from an imaging diagnostic device 37. Descriptions of parts common to the first embodiment will be omitted.

[0125] Fig. 16 is an example of a screen displayed by the catheter system 10 of embodiment 5. The screen shown in Fig. 16 includes an other device image field 59. In the other device image field 59, medical images captured by the image diagnostic device 37 are displayed.

[0126] A scan area 591 indicating the position of the tomographic image captured by the diagnostic imaging catheter 40 is displayed as a rectangle indicating the outline of the longitudinal tomographic image superimposed on the other device image field 59. The control unit 21 may display the longitudinal tomographic image or the longitudinal object arrangement image in real time inside the scan area 591. The control unit 21 may receive a selection regarding the display form of the scan area 591 from the user.

[0127] An outline of a method for displaying the scan area 591 will be described below, taking as an example a case where the image diagnostic device 37 is an X-ray angiography device. The sensor 42 is mounted on a sensor marker that is opaque to X-rays. Since the tip marker 44 and the sensor marker are opaque to X-rays, they are clearly displayed in the medical image captured by the X-ray angiography device.

[0128] The control unit 21 detects the tip marker 44 and the sensor marker from the medical image. The detected sensor marker indicates the position of the sensor 42. For example, when a set of transverse slice images 485 is generated using a pullback operation by the MDU 33, both ends of the operating range of the sensor 42 at the time of image acquisition correspond to the positions of the short sides of the scan area 591.

[0129] The control unit 21 determines the length of the short side of the scan area 591 based on the display range of the transverse cross-sectional image 485 and the scale of the other device image. The control unit 21 superimposes and displays the rectangular scan area 591 determined from the position and length of the short side on the other device image field 59 displaying the medical image.

[0130] By the above processing, even if the tip portion of the diagnostic imaging catheter 40 is not parallel to the projection surface of the diagnostic imaging device 37, the control unit 21 can display the scan area 591 at the correct position in the other device image field 59.

[0131] Returning to Fig. 16, the explanation will be continued. A plurality of transverse slice position markers 551 are displayed within a scan area 591. A transverse object arrangement image field 515 displaying an object arrangement image 482 corresponding to each of the transverse slice position markers 551 is displayed around the other device image field 59. The user can appropriately change the slice position of the object arrangement image 482 by moving the transverse slice position marker 551 via the input device 32. The control unit 21 may also receive voice input from the user.

[0132] The control unit 21 may switch between the object arrangement image 482 and the transverse cross-sectional image 485 based on an instruction from the user. The control unit 21 may display the object arrangement image 482 and the transverse cross-sectional image 485 side by side. The control unit 21 may display a longitudinal cross-sectional image or a longitudinal object arrangement image.

[0133] In the other device image field 59, a schematic diagram of a hollow organ may be displayed instead of a medical image captured by the diagnostic imaging device 37. This provides the catheter system 10 that allows the user to more easily grasp the position of the tomographic image.

[0134] 17 is a flowchart explaining the flow of processing of the program of embodiment 5. Controller 21 acquires a transverse cross-sectional image 485 and a medical image from catheter controller 271 and image diagnostic device 37, respectively, during one image acquisition (step S751).

[0135] The control unit 21 detects the tip marker 44 and the sensor marker from the medical image (step S752). For example, when a set of transverse slice images 485 is generated using a pullback operation by the MDU 33, the control unit 21 determines the position and dimensions of the scan area 591 based on the positions of the sensor markers detected at both ends of the pullback operation. The control unit 21 determines the position of the transverse slice position marker 551 based on the scan area 591 (step S753).

[0136] It is preferable that the control unit 21 traces and displays the position corresponding to the scan area 591 on a medical image captured thereafter in real time.

[0137] The control unit 21 inputs the cross-sectional image 485 acquired in step S751 into the first model 61, and acquires an object arrangement image 482 that associates the types of multiple objects included in the cross-sectional image 485 with the range of each object, and the probability that the object determination for each pixel is correct (step S754).

[0138] The control unit 21 generates a vertical object arrangement image based on one set of object arrangement images 482 (step S755). The control unit 21 records the generated vertical object arrangement image in the auxiliary storage device 23 so that the vertical object arrangement image based on the specified cross section can be quickly displayed in response to the user's operation of the vertical section position marker 552.

[0139] The control unit 21 displays the screen described with reference to Fig. 16 on the display device 31 (step S756). After that, the control unit 21 ends the process.

[0140] According to this embodiment, it is possible to provide a catheter system 10 that superimposes and displays a tomographic image captured using the diagnostic imaging catheter 40 or the position of an object arrangement image 482 generated based on a tomographic image on a medical image captured by an image diagnostic device 37. A user can easily change the position of a transverse slice image 485 to be displayed by operating the transverse slice position marker 551. As described above, it is possible to provide a catheter system 10 that allows a user to easily grasp the positional relationship between a tomographic image and the surrounding organs.

[0141] The diagnostic imaging device 37 is not limited to an X-ray angiography device. For example, an ultrasonic diagnostic device combined with an external probe or a TEE (Transesophageal Echocardiography) probe can capture a tomographic image different from that of the diagnostic imaging catheter 40 in real time.

[0142] When the diagnostic imaging catheter 40 is equipped with both an ultrasonic sensor 42 and an OCT sensor 42, a transverse cross-sectional image 485 by ultrasonic waves and a transverse cross-sectional image 485 by OCT can be captured at approximately the same cross section.

[0143] The control unit 21 may superimpose and display the object arrangement image 482 obtained from the transverse cross-sectional image 485 by OCT, which has excellent resolution, on the transverse cross-sectional image 485 by ultrasound, which has a deeper penetration depth than OCT. In addition, the control unit 21 may appropriately combine and display the transverse cross-sectional image 485 by OCT and the object arrangement image 482, or the transverse cross-sectional image 485 by ultrasound and the object arrangement image 482. It is possible to provide a catheter system 10 that displays information that makes the most of the advantages of both.

[0144] The medical image is not limited to a medical image captured in real time. The control unit 21 may superimpose the scan area 591 on a medical image captured by any image diagnostic device, such as a CT, MRI, PET, X-ray angiography device, or ultrasonic diagnostic device, and recorded in an electronic medical record, etc. The control unit 21 determines the position of the scan area 591 based on the branching of blood vessels, the position of the heart, etc. included in each image.

[0145] The processing of this embodiment may be executed on the side of the diagnostic imaging device 37 and displayed on a display device connected to the diagnostic imaging device 37 .

[0146] [Embodiment 6] This embodiment relates to a program for generating a first model 61 and a second model 62. Explanation of parts common to the fourth embodiment will be omitted.

[0147] FIG. 18 is an explanatory diagram illustrating the record layout of a training data DB (database). The training data DB is a database in which inputs and correct labels are recorded in association with each other, and is used for training a model by machine learning. The training data DB has a tomographic image data field, a color-coded data field, and a findings field. The findings field has fields corresponding to findings output by the second model 62, such as a necessity for treatment field, a blood flow stasis field, and a bifurcation field.

[0148] The tomographic image data field records a set of transverse images 485 from which longitudinal images can be generated. The color-coded data field records a set of images in which an expert has colored the transverse images 485 with different colors or background patterns for each object. That is, the color-coded data field records the objects to which each pixel constituting the transverse image 485 corresponds.

[0149] The necessity of treatment field displays the result of a specialist's judgment on the necessity of treatment based on the cross-sectional image 485 recorded in the tomographic image data field. Similarly, the blood flow stagnation field records the presence or absence of blood flow stagnation, and the branching field records the presence or absence of branching.

[0150] The training data DB records a large number of combinations of sets of transverse cross-sectional images 485 taken using the diagnostic imaging catheter 40, sets of colored images judged by a specialist doctor, etc., and the necessity of treatment. In the following explanation, an example will be given in which the intervals between the transverse cross-sectional images 485 constituting a set are constant.

[0151] When generating the first model 61, the set of cross-sectional images 485 recorded in the tomographic image data field is used as input data, and the set of images recorded in the color-coded data field is used as correct answer data. When generating the second model 62, the set of cross-sectional images 485 recorded in the tomographic image data field is used as input data, and the data recorded in each of the findings fields is used as correct answer labels.

[0152] The training data DB may be created, for example, separately into a database for generating the first model 61 and a database for generating the second model 62.

[0153] 19 is a flowchart for explaining the flow of processing of the program according to the embodiment 5. A case where machine learning of the first model 61 is performed using the information processing device 20 will be explained as an example.

[0154] 19 may be executed by hardware separate from the information processing device 20, and the first model 61 for which machine learning has been completed may be copied to the auxiliary storage device 23 via a network. The first model 61 trained by one piece of hardware may be used by multiple information processing devices 20.

[0155] Prior to the execution of the program in FIG. 19, an unlearned model, such as a U-Net structure that realizes semantic segmentation, is prepared. As described above, the U-Net structure is composed of a multi-layered encoder layer and a multi-layered decoder layer connected thereto. The program in FIG. 19 adjusts each parameter of the prepared model and performs machine learning.

[0156] The control unit 21 acquires training records to be used for one epoch of training from the training data DB (step S761). The number of training records to be used for one epoch of training is a so-called hyperparameter, and is appropriately determined.

[0157] The control unit 21 generates an input image from the input data included in each acquired training record (step S762). Specifically, the control unit 21 generates a single image by combining the transverse images 485 included in the tomographic image field in the scanning order. The combined transverse images may be recorded in the tomographic image data field.

[0158] The control unit 21 generates a correct answer image from the color-coded data included in each acquired training record (step S763). Specifically, the control unit 21 generates a single image by combining the color-coded images included in the color-coded data fields in the scanning order. Note that the combined color-coded images may be recorded in the color-coded data field.

[0159] The control unit 21 adjusts the parameters of the model so that, when an input image is input to the input layer of the model, a correct image label is output from the output layer (step S764).

[0160] The control unit 21 determines whether to end the process (step S765). For example, the control unit 21 determines to end the process when learning for a predetermined number of epochs is completed. The control unit 21 may obtain test data from the training data DB, input the test data to the model under machine learning, and determine to end the process when an output of a predetermined accuracy is obtained.

[0161] If it is determined that the process should not be terminated (NO in step S765), the control unit 21 returns to step S761. If it is determined that the process should be terminated (YES in step S765), the control unit 21 records the parameters of the trained model in the auxiliary storage device 23 (step S767). Thereafter, the control unit 21 terminates the process. Through the above process, the trained model is generated.

[0162] Fig. 20 is a flowchart for explaining the flow of processing of the program according to the embodiment 6. As in Fig. 19, the case where machine learning of the second model 62 is performed using the information processing device 20 will be explained as an example.

[0163] Prior to the execution of the program of FIG. 20, an untrained model such as a CNN having a neural network 629 including a convolution layer, a pooling layer, and a fully connected layer, and a softmax layer 625 is prepared. The untrained model is not limited to a CNN. Any type of model such as a decision tree or a random forest can be used. The program of FIG. 20 adjusts each parameter of the prepared model to perform machine learning.

[0164] The control unit 21 acquires training records to be used for training one epoch from the training data DB (step S771). The control unit 21 generates an input image from input data included in each acquired training record (step S772).

[0165] The control unit 21 adjusts the parameters of the model so that, when an input data vector is input to the input layer of the model, the correct label recorded in the finding field is output from the output layer (step S773).

[0166] The control unit 21 determines whether or not to end the process (step S774). If it is determined that the process should not be ended (NO in step S774), the control unit 21 returns to step S771. If it is determined that the process should be ended (YES in step S774), the control unit 21 records the parameters of the trained model in the auxiliary storage device 23 (step S775). Thereafter, the control unit 21 ends the process. Through the above process, a trained model is generated.

[0167] According to this embodiment, the first learning model 651 and the second learning model 652 can be generated by machine learning.

[0168] [Embodiment 7] This embodiment relates to a catheter system 10 that allows a user to correct findings output by a learning model 65. Descriptions of parts common to the first embodiment will be omitted.

[0169] 21 is an explanatory diagram for explaining the record layout of the first correction DB. The first correction DB is a database that records correction information that associates the object arrangement image 482 output by the catheter system 10 with corrections made by the user.

[0170] The first correction DB has a tomographic image data field, an output data field, and a correction data field. A set of transverse images 485 from which a longitudinal image can be generated is recorded in the tomographic image data field. An object arrangement image 482 output by the control unit 21 to the display device 31 is recorded in the output data field. An object arrangement image 482 corrected by the user is recorded in the correction data field. The first correction DB has one record for each correction made by the user to the set of transverse images 485.

[0171] Fig. 22 is an example of a screen displayed by the catheter system 10 of the seventh embodiment. Fig. 22 is a screen displayed by the control unit 21 on the display device 31 when the user instructs correction of the object arrangement image 482 while the screen described using Fig. 6 is being displayed, for example.

[0172] 22 includes a horizontal object arrangement image field 515, a candidate label field 571, a correct label field 572, and a shape correction button 574. The candidate label field 571 displays candidates for labels indicating the type of object. The correct label field 572 displays a label that the user judges to be correct.

[0173] The user designates an object whose type is to be changed by using a cursor 575. After that, the user inputs the correct label by dragging and dropping the label displayed in the candidate label column 571 into the correct label column 572.

[0174] The control unit 21 replaces the label displayed in the correct label column 572 with the label specified by the user. The control unit 21 repaints the object specified by the cursor 575 with a color or background pattern corresponding to the label selected by the user.

[0175] The control unit 21 may accept input by voice input. For example, when the user utters "change calcification to hematoma," the control unit 21 repaints an object displayed in the color or background pattern of "calcification" to the color or background pattern of "hematoma."

[0176] When the user selects the shape correction button 574, the control unit 21 accepts correction of the painted shape of the object arrangement image 482 through a paint software-like user interface. Paint software has been used for a long time, so a detailed description thereof will be omitted.

[0177] In this way, the user can appropriately modify the object arrangement image 482. The control unit 21 records the object arrangement image 482 modified by the user in the first modification DB together with a set of the object arrangement image 482 and the transverse cross-sectional image 485 before modification.

[0178] 23 is an explanatory diagram for explaining the record layout of the second revision DB. The second revision DB is a database that records revision information that associates findings output by the catheter system 10 with revisions made by the user. The revision DB has one record for each revision made by the user to the set of transverse images 485.

[0179] The correction DB has a tomographic image data field, an output field, and a correction field. In the tomographic image data field, a set of transverse images 485 from which longitudinal images can be generated is recorded. In the output field, findings output by the control unit 21 to the display device 31 are recorded. In the correction field, findings corrected by the user are recorded.

[0180] Fig. 24 is an example of a screen displayed by the catheter system 10 of the seventh embodiment. Fig. 24 is a screen displayed by the control unit 21 on the display device 31 when the user instructs correction of the findings field 53 while the screen described using Fig. 14 is being displayed, for example.

[0181] 24 includes a cross-sectional image field 51, a candidate label field 571, a correct label field 572, and a free description field 573. The candidate label field 571 displays candidates for labels indicating qualitative findings. The correct label field 572 displays a label that the user judges to be correct.

[0182] The user can input the correct finding by dragging and dropping the label between the candidate label column 571 and the correct label column 572. The control unit 21 may also accept input by voice input. For example, when the user utters "Correct, hematoma," the control unit 21 moves the label "hematoma" in the first line of the candidate label column 571 to the correct label column 572.

[0183] If a label that represents an appropriate finding cannot be found in the candidate label field 571, the user can input any finding using the free description field 573. Although not shown in the figure, the user can also appropriately change quantitative findings in the same way.

[0184] The control unit 21 records the correction content input by the user using the screen of FIG. 24 in the correction DB described with reference to FIG.

[0185] The content corrected by the user using the screens of Figures 22 and 24 may be attached to an electronic medical record that records the results of medical treatment of the patient. The data recorded in the first correction DB and the second correction DB are used for re-learning the first model 61 and the second model 62, and for correcting hyperparameters used during machine learning by a machine learning engineer.

[0186] [Embodiment 8] 25 is a functional block diagram of a catheter system 10 according to the eighth embodiment. The catheter system 10 includes an acquisition unit 86 and an output unit 87. The acquisition unit 86 acquires a tomographic image generated using a diagnostic imaging catheter inserted into a hollow organ. When the acquisition unit 86 acquires the tomographic image, the output unit 87 inputs the acquired tomographic image to a model 65 that outputs the types of multiple objects included in the tomographic image in association with the ranges of each object, and outputs the types and ranges of the objects output from the model 65.

[0187] [Embodiment 9] Fig. 26 is an explanatory diagram for explaining the configuration of a catheter system according to a ninth embodiment. This embodiment relates to a form in which an information processing device 20 according to this embodiment is realized by combining and operating a general-purpose computer 90 and a program 97. Fig. 26 is an explanatory diagram showing the configuration of the information processing device 20 according to the ninth embodiment. Explanations of parts common to the second embodiment will be omitted.

[0188] The catheter system 10 of this embodiment includes a computer 90. The computer 90 includes a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, a smartphone, or a server computer.

[0189] The program 97 is recorded in a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read out the program 97 stored in a semiconductor memory 98 such as a flash memory implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown), and store it in the auxiliary storage device 23.

[0190] The program 97 is installed as a control program for the computer 90, and is loaded into and executed by the main storage device 22. This causes the computer 90 to function as the information processing device 20 described above.

[0191] The technical features (constitutive elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0192] 10 Catheter System 20 Information processing device 21 Control section 22 Main storage 23 Auxiliary storage device 24 Communications Department 25 Display section 26 Input section 27 Catheter Control Device 271 Catheter control section 29 Reading unit 31 Display device 32 Input Devices 33 MDU 37 Diagnostic Imaging Devices 40 Diagnostic Imaging Catheter 41 Probe section 42 Sensors 43 Shaft 44 Tip Marker 45 Connector part 482 Object Arrangement 485 Cross-sectional image (tomogram) 51 Cross section image column 515 Horizontal Object Arrangement Image Column 52 Longitudinal Section 525 Vertical Object Arrangement Image Column 53 Findings column 538 Value Labels 551 Cross-Section Marker 552 Longitudinal fault position marker 561 Grounds Marker 571 Candidate label field 572 Correct answer label field 573 Free text section 574 Shape modification button 575 Cursor 591 Scan Area 61 1st Model 62 2nd Model 625 Softmax Layer 629 Neural Networks 65 Learning Model (Model) 651 First Learning Model 652 Second Learning Model 86 Acquisition Department 87 Output section 90 Computer 96 Portable recording media 97 Program 98 Semiconductor Memory

Claims

1. A tomographic image is obtained using an imaging catheter inserted into the tubular organ; inputting the acquired tomographic image into a first model which, when a tomographic image is input, outputs the types of a plurality of objects included in the tomographic image in association with the ranges of each of the objects, and acquires the types and ranges of the objects output from the first model; the tomographic image is a transverse image showing a cross section of the hollow organ and the periphery of the hollow organ cut in a direction intersecting with a longitudinal direction of the hollow organ, Mapping and storing the type and range of the object for each of the plurality of cross-sectional images; The type and range of the object are mapped onto a longitudinal section image, which is generated based on the multiple transverse section images and shows a cross section of the hollow organ and its surroundings cut in a direction parallel to the longitudinal direction of the hollow organ, and output. A program that causes a computer to carry out processing.

2. The type and range of the object are displayed in a different manner for each type of the object. The program according to claim 1.

3. outputting the longitudinal tomographic image and a selected tomographic image selected from the plurality of transverse tomographic images at the same time; A marker indicating the position of the selected tomographic image is displayed on the longitudinal tomographic image. The program according to claim 1 or 2.

4. A marker indicating the position of the acquired tomographic image is superimposed on a medical image or a schema different from the acquired tomographic image. The program according to any one of claims 1 to 3.

5. Calculating a size, area or volume of the object based on the extent of the object; Output calculated dimensions, areas or volumes The program according to any one of claims 1 to 4.

6. calculating an angle of a range in which the hollow organ is in a state corresponding to one object selected from the objects, based on the range of the one object selected from the objects; Output the calculated angle The program according to any one of claims 1 to 4.

7. A tomographic image is obtained using an imaging catheter inserted into the tubular organ; inputting the acquired tomographic image into a first model which, when a tomographic image is input, outputs the types of a plurality of objects included in the tomographic image in association with the ranges of each of the objects, and acquires the types and ranges of the objects output from the first model; calculating an angle of a range in which the hollow organ is in a state corresponding to one object selected from the objects, based on the range of the one object selected from the objects; Output the calculated angle A program that causes a computer to carry out processing.

8. Outputting the type and range of the object output from the first model together with grounds information regarding the type and range of the object. The program according to any one of claims 1 to 7.

9. The first model outputs a probability that each pixel constituting the input tomographic image corresponds to one of the objects; The type and range of the object are displayed in a hue determined for each type of object, and the difference in the probability of the object being included is displayed by the difference in brightness or saturation of each pixel. The program according to any one of claims 1 to 8.

10. Accepts modifications to the type or scope of the object The program according to any one of claims 1 to 9.

11. Recording correction information that associates the acquired tomographic image with a correction regarding the type or range of the object. The program according to claim 10.

12. Re-learning the first model based on the correction information. The program according to claim 11.

13. The acquired tomographic image is input to a second model that outputs information on the state of the hollow organ or the surroundings of the hollow organ when the tomographic image is input, and the information output from the second model is further output. The program according to any one of claims 1 to 10.

14. Accepting a modification to the state of the hollow organ or the surroundings of the hollow organ The program according to claim 13.

15. and recording correction information that associates the acquired tomographic image with a correction to the state of the hollow organ or the surroundings of the hollow organ. The program according to claim 14.

16. Re-learning the second model based on the correction information. The program according to claim 15.

17. A tomographic image is obtained using an imaging catheter inserted into the tubular organ; inputting the acquired tomographic image into a first model which, when a tomographic image is input, outputs the types of a plurality of objects included in the tomographic image in association with the ranges of each of the objects, and acquiring the types and ranges of the objects output from the first model; the tomographic image is a transverse image showing a cross section of the hollow organ and the periphery of the hollow organ cut in a direction intersecting with a longitudinal direction of the hollow organ, Mapping and storing the type and range of the object for each of the plurality of cross-sectional images; The type and range of the object are mapped onto a longitudinal section image, which is generated based on the multiple transverse section images and shows a cross section of the hollow organ and its surroundings cut in a direction parallel to the longitudinal direction of the hollow organ, and output. An information processing method in which processing is performed by a computer.

18. A tomographic image is obtained using an imaging catheter inserted into the tubular organ; inputting the acquired tomographic image into a first model which, when a tomographic image is input, outputs the types of a plurality of objects included in the tomographic image in association with the ranges of each of the objects, and acquiring the types and ranges of the objects output from the first model; calculating an angle of a range in which the hollow organ is in a state corresponding to one object selected from the objects, based on the range of the one object selected from the objects; Output the calculated angle An information processing method in which processing is performed by a computer.

19. an acquisition unit that acquires a tomographic image generated using a diagnostic imaging catheter inserted into a hollow organ; a second acquisition unit that, when the tomographic image acquired by the acquisition unit is input, inputs the acquired tomographic image into a model that outputs a correspondence between the types of a plurality of objects included in the tomographic image and the ranges of each of the objects, and acquires the types and ranges of the objects output from the model; the tomographic image is a transverse image showing a cross section of the hollow organ and the periphery of the hollow organ cut in a direction intersecting with a longitudinal direction of the hollow organ, A storage unit that maps and stores the type and range of the object for each of the plurality of cross-sectional images acquired by the second acquisition unit; and an output unit that maps the type and range of the object onto a longitudinal section image that shows a cross section of the hollow organ and its periphery cut in a direction parallel to the longitudinal direction of the hollow organ, the longitudinal section image being generated based on the plurality of transverse section images, and outputs the mapped type and range of the object. Information processing system.

20. an acquisition unit that acquires a tomographic image generated using a diagnostic imaging catheter inserted into a hollow organ; a second acquisition unit that inputs the acquired tomographic image into a model that outputs a model in which the types of a plurality of objects included in the tomographic image are associated with the ranges of each of the objects when the tomographic image acquired by the acquisition unit is input, and acquires the types and ranges of the objects output from the model; a calculation unit that calculates an angle of a range in which the tubular organ is in a state corresponding to one object selected from the objects, based on the range of the one object; An output section that outputs the calculated angle; An information processing system comprising:

Citation Information

Patent Citations

  • Radial image processing device

    JP1991102477A

  • Device and method for imaging lumen

    JP2007007410A

  • System and method for image segmentation using a multi-stage classifier

    JP2008535566A

  • Discrimination of nature of tissue using sparse coding method

    JP2010075616A

  • Device, method and program for determination of recognition result

    JP2010220902A