Program products, information processing methods and information processing systems

Abstract

The program provides a procedure that enables easy use of the catheter system. The program enables a computer to perform the following processing: acquire a tomographic image (485) generated using an image diagnostic catheter inserted into a luminal organ, input the acquired tomographic image (485) into a first model (61), and output the types and ranges of objects output from the first model (61), which, when the tomographic image (485) is input, outputs the types of multiple objects contained in the tomographic image (485) in association with the ranges of each object.

Description

Technical Field

[0001] This invention relates to programs, information processing methods, methods for generating learning models, methods for relearning learning models, and information processing systems. Background Technology

[0002] A catheter system that uses a diagnostic catheter to take tomographic images of luminal organs such as blood vessels (Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2017 / 164071 Summary of the Invention

[0006] Proficiency is required in the diagnostic use of tomographic images obtained through catheter-based imaging systems. Therefore, extensive training is necessary to effectively utilize these catheter systems.

[0007] One aspect is aimed at providing procedures that make it easy to use catheter systems.

[0008] The program causes the computer to perform the following processing: acquire a tomographic image generated using an image diagnostic catheter inserted into a luminal organ, input the acquired tomographic image into a first model, and output the types and extents of objects output from the first model, which, in the case of an input tomographic image, outputs the types of multiple objects contained in the tomographic image in association with the extents of each of the objects.

[0009] Invention Effects

[0010] In one aspect, it is possible to provide procedures that make the catheter system easy to use. Attached Figure Description

[0011] Figure 1 This is an explanatory diagram illustrating the general outline of the catheter system.

[0012] Figure 2 This is an explanatory diagram illustrating the outline of a catheter used for image-based diagnostics.

[0013] Figure 3 This is an explanatory diagram illustrating the structure of a catheter system.

[0014] Figure 4 This is an explanatory diagram illustrating the first model.

[0015] Figure 5 This is an example of a screen displayed by a catheter system.

[0016] Figure 6 This is an example of a screen displayed by a catheter system.

[0017] Figure 7 This is an example of a screen displayed by a catheter system.

[0018] Figure 8 It is a flowchart illustrating the process of the program.

[0019] Figure 9 This is an explanatory diagram illustrating the configuration of the conduit system in Embodiment 2.

[0020] Figure 10 This is an example of a screen displayed by the conduit system in Embodiment 3.

[0021] Figure 11 This is an example of a screen displayed by the conduit system in Embodiment 3.

[0022] Figure 12 This is a flowchart illustrating the processing flow of the procedure in Implementation Method 3.

[0023] Figure 13 This is an explanatory diagram illustrating the structure of the second model.

[0024] Figure 14 This is an example of a screen displayed by the conduit system in Embodiment 4.

[0025] Figure 15 This is a flowchart illustrating the processing flow of the program in Implementation Method 4.

[0026] Figure 16 This is an example of a screen displayed by the conduit system in Embodiment 5.

[0027] Figure 17 This is a flowchart illustrating the processing flow of the procedure in Implementation Method 5.

[0028] Figure 18 This is an illustration of the record layout of the training data database (DB).

[0029] Figure 19 This is a flowchart illustrating the processing flow of the program in Implementation Method 6.

[0030] Figure 20 This is a flowchart illustrating the processing flow of the program in Implementation Method 6.

[0031] Figure 21 This is an explanatory diagram illustrating the record layout of the first modified DB.

[0032] Figure 22 This is an example of a screen displayed by the conduit system in Embodiment 7.

[0033] Figure 23This is an explanatory diagram illustrating the record layout of the second revised DB.

[0034] Figure 24 This is an example of a screen displayed by the conduit system in Embodiment 7.

[0035] Figure 25 This is a functional block diagram of the conduit system in Implementation Method 8.

[0036] Figure 26 This is an explanatory diagram illustrating the configuration of the conduit system in Embodiment 9. Detailed Implementation

[0037] [Implementation Method 1]

[0038] Figure 1 This is an explanatory diagram illustrating the general structure of the catheter system 10. The catheter system 10 includes an image diagnostic catheter 40, an MDU (Motor Driving Unit) 33, and an information processing device 20. The image diagnostic catheter 40 is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 can be, for example, a keyboard, mouse, trackball, or microphone. The display device 31 and the input device 32 can also be integrally stacked to form a touch panel. The input device 32 and the information processing device 20 can also be integrally formed.

[0039] Figure 2 This is an explanatory diagram illustrating the general outline of the image diagnostic catheter 40. The image diagnostic catheter 40 has a probe portion 41 and a connector portion 45 disposed at the end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the image diagnostic catheter 40 furthest from the connector portion 45 is referred to as the front end side.

[0040] A shaft 43 is inserted inside the probe section 41. A sensor 42 is connected to the front end of the shaft 43. A ring-shaped front end mark 44 is fixed near the front end of the probe section 41.

[0041] Through the function of MDU33, sensor 42 and shaft 43 can rotate and retract simultaneously inside probe section 41. By pulling sensor 42 back while simultaneously rotating it towards MDU33 at a constant speed, multiple transverse tomographic images 485 (see reference) are continuously captured at predetermined intervals, approximately perpendicular to probe section 41, centered on probe section 41. Figure 4 ).

[0042] Sensor 42 is an ultrasonic transducer for transmitting and receiving ultrasound waves, or a transceiver unit for OCT (Optical Coherence Tomography) for receiving near-infrared light irradiation and reflected light. The luminal organs into which the catheter 40 is inserted for image diagnostics are, for example, blood vessels, pancreatic ducts, bile ducts, or bronchi.

[0043] Figure 2 This describes an example of an imaging diagnostic catheter 40 used for intravascular ultrasound (IVUS) when taking ultrasound tomographic images from the inside of a blood vessel. The following description uses the case where the imaging diagnostic catheter 40 is an IVUS catheter as an example.

[0044] Furthermore, the image diagnostic catheter 40 is not limited to a mechanical scanning method that involves mechanical rotation and retraction. It can also be an electronic radial scanning type image diagnostic catheter 40 that uses a sensor 42 with multiple ultrasonic transducers arranged in a ring.

[0045] The diagnostic catheter 40 may also have a so-called linear scanning sensor 42 in which multiple ultrasonic transducers are arranged in a row along the length direction. The diagnostic catheter 40 may also have a so-called two-dimensional array sensor 42 in which multiple ultrasonic transducers are arranged in a matrix.

[0046] The diagnostic imaging catheter 40 can capture tomographic images of not only the walls of blood vessels and other luminal walls, but also reflectors inside the lumen of luminal organs, such as red blood cells, and organs outside the luminal organs, such as the pericardium and heart.

[0047] Figure 3 This is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 includes an information processing unit 20, an MDU 33, and an image diagnostic catheter 40. The information processing unit 20 includes a control unit 21, a main storage unit 22, an auxiliary storage unit 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

[0048] The control unit 21 is an operational control device that executes the program of this embodiment. One or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), or multi-core CPUs are used in the control unit 21. The control unit 21 is connected to the various hardware components constituting the information processing device 20 via a bus.

[0049] The main storage device 22 is a storage device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. Information needed during processing by the control unit 21 and programs being executed by the control unit 21 are temporarily stored in the main storage device 22.

[0050] The auxiliary storage device 23 is a storage device such as SRAM, flash memory, hard disk, or magnetic tape. The auxiliary storage device 23 stores the program executed by the control unit 21, the first model 61, and various data required for program execution. The communication unit 24 is the interface for communication between the information processing device 20 and the network.

[0051] Display unit 25 is the interface connecting display device 31 and the bus. Input unit 26 is the interface connecting input device 32 and the bus. The catheter control unit 271 controls MDU 33, sensor 42, and generates transverse and longitudinal tomographic images 485 based on signals received from sensor 42. The function and configuration of catheter control unit 271 are the same as those of conventional ultrasound diagnostic devices, therefore, description is omitted. Furthermore, control unit 21 can also perform the functions of catheter control unit 271.

[0052] The information processing device 20 is connected to various image diagnostic devices 37, such as X-ray angiography devices, X-ray CT (Computed Tomography) devices, MRI (Magnetic Resonance Imaging) devices, PET (Positron Emission Tomography) devices, or ultrasound diagnostic devices, via a HIS (Hospital Information System).

[0053] The information processing device 20 in this embodiment is a dedicated ultrasound diagnostic device, or a personal computer, tablet computer, or smartphone with ultrasound diagnostic device functionality.

[0054] Figure 4 This is an explanatory diagram illustrating Model 61. Model 61 is a model that receives a tomographic image 485 and outputs an object configuration image 482 that maps the types of multiple objects contained in the tomographic image 485 to the range of each object. Model 61 is generated through machine learning.

[0055] exist Figure 4In the object configuration shown in image 482, the vertical shading lines represent the cross-section of the "diagnostic catheter 40", the horizontal shading lines represent the "wall of the lumen organ", the shading lines sloping downward to the right represent the "inner side of the lumen organ", the shading lines sloping downward to the left represent the "guide wire", and the fine grid-like shading lines represent "calcification".

[0056] "Guidewire" includes the guidewire itself, the multiple echoes generated by the guidewire, and the acoustic shadow generated by the guidewire. Similarly, "calcification" includes the calcified portion itself, the multiple echoes generated by the calcified portion, and the acoustic shadow generated by the calcified portion.

[0057] Figure 4 The shading lines in the diagram schematically represent the use of different colors to distinguish each shell. Distinguishing objects by color is one example of a method of differentiating objects. Alternatively, any shape, such as the outer edge surrounding each object, can be used to distinguish them from other objects.

[0058] Here, "the cross-section of the diagnostic catheter 40," "the wall of the lumen organ," "the inner side of the lumen organ," "the guidewire," and "calcification" are examples of objects included in the tomographic image 485. For example, "the guidewire itself," "multiple echoes generated by the guidewire," and "acoustic shadows generated by the guidewire" can also be classified as different objects. Similarly, lesions such as "plaques" and "dissociations" generated in the wall of the lumen organ can also be classified as different objects.

[0059] In the following description, a group of multiple transverse images 485 capable of generating longitudinal tomographic images will be referred to as a set of transverse tomographic images 485. Similarly, acquiring a set of transverse tomographic images 485 capable of generating longitudinal tomographic images using the image diagnostic conduit 40 will be referred to as a single image acquisition. The following describes an example of a first model 61 that accepts a set of transverse tomographic images 485 as input and outputs an object configuration image 482 corresponding to each transverse tomographic image 485.

[0060] A set of transverse images 485 is acquired, for example, through a single pull-back operation based on MDU33. A set of transverse images 485 can also be acquired during a user's manual push-pull operation of the image diagnostic catheter 40. Here, the push-pull operation of the image diagnostic catheter 40 includes both pushing and pulling the probe section 41 and pushing and pulling the sensor 42 inside the probe section 41.

[0061] For example, the user performs an operation to pull back the sensor 42 at a roughly constant speed or to push the sensor 42 in. The tomographic images 485 acquired during the period from the start of acquisition by the user using instructions such as voice input to the end of acquisition constitute a set of tomographic images 485.

[0062] Sensors may also be provided to detect the amount by which the user pushes or pulls the sensor 42. Images acquired during the period when the user pulls the sensor 42 back or pushes it in within the entire specified range constitute a set of tomographic images 485.

[0063] When the position of sensor 42 can be detected, the user can push and pull sensor 42 at any speed and orientation. A set of transverse tomographic images 485 arranged sequentially along the length of probe portion 41 constitutes a set of transverse tomographic images 485. Even when the interval between transverse tomographic images 485 is not constant, positional information along the length of probe portion 41 is recorded in association with each transverse tomographic image 485. Furthermore, in the following description, the case where the interval between transverse tomographic images 485 is constant will be used as an example.

[0064] As previously mentioned, the first model 61 can be either a model that accepts a set of transverse images 485 obtained through a pull-back operation based on MDU33, or a model that accepts input of a set of transverse images 485 obtained by manually moving the sensor 42 forward and backward. The first model 61 can also be a model that accepts input of a single transverse image 485. Furthermore, the first model 61 can be a model that accepts input of half or one-third of a transverse image 485 obtained through a single pull-back operation.

[0065] Model 61 is, for example, a semantic segmentation model, comprising an input layer, a neural network, and an output layer. The neural network has, for example, a U-Net structure for implementing semantic segmentation. The U-Net structure consists of multiple encoder layers and multiple decoder layers connected after these encoder layers. Semantic segmentation is used to assign labels representing the category of objects to each pixel constituting the input image.

[0066] The control unit 21 determines the display method for each pixel according to the label, such as... Figure 4 As shown in Figure 482, the object configuration can generate an output image by mapping objects according to their type with different colors or textures.

[0067] Model 61 can also be a Mask R-CNN (Regions with Convolutional Neural Networks) model, or any other model that performs image segmentation based on machine learning algorithms.

[0068] Model 61 can also be a model that performs object detection, such as R-CNN. When using a model that performs object detection without segmentation, the control unit 21 surrounds the detected object with bounding boxes and displays text such as "calcification" near the bounding boxes.

[0069] Furthermore, by using a set of transverse tomographic images 485 as input data, information from adjacent transverse tomographic images 485 is reflected in the object configuration image 482. Therefore, the first model 61, which is less susceptible to noise or other disturbances in individual transverse tomographic images 485, can accurately output the extent of the object.

[0070] Figure 5 This is an example of the screen displayed by the catheter system 10. Figure 5 The displayed image has a transverse section image bar 51, a horizontal object configuration image bar 515, and a vertical object configuration image bar 525. The aforementioned transverse section image 485 is displayed in the transverse section image bar 51. The object configuration image 482 corresponding to the transverse section image 485 displayed in the horizontal object configuration image bar 515 is displayed.

[0071] The longitudinal object configuration image 525 displays an object configuration image 482 corresponding to the longitudinal fault image. The object configuration image 482 corresponding to the longitudinal fault image is formed, similarly to the longitudinal fault image, based on the object configuration image 482 corresponding to a set of transverse fault images 485.

[0072] Specifically, pixels corresponding to the longitudinal section images are extracted from each object configuration image 482, interpolation is performed, and reconstruction is carried out to form an object configuration image corresponding to the longitudinal section image. This process is the same as the method for forming a longitudinal section image from a set of transverse section images 485, so details are omitted.

[0073] Alternatively, three-dimensional semantic segmentation can be performed as follows: three-dimensional data is generated based on a set of transverse tomographic images 485, and each voxel is labeled with a label representing the type of object. From the three-dimensional object configuration images generated based on the three-dimensional semantic segmentation results, an object configuration image 482 corresponding to the longitudinal tomographic images can be generated.

[0074] A transverse fault position indicator 551 is displayed at the edge of the longitudinal image frame 525, indicating the position of the transverse fault image 485 displayed in the transverse image frame 51 and the transverse image frame 515. A longitudinal fault position indicator 552 is displayed near the edge of the transverse image frame 51 and the transverse image frame 515, indicating the position of the longitudinal fault image displayed in the longitudinal image frame 52.

[0075] Users can also change the positions of the transverse section location marker 551 and the longitudinal section location marker 552 by operating the input device 32, thereby appropriately changing the displayed cross section. In addition, the control unit 21 can also accept voice input from the user.

[0076] Users can based on Figure 5 The image confirms the condition of the lesion along its long axis. Furthermore, Figure 5This is one example of screen display, but it is not limited to this. For example, a three-dimensional display of an assembly as a cross-section can also be performed.

[0077] Figure 6 This is an example of the screen displayed by the conduit system 10. When the user instructs the control unit 21 to display the correct probability of each pixel representing the object of "calcification" showing "calcification," the control unit 21 displays... Figure 6 The image shown. Figure 6 The upper part shows an enlarged view of the portion representing "calcification".

[0078] Furthermore, the probability of correctly determining each pixel varies depending on the usage. Figure 4 The output of the first model 61, as described above, is shown. The control unit 21 distinguishes the colors of pixels based on the probability that each pixel is classified as a "calcified" object.

[0079] For example, by keeping the hue representing "calcification" common and applying a gradient to the brightness or chroma, the control unit 21 can represent the probability that the determination of "calcification" is correct. The control unit 21 can also display the entire horizontal object configuration pixel block 515 by color differentiation based on the determination probability, where the determination probability represents the correctness of the object determination for each pixel.

[0080] Users can based on Figure 6 The degree to which the image recognition of "calcification" can be relied upon is unclear. The expectation is that, assuming the output of "calcification" has a relatively low probability, the user, from a professional standpoint, will fully observe the tomographic image 485 displayed in the tomographic image column 51.

[0081] Figure 7 This is an example of the screen displayed by the catheter system 10. When the user gives instructions in a manner that shows the basis for indicating "calcification," the control unit 21 displays... Figure 7 The image shown. A cross-sectional image 485 is displayed in the cross-sectional image column 51, which is formed by overlapping a basis identifier 561, representing the basis for determining "calcification". The basis identifier 561 is an example of basis information related to the basis of the object displayed in the cross-sectional image column 515.

[0082] The control unit 21 extracts the basis regions using model visualization methods such as Grad-CAM (Gradient-weighted Class Activation Mapping) or Grad-CAM++. The basis regions are areas in the multiple cross-sectional images 485 input to the learning model 65 that strongly influence the output of pixels judged as "calcified". Regarding the basis identifier 561, areas with higher influence on the output are displayed using finer shaded lines.

[0083] Users can based on Figure 7 The screen shown is used to judge, based on a professional standpoint, whether the basis for the judgment made by the control unit 21 is appropriate.

[0084] Figure 8 This is a flowchart illustrating the processing flow of the procedure. Control unit 21 acquires a set of transverse tomographic images 485 from conduit control unit 271 (step S701). Control unit 21 inputs the acquired transverse tomographic images 485 into the first model 61, and acquires an object configuration image 482 obtained by associating the types of multiple objects contained in the transverse tomographic image 485 with the range of each object, and the probability of correctly identifying the object for each pixel (step S702).

[0085] The control unit 21 generates a longitudinal section image of the object configuration image 482 based on a set of object configuration images 482 (step S703). In the following description, the longitudinal section image of the object configuration image 482 will be recorded as a longitudinal object configuration image. The control unit 21 records the generated longitudinal object configuration image in the auxiliary storage device 23 in a manner that can quickly display the longitudinal object configuration image based on the specified section in accordance with the user's operation on the longitudinal section position marker 552.

[0086] Control unit 21 will use Figure 5 The explained screen is displayed on the display device 31 (step S704). The control unit 21 determines whether the user has received an instruction on the probability of displaying an object (step S705). The user can input the instruction for displaying the probability by, for example, double-clicking an object in the horizontal object arrangement bar 515. The control unit 21 can also accept voice input from the user.

[0087] If the instruction to display the probability is deemed accepted (Yes in step S705), the control unit 21 displays the probability of correct object determination for each pixel based on the probability obtained in step S702. Figure 6 The illustrated image (step S706).

[0088] If the instruction to display the probability is determined not to have been accepted (No in step S705), or after step S706, the control unit 21 determines whether the instruction to display the basis has been accepted from the user (step S707). The user can input the instruction for display basis by, for example, sliding objects in the horizontal object configuration bar 515. The control unit 21 can also accept voice input from the user.

[0089] If the instruction to display the basis is deemed accepted (Yes in step S707), the control unit 21 acquires the item for displaying the basis (step S708). The control unit 21 uses a model visualization method such as Grad-CAM or Grad-CAM++ to extract the basis region associated with the item acquired in step S708 (step S709).

[0090] Control unit 21 uses Figure 7 The screen that has been explained displays a cross-sectional image 485 overlaid with the reference identifier 561 in the cross-sectional image column 51 (step S710). If it is determined that the instruction to display the reference has not been accepted (no in step S707), or after step S710 is completed, the control unit 21 ends the processing.

[0091] According to this embodiment, a catheter system 10 can be provided that can be easily used even by users who are not very skilled in the diagnosis of tomographic images.

[0092] According to this embodiment, a catheter system 10 that utilizes an object configuration image 482 to assist the user in image diagnosis can be provided. The user can quickly grasp where and what is displayed on the tomographic image, thus enabling them to focus their attention on observations of parts important for diagnosis and treatment.

[0093] According to this embodiment, a conduit system 10 can be provided that displays the probability of object determination for each pixel. A conduit system 10 can be provided that allows a user to ascertain the degree of reliability of the determination based on the first model 61.

[0094] According to this embodiment, a catheter system 10 that displays the basis for determining an object can be provided. A catheter system 10 that allows a user to confirm the basis for determining the object based on the first model 61 can also be provided.

[0095] [Implementation Method 2]

[0096] This embodiment relates to a catheter system 10 consisting of a catheter control device 27 and an information processing device 20. Descriptions of parts common to Embodiment 1 are omitted.

[0097] Figure 9This is an explanatory diagram illustrating the configuration of the catheter system 10 according to Embodiment 2. The catheter system 10 of this embodiment includes an information processing device 20, a catheter control device 27, an MDU 33, and an image diagnostic catheter 40. The information processing device 20 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, and a bus.

[0098] The catheter control device 27 is an ultrasound diagnostic device for IVUS that controls the MDU 33, controls the sensor 42, and generates transverse and longitudinal tomographic images 485 based on signals received from the sensor 42. The function and configuration of the catheter control device 27 are the same as those of conventional ultrasound diagnostic devices, therefore, a description is omitted.

[0099] The conduit control device 27 and the information processing device 20 can be directly connected via cable or wireless communication, or they can be connected via a network.

[0100] The information processing device 20 in this embodiment can be a general-purpose personal computer, tablet computer, smartphone, mainframe computer, virtual machine operating on a mainframe computer, cloud computing system, or quantum computer. The information processing device 20 can also be multiple personal computers performing distributed processing, etc.

[0101] [Implementation Method 3]

[0102] This embodiment relates to a catheter system 10 that displays quantitative information such as the length and area of ​​the portion depicted in a tomographic image. Descriptions of portions common to Embodiment 1 are omitted.

[0103] Figures 10 to 12 This is an example of a screen displayed by the conduit system in Embodiment 3. Figure 10 In the image shown, when using Figure 5 The horizontal object configuration image bar 515 and the vertical object configuration image bar 525 of the screen respectively display value labels 538.

[0104] The control unit 21 calculates quantitative information such as length, area, and volume associated with the object specified by the user, based on an object configuration image 482 corresponding to a set of tomographic images 485. In calculating, for example, area, the control unit 21 multiplies the number of pixels constituting the object by the area of ​​each pixel. In calculating volume, the control unit 21 further multiplies the thickness of each slice by the calculated area. Methods for calculating quantitative information from arbitrary graphics are well-known, therefore detailed descriptions are omitted.

[0105] The following explanation will continue with an example where the designated object is the "inner side of a luminal organ," and a stenotic lesion is included in a set of transverse tomographic images 485. The control unit 21 determines the position with the smallest diameter, i.e., the narrowest position, based on the diameter of the "inner side of the luminal organ" calculated from each object configuration image 482. The control unit 21 determines the two ends of the lesion based on a predetermined determination criterion.

[0106] exist Figure 10 In the value label 538 displayed in the lower left of the horizontal object configuration image bar 515, the area, minimum diameter, and maximum diameter of the "inner side of the lumen organ" are displayed respectively, indicated by a shading line sloping downwards to the right.

[0107] The vertical column configuration panel 525 displays a total of four vertical lines, indicating the positions of the "reference vessel," the two "lesion ends," and the "minimum diameter section." The "reference vessel" refers to the part without lesions.

[0108] At the ends of each longitudinal line, a corresponding value label 538 is displayed. The value label 538 displays the minimum and maximum diameter of the "inner side of the lumen organ" at each location, as well as the plaque load. The "lesion length" is displayed between the two longitudinal lines representing the two "lesion ends".

[0109] If, for example, the user clicks near the vertical line indicating the "minimum diameter" location, the transverse section location marker 551 moves to the "minimum diameter" location. The transverse section image 485 of the "minimum diameter" and the transverse object configuration image bar 515 are displayed in the transverse section image bar 51 and the transverse object configuration image bar 515. The user can confirm the status of the "minimum diameter".

[0110] Similarly, when the user clicks near the vertical line indicating the location of the "reference vessel" or "lesion end," the transverse tomographic location marker 551 moves to the corresponding position. The user can easily confirm the state of the part of the point used for judgment.

[0111] Users can perform diagnosis and treatment based on the tissue characteristics and values ​​displayed on the value label 538, as well as the judgment criteria determined by the society, etc. For example, users can determine the need for vasodilator surgery and the surgical procedure based on the vessel diameter, the degree of stenosis, the length of the lesion, or the distribution of calcification. The control unit 21 can also display information related to the associated judgment criteria together with the value label 538.

[0112] Figure 11 This indicates the state after the user moves the transverse section position marker 551 to the right of the longitudinal section image frame 525. The transverse section image frame 51 displays [the image frame]. Figure 10Different transverse section images 485. Object configuration images 482 corresponding to transverse section image columns 515 are displayed in the transverse object configuration image column 51.

[0113] exist Figure 11 The horizontal image 515 displays the cross-section of the diagnostic catheter 40 (represented by vertical shading lines), the inner side of the luminous organ (represented by sloping downwards to the right shading lines), and the calcified object (represented by a grid-like shading line). A tomographic image 485 is displayed in areas where the object is not shown. The user can confirm the condition of the luminous organ surrounding the calcified object.

[0114] The object representing "calcification" is displayed in an arc shape. The control unit 21 calculates the angle of "calcification" with the center of the "vascular wall" as a reference and displays it on the value label 538.

[0115] A specific example illustrating the method for calculating angles is provided. Control unit 21 calculates the outer periphery of the "inner side of the lumen organ," that is, a circle whose shape approximates the inner surface of the lumen organ wall. Control unit 21 extracts vectors extending from the center of the calculated circle to both ends of the object representing "calcification." Control unit 21 calculates the angle between the two extracted vectors. Furthermore, the method for calculating angles is arbitrary and not limited to this.

[0116] For cylindrical objects such as "the wall of a tubular organ" or "the inner side of a tubular organ," control unit 21 calculates the inner diameter, outer diameter, minimum inner diameter, and maximum inner diameter and displays them on value label 538. Figure 11 The angle of the object shown as "calcification" is calculated and displayed on value label 538. Furthermore, the control unit 21 can also... Figure 11 The objects shown as “calcified” exhibit thickness and volume, among other characteristics.

[0117] Users can also specify which item to display in value label 538 each time. Alternatively, the default item to be displayed in value label 538 can be determined based on the type, shape, etc. of the object.

[0118] Figure 12 This is a flowchart illustrating the processing flow of the procedure in Implementation Method 3. When using... Figure 8 In the explained procedure S704, the control unit 21 displays the use of... Figure 5 After the explanatory screen, and with the user indicating the display of value label 538, the process is executed. Figure 12 The program.

[0119] Users can instruct the display of value labels 538 by clicking on objects displayed in the horizontal object configuration bar 515. In addition, the control unit 21 can also accept voice input from the user.

[0120] The control unit 21 acquires the designation of the object of the display value label 538 based on the user's instruction (step S721). The control unit 21 determines the item to be calculated based on the shape and the like of the designated object (step S722).

[0121] The control unit 21 calculates the item determined in step S722 (step S723). The control unit 21 displays the display value label 538 as described using Figure 10 and Figure 11 (step S724). After that, the control unit 21 ends the process.

[0122] [Embodiment 4]

[0123] This embodiment relates to the catheter system 10 using the second model 62, which outputs information related to the state of the luminal organ or the periphery of the luminal organ when a tomographic image is input. For parts common to Embodiment 1, the description is omitted.

[0124] Figure 13 FIG. is an explanatory diagram for explaining the configuration of the second model 62. The second model 62 is a model that receives a set of tomographic images 485 and outputs medical findings (Japanese: 所見) related to the state of the luminal organ or the periphery of the luminal organ, such as the necessity of treatment, the presence or absence of blood flow stagnation, or the presence or absence of branches. In addition, the "necessity of treatment" can be either the necessity of IVR (Interventional Radiology) for treating inside the luminal organ or the necessity of general treatment including medication and diet therapy.

[0125] The medical findings output by the second model 62 are probabilities related to predetermined options such as "present" and "absent" for each of a plurality of items. Examples of the items for which the second model 62 outputs probabilities are shown in Tables 1 to 5. One row in Tables 1 to 5 represents one item. The second model 62 outputs the probability of the option for each item. Examples of the items included in the medical findings output by the second model 62 are shown in Tables 1 to 5.

[0126] Table 1 shows information related to the necessity of treatment.

[0127] [Table 1]

[0128]

[0129] Table 2 shows items related to blood flow information.

[0130] [Table 2]

[0131]

[0132] Table 3 shows items related to qualitative shape information about luminal organs and the area surrounding them.

[0133] [Table 3]

[0134]

[0135] Table 4 shows items related to phenotypic information representing the characteristics of luminal organs and the features surrounding luminal organs.

[0136] [Table 4]

[0137]

[0138] Table 4 shows "in-stent stenosis," indicating the presence or absence of stenosis in a stent placed in a lumen organ, for example, several months to several years ago. In the case of a tomographic image 485 taken immediately after stent placement, it indicates whether or not the placed stent has stenosis. That is, the tomographic image 485 can be a tomographic image of the lumen organ before treatment, a tomographic image of the lumen organ during long-term observation after treatment, or a tomographic image of the lumen organ taken immediately after a series of in-lumen organ treatments.

[0139] Table 5 shows items related to equipment information, which indicates the status of indwelling devices such as stents configured in luminal organs.

[0140] [Table 5]

[0141]

[0142] The items shown in Tables 1 through 5 are illustrative. Model 62 can also output the probability for a subset of the items shown in Tables 1 through 5. Model 62 can also output the probability for items other than those shown in Tables 1 through 5.

[0143] The options for each item shown in Tables 1 to 5 are illustrative. For items displayed as "Yes" or "No" in the tables, options of three or more, such as "Large," "Small," or "No," can also be used.

[0144] In the following description, a second model 62 will be used as an example, which accepts a set of tomographic images 485 obtained through a single image acquisition as input and outputs medical findings related to the state of a luminal organ or the state surrounding that luminal organ. Alternatively, the second model 62 could also be a model that accepts one tomographic image 485 as input and outputs medical findings related to the state of a luminal organ or the state surrounding that luminal organ.

[0145] As mentioned earlier, an image can also be acquired in a single pull-back operation based on MDU33. The second model 62 can also be a model that takes as input a portion of a cross-sectional image 485, such as half or one-third, obtained through a single pull-back operation, and outputs medical findings related to the state of a lumen organ or the state of its surroundings.

[0146] Model 2 62 has an input layer, a neural network 629, multiple softmax layers 625, and an output layer. The neural network 629 is a CNN (Convolutional Neural Network) with, for example, multiple convolutional and pooling layers, as well as fully connected layers. For each row shown in Tables 1 to 5, there is one softmax layer 625.

[0147] The input layer is fed an image that combines a set of transverse images 485 in scanning order into a single image. The output layer is then fed probabilities for each item shown in Tables 1 to 5 using a neural network 629 and a Softmax layer 625.

[0148] For example in Figure 13 In the model, the probability of "no" for "need for treatment" is 95%; the probability of "no" for "blood stasis" is 90%; and the probability of "present" for "branch" is 90%. Furthermore, Model 62 can also be separated for Tables 1 to 5 separately. Model 62 can also be separated according to the output items.

[0149] Alternatively, a selection layer can be set at the end of the Softmax layer 625 to output the option with the highest selection probability.

[0150] It is also possible to input acoustic data, such as data obtained by the duct control unit 271 from the sensor 42, and data from the previous stage of forming the transverse image 485 into the second model 62.

[0151] Figure 14 This is an example of a screen displayed by the conduit system 10 in embodiment 4. Figure 14 The displayed screen includes a tomographic image bar 51, a horizontal object configuration image bar 515, and a medical discovery bar 53. For the tomographic image bar 51 and the horizontal object configuration image bar 515, compared with using... Figure 5 The screen displayed is the same as that of the catheter system 10 in Embodiment 1, which has been described, so the description is omitted.

[0152] Medical findings are displayed in the medical findings column 53. The control unit 21 selects medical findings with a probability higher than a specified threshold from the medical findings output by the second model 62 and displays them in the medical findings column 53.

[0153] The control unit 21 can also select medical findings associated with the transverse image 485 displayed in the transverse image column 51 and display them in the medical findings column 53. Although in Figure 14 The diagram is omitted, but the control unit 21 can also be connected to... Figure 5 Similarly, column 525 is displayed for the vertical object configuration.

[0154] Figure 15 This is a flowchart illustrating the processing flow of the procedure in Implementation Method 4. Up to step S703, the process involves... Figure 8 The procedure for Implementation Method 1, which has been described, is the same, so the description is omitted.

[0155] The control unit 21 inputs the transverse tomographic image 485 obtained in step S701 to the second model 62 and obtains medical findings (step S731). The control unit 21 will use... Figure 14 A screen with explanations is displayed on the display device 31 (step S732). The control unit 21 determines whether the user has accepted the instruction regarding the probability of determining the displayed object (step S705). Subsequent processing and use. Figure 8 The procedure for Implementation Method 1, which has been described, is the same, so the description is omitted.

[0156] According to this embodiment, a catheter system 10 that displays not only the object configuration image 482, but also medical findings can be provided.

[0157] Furthermore, the control unit 21 can also display a basis identifier 561, which indicates the basis for the medical findings displayed in the medical findings section 53. The control unit 21 can extract the basis area related to the medical findings output from the second model 62 using model visualization methods such as Grad-CAM or Grad-CAM++.

[0158] Alternatively, in addition to the tomographic image 485, the first model 61 and the second model 62 may also input real-time medical information such as images taken using the image diagnostic device 37, blood pressure, heart rate, or oxygen saturation. Alternatively, in addition to the tomographic image 485, the first model 61 and the second model 62 may also input medical information obtained from electronic medical records, such as past medical conditions, height, weight, and images previously taken using the image diagnostic device 37.

[0159] In this case, Model 61 receives the tomographic image 485 and medical information, and outputs an object configuration image 482 that associates and maps the types of multiple objects contained in the tomographic image 485 with the range of each object. Similarly, Model 62 receives the tomographic image 485 and medical information, and outputs medical findings related to the state of a lumen organ or the state of its surroundings.

[0160] By incorporating medical information other than the tomographic image 485 into the input data of the first model 61, a catheter system 10 can provide object classification with good accuracy. By incorporating medical information other than the tomographic image 485 into the input data of the second model 62, a catheter system 10 can provide high-precision medical findings.

[0161] [Implementation Method 5]

[0162] This embodiment relates to a catheter system 10 that overlays the position of a tomographic image taken using an image diagnostic catheter 40 onto an image acquired from an image diagnostic device 37. Descriptions of parts common to Embodiment 1 are omitted.

[0163] Figure 16 This is an example of a screen displayed by the conduit system 10 in embodiment 5. Figure 16 The screen shown includes the Other Devices Image Bar 59. The Other Devices Image Bar 59 displays medical images captured by the image diagnostic device 37.

[0164] The scan area 591, which indicates the location of the tomographic image captured by the diagnostic catheter 40, is displayed as a rectangle overlaid on the image bar 59 of other devices, representing the shape of the longitudinal section image. The control unit 21 can also display the longitudinal section image or the longitudinal body configuration image in real time inside the scan area 591. The control unit 21 can also allow the user to make selections related to the display mode of the scan area 591.

[0165] Taking the case where the image diagnostic device 37 is an X-ray angiography device as an example, the outline of the method for displaying the scan area 591 will be explained. The sensor 42 is mounted on a sensor label that prevents X-rays from passing through. The front label 44 and the sensor label prevent X-rays from passing through, and therefore are clearly displayed in the medical image captured by the X-ray angiography device.

[0166] The control unit 21 detects the medical image front-end identifier 44 and the sensor identifier. The detected sensor identifier indicates the position of the sensor 42. In the case of generating a set of tomographic images 485, for example, using a pull-back operation based on MDU33, the two ends of the operating range of the sensor 42 during image acquisition correspond to the positions of the short sides of the scanning area 591.

[0167] Based on the display range of the tomographic image 485 and the scale of other device images, the control unit 21 determines the length of the short side of the scanning area 591. The control unit 21 causes the rectangular scanning area 591, determined by the position and length of the short side, to be superimposed on the image bar 59 of other devices displaying medical images.

[0168] Through the above processing, even when the front end of the image diagnostic catheter 40 is not parallel to the projection plane of the image diagnostic device 37, the control unit 21 can still display the scanning area 591 in the correct position of the image bar 59 of other devices.

[0169] return Figure 16 Continuing the explanation, multiple transverse section location markers 551 are displayed within the scanning area 591. A transverse object configuration image bar 515, displaying the object configuration image 482 corresponding to each transverse section location marker 551, is displayed around the image bar 59 of other devices. The user can appropriately change the transverse section location of the object configuration image 482 by moving the transverse section location markers 551 using the input device 32. Furthermore, the control unit 21 can also accept voice input from the user.

[0170] The control unit 21 can also switch between displaying the object configuration image 482 and the transverse section image 485 based on the user's instructions. The control unit 21 can also display the object configuration image 482 and the transverse section image 485 side by side. The control unit 21 can also display a longitudinal section image or a longitudinal object configuration image.

[0171] Alternatively, a schematic diagram of the luminal organ can be displayed in the image panel 59 of another device to replace the medical image captured by the image diagnostic device 37. This provides the catheter system 10 with an ability for the user to more easily grasp the position of the tomographic image.

[0172] Figure 17 This is a flowchart illustrating the processing flow of the procedure in Embodiment 5. During an image acquisition, the control unit 21 acquires a cross-sectional image 485 and a medical image from the catheter control unit 271 and the image diagnostic device 37, respectively (step S751).

[0173] The control unit 21 detects the front-end identifier 44 and the sensor identifier from the medical image (step S752). In the case of generating a set of tomographic images 485, for example, using a pull-back operation based on MDU33, the control unit 21 determines the position and size of the scanning area 591 based on the positions of the sensor identifiers detected at both ends of the pull-back operation. The control unit 21 determines the position of the tomographic position identifier 551 based on the scanning area 591 (step S753).

[0174] Furthermore, it is desirable that the control unit 21 displays the position corresponding to the scanning area 5 in real time on the medical images subsequently captured.

[0175] The control unit 21 inputs the tomographic image 485 obtained in step S751 to the first model 61, and obtains the object configuration image 482 that establishes the association between the types of multiple objects contained in the tomographic image 485 and the range of each object, as well as the probability of correctly determining the object for each pixel (step S754).

[0176] The control unit 21 generates a longitudinal object configuration image based on a set of object configuration images 482 (step S755). The control unit 21 records the generated longitudinal object configuration image in the auxiliary storage device 23 in a manner that can quickly display the longitudinal object configuration image based on the specified section in accordance with the user's operation on the longitudinal section location marker 552.

[0177] Control unit 21 will use Figure 16 The explanatory screen is displayed on the display device 31 (step S756). After that, the control unit 21 ends the processing.

[0178] According to this embodiment, a catheter system 10 can be provided that overlays the position of a tomographic image taken using an imaging diagnostic catheter 40 or an object configuration image 482 generated based on the tomographic image onto a medical image taken by an imaging diagnostic device 37. The user can easily change the position of the tomographic image 485 to be displayed by manipulating the tomographic position marker 551. Based on the above, a catheter system 10 can be provided that allows the user to easily grasp the positional relationship between the tomographic image and its surrounding organs.

[0179] Furthermore, the imaging diagnostic device 37 is not limited to an X-ray angiography device. For example, even an ultrasound diagnostic device combined with an external probe or a TEE (transesophageal echocardiography) probe can capture tomographic images in real time that are different from those captured by the imaging diagnostic catheter 40.

[0180] When the image diagnostic catheter 40 is equipped with both an ultrasound sensor 42 and an OCT sensor 42, it is possible to capture ultrasound-based tomographic images 485 and OCT-based tomographic images 485 in approximately the same cross section.

[0181] The control unit 21 can also overlay the object configuration image 482 obtained from the high-resolution OCT-based transverse image 485 onto the ultrasound-based transverse image 485 (which has superior depth compared to OCT). Furthermore, the control unit 21 can also appropriately combine and display the OCT-based transverse image 485, the object configuration image 482, the ultrasound-based transverse image 485, and the object configuration image 482. The catheter system 10 is capable of providing information that leverages the advantages of both methods.

[0182] Medical images are not limited to real-time captured medical images. Alternatively, the control unit 21 may overlay the scan area 591 onto a medical image captured and recorded in an electronic medical record by any image diagnostic device such as a CT, MRI, PET, X-ray angiography device, or ultrasound diagnostic device. The control unit 21 determines the position of the scan area 591 based on the branches of blood vessels, the location of the heart, etc., contained in each image.

[0183] The processing described in this embodiment can also be performed on the image diagnostic device 37 side and displayed on a display device connected to the image diagnostic device 37.

[0184] [Implementation Method 6]

[0185] This embodiment relates to procedures for generating the first model 61 and the second model 62, respectively. Descriptions of parts common to embodiment 4 are omitted.

[0186] Figure 18 This is an illustrative diagram illustrating the record layout of the training data DB (Database). The training data DB is a database that records the inputs and correct answer labels in a related manner, used in training machine learning-based models. The training data DB has tomographic data fields, color-coded data fields, and medical discovery fields. The medical discovery fields include fields such as treatment necessity, blood flow stasis, and branching, corresponding to the medical discoveries output by Model 2.

[0187] The tomographic image data field records sets of transverse tomographic images 485 that can generate longitudinal tomographic images. The color differentiation data field records sets of images created by experts color differentiating each object in the transverse tomographic image 485 using different colors or textures. That is, the color differentiation data field records the objects corresponding to each pixel that constitutes the transverse tomographic image 485.

[0188] The "Necessity of Treatment" field displays the results of expert judgments on the necessity of treatment based on the transverse tomographic image 485 recorded in the tomographic image data field. Similarly, the "Blood Stasis" field records the presence or absence of blood stasis, and the "Branching" field records the presence or absence of branching.

[0189] The training data DB contains a large number of sets of transverse tomographic images 485 taken using the image diagnostic catheter 40, sets of images that have been color-coded and differentiated by professional physicians, and combinations of treatment needs. Furthermore, the following explanation will use the case where the intervals between the transverse tomographic images 485 constituting the set are constant as an example.

[0190] When generating the first model 61, a set of transverse tomographic images 485 recorded in the tomographic image data field is used as input data, and a set of images recorded in the color discrimination data field is used as forward resolution data. When generating the second model 62, a set of transverse tomographic images 485 recorded in the tomographic image data field is used as input data, and the data recorded in each of the medical findings fields is used as forward resolution labels.

[0191] The training data database can also be created by dividing it into, for example, a database used to generate the first model 61 and a database used to generate the second model 62.

[0192] Figure 19 This is a flowchart illustrating the processing flow of the procedure in Implementation Method 5. The example given is the case where machine learning of the first model 61 is performed using the information processing device 20.

[0193] It could also be, Figure 19 The program is executed by hardware outside the information processing device 20, and the first model 61, after machine learning, is copied to the auxiliary storage device 23 via a network. The first model 61, learned by one hardware, can be used in multiple information processing devices 20.

[0194] exist Figure 19 Before the program executes, an unlearned model is prepared, such as the U-Net construction for semantic segmentation. As mentioned earlier, the U-Net construction consists of multiple encoder layers and multiple decoder layers connected to them. It is possible to utilize... Figure 19 The program adjusts the parameters of the prepared model to perform machine learning.

[0195] Control unit 21 retrieves the training records used in one epoch of training from the training data DB (step S761). The number of training records used in one epoch of training is a so-called hyperparameter, which can be appropriately determined.

[0196] The control unit 21 generates an input image based on the input data contained in each acquired training record (step S762). Specifically, the control unit 21 generates an image in which the transverse tomographic images 485 contained in the tomographic image field are combined into one image according to the scanning order. Alternatively, the combined transverse tomographic image may also be recorded in the tomographic image data field.

[0197] The control unit 21 generates a correct image based on the color discrimination data contained in each acquired training record (step S763). Specifically, the control unit 21 generates an image in which the color discrimination images contained in the color discrimination data field are combined into one image according to the scanning order. Alternatively, the combined color discrimination image may also be recorded in the color discrimination data field.

[0198] When an image is input to the input layer of the model, the control unit 21 adjusts the parameters of the model to output the correct solution image label from the output layer (step S764).

[0199] The control unit 21 determines whether to end the process (step S765). For example, if the control unit 21 has completed the learning of the prescribed number of epochs, it determines that the process is to end. Alternatively, the control unit 21 may obtain test data from the training data DB and input it into the machine learning model, and determine that the process is to end if an output with the prescribed accuracy is obtained.

[0200] If the process is determined not to end (No in step S765), the control unit 21 returns to step S761. If the process is determined to end (Yes in step S765), the control unit 21 records the parameters of the learned model to the auxiliary storage device 23 (step S767). After that, the control unit 21 ends the process. Through the above processes, a learned model is generated.

[0201] Figure 20 This is a flowchart illustrating the processing flow of the procedure in Implementation Method 6. Figure 19 Similarly, the case of using information processing device 20 to perform machine learning on the second model 62 will be explained as an example.

[0202] exist Figure 20 Before the program executes, an unlearned model is prepared, such as a CNN (Neural Network) with convolutional layers, pooling layers, fully connected layers (629), and softmax layers (625). The unlearned model is not limited to CNNs. Any type of model, such as decision trees or random forests, can be used. Figure 20 The program adjusts the parameters of the prepared model to perform machine learning.

[0203] The control unit 21 retrieves the training records used in one epoch of training from the training data DB (step S771). The control unit 21 generates an input image based on the input data contained in each retrieved training record (step S772).

[0204] When the input data vector is input to the input layer of the model, the control unit 21 adjusts the parameters of the model to output the correct answer label recorded in the medical discovery field from the output layer (step S773).

[0205] The control unit 21 determines whether to end the process (step S774). If it determines that the process should not be ended (No in step S774), the control unit 21 returns to step S771. If it determines that the process should be ended (Yes in step S774), the control unit 21 records the parameters of the learned model to the auxiliary storage device 23 (step S775). After that, the control unit 21 ends the process. Through the above process, a learned model is generated.

[0206] According to this embodiment, a first learning model 651 and a second learning model 652 can be generated using machine learning.

[0207] [Implementation Method 7]

[0208] This embodiment relates to a catheter system 10 in which the user can modify medical findings output by the learning model 65. Descriptions of parts common to Embodiment 1 are omitted.

[0209] Figure 21 This is an explanatory diagram illustrating the record layout of the first correction DB. The first correction DB is a database that records correction information that associates the object configuration image 482 output by the conduit system 10 with corrections made by the user.

[0210] The first correction database has a tomographic image data field, an output data field, and a correction data field. The tomographic image data field records a set of transverse tomographic images 485 capable of generating longitudinal tomographic images. The output data field records the object configuration image 482 output by the control unit 21 to the display device 31. The correction data field records the object configuration image 482 corrected by the user. The first correction database has one record for each correction made by the user to the set of transverse tomographic images 485.

[0211] Figure 22 This is an example of a screen displayed by the conduit system 10 in embodiment 7. Figure 22 In using, for example Figure 6 During the display of the explained screen, if the user instructs the user to correct the object configuration image 482, the control unit 21 displays a screen on the display device 31.

[0212] Figure 22 The screen shown includes a horizontal object configuration bar 515, a candidate label bar 571, a correct answer label bar 572, and a shape correction button 574. The candidate label bar 571 displays candidate labels indicating the type of object. The correct answer label bar 572 displays labels that the user has determined to be the correct answer.

[0213] The user uses pointer 575 to specify the object whose type needs to be changed. Then, the user enters the correct label by dragging and dropping the label displayed in the candidate label bar 571 to the correct label bar 572.

[0214] The control unit 21 will replace the label displayed in the correct label bar 572 with the label specified by the user. The control unit 21 will recolor the object specified by the pointer 575 with a color or background corresponding to the label selected by the user.

[0215] In addition, the control unit 21 can also accept voice-based input. For example, if the user says "change calcification to hematoma", the control unit 21 will recolor the object displayed in the color or background of "calcification" to the color or background of "hematoma".

[0216] When the user selects the shape correction button 574, the control unit 21 uses a user interface such as drawing software to handle corrections to the object configuration, such as the color-coded shape of 482. The drawing software is the one that has been used in the past, so detailed explanation is omitted.

[0217] Through the above steps, the user can appropriately correct the object configuration image 482. The control unit 21 records the object configuration image 482 corrected by the user, together with the object configuration image 482 before correction and the transverse section image 485, into the first correction DB.

[0218] Figure 23 This is an explanatory diagram illustrating the record layout of the second correction DB. The second correction DB is a database that records correction information that establishes a correlation between medical findings output by the catheter system 10 and corrections made by the user. The correction DB has one record for each correction made by the user to the set of sectional images 485.

[0219] The modified DB has a tomographic image data field, an output field, and a correction field. The tomographic image data field records a set of transverse tomographic images 485 capable of generating longitudinal tomographic images. The output field records medical findings output by the control unit 21 to the display device 31. The correction field records the medical findings corrected by the user.

[0220] Figure 24 This is an example of a screen displayed by the conduit system 10 in embodiment 7. Figure 24 In, for example, using Figure 14 During the display of the explained screen, the screen displayed on the display device 31 by the control unit 21 when the user instructs the medical discovery section 53 to be corrected.

[0221] Figure 24The displayed screen includes a tomographic image panel 51, a candidate label panel 571, a correct answer label panel 572, and a free comment panel 573. The candidate label panel 571 displays candidate labels representing qualitative medical findings. The correct answer label panel 572 displays labels that the user has determined to be correct.

[0222] Users can input correct medical findings by dragging and dropping labels between the candidate label bar 571 and the correct answer label bar 572. Furthermore, the control unit 21 can also accept voice-based input. For example, if the user utters the words "correct answer, hematoma," the control unit 21 moves the "hematoma" label from the first row of the candidate label bar 571 to the correct answer label bar 572.

[0223] If no label indicating an appropriate medical finding is found in the candidate label field 571, the user can enter any medical finding using the free description field 573. Although an illustration is omitted, the user can also make appropriate changes for quantitative medical findings.

[0224] Control unit 21 will use by the user Figure 24 The corrections entered on the screen are recorded in the user's browser. Figure 23 The DB has been revised with explanations provided.

[0225] User use Figure 22 and Figure 24 The corrected content of the images can also be attached to the electronic medical record that records the patient's diagnosis and treatment results. The data recorded in the first correction DB and the second correction DB can be flexibly used for the relearning of the first model 61 and the second model 62, as well as for the correction of hyperparameters used in machine learning by machine learning engineers.

[0226] [Implementation Method 8]

[0227] Figure 25 This is a functional block diagram of the catheter system 10 according to Embodiment 8. The catheter system 10 includes an acquisition unit 86 and an output unit 87. The acquisition unit 86 acquires a tomographic image generated using an image diagnostic catheter inserted into a luminal organ. The output unit 87 inputs the acquired tomographic image to a model 65 and outputs the types and ranges of objects output from the model 65. The model 65, when the tomographic image acquired by the acquisition unit 86 is input, outputs the types of multiple objects included in the tomographic image in association with the range of each object.

[0228] [Implementation Method 9]

[0229] Figure 26This is an explanatory diagram illustrating the configuration of the conduit system in Embodiment 9. This embodiment relates to a configuration in which the information processing apparatus 20 of this embodiment is implemented by combining a general-purpose computer 90 and a program 97. Figure 26 This is an explanatory diagram showing the configuration of the information processing apparatus 20 in Embodiment 9. Descriptions of parts common to Embodiment 2 are omitted.

[0230] The conduit system 10 of this embodiment includes a computer 90. The computer 90 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is a general-purpose information device such as a personal computer, tablet computer, smartphone, or server computer.

[0231] Program 97 is recorded on portable recording medium 96. Control unit 21 reads program 97 via reading unit 29 and saves it to auxiliary storage device 23. Alternatively, control unit 21 can also read program 97 stored in semiconductor memory 98 such as flash memory installed in computer 90. Furthermore, control unit 21 can also download program 97 from other server computer (not shown) connected via communication unit 24 and a network (not shown) and save it to auxiliary storage device 23.

[0232] Program 97 is installed as the control program of computer 90, loaded into main storage device 22, and executed. Thus, computer 90 functions as the aforementioned information processing device 20.

[0233] The technical features (constituent elements) described in each embodiment can be combined with each other to form new technical features.

[0234] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the invention is defined not by the foregoing but by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0235] Explanation of reference numerals in the attached figures

[0236] 10. Catheter system

[0237] 20 Information processing devices

[0238] 21 Control Department

[0239] 22 Main storage device

[0240] 23. Auxiliary storage device

[0241] 24 Ministry of Communications

[0242] 25 Display Section

[0243] 26 Input Section

[0244] 27. Catheter control device

[0245] 271 Catheter Control Section

[0246] 29 Reading Department

[0247] 31 Display devices

[0248] 32 Input Device

[0249] 33 MDU

[0250] 37 Image Diagnostic Device

[0251] 40 Diagnostic catheters

[0252] 41. Probe section

[0253] 42 sensors

[0254] 43 axis

[0255] 44 Front-end identifier

[0256] 45 Connector Section

[0257] 482 Object Configuration Image

[0258] 485 Transverse tomographic image (fault image)

[0259] 51 Transverse Frame

[0260] 515 Horizontal Object Configuration Image Column

[0261] 52 Longitudinal Fault Image Column

[0262] 525 Vertical Object Configuration Image Column

[0263] 53 Medical Discoveries

[0264] 538 value tags

[0265] 551 Cross-fault location marker

[0266] 552 Longitudinal Fault Location Marking

[0267] 561 According to the label

[0268] 571 Candidate Tag Bar

[0269] 572 Correct Answer Tag Bar

[0270] 573 Free Comments Section

[0271] 574 Shape Correction Button

[0272] 575 pointer

[0273] 591 Scan Area

[0274] 61 Model 1

[0275] 62 Model 2

[0276] 625 Softmax layer

[0277] 629 Neural Network

[0278] 65 Learning Model (Model)

[0279] 651 First Learning Model

[0280] 652 Second Learning Model

[0281] 86 Acquisition Department

[0282] 87 Output Section

[0283] 90 Computers

[0284] 96 Portable recording media

[0285] 97 Program

[0286] 98 Semiconductor memory

Claims

1. A program product comprising a program that causes a computer to perform the following processes: Acquiring tomographic images generated using image diagnostic catheters inserted into luminal organs; The acquired tomographic image is input into the first model, and the types and ranges of objects output from the first model are output. The first model outputs the types of multiple objects contained in the tomographic image in association with the ranges of each object when the tomographic image is input. The angle of the range of states in which the luminal organ is in relation to the object is calculated based on the range of an object selected from the object; as well as Output the calculated angle.

2. The program product according to claim 1, wherein, The types and scope of the objects are displayed in different forms according to the types of the objects.

3. The program product according to claim 1 or 2, wherein, The type and extent of the object are mapped and stored in association with the location of the tomographic images in the luminal organs.

4. The program product according to claim 3, wherein, The tomographic image represents a cross-sectional image obtained by cutting through the lumen and its periphery in a direction intersecting the length direction of the lumen. For each of the multiple tomographic images, a mapping is established and stored to represent the type and extent of the object. The type and extent of the object are mapped to a longitudinal tomographic image generated from multiple transverse tomographic images, representing a cross-section obtained by cutting the lumen organ and its periphery in a direction parallel to the length direction of the lumen organ, and then output.

5. The program product according to claim 4, wherein, Simultaneously output the longitudinal tomographic image and the selected tomographic image chosen from multiple transverse tomographic images. An identifier indicating the location of the selected fault image is displayed on the longitudinal fault image.

6. The program product according to claim 1 or 2, wherein, The identifier indicating the location of the tomographic image is overlaid on a medical image or diagram that is different from the acquired tomographic image.

7. The program product according to claim 1 or 2, wherein, The size, area, or volume of the object is calculated based on the scope of the object. Output the calculated dimensions, area, or volume.

8. The program product according to claim 1 or 2, wherein, Along with the types and ranges of objects output from the first model, output information relating to the types and ranges of objects.

9. The program product according to claim 1 or 2, wherein, The first model outputs the probability of which of the objects each pixel in the input tomographic image corresponds to. The type and range of the object are displayed by a hue determined according to the type of the object, and the different probabilities of matching the object are displayed by the difference in brightness or chroma of each pixel.

10. The program product according to claim 1 or 2, wherein, Acceptance of amendments concerning the type or scope of the object.

11. The program product according to claim 10, wherein, The record establishes correction information by associating the acquired tomographic images with corrections related to the type or extent of the object.

12. The program product according to claim 11, wherein, The first model is relearned based on the corrected information.

13. The program product according to claim 1 or 2, wherein, The acquired tomographic image is input into the second model, and information output from the second model is further output. The second model outputs information related to the state of the lumen organ or the area surrounding the lumen organ when the tomographic image is input.

14. The program product according to claim 13, wherein, The authority accepts corrections for the condition of the lumen organ or the area surrounding the lumen organ.

15. The program product according to claim 14, wherein, The record establishes correction information by associating the acquired tomographic images with corrections for the state of the lumen organ or the area surrounding the lumen organ.

16. The program product according to claim 15, wherein, The second model is relearned based on the corrected information.

17. An information processing method that causes a computer to perform the following processing: Acquiring tomographic images generated using image diagnostic catheters inserted into luminal organs; The acquired tomographic image is input into the first model, and the types and ranges of objects output from the first model are output. The first model outputs the types of multiple objects contained in the tomographic image in association with the ranges of each object when the tomographic image is input. The angle of the range of states in which the luminal organ is in relation to the object is calculated based on the range of an object selected from the object; as well as Output the calculated angle.

18. An information processing system, comprising: The acquisition unit acquires tomographic images generated using an image diagnostic catheter inserted into a luminal organ; and The output unit inputs the acquired tomographic image into the model and outputs the types and ranges of objects output from the model. The model, given the tomographic image acquired by the acquisition unit, outputs the types of multiple objects contained in the tomographic image in association with the ranges of each object. The information processing system calculates the angle of the range of the luminal organ in the state corresponding to the selected object from the range of the object, and outputs the calculated angle.

Images