Computer program, information processing method, and information processing apparatus

Abstract

To provide computer program, an information processing method, and an information processing device capable of presenting information on an anatomical feature acquired through an entire examination in such a manner that an occurrence of a problem can be recognized even when there is a problem with an image.SOLUTION: A computer program causes a computer for acquiring a plurality of cross-sectional images of a hollow organ based on a signal detected by an imaging device equipped to a catheter inserted into the hollow organ to execute processing for: calculating data indicating an anatomical feature in the hollow organ for each of a plurality of cross-sectional images in which it is determined that there is no problem in the detection by the imaging device in the cross-sectional images of the above plurality of cross-sectional images; displaying the distribution of data indicating the anatomical feature in a longitudinal direction of the hollow organ; and displaying an object indicating a place corresponding to the cross-sectional image in which a problem is determined to be occurring on the displayed distribution.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to a computer program, an information processing method, and an information processing device related to processing of medical images. [Background technology]

[0002] In medical examinations, images are used for diagnosis by either directly capturing an image of the subject or by imaging the results of measurements using electromagnetic waves. In particular, in the examination of hollow organs, various techniques are used that use images obtained by moving an imaging element inside the organ.

[0003] Diagnostic imaging of vascular organs, particularly vascular organs, is essential for the safe and reliable performance of procedures such as percutaneous coronary intervention (PCI). For this reason, in addition to angiography, which uses contrast media to capture images from outside the body, intravascular imaging techniques such as catheter-based IVUS (Intravascular Ultrasound), OCT (Optical Coherence Tomography) / OFDI (Optical Frequency Domain Imaging) are becoming increasingly common.

[0004] In the above-mentioned image diagnosis, it is not easy to obtain accurate diagnostic information from the captured medical images. To assist in the interpretation of medical images, various techniques have been proposed that use image analysis or machine learning to correct images or add information (Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-075702 Summary of the Invention [Problem to be solved by the invention]

[0006] In catheter-based imaging technology, problems can occur in the tomographic images obtained from the signals acquired by the image sensor due to noise, air traps, broken wires, and rotational obstructions of the image sensor. For example, problems can result in the tomographic image being dark overall, making it impossible to distinguish the organ parts that should be visible in the tomographic image, or the generation of indistinguishable artifacts. Even before the examination begins, if a problem occurs that prevents medical images from being obtained throughout the entire examination, such as a poor connection or broken wire, it is easy for the examination operator to notice it. However, if a problem occurs only partially, it is difficult for the examination operator to detect it, and it is extremely tedious to check all the images.

[0007] The object of the present disclosure is to provide a computer program, an information processing method, and an information processing device that present information about anatomical features obtained through the entire examination in a recognizable manner, even if there is a problem with the images obtained using a catheter. [Means for solving the problem]

[0008] The computer program of the present disclosure causes a computer that acquires multiple tomographic images of a tubular organ based on signals detected by an imaging device provided on a catheter inserted into the tubular organ to perform the following process: calculate data indicating anatomical features of the tubular organ for each of multiple tomographic images among the multiple tomographic images that are determined to have no problems in detection by the imaging device; display a distribution of the data indicating the anatomical features in the longitudinal direction of the tubular organ; and display, on the displayed distribution, an object indicating a location corresponding to the tomographic image that is determined to have a problem.

[0009] The information processing method disclosed herein involves a computer that acquires multiple tomographic images of a tubular organ based on signals detected by an imaging device provided on a catheter inserted into the organ, calculates data indicating anatomical features of the tubular organ for each of the multiple tomographic images that are determined to have no problems in detection by the imaging device, displays a distribution of the data indicating the anatomical features in the longitudinal direction of the tubular organ, and displays, on the displayed distribution, an object indicating a location corresponding to the tomographic image that is determined to have a problem.

[0010] The information processing device of the present disclosure is an information processing device that acquires multiple tomographic images of a tubular organ based on signals detected by an imaging device provided on a catheter inserted into the tubular organ, and includes a processing unit that performs image processing on each of the multiple tomographic images.The processing unit calculates data indicating anatomical features of the tubular organ for each of the multiple tomographic images that are determined to have no problems in detection by the imaging device, displays a distribution of the data indicating the anatomical features in the longitudinal direction of the tubular organ, and displays an object on the displayed distribution that indicates the location corresponding to the tomographic image that is determined to have a problem. [Effects of the Invention]

[0011] According to the present disclosure, even if there is a problem with a tomographic image of a hollow organ, information on anatomical features obtained through the entire examination can be presented, and an operator who visually checks the displayed information can recognize the occurrence of the problem and its location. Furthermore, by recognizing the location of the problem, the operator can determine whether to re-acquire the information or check the details of the relevant area using another modality (such as an angiography device) that is also being used. [Brief explanation of the drawings]

[0012] [Figure 1]FIG. 1 is a diagram illustrating an example of the configuration of an imaging diagnostic apparatus. [Figure 2] FIG. 10 is an explanatory diagram showing the operation of the catheter. [Figure 3] FIG. 1 is a block diagram showing a configuration of an image processing device. [Figure 4] This is an overview of the trained first model. [Figure 5] FIG. 10 is a diagram showing detected boundaries (contours). [Figure 6] This is an overview of the trained second model. [Figure 7] 10 is a flowchart illustrating an example of an information processing procedure performed by the image processing device. [Figure 8] FIG. 1 is a schematic diagram of a method for calculating data indicating anatomical features. [Figure 9] 1 shows an example of a screen displayed on a display device. [Figure 10] 10 is a flowchart showing an example of an information processing procedure by an image processing device according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of an information processing procedure by an image processing device according to a second embodiment. [Figure 12] 10 shows an example of a screen displayed on a display device in the second embodiment. [Figure 13] 11 is a flowchart showing an example of an information processing procedure by an image processing device according to a third embodiment. [Figure 14] 11 is a flowchart showing an example of an information processing procedure by an image processing device according to a third embodiment. [Figure 15] FIG. 10 is a schematic diagram of another example of a method for estimating data indicative of anatomical features. [Figure 16] FIG. 10 is a schematic diagram of another example of a method for estimating data indicative of anatomical features. [Figure 17] FIG. 1 is a schematic diagram of an image generation model. [Figure 18] 10 is a flowchart showing an example of an information processing procedure by an image processing device according to a fourth embodiment. [Figure 19]10 is a flowchart showing an example of an information processing procedure by an image processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Specific examples of a computer program, an information processing method, and an information processing device according to embodiments of the present invention will be described below with reference to the drawings.

[0014] (First embodiment) 1 is a diagram showing an example of the configuration of an imaging diagnostic device 100. The imaging diagnostic device 100 is a device for generating medical images including ultrasonic tomographic images of blood vessels (hollow organs) by the IVUS method and for performing intravascular ultrasound examination and diagnosis.

[0015] The diagnostic imaging apparatus 100 includes a catheter 1, an MDU (Motor Drive Unit) 2, an image processing device (information processing device) 3, a display device 4, and an input device 5.

[0016] The catheter 1 is a flexible tube for medical use. The catheter 1 is particularly known as an imaging catheter, which has an imaging device 11 at its tip and rotates in a circumferential direction by being driven from its base end. In the case of the IVUS method, the imaging device 11 is an ultrasound probe including an ultrasound transducer and an ultrasound sensor. In the case of OCT, the imaging device 11 is an OCT device including a near-infrared laser and a near-infrared sensor. The imaging device 11 may also be another device that uses electromagnetic waves of other wavelengths, such as visible light.

[0017] The MDU 2 is a driving device attached to the base end of the catheter 1, and controls the operation of the catheter 1 by driving an internal motor in response to the operation of the examination operator.

[0018] The image processing device 3 generates a plurality of medical images, such as cross-sectional images of blood vessels, based on the signals output from the imaging device 11 of the catheter 1. The configuration of the image processing device 3 will be described in detail later.

[0019] The display device 4 uses a liquid crystal display panel, an organic EL display panel, etc. The display device 4 displays the medical image generated by the image processing device 3 and information related to the medical image.

[0020] The input device 5 is an input interface that accepts operations for the image processing device 3. The input device 5 may be a keyboard, a mouse, or the like, or may be a touch panel, soft keys, hard keys, or the like built into the display device 4.

[0021] Figure 2 is an explanatory diagram showing the operation of the catheter 1. In Figure 2, the catheter 1 is inserted into a tubular blood vessel L by an examination operator along a guide wire W inserted into a coronary artery shown in the figure. In the enlarged view of the blood vessel L in Figure 2, the right side corresponds to the distal side from the insertion point of the catheter 1 and the guide wire W, and the left side corresponds to the proximal side.

[0022] The catheter 1 is driven by the MDU 2 to move from the distal end to the proximal end within the blood vessel L as indicated by the arrow in the figure, and while rotating in the circumferential direction, the imaging device 11 scans the inside of the blood vessel in a spiral manner.

[0023] In the diagnostic imaging apparatus 100 of this embodiment, the image processing device 3 acquires signals for each scan output from the imaging device 11 of the catheter 1. Each scan involves emitting a detection wave from the imaging device 11 in the radial direction and detecting reflected light, resulting in a spiral scan. The image processing device 3 generates a tomographic image (cross-sectional image) obtained by polar coordinate conversion of the signals for each scan for every 360 degrees (I1 in FIG. 2). The tomographic image I1 is also referred to as a frame image. The reference point (center) of the tomographic image I1 corresponds to the range of the catheter 1 (not imaged). The image processing device 3 further generates a long-axis image (longitudinal cross-sectional image) in which pixel values ​​on a line passing through the reference point of the tomographic image I1 are arranged along the length (long-axis direction) of the blood vessel by the catheter 1 (I2 in FIG. 2). The image processing device 3 analyzes and processes the branching structure of the blood vessel based on the obtained tomographic image I1 and long-axis image I2, and outputs a two-dimensional or three-dimensional image showing the blood vessel structure so that it can be viewed by an examination operator or other medical personnel. In the image diagnostic apparatus 100 of the present disclosure, when there is a problem with the tomographic image I1 created from the signal obtained from the imaging device 11, the problem is displayed recognizably on the display device 4. Hereinafter, the display process performed by the image processing device 3 when there is a problem with the tomographic image I1 will be described in detail.

[0024] 3 is a block diagram showing the configuration of the image processing device 3. The image processing device 3 is a computer, and includes a processing unit 30, a storage unit 31, and an input / output I / F 32.

[0025] The processing unit 30 includes one or more central processing units (CPUs), micro-processing units (MPUs), graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), tensor processing units (TPUs), etc. The processing unit 30 has a built-in non-transitory storage medium such as random access memory (RAM), and performs calculations based on a computer program 3P stored in the storage unit 31 while storing data generated during processing in the non-transitory storage medium.

[0026] The storage unit 31 is a non-volatile storage medium such as a hard disk, a flash memory, etc. The storage unit 31 stores a computer program 3P, setting data, etc. that are read by the processing unit 30. The storage unit 31 also stores a first model 31M and a second model 32M that have been trained.

[0027] The computer program 3P, the first model 31M, and the second model 32M may be copies of the computer program 9P, the first model 91M, and the second model 92M stored in a non-temporary storage medium 9 outside the device, read out via the input / output I / F 32. The computer program 3P, the first model 31M, and the second model 32M may be distributed by a remote server device, acquired by the image processing device 3 via a communication unit (not shown), and stored in the storage unit 31.

[0028] The input / output I / F 32 is an interface to which the catheter 1, the display device 4, and the input device 5 are connected. The processing unit 30 acquires a signal (digital data) output from the imaging device 11 via the input / output I / F 32. The processing unit 30 outputs screen data for a screen including the generated tomographic image I1 and / or long-axis image I2 to the display device 4 via the input / output I / F 32. The processing unit 30 accepts operation information input to the input device 5 via the input / output I / F 32.

[0029] 4 is a schematic diagram of a trained first model 31M. The first model 31M in the present disclosure is a model trained to output, when an image is input, an image showing the regions of one or more objects appearing in the image. The first model 31M is, for example, a model that performs semantic segmentation. The first model 31M is designed to output an image in which each pixel in the input image is tagged with data indicating which object each pixel is in the range of.

[0030] The first model 31M may use, for example, a so-called U-net in which a convolution layer, a pooling layer, an upsampling layer, and a softmax layer are symmetrically arranged, as shown in FIG. 4. When a tomographic image I1 created using a signal from the catheter 1 is input, the first model 31M outputs a tag image IS. The tag image IS is obtained by tagging the lumen area of ​​the blood vessel, the membrane area corresponding to the area between the lumen boundary of the blood vessel and the blood vessel boundary, including the tunica media, the area in which the guidewire W and its reflection are captured, and the area corresponding to the catheter 1, with pixels at those positions each having a different pixel value (shown by different types of hatching and solid color in FIG. 4). The first model 31M may further be able to identify areas such as fibrous plaque, lipid plaque, and calcified plaque formed in the blood vessel.

[0031] As described above, the first model 31M is exemplified by semantic segmentation and U-net, but it goes without saying that the first model 31M is not limited to this. Alternatively, the first model 31M may be a model that realizes individual recognition processing using instance segmentation, etc. The first model 31M is not limited to being based on U-net, and may be a model based on SegNet, R-CNN, or an integrated model with other edge extraction processing, etc.

[0032] The processing unit 30 can detect the edges of the lumen boundary and vascular boundary of the blood vessel shown in the tomographic image I1 by using pixel values ​​in the tag image IS obtained by inputting the tomographic image I1 into the first model 31M and their coordinates within the image. Strictly speaking, the vascular boundary is the external elastic membrane (EEM) between the tunica media and adventitia of the blood vessel, and is shown relatively clearly and with low brightness in the image I1 using the IVUS method. Figure 5 shows the detected boundary (contour). Figure 5 shows the tomographic image I1 shown in Figure 4, with a curve B1 indicating the lumen boundary and a curve B2 indicating the vascular boundary superimposed on it.

[0033] By generating the tomographic image I1 as described above and displaying the results of the range identification as an auxiliary function, the examination operator and other medical personnel can more easily understand the inside of the luminous organ. However, if the tomographic image I1 input to the first model 31M is of poor quality, it is extremely difficult to identify the above-mentioned range. For example, if air (air bubbles) adhere to the surface while the imaging device 11 is rotating spirally during scanning, the generated image will be a dark image with a pixel value (brightness) close to zero. If there are intermittent disconnections, the tomographic image I1 generated based on the signal from the imaging device 11 will also be a dark image. If the rotation of the imaging device 11 is obstructed (for example, when the imaging device 11 passes through a highly tortuous blood vessel), the image may be partially dark or may have geometric patterns appearing in part or all of the image.

[0034] Therefore, before inputting the tomographic image I1 obtained from the signal from the catheter 1 into the first model 31M for segmentation, the image processing device 3 of the present disclosure uses the second model 32M to detect whether a problem has occurred during scanning of the tomographic image I1. If a problem is detected, the image processing device 3 of the present disclosure first generates a screen displaying the location on the long axis corresponding to the tomographic image I1. The image processing device 3 avoids deriving data indicating anatomical features from the tomographic image I1 where the problem has occurred. The image processing device 3 may also predict the anatomical features of the location where the problem has occurred using tomographic images I1 scanned before and after. The detailed processing procedure will be described below.

[0035] FIG. 6 is a schematic diagram of a trained second model 32M. The second model 32M in the present disclosure is a model trained to output, when an image is input, a numerical value (accuracy) indicating the probability of whether or not a problem occurs during scanning of the image. The second model 32M is a model that derives image features using, for example, a convolutional neural network (CNN) including a convolutional layer, a pooling layer, and the like. The second model 32M has an input layer 321, an intermediate layer 322, and an output layer 323. The second model 32M has trained parameters in the intermediate layer 322 by providing training data in which a tag of "problem occurred: 1" is given to images that have been previously determined to have a problem, and a tag of "no problem: 0" is given to images that have been previously determined to have no problem.

[0036] The second model 32M may be a classification model that classifies the type of problem that has occurred. The problem types are "air trap," "wire break," "rotation obstruction," etc., and the image processing device 3 may assign a different label to each of these results, learn the tomographic image I1 in advance, and train the parameters of the intermediate layer 322 so as to output the accuracy for each type.

[0037] In the present disclosure, the first model 31M and the second model 32M are described as separate models. However, the output of the second model 32M (probability of whether a problem occurs) may be replaced by the accuracy obtained from the first model 31M to which the tomographic image I1 is input. Since the accuracy corresponds to the reliability of whether the segmentation was successful, even if the first model 31M is applied to a tomographic image in which a problem occurs, it is estimated that the segmentation will not be successful and the output accuracy will be low. In other words, the single first model 31M may serve as both the first model 31M and the second model 32M. In this case, the image processing device 3 acquires the tag image IS and accuracy obtained by inputting the tomographic image I1 to the first model 31M, and does not use the tag image IS if the accuracy is less than a predetermined value.

[0038] The following describes the processing procedure by the image processing device 3 using the first model 31M and the second model 32M. Fig. 7 is a flowchart showing an example of the information processing procedure by the image processing device 3. When a signal is output from the imaging device 11 of the catheter 1, the processing unit 30 of the image processing device 3 starts the following processing.

[0039] Each time the processing unit 30 acquires a predetermined amount (e.g., 360 degrees) of signals (data) from the imaging device 11 of the catheter 1 (step S301), it performs polar coordinate conversion (inverse conversion) on the rectangularly arranged signals to generate a tomographic image I1 (step S302). The processing unit 30 outputs the generated tomographic image I1 so that it can be displayed in real time on the screen of the display device 4 (step S303). The processing unit 30 stores the signal data acquired in step S301 and the tomographic image I1 in the storage unit 31 in association with positions on the long axis (step S304). In step S304, the processing unit 30 may also store the scanning angle of the imaging device 11.

[0040] The processing unit 30 inputs the tomographic image I1 to the second model 32M (step S305). The processing unit 30 determines whether or not there is a problem in the tomographic image I1 generated in step S302 based on the probability (accuracy) of whether or not a problem has occurred, which is output from the first model 31M (step S306).

[0041] In step S306, the processing unit 30 may determine whether or not there is a problem without using the second model 32M. For example, the processing unit 30 determines whether or not there is a problem based on the pixel values ​​(brightness values) of the tomographic image I1, depending on whether the pixel values ​​of 80% or more of the pixels are equal to or less than a predetermined pixel value, that is, whether the image is dark.

[0042] If it is determined that there is no problem (S306: NO), the processing unit 30 inputs the tomographic image I1 to the first model 31M (step S307). The processing unit 30 calculates data indicating anatomical features obtained from the tomographic image I1 based on the tag image IS output from the first model 31M (step S308).

[0043] In step S308, in a first example, the processing unit 30 calculates values ​​such as the maximum diameter, minimum diameter, and average inner diameter of the lumen area from the lumen boundary to the inside. The processing unit 30 may further calculate the outer contour of the membrane area as the blood vessel boundary, and calculate the maximum diameter, minimum diameter, and average diameter.

[0044] In step S308, in the second example, the processing unit 30 calculates the lumen boundary and the vascular boundary as in the first example, and may also determine the fibrous plaque range, the lipid plaque range, the calcified plaque range, etc. based on pixel values ​​between the lumen boundary and the vascular boundary in the tomographic image I1. When the processing unit 30 has been able to determine the fibrous plaque range, the lipid plaque range, or the calcified plaque range, it may calculate the ratio of the cross-sectional area to the area inside the vascular boundary (plaque burden).

[0045] The processing unit 30 stores the data indicating the anatomical features calculated in step S308 in the storage unit 31 in association with the position on the long axis corresponding to the tomographic image I1 (step S309).

[0046] The processing unit 30 outputs the data indicating the anatomical features calculated in step S308 onto the screen being displayed on the display device 4 (step S310). In step S310, the processing unit 30 may output a graph indicating the progression of the data in the long axis direction, or may output the numerical values ​​of the data.

[0047] The processing unit 30 determines whether scanning by the imaging device 11 of the catheter 1 has been completed (step S311). If it is determined that scanning has not been completed (S311: NO), the processing unit 30 returns the process to step S301 and generates the next tomographic image I1.

[0048] If it is determined that scanning is completed (S311: YES), the processing unit 30 displays again the distribution of data for the entire longitudinal axis direction of the scanned blood vessel (step S312), and ends the processing. The processing unit 30 may also display the numerical values ​​of the data in addition to the distribution of the data.

[0049] If it is determined in step S306 that there is a problem (S306: YES), the processing unit 30 outputs text or an image indicating that there is a problem in the target tomographic image I1 to the screen being displayed on the display device 4 (step S313). The processing unit 30 stores the fact that there is a problem in the storage unit 31 in association with the position on the long axis corresponding to the tomographic image I1 (step S314), and proceeds to step S311.

[0050] In step S313, the processing unit 30 displays, for example, an image indicating the occurrence of a problem on a graph showing the progression of data indicating anatomical features relative to the position on the longitudinal axis of the tubular organ, at a position corresponding to the tomographic image I1 in which the problem occurs (see Figure 9).

[0051] An example of the calculation of data indicating anatomical features in step S308 will be described in detail. FIG. 8 is a schematic diagram of a method for calculating data indicating anatomical features. In one example, the image processing device 3 calculates the average lumen diameter and the plaque cross-sectional area. The processing unit 30 of the image processing device 3 may calculate the maximum and minimum diameters passing through the center of gravity of the inner region of the lumen boundary shown in FIG. 5 and calculate the average of the maximum and minimum diameters, or may calculate the average of each line segment passing through the center of gravity and connecting points on the lumen boundary. The processing unit 30 determines the area with low pixel values ​​(high brightness) within the range outside the curve B1 of the lumen boundary and inside the curve B2 of the vascular boundary as the area where plaque is captured. In FIG. 8, the area where plaque is captured is indicated by hatching. The processing unit 30 calculates the proportion of the plaque area (Plaque Burden) to the area of ​​the range outside the curve B1 of the lumen boundary and inside the curve B2 of the vascular boundary. In the example of FIG. 8, the processing unit 30 calculates, for example, 55%.

[0052] FIG. 9 shows an example of a screen 400 displayed on the display device 4. The screen 400 shown in FIG. 8 includes a cursor 401 for selecting a position on the long axis and a tomographic image I1 at the position corresponding to the cursor 401. The screen 400 includes a graph 402 and a graph 403 of data indicating anatomical features. The graph 402 shows the distribution of the average lumen diameter with respect to the position on the long axis. The graph 402 shows the position on the long axis on the horizontal axis, and the average inner diameter (average lumen diameter) of the luminal organ at each position on the vertical axis. The graph 403 shows the distribution of the plaque extent ratio with respect to the position on the long axis. The graph 403 shows the position on the long axis on the horizontal axis, and the plaque extent ratio at each position as a percentage on the vertical axis.

[0053] On screen 400 in Figure 9, image 404 is displayed superimposed on graphs 402 and 403, indicating that a problem occurred in tomographic image I1. Image 404 shows that a problem occurred in tomographic image I1 at that position on the long axis, and that data indicating anatomical features could not be calculated. A medical professional viewing screen 400 can recognize the area where the problem occurred, and can also estimate the lumen diameter and plaque area ratio within that area from the continuity with the areas before and after. By being able to recognize the location of the problem, the examination operator can determine whether to re-acquire information or can use another modality (such as an angiography device) in combination to check the area in detail.

[0054] (Second embodiment) In the first embodiment, if a problem occurs in a tomographic image I1, data indicating anatomical features corresponding to that image is not calculated. In contrast, the image processing device 3 of the second embodiment calculates data for a tomographic image I1 in which no problem occurs, and then estimates data corresponding to the tomographic image I1 in which a problem occurs from the calculated data.

[0055] The configurations of the image diagnostic apparatus 100 and the image processing apparatus 3 in the second embodiment are the same except for some of the processing procedures by the image processing apparatus 3, so the same reference numerals are used and detailed description will be omitted.

[0056] 10 and 11 are flowcharts showing an example of an information processing procedure by the image processing device 3 in the second embodiment. Among the processing procedures shown in the flowcharts of Fig. 10 and Fig. 11, the same steps as those shown in the flowchart of Fig. 7 are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0057] In the second embodiment, when it is determined in step S306 that a problem has occurred (S306: YES), the processing unit 30 of the image processing device 3 stores in the memory unit 31 (S314) characters or images indicating that a problem has occurred in the target tomographic image I1 without outputting them to the screen currently being displayed on the display device 4, and proceeds to step S311.

[0058] In the second embodiment, when it is determined in step S311 that scanning is completed (S311: YES), the processing unit 30 of the image processing device 3 reads out the position of the tomographic image I1 where the occurrence of a problem is stored (step S321). The processing unit 30 reads out a plurality of data calculated at a position distal to and / or proximal to the read out position (step S322). The processing unit 30 executes a process of inferring data indicating anatomical features at the position of the tomographic image I1 where the occurrence of a problem is stored from the plurality of read out data (step S323).

[0059] In step S323, the processing unit 30 may estimate data by connecting adjacent data on the distal and proximal sides using a spline. The processing unit 30 may estimate data by creating an approximation curve (straight line) on the distal or proximal side and extending it to the target position. When adjacent data on the distal and proximal sides are input, the processing unit 30 may separately learn a prediction model that predicts uncalculated data and make predictions using the prediction model.

[0060] The processing unit 30 complements the data distribution for the entire longitudinal direction of the scanned blood vessel with the estimated data (step S324) and displays the distribution (S312). The processing unit 30 superimposes an image that highlights the complemented portion of the distribution so that the range of the estimated values ​​can be visually confirmed (step S325), and then ends the processing.

[0061] In the above description, the image processing device 3 executes the process of step S323 after it is determined in step S311 that scanning has been completed. However, the timing for executing the data estimation process of step S323 is not limited to this. For example, when it is determined that a problem has occurred, the image processing device 3 may execute data estimation using a tomographic image I1 (distal-side tomographic image I1) generated before that time. Alternatively, the image processing device 3 may execute data estimation when it generates (acquires) a tomographic image I1 that is determined not to have a problem after it has determined that a problem has occurred.

[0062] Fig. 12 shows an example of a screen 400 displayed on the display device 4 in the second embodiment. Like the screen 400 in the first embodiment, the screen 400 in Fig. 12 includes a cursor 401, a tomographic image I1, and graphs 402 and 403. As described above, in the graphs 402 and 403 in the second embodiment, data is not missing but is supplemented by estimation. Furthermore, a frame image 405 indicating that the values ​​are estimations is superimposed.

[0063] By viewing image 405, the examination operator and other medical personnel can see that a problem has occurred in the tomographic image I1 at the position on the long axis where image 405 is displayed, and that data indicating anatomical features could not be calculated. By being able to recognize the location of the problem, the examination operator can decide whether to re-acquire the information or can check the relevant area in detail using another modality (such as an angiography device) that is also being used. Furthermore, in the second embodiment, the examination operator can recognize that the data of the portion highlighted in image 405 is estimated data that could not be calculated. Medical personnel can use the estimated data as a reference.

[0064] (Third embodiment) The image processing device 3 of the third embodiment uses a tomographic image I1 in which no problem occurs to estimate the original tomographic image I1 at a position corresponding to the tomographic image I1 in which a problem occurs, and calculates data indicating anatomical features from the estimated tomographic image I1.

[0065] The configurations of the image diagnostic apparatus 100 and the image processing apparatus 3 in the third embodiment are the same except for some of the processing procedures by the image processing apparatus 3, so the same reference numerals are used and detailed description will be omitted.

[0066] 13 and 14 are flowcharts showing an example of an information processing procedure by the image processing device 3 in the third embodiment. Among the processing procedures shown in the flowcharts of Fig. 13 and 14, the same steps as those shown in the flowchart of Fig. 7 are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0067] In the third embodiment, the image processing device 3 inputs the tomographic image I1 to the first model 31M (S307), and stores the tag image IS obtained from the first model 31M in the storage unit 31 in association with the position on the long axis corresponding to the tomographic image I1 (step S378). The processing unit 30 calculates data based on the tag image IS (S308), stores the data in association with the position on the long axis (S309), and outputs the data (S310).

[0068] In the third embodiment, when it is determined in step S306 that a problem has occurred (S306: YES), the processing unit 30 of the image processing device 3 stores in the memory unit 31 (S314) characters or images indicating that a problem has occurred in the target tomographic image I1 without outputting them to the screen currently being displayed on the display device 4, and proceeds to step S311.

[0069] If it is determined in step S311 that scanning is complete (S311: YES), the processing unit 30 reads out the position of the tomographic image I1 for which it is stored that a problem has occurred (step S331). The processing unit 30 reads out the output (tag image IS) or the calculated contour when the tomographic image I1 at a position distal to or proximal to the read-out position, or at either one of the positions, is input to the first model 31M (step S332). The processing unit 30 estimates the output for the tomographic image I1 for which it is stored that a problem has occurred from the read-out output (step S333).

[0070] In step S333, the processing unit 30 estimates, as an output, a range intermediate between the distal tag image IS and the proximal tag image IS, as output as shown in the upper right of Fig. 4. The processing unit 30 may obtain a prediction based on the displacement of multiple outputs (ranges) only on the distal side, or based on the displacement of multiple outputs (ranges) only on the proximal side. In step S333, the processing unit 30 may obtain a contour intermediate between the distal side and the proximal side for the contour obtained based on the output from the first model 31M (Fig. 5) (see Fig. 15).

[0071] The processing unit 30 executes a process of inferring data indicating anatomical features at the position of the tomographic image I1 where the occurrence of a problem is stored, from the output inferred in step S333 (step S334). In step S334, the processing unit 30 infers data from the inferred output, as shown in FIG.

[0072] The processing unit 30 complements the data distribution for the entire longitudinal direction of the scanned blood vessel with the estimated data (step S335) and displays the distribution (S312). The processing unit 30 superimposes an image that highlights the complemented portion of the distribution so that the range of the estimated values ​​can be visually recognized (step S336), and then ends the processing.

[0073] Fig. 15 is a schematic diagram of another example of a method for estimating data indicating anatomical features. In the example of Fig. 15, the processing unit 30 estimates the output (range) of a tomographic image I1 that could not be input to the first model 31M due to a problem, from the output of the tomographic image I1 on the distal side and / or the proximal side. In the example shown in Fig. 15, the processing unit 30 estimates the intermediate (average) between the output for the tomographic image I1 on the one distal side and the output for the tomographic image I1 on the one proximal side as the output that would have been obtained by inputting the tomographic image I1 with the problem into the first model 31M.

[0074] FIG. 16 is a schematic diagram of another example of a method for estimating data indicating anatomical features. In the example of FIG. 16, the processing unit 30 may estimate a contour for a tomographic image I1 that could not be input to the first model 31M due to a problem, from contours calculated for the tomographic image I1 on the distal side and / or the proximal side. In this case, the processing of output from the first model 31M in step S378 may be omitted. In the example shown in FIG. 15, the processing unit 30 estimates an intermediate contour between the contour obtained from the tomographic image I1 on the distal side and the contour obtained from the tomographic image I1 on the proximal side as the contour that should have been obtained from the tomographic image I1 on the problematic side.

[0075] In the above description, the image processing device 3 executes the process of step S323 after it is determined in step S311 that scanning has been completed. However, the timing for executing the data estimation process of step S323 is not limited to this. For example, when it is determined that a problem has occurred, the image processing device 3 may execute data estimation using a tomographic image I1 (distal-side tomographic image I1) generated before that time. Alternatively, the image processing device 3 may execute data estimation when it generates (acquires) a tomographic image I1 that is determined not to have a problem after it has determined that a problem has occurred.

[0076] An example of the content of the screen 400 output in the third embodiment is the same as that described in the second embodiment with reference to Fig. 12. In the third embodiment, the screen 400 also outputs plots corresponding to estimated data in graphs 402 and 403, and also displays a superimposed image 405 of a frame indicating that the plots are estimated values.

[0077] This allows the examination operator and other medical personnel to understand that a problem occurred in the tomographic image I1 at the position on the long axis where the image 405 is displayed, and that data indicating anatomical features could not be calculated. By being able to recognize the location of the problem, the examination operator can decide whether to re-acquire the information or can check the relevant area in detail using another modality (such as an angiography device) that is being used in combination. In the third embodiment, too, it can be recognized that the data for the portion highlighted in the image 405 is estimated data that could not be calculated. Medical personnel can use the estimated data as a reference.

[0078] (Fourth embodiment) In the fourth embodiment, a tomographic image I1 determined to have a problem is input and image transformation (GAN) is performed to generate a pseudo tomographic image I3 that would have been generated without any problems, and this is input to the first model 31M.

[0079] The configurations of the image diagnostic apparatus 100 and the image processing apparatus 3 in the fourth embodiment are the same except for some of the processing contents by the image processing apparatus 3, so the same reference numerals are used and detailed description will be omitted.

[0080] FIG. 17 is a schematic diagram of the image generation model 33M. The processing unit 30 in the fourth embodiment generates a pseudo tomographic image I3 that would have been generated if a problem had not occurred, using the image generation model 33M. The image generation model 33M is configured based on a convolutional neural network to output an image by appropriately combining networks such as a transposed convolutional layer, a convolutional layer, and upsampling. The image generation model 33M is trained to output a pseudo tomographic image I3 when seed data (any data such as data called a latent variable, an image, or text data) is input. The seed data may be output from the first model 31M (the probability or type of whether a problem has occurred). The image generation model 33M may be an autoencoder trained to output a pseudo tomographic image I3 at a target position when an image proximal or distal to the tomographic image I1 in which a problem has occurred is input.

[0081] The image generation model 33M is trained by configuring a GAN (Generative Adversarial Network) together with a discrimination model 34M that is trained to distinguish between a pseudo tomographic image I3 generated by the image generation model 33M and a tomographic image I1 generated without any problems. The discrimination model 34M may be configured to include multiple stages of convolution layers defined by trained parameters, or may further include a pooling layer, a fully connected layer, etc.

[0082] Image generation model 33M is trained until it is determined that the discrimination accuracy of discriminant model 34M is reduced to half, i.e., until it is determined that pseudo tomographic image I3 derived from image generation model 33M can no longer be accurately discriminated from tomographic image I1. This makes it possible to generate pseudo tomographic image I3 that does not affect tomographic image I1 and data indicating anatomical features.

[0083] The following describes a process of complementing data indicating anatomical features for the tomographic image I1 in which a problem has occurred, using the image generation model 33M.

[0084] 18 and 19 are flowcharts showing an example of an information processing procedure by the image processing device 3 in the fourth embodiment. Among the processing procedures shown in Fig. 18 and 19, steps common to the processing procedures shown in the flowchart of Fig. 7 are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0085] In the fourth embodiment, when it is determined in step S306 that a problem has occurred (S306: YES), the processing unit 30 of the image processing device 3 stores in the storage unit 31 (S314) text or an image indicating that a problem has occurred in the target tomographic image I1 without outputting it to the screen currently being displayed on the display device 4. The processing unit 30 acquires seed data for the image generation model 33M, such as probability data output from the first model 31M for the target tomographic image I1 or a previously generated tomographic image I1 without any problem (step S341). The processing unit 30 inputs the acquired data to the image generation model 33M (step S342) and acquires a pseudo tomographic image I3 from the image generation model 33M (step S343). The processing unit 30 inputs the pseudo tomographic image I3 to the first model 31M (step S344), and the process proceeds to step S308.

[0086] In the fourth embodiment, the processing unit 30 of the image processing device 3 determines that scanning is complete (S311: YES), and redisplays the distribution of data for the entire longitudinal direction of the scanned blood vessel (S312). The processing unit 30 superimposes an image emphasizing that the data is an estimated value in the range of the displayed distribution corresponding to the position of the tomographic image I1 where a problem has been stored (step S345), and ends the processing.

[0087] An example of the content of the screen 400 output in the fourth embodiment is the same as that described in the second embodiment with reference to Fig. 12. In the fourth embodiment, the screen 400 also displays plots of data indicating anatomical features calculated from the pseudo tomographic image I3 in graphs 402 and 403, and also displays a frame image 405 superimposed thereon to indicate that the plots are estimated values.

[0088] According to the fourth embodiment, even if a problem occurs partially during scanning of a blood vessel, causing the tomographic image I1 to appear almost entirely dark, data indicating the anatomical features of the area where the problem occurred can be supplemented. Furthermore, at the position on the long axis where the image 405 is displayed, the problem occurs in the tomographic image I1, and the examination operator and other medical personnel can see that this is an estimated value. Medical personnel can use the estimated data as a reference.

[0089] In the first to fourth embodiments, the first model 31M, the second model 32M, and the image generation model 33M have been described as being stored in and used by the storage unit 31 of the image processing device 3. However, the image processing device 3 may be configured to be able to use any of the first model 31M, the second model 32M, and the image generation model 33M via a local network or an external network.

[0090] In the first to fourth embodiments, the image processing device 3 connected to the catheter 1 generates a tomographic image I1 in almost real time based on a signal from the imaging device 11, calculates data indicating anatomical features for a tomographic image without any problems, and displays the data on the display device 4. However, the processing by the image processing device 3 described above may be performed after the fact on a previously generated tomographic image I1. In other words, the image processing device 3 is not necessarily directly connected to the imaging device 11 of the catheter 1, as long as it can acquire a signal from the imaging device 11. The image processing device 3 may be a device, such as a server device, that can read a storage device that stores a signal from the imaging device 11 via a network. In other words, the processing procedure of steps S301-S304 shown in the flowchart of FIG. 7 may be performed by an existing processing device, and the processing of steps S305-S314 may be performed by an image processing device 3 connected to the existing processing device, and the data may be displayed on the display device 4 via the processing device.

[0091] In the first to fourth embodiments, medical images have been described using IVUS images of coronary arteries as examples. However, the application is not limited to this, and OCT / OFDI and the like may also be used, and the hollow organ is not limited to blood vessels.

[0092] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0093] 1 catheter 11 Imaging Devices 3. Image processing device (information processing device) 30 Processing section 31 Storage section 3P Computer Program 31M 1st Model 32M 2nd Model 33M image generation model 4 Display device 400 screens 401 cursor 402,403 graphs 404,405 images I1 Tomographic image I3 Pseudo-tomographic image

Claims

1. a computer that acquires a plurality of tomographic images of a hollow organ based on signals detected by an imaging device provided in a catheter inserted into the hollow organ; calculating data indicating anatomical features of the hollow organ for each of a plurality of tomographic images among the plurality of tomographic images that are determined to have no problems in detection by the imaging device; displaying a distribution of data indicating the anatomical features relative to a longitudinal axis of the luminal organ; On the displayed distribution, an object indicating a location corresponding to the tomographic image where a problem is determined to have occurred is displayed. A computer program that executes a process.

2. The computer, Inferring data indicating anatomical features corresponding to a tomographic image determined to have a problem from a tomographic image on the distal or proximal side of the long axis of the tubular organ on which no problem has occurred relative to the tomographic image, Interpolating the distribution of the data using the estimated data The computer program product of claim 1 , which executes a process.

3. The computer, On the distribution of the data, an interpolated portion and a non-interpolated portion of the distribution are displayed in a distinguishable manner.

3. The computer program product according to claim 2, which executes a process.

4. The computer, The data indicating the anatomical features corresponding to the tomographic image determined to have a problem is estimated to be continuous with the data calculated for the tomographic image on the distal side or the proximal side without the problem.

4. A computer program according to claim 1, which causes a process to be executed.

5. The computer, As a process of estimating data indicating anatomical features corresponding to a tomographic image determined to have a problem, Acquire data in which a tomographic image without any problem on the distal or proximal side of the tomographic image is divided into different ranges including the lumen and membrane of the hollow organ; From the data, a division of the range in the tomographic image in which it is determined that a problem has occurred is estimated; Data indicating anatomical features corresponding to the tomographic image in which the problem is determined to occur is estimated from the shape or size of each estimated range.

4. A computer program according to claim 1, which causes a process to be executed.

6. The computer, An image generation model is used that has been trained to output a pseudo-tomographic image that is determined to be free of problems when any data is input. For a tomographic image determined to have a problem, input data corresponding to the tomographic image into the image generation model to generate a pseudo tomographic image; Calculating data indicating anatomical features of the hollow organ for the generated pseudo-tomographic image.

4. A computer program according to claim 1, which causes a process to be executed.

7. The computer, a first model that, when a tomographic image is input, outputs data that is divided into different ranges including the lumen and membrane of the hollow organ in the tomographic image; Each of the plurality of tomographic images is input to the first model, and data indicating the anatomical characteristics of the hollow organ is calculated based on the dimensions or shapes of the divided ranges, based on the data output from the first model.

7. A computer program product according to claim 1, which causes a process to be executed.

8. The computer, using a second model that has been trained to output data indicating whether or not a problem has occurred in a tomographic image when the tomographic image is input; The computer Each of the plurality of tomographic images is input to the second model, and it is determined whether a problem has occurred based on an output from the second model. A computer program according to any one of claims 1 to 7, which causes a process to be executed.

9. a computer that acquires a plurality of tomographic images of a hollow organ based on signals detected by an imaging device provided in a catheter inserted into the hollow organ, calculating data indicating anatomical features of the hollow organ for each of a plurality of tomographic images among the plurality of tomographic images that are determined to have no problems in detection by the imaging device; displaying a distribution of data indicating the anatomical features relative to a longitudinal axis of the luminal organ; On the displayed distribution, an object indicating a location corresponding to the tomographic image where a problem is determined to have occurred is displayed. Information processing methods.

10. 1. An information processing device that acquires a plurality of tomographic images of a hollow organ based on signals detected by an imaging device provided in a catheter inserted into the hollow organ, a processing unit that performs image processing on each of the plurality of tomographic images, The processing unit calculating data indicating anatomical features of the hollow organ for each of a plurality of tomographic images among the plurality of tomographic images that are determined to have no problems in detection by the imaging device; displaying a distribution of data indicating the anatomical features relative to a longitudinal axis of the luminal organ; On the displayed distribution, an object indicating a location corresponding to the tomographic image where a problem is determined to have occurred is displayed. Information processing device.

Images