Computer program, information processing method, and information processing apparatus

Abstract

Provided are a model generation method for a model that outputs data relating to an angle for capturing an anatomical feature of a luminal organ, a learned learning model, a computer program using the learning model, an information processing method, and an information processing device.SOLUTION: In the model generation method, a computer generates a learning model that outputs an existence probability of a lesion area or a medical instrument for each rotation angle of a catheter when an image of a luminal organ is input. The trained model is used in a computer together with a model that outputs data in which an image of a hollow organ is classified into different regions including a lumen and a tunica of the hollow organ, and is used in a process of calculating an angle range in which a lesion area or a medical instrument is captured from the center of the hollow organ.SELECTED DRAWING: Figure 13B

Description

[Technical Field]

[0001] The present invention relates to a model generation method for a model used to capture anatomical features of a hollow organ from a medical image of the hollow organ, a trained learning model, a computer program using the learning model, an information processing method, and an information processing device. [Background technology]

[0002] Diagnostic imaging of vascular organs, particularly vascular organs, is essential for the safe and reliable performance of procedures such as percutaneous coronary intervention (PCI). For this reason, in addition to angiography, which uses contrast media to capture images from outside the body, intravascular imaging techniques such as catheter-based IVUS (Intravascular Ultrasound), OCT (Optical Coherence Tomography) / OFDI (Optical Frequency Domain Imaging) are becoming increasingly common.

[0003] In the above-mentioned image diagnosis, it is not easy to obtain accurate diagnostic information from the captured medical images. To assist in the interpretation of medical images, various techniques have been proposed that use image analysis or machine learning to correct images or add information (Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-075702 Summary of the Invention [Problem to be solved by the invention]

[0005] Interpretation of medical images requires accurate understanding of anatomical features from the images. Therefore, it is necessary to capture medical images accurately and present information about anatomical features in a more easily understandable and visible manner. In particular, for procedures and examinations performed using catheters, data on the angle at which the target object is located from the catheter insertion point is useful.

[0006] The object of the present disclosure is to provide a model generation method for a model that outputs data regarding angles for capturing anatomical features of tubular organs based on images obtained using a catheter, a trained learning model, a computer program that uses the learning model, an information processing method, and an information processing device. [Means for solving the problem]

[0007] The model generation method according to the present disclosure involves a computer acquiring scanning signals from an imaging device attached to a catheter that is inserted into a tubular organ and moves longitudinally while rotating around the longitudinal axis of the tubular organ, and generating a learning model that outputs the probability of the presence of a lesion or medical device for each rotation angle of the catheter when an image of the tubular organ based on the scanning signals is input.

[0008] The learning model according to the present disclosure comprises an input layer to which an image based on a scanning signal from an imaging device provided on a catheter that is inserted into a tubular organ and moves longitudinally while rotating around the longitudinal direction of the tubular organ as an axis is input; an output layer that outputs the probability of the presence of a lesion or medical device for each rotation angle of the catheter; and an intermediate layer that is trained based on training data including the images and data indicating the presence or absence of a lesion or medical device for each angle within the images, and causes a computer to function by providing an image based on the scanning signal from the imaging device to the input layer, performing calculations based on the intermediate layer, and outputting the presence probability corresponding to the image from the output layer.

[0009] The computer program of the present disclosure causes a computer to acquire scanning signals from an imaging device attached to a catheter that is inserted into a tubular organ and moves longitudinally while rotating around the longitudinal axis of the tubular organ, and when an image of the tubular organ based on the scanning signals is input, the computer uses a first model that outputs data identifying different regions in the image, including the lumen and membrane of the tubular organ, and a second model that, when the image is input, outputs the probability of the presence of a lesion or a medical device for each rotation angle of the catheter of the imaging device, and executes a process of calculating the angular range in which a lesion or a medical device can be captured from the center of the tubular organ, based on parameters indicating the anatomical features of the tubular organ based on the regions identified by the first model and the probability of presence for each rotation angle obtained from the second model.

[0010] The information processing method of the present disclosure involves a computer that acquires scanning signals from an imaging device attached to a catheter that is inserted into a tubular organ and moves longitudinally while rotating around the longitudinal axis of the tubular organ, using a first model that, when an image of the tubular organ based on the scanning signals is input, outputs data identifying different areas in the image, including the lumen and membrane of the tubular organ, and a second model that, when the image is input, outputs the probability of the presence of a lesion or a medical device for each rotation angle of the catheter of the imaging device, and calculates the angular range for capturing the lesion or medical device from the center of the tubular organ based on parameters indicating the anatomical features of the tubular organ based on the areas identified by the first model and the probability of presence for each rotation angle obtained from the second model.

[0011] The information processing device according to the present disclosure acquires scanning signals from an imaging device provided on a catheter that is inserted into a tubular organ and moves longitudinally while rotating around the longitudinal axis of the tubular organ, and includes a memory unit that stores a first model that, when an image of the tubular organ based on the scanning signals is input, outputs data identifying different regions in the image, including the lumen and membrane of the tubular organ, and a second model that, when the image is input, outputs the probability of the presence of a lesion or a medical instrument for each rotation angle of the catheter of the imaging device, and a processing unit that performs image processing based on the scanning signals, and the processing unit calculates an angular range for capturing the lesion or medical instrument from the center of the tubular organ based on parameters indicating the anatomical features of the tubular organ based on the regions identified in the first model and the probability of presence for each rotation angle obtained from the second model. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an imaging diagnostic apparatus. [Figure 2] FIG. 10 is an explanatory diagram showing the operation of the catheter. [Figure 3] FIG. 1 is a block diagram showing a configuration of an image processing device. [Figure 4] This is an overview of the trained first model. [Figure 5] FIG. 10 is a diagram showing detected boundaries (contours). [Figure 6] FIG. 1 is a schematic diagram of a second model. [Figure 7] FIG. 10 is a diagram illustrating an overview of learning of a second model. [Figure 8] 10 is a flowchart illustrating an example of a process for generating a second model. [Figure 9] FIG. 10 is a diagram showing the deviation between the center of the catheter and the center of the blood vessel (center of gravity of the cross section). [Figure 10] 10 is a flowchart illustrating an example of a processing procedure performed by a processing unit of the image processing apparatus. [Figure 11] 10 is a flowchart illustrating an example of a processing procedure performed by a processing unit of the image processing apparatus. [Figure 12] 10 is a flowchart illustrating an example of a procedure for calculating an angle range. [Figure 13A] FIG. 10 is an explanatory diagram of a calculation process of an angle range. [Figure 13B] FIG. 10 is an explanatory diagram of a calculation process of an angle range. [Figure 14] FIG. 10 is an explanatory diagram of a correction process for a lumen boundary. [Figure 15] 1 shows an example of a screen displayed on a display device. [Figure 16] FIG. 10 is a diagram showing another example of a screen on the display device. [Figure 17] An example of calculating the angle for capturing a lesion occurring only on the luminal surface is shown below. [Figure 18] An example of calculating the angle at which calcified plaque is captured will be shown. [Figure 19] An example of calculating the angle at which attenuating plaque is captured is shown. [Figure 20] An example of calculating the angle at which a stent placed in a blood vessel is captured will be shown. [Figure 21] An example of calculating the angle at which a guide wire inserted together with a catheter is captured will be shown. DETAILED DESCRIPTION OF THE INVENTION

[0013] Specific examples of a computer program, an information processing method, and an information processing device according to embodiments of the present invention will be described below with reference to the drawings.

[0014] 1 is a diagram showing an example of the configuration of an imaging diagnostic device 100. The imaging diagnostic device 100 is a device that generates medical images including ultrasonic tomographic images of blood vessels (hollow organs) by the IVUS method and is used for intravascular ultrasound examination and diagnosis.

[0015] The diagnostic imaging apparatus 100 includes a catheter 1, an MDU (Motor Drive Unit) 2, an image processing device (information processing device) 3, a display device 4, and an input device 5.

[0016] The catheter 1 is a flexible tube for medical use. The catheter 1 is particularly known as an imaging catheter, which has an imaging device 11 at its tip and rotates in a circumferential direction by being driven from its base end. In the case of the IVUS method, the imaging device 11 is an ultrasound probe including an ultrasound transducer and an ultrasound sensor. In the case of OCT, the imaging device 11 is an OCT device including a near-infrared laser and a near-infrared sensor. The imaging device 11 may also be another device that uses electromagnetic waves of other wavelengths, such as visible light.

[0017] The MDU 2 is a driving unit attached to the base end of the catheter 1, and controls the operation of the catheter 1 by driving an internal motor in response to the operation of a medical professional.

[0018] The image processing device 3 generates a plurality of medical images, such as cross-sectional images of blood vessels, based on the signals output from the imaging device 11 of the catheter 1. The configuration of the image processing device 3 will be described in detail later.

[0019] The display device 4 uses a liquid crystal display panel, an organic EL display panel, etc. The display device 4 displays the medical image generated by the image processing device 3 and information related to the medical image.

[0020] The input device 5 is an input interface that accepts operations for the image processing device 3. The input device 5 may be a keyboard, a mouse, or the like, or may be a touch panel, soft keys, hard keys, or the like built into the display device 4.

[0021] Figure 2 is an explanatory diagram showing the operation of the catheter 1. In Figure 2, the catheter 1 is inserted into a tubular blood vessel L by a medical professional along a guidewire W inserted into a coronary artery shown in the figure. In the enlarged view of the blood vessel L in Figure 2, the right side corresponds to the distal side from the insertion point of the catheter 1 and the guidewire W, and the left side corresponds to the proximal side.

[0022] Driven by the MDU 2, the catheter 1 moves from the distal end to the proximal end within the blood vessel L as indicated by the arrow in the figure, while rotating around its axis in the longitudinal direction. As a result, the imaging device 11 scans the inside of the blood vessel L in a spiral manner.

[0023] In the diagnostic imaging apparatus 100 of this embodiment, the image processing device 3 acquires signals for each scan output from the imaging device 11 of the catheter 1. During each scan, the imaging device 11 emits a detection wave in the radial direction and detects the reflected wave. The imaging device 11 performs this scan tens to thousands of times for each angle while rotating 360 degrees. The image processing device 3 can acquire the radial distribution of the detected reflected wave for each angle. The image processing device 3 generates a tomographic image (cross-sectional image) (I1 in FIG. 2) by polar coordinate transformation (inverse transformation) for each 360 degrees of a rectangular image (I0 in FIG. 2) in which the signals for each scan are aligned in the radial direction and arranged in a rectangular shape. The tomographic image I1 is also called a frame image. The reference point (center) of the tomographic image I1 corresponds to the range of the catheter 1 (not imaged). The image processing device 3 can output information including an image showing the structure of the blood vessels based on the acquired tomographic image I1 so that a medical professional can visually recognize it. In particular, when a scanning signal corresponding to each angle or a tomographic image converted from the scanning signal is input, the diagnostic imaging device 100 of the present disclosure outputs a range (angle range) of a lesion present in a blood vessel or a medical device such as a stent as viewed from a side surface of a catheter. The angle range and the output method will be described in detail below.

[0024] 3 is a block diagram showing the configuration of the image processing device 3. The image processing device 3 is a computer, and includes a processing unit 30, a storage unit 31, and an input / output I / F 32.

[0025] The processing unit 30 includes one or more central processing units (CPUs), micro-processing units (MPUs), graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), tensor processing units (TPUs), etc. The processing unit 30 has a built-in non-transitory storage medium such as random access memory (RAM), and performs calculations based on a computer program 3P stored in the storage unit 31 while storing data generated during processing in the non-transitory storage medium.

[0026] The storage unit 31 is a non-volatile storage medium such as a hard disk, a flash memory, etc. The storage unit 31 stores a computer program 3P, setting data, etc. that are read by the processing unit 30. The storage unit 31 also stores a first model 31M and a second model 32M that have been trained.

[0027] The computer program 3P, the first model 31M, and the second model 32M may be copies of the computer program 9P, the first model 91M, and the second model 92M stored in a non-temporary storage medium 9 outside the device, which are read by the processing unit 30 via the input / output I / F 32. The computer program 3P, the learned first model 31M, and the learned second model 32M may be distributed by a remote server device, acquired by the image processing device 3 via a communication unit (not shown), and stored in the storage unit 31.

[0028] The input / output I / F 32 is an interface to which the catheter 1, the display device 4, and the input device 5 are connected. The processing unit 30 acquires a signal (digital data) output from the imaging device 11 via the input / output I / F 32. The processing unit 30 outputs screen data for a screen including the generated tomographic image I1 and / or long-axis image I2 to the display device 4 via the input / output I / F 32. The processing unit 30 accepts operation information input to the input device 5 via the input / output I / F 32.

[0029] FIG. 4 is a schematic diagram of a trained first model 31M. The first model 31M is a model trained to output an image showing the regions of one or more objects appearing in the tomographic image I1 when the tomographic image I1 obtained by polar coordinate conversion of a scanning signal is input. The first model 31M is, for example, a model that performs semantic segmentation. The first model 31M is trained to output a tag image IS in which each pixel of the input tomographic image I1 is tagged with the range of the object to which each pixel belongs.

[0030] The first model 31M uses, for example, a so-called U-net, which has a convolutional layer, a pooling layer, an upsampling layer, and a softmax layer symmetrically arranged, as shown in FIG. 4. When a tomographic image I1 is input, the first model 31M outputs a tag image IS indicating the area identified within the image. The output tag image IS is obtained by tagging the area of ​​the blood vessel lumen, the area of ​​the membrane corresponding to the area between the lumen boundary of the blood vessel and the blood vessel boundary, including the tunica media, the area showing the guidewire W and its reflection, the area corresponding to the catheter 1, and lesions (e.g., plaque, calcification), etc., with different pixel values ​​(shown in FIG. 4 with different types of hatching and solid color). The area of ​​the lesion may also include the area of ​​an artificial object, particularly a medical device, such as a stent.

[0031] As described above, the first model 31M is exemplified by semantic segmentation and U-net, but it goes without saying that the first model 31M is not limited to this. Alternatively, the first model 31M may be a model that realizes individual recognition processing using instance segmentation or the like. The first model 31M is not limited to being based on U-net, and may be a model based on SegNet, R-CNN, or an integrated model with other edge extraction processing.

[0032] The processing unit 30 can detect the edges of the lumen boundary and vascular boundary of the blood vessel inspected using the catheter 1 based on the pixel values ​​in the tag image IS obtained by inputting the tomographic image I1 into the first model 31M and the coordinates of the pixels in that image. Strictly speaking, the vascular boundary is the external elastic membrane (EEM) between the tunica media and adventitia of the blood vessel, and is shown relatively clearly and with low brightness in the tomographic image I1 using the IVUS method. Figure 5 shows the detected boundary (contour). Figure 5 shows the tomographic image I1 shown in Figure 4, with a curve B1 indicating the lumen boundary and a curve B2 indicating the vascular boundary superimposed on it.

[0033] The image processing device 3 in this embodiment stores a second model 32M in the storage unit 31 for use in conjunction with the first model 31M. The second model 32M is a model that outputs the presence or absence of a lesion or a medical instrument for each scanning signal, in association with angle information. FIG. 6 is a schematic diagram of the second model 32M. The second model 32M is a model using a neural network including an input layer 321, an intermediate layer 322, and an output layer 323. The input layer 321 inputs two-dimensional signal distribution, i.e., image data. The output layer 323 outputs the probability of the presence of a lesion or a medical instrument for each angle data (e.g., 1° to 360°). For example, the output layer 323 outputs an array of 180 probabilities, each 2° apart from 0°, 2°, 4°, ..., 356°, and 358°. The number of probabilities is not limited to 180, and may be, for example, 360, 120 at 3° intervals, 90 at 4° intervals, or more.

[0034] The processing unit 30 inputs the tomographic image I1 to the input layer 321 or inputs the rectangular image I0 to the input layer 321, and can obtain an array of output probabilities. The processing unit 30 obtains an array of probabilities of the presence of a lesion or medical instrument at each angle output from the second model 32M, and can obtain a continuous portion where the probability is equal to or greater than a predetermined value as the angle range of the lesion or medical instrument. Note that the probability of the presence of a lesion or medical instrument at each angle output from the second model 32M is the probability for each angle relative to the imaging device 11.

[0035] The second model 32M is created in advance by the image processing device 3 or another processing device and is considered to have been trained. Fig. 7 is a diagram showing an overview of the training of the second model 32M. In the following, it is assumed that the image processing device 3 executes the training process in advance, but this is merely an example and is not limiting. The training may also be performed in advance by another processing device.

[0036] The training data is an annotated tomographic image I1 or rectangular image I0. The annotations indicate the presence or absence of a lesion or a medical device, and the probability of its presence at each angle relative to the tomographic image I1 or rectangular image I0, whose location within the image is known. The training data may be prepared for each type of lesion or type of medical device. For example, the second model 32M may be trained separately using different training data for calcified plaque, attenuating plaque, a stent, or a guidewire.

[0037] In FIG. 7, the presence or absence of a lesion or medical device for each angle in the training data is indicated by marks arranged at the same distance from the tomographic image I1. The presence or absence of a lesion or medical device is indicated by hatching of the marks. The annotation is, for example, an array of probabilities associated with the tomographic image I1, with "1.0" assigned if a lesion or medical device is present and "0.0" assigned if it is not. In the example of FIG. 7, plaque is present in the range of 134° to 230°, and the probability of presence in that angle range is set to "1.0."

[0038] The processing unit 30 of the image processing device 3 learns and generates the second model 32M based on the training data of images to which such known existence probabilities have been assigned. Figure 8 is a flowchart showing an example of the process of generating the second model 32M.

[0039] The processing unit 30 acquires a previously generated tomographic image I1 or rectangular image I0 based on a signal obtained from the imaging device 11 (step S201). The processing unit 30 accepts the presence or absence of a lesion or a medical device for each angle based on the image center (center of the catheter 1) in the acquired image (step S202). The acceptance method may be realized by the processing unit 30 displaying the tomographic image I1 and a mark such as that shown in FIG. 7 on the display device 4 or the like, and controlling the mark to change to a color, pattern, etc. corresponding to "present (1.0)" or "absent (0.0)" based on the judgment of a medical professional (doctor, engineer, etc.). Alternatively, the acceptance method may be a method of accepting an arrangement of "present (1.0)" and "absent (0.0)" for each angle.

[0040] The processing unit 30 stores the image acquired in step S201 and training data including data indicating the presence or absence of the object for each angle received in step S202 for the image (step S203). It is desirable to perform the processes of steps S201 to S203 until as much data as possible is collected.

[0041] The tomographic image I1 or rectangular image I0 used as training data may be reduced before being used for learning. When inputting a rectangular image I0, it is desirable to use a rectangular image I0 of 540°, which is the target 360° of data plus 90° in front and behind (see Figure 6). By inputting an image that maintains continuity with adjacent scanning signals, the accuracy of learning can be maintained.

[0042] The processing unit 30 inputs the stored teacher data image to the input layer 321 of the second model 32M before the completion of learning (step S204). The processing unit 30 calculates a loss using the existence probability for each angle output from the output layer 323 of the second model 32M and data indicating the presence or absence of each accuracy corresponding to the input image, and thereby learns (updates) the parameters of the intermediate layer 322 (step S205).

[0043] The processing unit 30 determines whether the learning conditions are met (step S206), and if it is determined that the learning conditions are not met (S206: NO), the processing returns to step S201 and continues learning.

[0044] If it is determined that the learning conditions are met (S206: YES), the processing unit 30 stores description data indicating the network configuration and conditions of the second model 32M and the parameters of the intermediate layer 322 in the storage unit 31 or another storage medium (step S207), and ends the model generation process. Note that the processing unit 30 may accept the processes of steps S201-S203 in advance, and then execute the processes of steps S204-S207 for the collected teacher data.

[0045] As a result, when an image is input, the second model 32M is generated so as to output a circumferential probability distribution indicating the presence or absence of a lesion or a medical instrument for each angle relative to the imaging device 11. It should be noted here that the probability indicating the presence or absence of a lesion or a medical instrument for each angle output from the generated second model 32M is the circumferential angle range in which a lesion or the like is captured with the catheter 1 at the center. As shown in FIG. 7, the circumferential probability distribution output from the second model 32M is based on the center of the tomographic image I1, i.e., the center of the catheter 1.

[0046] However, the position of the catheter 1 within the blood vessel is not always constant; it passes through the center of the blood vessel or the edge of the blood vessel lumen due to the influence of pulsation. FIG. 9 is a diagram showing the deviation between the center of the catheter 1 and the center of the blood vessel (the center of gravity of the cross-section). As shown in FIG. 9, the center of the tomographic image I1, indicated by the symbol x (one long side of the rectangular image I0 on the catheter 1 side), is deviated from the center of the blood vessel, indicated by the symbol o in FIG. 9. Therefore, when the catheter 1 is deviated from the center of the blood vessel, data capturing a lesion with the catheter 1 as its center is dissociated from the angle information from the center of the blood vessel that should be considered clinically, making it difficult to refer to this data. Therefore, the image processing device 3 in this embodiment uses the first model 31M to recalculate the angle at which an object such as a lesion or a medical instrument centered on the catheter 1, which is derived based on the second model 32M, into data redescribed using an angle based on the anatomical features of the blood vessel.

[0047] The processing procedure by the image processing device 3 using the first model 31M and the second model 32M will be described in detail below. Figures 10 and 11 are flowcharts showing an example of the processing procedure by the processing unit 30 of the image processing device 3. When a signal is output from the imaging device 11 of the catheter 1, the processing unit 30 of the image processing device 3 starts the following processing.

[0048] Each time the processing unit 30 acquires a predetermined amount (e.g., 360 degrees) of signals (data) from the imaging device 11 of the catheter 1 (step S101), it performs polar coordinate conversion (inverse conversion) on an image in which radial signals are arranged in a rectangle to generate a tomographic image I1 (step S102) (see FIG. 2). The processing unit 30 outputs the generated tomographic image I1 so that it can be displayed in real time on the screen of the display device 4 (step S103). The processing unit 30 stores the signal data acquired in step S101 and the tomographic image I1 in the memory unit 31 in association with the position (position on the long axis, angle) of the imaging device 11 (step S104).

[0049] The processing unit 30 inputs the tomographic image I1 to the first model 31M (step S105). Based on the tag image IS obtained from the first model 31M, the processing unit 30 calculates data on the lumen boundary and the vascular boundary in the tomographic image I1 (step S106). In step S106, the processing unit 30 calculates the lumen range, the inner membrane range including the tunica media of the blood vessel, and the contour (edge) of the lumen range as the lumen boundary, and the outer contour of the membrane range as the vascular boundary, output from the first model 31M. In step S106, the processing unit 30 may implement high-speed processing, such as reducing the size of the tomographic image I1 before inputting it to the first model 31M.

[0050] The processing unit 30 calculates the center of the blood vessel based on the data of the lumen boundary and the blood vessel boundary calculated in step S106 (step S107). In step S107, the processing unit 30 extracts a circle along the blood vessel boundary or the lumen boundary, for example, by Hough transform, and determines the center of the circle as the center of the blood vessel. In step S107, the processing unit 30 may obtain the center of gravity of the region inside the blood vessel boundary or the region inside the lumen boundary, and determine this center of gravity as the blood vessel center. The processing unit 30 stores the coordinates of the calculated blood vessel center in the tomographic image I1 (step S108).

[0051] The processing unit 30 inputs the tomographic image I1 or the rectangular image I0 (see FIGS. 2 and 9) obtained by adding the scanning results for 90° forward and backward from the signal data acquired in step S101 to the second model 32M (step S109). The processing unit 30 obtains the existence probability of the lesion or the medical instrument for each angle obtained from the second model 32M (step S110).

[0052] The processing unit 30 executes a process of calculating an angle range in which a lesion or a medical instrument exists, based on the blood vessel center, based on the existence probability of the lesion or the medical instrument for each angle acquired in step S110 and the coordinates of the blood vessel center stored in step S108 (step S111). The process in step S111 will be described later.

[0053] The processing unit 30 superimposes an image indicating the angle range in which the lesion or medical instrument is present obtained as a result of the calculation process in step S111 on the tomographic image I1 (step S112). The processing unit 30 stores the data on the angle range in which the lesion or medical instrument is present in association with the tomographic image I1 stored in step S104 (step S113).

[0054] The processing unit 30 corrects the range of the lesion or medical instrument identified based on the tag image IS obtained in step S105 based on the angle range calculated in step S111 (step S114). In step S114, the processing unit 30 may, for example, leave the range included in the angle range and remove the other range as noise.

[0055] The processing unit 30 causes the display device 4 to display data indicating the position of the target's tomographic image I1 in the long axis direction (step S115). In step S115, the processing unit 30 causes the display device 4 to display the position of the target's tomographic image I1 in real time during scanning. In this case, when the processing unit 30 has associated the target's tomographic image I1 with an angle range in which a lesion or a medical instrument exists by the processing of step S111, it is preferable that the processing unit 30 superimposes a mark or the like indicating this on the image indicating the long axis direction.

[0056] The processing unit 30 determines whether scanning by the imaging device 11 of the catheter 1 has been completed (step S116). If it is determined that scanning has not been completed (S116: NO), the processing unit 30 returns the process to step S101 and generates the next tomographic image I1.

[0057] If it is determined that the scanning is completed (S116: YES), the processing unit 30 ends the processing.

[0058] 12 is a flowchart illustrating an example of a calculation process for an angle range. The process shown in the flowchart of FIG. 12 corresponds to the detailed procedure of step S111 in the flowcharts of FIGS.

[0059] The processing unit 30 calculates, from the existence probability for each angle obtained from the second model 32M, a range where the existence probability is continuously equal to or greater than a predetermined probability value as a range where a lesion or a medical instrument exists (step S301).

[0060] The processing unit 30 refers to pixel values ​​in the tomographic image I1 and extracts areas between the blood vessel boundary and the lumen boundary where pixel values ​​are higher (or lower) than others (step S302). That is, in step S302, the processing unit 30 extracts areas in the image that are brighter than the surrounding area or darker areas with lower brightness. In step S302, the processing unit 30 may also extract areas inside the lumen boundary (because there may be lesions or medical devices that extend into the lumen). Furthermore, the processing of step S302 may be omitted because extraction is not possible depending on the type of lesion.

[0061] The processing unit 30 adjusts the angle range from the center of the tomographic image I1 (the center of the catheter 1) of the range where the lesion or medical device is present, based on the range calculated in step S301 and the portion extracted in step S302 (step S303). In step S303, the processing unit 30 may exclude a portion of the portion extracted in step S302 that is outside the range calculated in step S301. When the range calculated in step S301 and the portion extracted in step S302 overlap, the processing unit 30 may adjust the angle range to a wider range (a range where OR is satisfied) or to a narrower range (only a range where AND is satisfied). The processing of steps S302 and S303 is not essential.

[0062] The processing unit 30 calculates the coordinates within the tomographic image I1 of the intersections between the line with the maximum angle and the line with the minimum angle in the angle range from the center of the tomographic image I1 where there is a high probability that a lesion or a medical device exists, and the calculated lumen boundary (or blood vessel boundary) (step S304).

[0063] The processing unit 30 calculates and stores the angles of the lines connecting the coordinates of the two points calculated in step S301 and the coordinates of the blood vessel center (step S305). In step S305, the processing unit 30 calculates the angles between each line and a line segment extending vertically upward from the center of the tomographic image I1.

[0064] Based on the coordinates of the two points stored in step S305, the processing unit 30 connects the intersection of the line and the lumen boundary or the blood vessel boundary with a spline curve, a circular arc, or the like, to correct the lumen boundary or the blood vessel boundary (step S306).

[0065] The processing unit 30 ends the calculation process of the angle range based on the blood vessel center, and returns the process to step S112 in the flowcharts of FIGS.

[0066] The processing unit 30 may execute the processing of step S302 of the processing procedure shown in the flowchart of Fig. 12 after the processing of step S305. In this case, the processing unit 30 executes the processing as follows. The processing unit 30 calculates the intersection of two straight lines extending from the center of the tomographic image I1 to both ends of the angle range with the lumen boundary or the vascular boundary (S304), and calculates the angle of the line connecting the intersection with the vascular center (S305). The processing unit 30 extracts a portion having a higher pixel value (whiter than the surrounding area) within a sector shape formed by at least two straight lines with the angle calculated in step S305 and the vascular boundary or the lumen boundary (S302). The processing unit 30 may change the angle of the two straight lines relative to the vascular center according to the size and shape of the extracted portion.

[0067] 13A and 13B are explanatory diagrams of the calculation process of the angle range. Fig. 13A shows the angle range from the center of the tomographic image I1 shown in Fig. 9 where the existence probability is continuously equal to or greater than a predetermined probability value. Fig. 13A shows a portion (plaque) where the pixel value is higher than others between the curve B2 of the vascular boundary and the curve B1 of the lumen boundary extracted in step S302. The processing unit 30 may make adjustments in step S303 so that a straight line tangent to this extracted range is tangent, as shown by the dashed line.

[0068] In Figure 13A, the intersections of the line with the maximum angle and the line with the minimum angle in the angle range from the center of the tomographic image I1 where there is a high probability of a lesion or medical device being present, and the calculated lumen boundary curve B1 and vascular boundary curve B2 are shown as black circles.

[0069] FIG. 13B shows the results of recalculating the angle range from the image center of the tomographic image I1 shown in FIG. 13A as the angle range from the blood vessel center. In FIG. 13B, the angle range from the image center of the tomographic image I1 is indicated by a dashed line. By changing the angle calculation reference from the image center to the blood vessel center, for example, the range in which a lesion or medical device exists, calculated as a range of 148° to 215° based on the probability distribution obtained from the second model 32M, is recalculated as 134° to 223°. This makes it possible to more accurately calculate the angle at which the lesion is captured based on data indicating anatomical features. Furthermore, data on the angle range based on the blood vessel center makes it possible to reproduce, based on signals from the imaging device 11, a three-dimensional image that more accurately shows the structure of the blood vessel and the lesion in the blood vessel. The processing unit 30 can also recalculate the blood vessel boundary and lumen boundary based on data on the angle range based on the blood vessel center.

[0070] FIG. 14 is an explanatory diagram of the correction process for the lumen boundary. FIG. 14A shows the lumen boundary curve B1 before correction, FIG. 14B shows the process during correction, and FIG. 14C shows the lumen boundary curve B1 after correction. The lumen boundary curve B1 is calculated as the boundary of a region as a result of performing segmentation on the tomographic image I1 (or rectangular image I0) using the first model 31M. However, depending on the vascular lesion or the like captured in the tomographic image I1, the pixel value may be low, making it difficult to distinguish the region, and the accuracy of the calculated boundary may be low. In FIG. 14A, the calculated lumen boundary curve B1 appears to have an ambiguous portion recessed toward the center due to the influence of the range of the lesion.

[0071] 14B shows the intersection points between the lumen boundary and a line indicating the angle range where there is a high probability that a lesion or a medical device exists for the image shown in FIG. 14A. The processing unit 30 deletes the lumen boundary between the two intersection points.

[0072] FIG. 14C shows the lumen boundary after correction. The curve B1 of the lumen boundary after correction shown in FIG. 14 is a curve connected by a spline curve between the intersection points shown in FIG. 14B to create a natural lumen boundary. This allows a natural-connected boundary to be drawn even in areas where pixel values ​​are low due to the presence of a lesion or medical device. Because blood vessels are elastic and membranes should have smoothly curved surfaces, and therefore boundaries, curves connected by a spline curve may have higher reproducibility. The correction process shown in FIG. 14 can be applied not only to lumen boundaries but also to vascular boundaries.

[0073] The curve B1 of the lumen boundary shown in Fig. 14C may be corrected not only to a spline curve but also to an arc of a circle centered at the specified vascular center o. Blood vessels have elasticity, and it is natural for their cross sections to be approximately circular.

[0074] Fig. 15 shows an example of a screen 400 displayed on the display device 4. The screen 400 shown in Fig. 15 includes a cursor 401 for selecting a position in the longitudinal direction of the blood vessel, and a tomographic image I1 at the position corresponding to the cursor 401. The screen 400 also includes a graph 402 of data indicating anatomical features. The graph 402 shows the distribution of the mean lumen diameter and the distribution of the plaque extent ratio with respect to the position on the longitudinal axis.

[0075] 15 shows an image (black circles, straight lines, and dashed lines) representing the angular range in which the lesion or medical device is located, relative to the center of the blood vessel. Below the tomographic image I1 on the screen 400, a numerical value indicating the angular range is displayed together with data indicating the anatomical features calculated from the tomographic image I1 at that position.

[0076] 15, the angle at which the lesion or medical instrument is captured can be rewritten as an angle based on the center of the blood vessel, and the result can be visually confirmed by the medical staff on the screen 400. This allows the medical staff to visualize the extent of the lesion when inserting a catheter for treatment.

[0077] Fig. 16 is a diagram showing another example of a screen on the display device 4. A screen 400 in Fig. 16 includes a three-dimensional image 403 of a blood vessel, and includes a three-dimensional cursor 401, which is an object that can be moved along the longitudinal axis, on the three-dimensional image 403. As indicated by the symbol E in Fig. 16, an image that connects, in the longitudinal direction, the angle ranges that capture the lesion identified for each tomographic image I1 is superimposed and displayed on the three-dimensional image 403. This makes it easier for medical personnel to stereoscopically capture the lesion or medical instrument within the blood vessel.

[0078] We will now explain examples where the above-mentioned processing procedure is applied to different types of lesions and medical devices. Figure 17 shows an example of calculating the angle for capturing a lesion that occurs only on the lumen surface. Figure 17A shows the lumen boundary curve B1 and the vascular boundary curve B2 obtained by inputting a tomographic image I1 showing dissection and protrusion into a first model 31M. Figure 17B shows the angular range in which a lesion or medical device exists, based on the image center obtained by inputting the tomographic image I1 into a second model 32M. Figure 17C shows the angular range based on the vascular center.

[0079] Dissection and protrusion occur only on the surface of the lumen. Therefore, in this case, partial extraction based on pixel values ​​within the range referencing the vascular boundary and lumen boundary by the process of S302 is not necessary. As shown in Figures 17A to 17C, the processing unit 30 calculates the intersections of two lines from the center of the tomographic image I1 to both ends of the angle range with the lumen boundary curve B1 or the vascular boundary curve B2, without performing extraction processing based on pixel values, and calculates the angle range from the vascular center by finding a line passing through these intersections and the vascular center. This makes it possible to obtain an angle that captures lesions occurring only on the lumen surface.

[0080] Figure 18 shows an example of calculating the angle at which calcified plaque is captured. Figure 18A shows the lumen boundary curve B1 and the blood vessel boundary curve B2 obtained by inputting a tomographic image I1 showing calcified plaque into a first model 31M. Figure 18B shows the angle range in which a lesion or medical device exists, based on the image center obtained by inputting the tomographic image I1 into a second model 32M. Figure 18C shows the angle range based on the blood vessel center.

[0081] Calcified plaque is found between the intima and EEM of a blood vessel. Therefore, in this case, the lesion is found between the lumen boundary and the vascular boundary. Regarding plaque, the processing unit 30 extracts a region corresponding to plaque on the tomographic image I1 based on pixel values ​​between the lumen boundary and the vascular boundary, which is a range where the probability of the lesion existence calculated in step S301 is equal to or greater than a predetermined probability value (S302). The processing unit 30 calculates the angle range from the center of the blood vessel that covers the extracted range, as shown in Figures 18B and 18C. This makes it possible to obtain an angle that covers the lesion occurring between the surface of the intima and the vascular boundary (external elastic lamina).

[0082] Correcting the vascular boundary is also effective in detecting calcified plaque. Figure 18C shows a corrected boundary obtained by correcting the vascular boundary using a thick arc. Calcified plaque does not transmit ultrasound, and ultrasound from the imaging device 11 may not reach the vascular boundary beyond the plaque. In this case, the area outside the calcified plaque becomes a dark area with significantly lower pixel values, reducing the accuracy of segmentation by the first model 31M. Therefore, as shown in Figure 19C, it is effective to correct the two points at which two straight lines from the image center of the tomographic image I1 to both ends of the angle range intersect with the vascular boundary curve B2 by connecting them with an arc of a circle around the vascular center or by connecting them with a spline curve. In this way, the data on the angle range capturing the lesion or medical device obtained from the second model 32M can also be used for boundary correction.

[0083] Figure 19 shows an example of calculating the angle at which attenuating plaque is captured. Figure 19A shows the curve B2 of the vascular boundary obtained by inputting a tomographic image I1 showing attenuating plaque into a first model 31M. Figure 19B shows the angular range in which a lesion or medical device exists, based on the image center obtained by inputting the tomographic image I1 into a second model 32M. Figure 19C shows the angular range based on the blood vessel center.

[0084] Attenuating plaques appear dark because they attenuate ultrasound. In this case, the pixel values ​​are low, resulting in low accuracy in region identification and boundary calculation. Regarding attenuating plaques, the processing unit 30 extracts a region corresponding to plaques with low pixel values ​​on the tomographic image I1 within a region inside the vascular boundary, where the probability of the presence of a lesion calculated in step S301 is equal to or greater than a predetermined probability value (S302). The processing unit 30 calculates the angular range from the center of the blood vessel that captures the extracted region, as shown in FIGS. 19B and 19C. The boundary can then be corrected (S306).

[0085] Fig. 20 shows an example of calculating the angle at which a stent placed in a blood vessel is captured. Fig. 20A shows the lumen boundary curve B1 and the blood vessel boundary curve B2 obtained by inputting a tomographic image I1 showing the stent into a first model 31M. Fig. 20B shows the angle range in which a lesion or medical device exists, based on the image center obtained by inputting the tomographic image I1 into a second model 32M. Fig. 20C shows the angle range based on the blood vessel center.

[0086] Stents may exist inside the lumen or may be incorporated into plaque that has grown from the lumen and reside between the lumen boundary and the vascular boundary. Stents reflect ultrasound waves and appear bright and highly luminous. Because stents are mesh-like, the continuity of the range in which the probability of lesion presence is equal to or greater than a predetermined probability value is not an issue. Regarding the stent, the processing unit 30 extracts a region with high pixel values ​​on the tomographic image I1 within the region inside the vascular boundary, which is the range in which the probability of lesion presence calculated in step S301 is equal to or greater than a predetermined probability value (S302). The processing unit 30 may remove noise based on the size of the extracted region from the cross-sectional size of the stent. As shown in Figures 20B and 20C, the processing unit 30 calculates the angular range from the center of the blood vessel to the mesh portion of each stent, capturing the extracted range.

[0087] Fig. 21 shows an example of calculating the angle at which the guidewire W inserted together with the catheter 1 is captured. Fig. 21A shows the curve B1 of the lumen boundary obtained by inputting a tomographic image I1 showing the guidewire W into a first model 31M. Fig. 21B shows the angle range in which a lesion or medical device exists, based on the image center obtained by inputting the tomographic image I1 into a second model 32M. Fig. 21C shows the angle range based on the blood vessel center.

[0088] The guidewire W is located inside the lumen. The guidewire W appears as a shadow in the tomographic image I1, with its main body and echoes having low pixel values ​​and low brightness. Regarding the guidewire W, the processing unit 30 extracts a portion of the tomographic image I1 with low pixel values, within a range where the probability of the presence of a lesion calculated in step S301 is equal to or greater than a predetermined probability value, and within a region inside the curve B1 of the lumen boundary (S302). The processing unit 30 can calculate the angle range for capturing the guidewire W from the center of the blood vessel, capturing the extracted range, as shown in FIGS. 21B and 21C.

[0089] 10 and 11, and the flowchart of Fig. 12, the angle range of the lesion or the like at the center of the blood vessel can be obtained using the second model 32M that outputs the angle distribution of the probability that a lesion or a medical device exists, and the first model 31M for identifying the lumen boundary or the blood vessel boundary. As shown in Fig. 17 to Fig. 21, the angle range can be obtained using one second model 32M for various lesions and medical devices, but to improve accuracy, it is preferable to train the second model 32M for each type of lesion and medical device.

[0090] In the present embodiment, the image processing device 3 connected to the catheter 1 generates a tomographic image I1 in almost real time based on signals from the imaging device 11, calculates the angular range in which the lesion or medical instrument is located, and displays the image on the display device 4. However, the processing by the image processing device 3 described above may be performed after the fact based on signal data separately obtained from the imaging device 11. In other words, the image processing device 3 is not necessarily directly connected to the imaging device 11 of the catheter 1, as long as it can acquire signals from the imaging device 11. The image processing device 3 may be a device, such as a server device, that can read a storage device that stores signals from the imaging device 11 via a network. In other words, for example, the processing procedures of steps S101-S104 shown in the flowcharts of FIGS. 10 and 11 may be performed by an existing processing device, and the processing of steps S006-S116 may be executed by the image processing device 3 connected to the processing device, and displayed on the display device 4 via the processing device.

[0091] In this embodiment, the medical image has been described using an image obtained by IVUS of a coronary artery as an example. However, the application is not limited to this, and may be OCT / OFDI or the like, and the hollow organ is not limited to a blood vessel.

[0092] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0093] 1 catheter 11 Imaging Devices 3. Image processing device (information processing device) 30 Processing section 31 Storage section 3P Computer Program 31M 1st Model 32M 2nd Model 4 Display device 400 screens I0 Rectangular image (image based on scanning signal) I1 Tomographic image (image based on scanning signal)

Claims

1. The computer acquiring a scanning signal by an imaging device provided in a catheter that is inserted into a hollow organ and moves in the longitudinal direction while rotating about an axis in the longitudinal direction of the hollow organ; When an image of the hollow organ based on the scanning signal is input, a learning model is generated that outputs the probability of the presence of a lesion or a medical device for each rotation angle of the catheter. Model generation method.

2. The computer generates the learning model for each type of lesion or each type of medical device. The model generation method of claim 1 .

3. The computer The learning model is generated using as training data rectangular images in which the scanning signals are arranged in order of increasing and decreasing rotation angles and data indicating the presence or absence of a lesion or a medical instrument for each rotation angle associated with the rectangular images. The model generation method according to claim 1 or 2.

4. The computer The learning model is generated using as training data the tomographic image of the hollow organ obtained based on the scanning signal and data indicating the presence or absence of a lesion or a medical instrument for each angle associated with the tomographic image. The model generation method according to claim 1 or 2.

5. an input layer to which an image based on a scanning signal from an imaging device provided on a catheter that is inserted into a hollow organ and moves in the longitudinal direction of the hollow organ while rotating around an axis in the longitudinal direction of the hollow organ; an output layer that outputs the probability of a lesion or a medical device being present for each rotation angle of the catheter; an intermediate layer trained based on training data including the image and data indicating the presence or absence of a lesion or a medical instrument for each angle in the image; Equipped with A learning model for causing a computer to function by providing an image based on a scanning signal from the imaging device to the input layer, performing calculations based on the intermediate layer, and outputting the existence probability corresponding to the image from the output layer.

6. a computer that acquires scanning signals from an imaging device provided in a catheter that is inserted into a hollow organ and moves in the longitudinal direction of the hollow organ while rotating about an axis in the longitudinal direction of the hollow organ; a first model that, when an image of the hollow organ based on the scanning signal is input, outputs data that identifies different regions in the image, including the lumen and membrane of the hollow organ; a second model that outputs a probability of existence of a lesion or a medical instrument for each rotation angle of the catheter of the imaging device when the image is input; Using Calculating an angular range for capturing the lesion or the medical instrument from the center of the hollow organ based on parameters indicating the anatomical features of the hollow organ based on the region identified in the first model and the existence probability for each rotation angle obtained from the second model. A computer program that executes a process.

7. The computer, Calculating a lumen boundary in the luminal organ or an organ boundary corresponding to the outside of the membrane based on the region identified in the first model; extracting the range of the lesion or medical device in the image, the range being inside the organ boundary, or the range being inside the organ boundary and outside the lumen boundary; The angle range for capturing the lesion or the medical instrument from the center of the hollow organ is corrected based on the extracted range.

7. A computer program product according to claim 6, which causes a process to be executed.

8. The computer, Calculating a lumen boundary in the luminal organ or an organ boundary corresponding to the outside of the membrane based on the region identified in the first model; Correcting a part of the calculated lumen boundary or organ boundary that is included in an angle range capturing the lesion or medical instrument from the center of the luminal organ.

7. A computer program product according to claim 6, which causes a process to be executed.

9. The computer, Calculate the angle range capturing the plaque from the center of the hollow organ 9. A computer program product according to claim 6, which causes a process to be executed.

10. The computer, Calculate the angle range in which the stent is captured from the center of the luminal organ 9. A computer program product according to claim 6, which causes a process to be executed.

11. The computer, Calculate the angle range for capturing the guidewire from the center of the hollow organ 9. A computer program product according to claim 6, which causes a process to be executed.

12. The computer, Based on the calculated angle range, the range of the lesion or medical instrument in the image of the hollow organ is depicted and displayed on a display device.

9. A computer program product according to claim 6, which causes a process to be executed.

13. The computer, Based on the calculated angle range, the range of the lesion or medical device is depicted in a three-dimensional image showing the structure of the hollow organ, and displayed on a display device.

9. A computer program product according to claim 6, which causes a process to be executed.

14. a computer that acquires scanning signals from an imaging device provided in a catheter that is inserted into a hollow organ and moves in the longitudinal direction while rotating about the longitudinal direction of the hollow organ; a first model that, when an image of the hollow organ based on the scanning signal is input, outputs data that identifies different regions in the image, including the lumen and membrane of the hollow organ; a second model that outputs a probability of existence of a lesion or a medical instrument for each rotation angle of the catheter of the imaging device when the image is input; Using Calculating an angular range for capturing the lesion or the medical instrument from the center of the hollow organ based on parameters indicating the anatomical features of the hollow organ based on the region identified in the first model and the existence probability for each rotation angle obtained from the second model. Information processing methods.

15. 1. An information processing device for acquiring scanning signals from an imaging device provided in a catheter that is inserted into a hollow organ and moves in the longitudinal direction of the hollow organ while rotating around an axis in the longitudinal direction of the hollow organ, a first model that, when an image of the hollow organ based on the scanning signal is input, outputs data that identifies different regions in the image, including the lumen and membrane of the hollow organ; a second model that outputs a probability of existence of a lesion or a medical instrument for each rotation angle of the catheter of the imaging device when the image is input; a storage unit that stores the a processing unit that performs image processing based on the scanning signal; Equipped with The processing unit calculates an angle range for capturing a lesion or a medical instrument from the center of the hollow organ based on parameters indicating anatomical features of the hollow organ based on the area identified in the first model and the existence probability for each rotation angle obtained from the second model. Information processing device.

Images