Information Processing Apparatus, Information Processing Method, and Program

Abstract

This information processing device comprises: an acquisition unit which acquires multiple transverse tomographic images in frames captured of a blood vessel of a patient; a calculation unit which calculates the the plaque burden in each frame by inputting the aforementioned acquired transverse tomographic images in each frame into a model trained so as to calculate plaque burden when transverse tomographic images are inputted; and a display unit which displays a vertical tomographic image based on the transverse tomographic images in the multiple frames, and a first object displayed in association with the vertical tomographic image and showing the size of the plaque burden at each position in the vertical tomographic image along the axial direction of the blood vessel.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is a technique for assisting image diagnosis related to vascular treatment such as PCI (Percutaneous Coronary Intervention). For example, in Patent Document 1, there is disclosed a medical image processing apparatus that converts three-dimensional CT (Computer Tomography) data into two-dimensional projection images, extracts a plaque region of a blood vessel from the three-dimensional CT data, calculates a stenosis rate of the blood vessel, and displays a projection image in which a color value of the plaque region is determined according to the calculated stenosis rate or the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the invention according to Patent Document 1 extracts a plaque region based on a rule from three-dimensional CT values, and does not necessarily extract the plaque region with high accuracy to calculate the stenosis rate. Further, the invention according to Patent Document 1 requires three-dimensional CT data and does not calculate a stenosis rate or the like from two-dimensional data (images).

[0005] In one aspect, an object is to provide an information processing apparatus or the like that can suitably assist image diagnosis related to vascular treatment.

Means for Solving the Problems

[0006] An information processing apparatus according to one aspect includes an acquisition unit that acquires cross-sectional images of a plurality of frames obtained by imaging a patient's blood vessel, a calculation unit that inputs the cross-sectional images to a learned model that calculates a plaque burden, inputs the cross-sectional images of the acquired respective frames, and calculates the plaque burden for each frame, a longitudinal sectional image based on the cross-sectional images of the plurality of frames, and a display unit that displays an object corresponding to the longitudinal sectional image and a first object that represents the magnitude of the plaque burden at each position of the longitudinal sectional image along the axial direction of the blood vessel.

Advantages of the Invention

[0007] In one aspect, it is possible to suitably support image diagnosis related to blood vessel treatment.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

BEST MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments. (Embodiment 1) FIG. 1 is an explanatory diagram showing a configuration example of an image diagnostic system. In the present embodiment, an image diagnostic system that calculates the plaque burden from a tomographic image of a patient's blood vessel and supports blood vessel treatment using a predetermined treatment device will be described. The image diagnostic apparatus 2 includes a server 1 and an image diagnostic apparatus (information processing apparatus) 2. Each apparatus is communicatively connected via a network N such as the Internet.

[0010] The server 1 is a server computer capable of various information processing and information transmission and reception. Note that the apparatus corresponding to the server 1 may be an apparatus such as a personal computer. The server 1 functions as a generation device that performs machine learning for learning predetermined training data and generates a calculation model 51 (see FIG. 4) for calculating the plaque burden with the tomographic image of the patient's blood vessel as an input.

[0011] Note that the plaque burden is the area ratio of the plaque in the cross-section of the blood vessel, which is the value obtained by dividing the cross-sectional areas of the plaque and the tunica media by the cross-sectional area of the blood vessel (the area of the region within the EEM (External Elastic Membrane)) (see Fig. 4).

[0012] The imaging device 2 is an imaging device that captures a medical image of a patient's blood vessel. For example, it is an IVUS (Intravascular Ultrasound) device that performs an ultrasonic examination using a catheter 201. The catheter 201 is a medical instrument inserted into a patient's blood vessel. It transmits an ultrasonic signal from an ultrasonic probe attached to the tip of the catheter 201 and receives a reflected wave. The imaging device 2 generates and displays an ultrasonic tomographic image based on the reflected wave received by the catheter 201.

[0013] In this embodiment, the imaging device 2 is described as an IVUS device, but it may be an optical imaging device such as an OCT (Optical Coherence Tomography) device. Also, the blood vessel tomographic image may be captured by a method other than IVUS and OCT.

[0014] The imaging device 2 has installed therein the data of the calculation model 51 generated by the server 1. The imaging device 2 inputs the blood vessel tomographic image captured using the catheter 201 into the calculation model 51 to calculate the plaque burden. Then, the imaging device 2 presents the calculated plaque burden to the user (medical staff). Further, the imaging device 2 identifies the lesion region of the blood vessel based on the calculated plaque burden and supports the determination of the placement position for placing a stent (treatment device) to be placed in the blood vessel.

[0015] In the present embodiment, the image diagnostic apparatus 2 calculates the plaque burden using the calculation model 51. However, the server 1 on the cloud may execute the process using the calculation model 51. Further, for example, a general-purpose computer connected to the image diagnostic apparatus 2 may perform the process. Thus, the processing entity that executes the series of processes is not particularly limited.

[0016] In the present embodiment, a stent is cited as an example of the treatment device. However, as will be described later, a balloon that dilates blood vessels, a rotablator that shaves stenotic lesions of blood vessels, etc. may also be used.

[0017] FIG. 2 is a block diagram showing a configuration example of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary storage unit 14. The control unit 11 has an arithmetic processing device such as one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), etc., and reads and executes the program P1 stored in the auxiliary storage unit 14 to perform various information processes, control processes, etc. The main memory unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processes, and transmits and receives information to and from the outside. The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory, hard disk, etc., and stores the program P1 and other data necessary for the control unit 11 to execute the process.

[0018] Note that the server 1 may be a multi-computer composed of a plurality of computers, or may be a virtual machine virtually constructed by software.

[0019] In addition, in this embodiment, the server 1 is not limited to the above configuration. For example, it may include an input unit that receives operation inputs, a display unit that displays images, and the like. Further, the server 1 may be provided with a reading unit that reads a portable storage medium 1a such as a CD (Compact Disk)-ROM or a DVD (Digital Versatile Disc)-ROM, and reads and executes the program P1 from the portable storage medium 1a. Alternatively, the server 1 may read the program P1 from the semiconductor memory 1b.

[0020] FIG. 3 is a block diagram showing a configuration example of the image diagnostic apparatus 2. The image diagnostic apparatus 2 includes a control unit 21, a main storage unit 22, a communication unit 23, a display unit 24, an input unit 25, an image processing unit 26, and an auxiliary storage unit 27. The control unit 21 is an arithmetic processing device such as one or more CPUs, MPUs, GPUs, etc., and performs various information processing, control processing, etc. by reading and executing the program P2 stored in the auxiliary storage unit 27. The main storage unit 22 is a temporary storage area such as a RAM, and temporarily stores data necessary for the control unit 21 to execute arithmetic processing. The communication unit 23 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside. The display unit 24 is a display screen such as a liquid crystal display, and displays images. The input unit 25 is an operation interface such as a keyboard or a mouse, and receives operation inputs from the user. The image processing unit 26 is an image processing module that processes signals transmitted and received via the catheter 201 and generates tomographic images.

[0021] The auxiliary storage unit 27 is a non-volatile storage area such as a hard disk or a large-capacity memory, and stores the program P2 and other data necessary for the control unit 21 to execute processing. Further, the auxiliary storage unit 27 stores the calculation model 51. The calculation model 51 is a machine learning model generated by learning predetermined training data, and is a learned model that calculates the plaque burden using the tomographic image of the blood vessel imaged by the image diagnostic apparatus 2 as an input. The calculation model 51 is assumed to be used as a program module constituting a part of artificial intelligence software.

[0022] Note that the image diagnostic apparatus 2 may be provided with a reading unit that reads a portable storage medium 2a such as a CD-ROM, and reads and executes the program P2 from the portable storage medium 2a. Alternatively, the image diagnostic apparatus 2 may read the program P2 from the semiconductor memory 2b.

[0023] FIG. 4 is an explanatory diagram regarding the calculation process of the plaque burden. In FIG. 4, the state of inputting each frame of the vascular tomogram imaged by the image diagnostic apparatus 2 into the calculation model 51 and calculating the plaque burden is conceptually illustrated.

[0024] The calculation model 51 is a machine learning model that has learned predetermined training data, and is, for example, a CNN (Convolution Neural Network). Note that the calculation model 51 may be any model that can calculate the plaque burden from the vascular tomogram, and may be a model based on other learning algorithms such as a neural network other than CNN, a decision tree, or an SVM (Support Vector Machine).

[0025] The calculation model 51 includes an input layer that receives the input of an image (tomogram), an intermediate layer that extracts the feature amount of the input image, and an output layer that calculates (outputs) the plaque burden. The input layer receives the input of the image and passes the data of the input image to the intermediate layer. The intermediate layer includes a convolutional layer that convolves the data of the input image, extracts the feature amount of the input image, and passes it to the output layer. The output layer calculates the plaque burden based on the feature amount extracted by the intermediate layer.

[0026] In the present embodiment, the process executed by the calculation model 51 is treated as a regression problem, and the plaque burden is predicted as a continuous value. Note that this process may be treated as a classification problem, and the plaque burden may be predicted (classified) within a certain numerical range, for example, "0 to X1", "X1 to X2", "X2 to X3"... (0 < X1 < X2 < X3...). Alternatively, without predicting the plaque burden numerically, it may be classified into multiple stages (for example, classified into a red zone, an orange zone, a yellow zone...).

[0027] Server 1 uses the data with the correct value of plaque burden assigned to the training vascular tomogram (tomogram) as training data (second training data) to generate the calculation model 51. The tomogram for training is a vascular tomogram taken with an actual patient as the subject, and the correct value is the actual value of the plaque burden in the tomogram of the patient. Server 1 inputs the tomogram for training into the calculation model 51 to calculate the plaque burden and compares it with the correct value. Server 1 optimizes parameters such as the weights between neurons so that the two are approximated, and generates the calculation model 51.

[0028] The image diagnostic apparatus 2 calculates the plaque burden in the blood vessels of the patient undergoing catheter treatment using the calculation model 51 generated by Server 1. Specifically, the image diagnostic apparatus 2 sequentially inputs a plurality of frames of tomograms taken in response to the pull-back operation of the catheter 201 into the calculation model 51, and calculates the plaque burden in each frame. Thereby, the image diagnostic apparatus 2 calculates the area ratio obtained by dividing the area of the image region corresponding to the plaque and the tunica media (hereinafter referred to as the "plaque region") by the area of the EEM region (the region obtained by adding the lumen region to the plaque region), that is, the plaque burden (see the lower right of FIG. 4).

[0029] The image diagnostic apparatus 2 presents to the user the plaque burden in each frame calculated above, that is, the plaque burden at each position of the blood vessel along the axial direction, and supports the determination of the stent implantation position.

[0030] FIG. 5 is an explanatory diagram showing an example of displaying the plaque burden. In the present embodiment, the image diagnostic apparatus 2 displays a predetermined object (first object) representing the magnitude of the plaque burden at each position of the blood vessel corresponding to the longitudinal tomogram 5 reconstructed from a plurality of frames of tomograms. Specifically, as shown in FIG. 5, the image diagnostic apparatus 2 displays a gradation bar 6 with the display color gradually changed according to the magnitude of the plaque burden above the longitudinal tomogram 5.

[0031] The longitudinal sectional image 5 is a longitudinal section that reproduces the state of the blood vessel along the axial direction (running direction) of the blood vessel, and is generated from a plurality of frames of tomographic images captured in response to the pull-back operation. For example, as shown in FIG. 5, the image diagnostic apparatus 2 displays the longitudinal sectional image 5 with the axial direction of the blood vessel as the horizontal direction.

[0032] The image diagnostic apparatus 2 displays a gradation bar 6 having substantially the same length as the length of the longitudinal sectional image 5 along the axial direction above the longitudinal sectional image 5. The gradation bar 6 is a bar-shaped object representing the magnitude of the plaque burden, and is a display bar whose display color is changed according to the magnitude of the plaque burden at each position of the corresponding longitudinal sectional image 5, that is, in the tomographic image of each frame. The image diagnostic apparatus 2 changes the display color at each position of the gradation bar 6 according to the magnitude of the plaque burden at each position on the longitudinal sectional image 5.

[0033] In FIG. 5, for the sake of convenience, an area where the plaque burden is large (an area above the threshold value) is illustrated by hatching. Also, actually, as described below, the gradation of the display color in the gradation bar 6 is finely set according to the numerical value of the plaque burden, but in FIG. 5, for the sake of convenience, an area where the plaque burden is large is illustrated by the same hatching.

[0034] In the present embodiment, the object representing the magnitude of the plaque burden is described as the gradation bar 6, but the object is not limited to the gradation bar 6. For example, the image diagnostic apparatus 2 may display a line graph representing the magnitude of the plaque burden above the longitudinal sectional image 5. Thus, the gradation bar 6 is an example of the object, and any object that appropriately represents the magnitude of the plaque burden may be used.

[0035] For example, the imaging diagnostic apparatus 2 refers to a predetermined threshold value (e.g., 50%) set by default to set the display color of the area with many plaques. Specifically, the imaging diagnostic apparatus 2 sets the display color of the area where the plaque burden is equal to or greater than the threshold value to a warm color (e.g., orange), and sets the area where the plaque burden is less than the threshold value to the default color (e.g., white). Then, for the area where the plaque burden is equal to or greater than the threshold value, the imaging diagnostic apparatus 2 changes the display color (e.g., increases the brightness) so that the greater the plaque burden, the more emphasized it is, and performs a gradation display.

[0036] Note that as a method of gradation display, for example, it is assumed that the brightness is changed, but the hue, saturation, etc. may be changed, and the specific display method is not particularly limited.

[0037] When displaying the gradation bar 6 as described above, it is preferable that the imaging diagnostic apparatus 2 does not display the area where the reliability of the calculated result of the plaque burden in the calculation model 51 is low (hereinafter referred to as the "low reliability area"). In FIG. 5, the low reliability area is shown in black. When the imaging diagnostic apparatus 2 calculates the plaque burden in each frame using the calculation model 51, together with the plaque burden in each frame, the imaging diagnostic apparatus 2 acquires the reliability of the calculated value (e.g., a probability value from 0 to 1) from the calculation model 51. When the reliability at each position (frame) of the blood vessel is equal to or less than a predetermined value, that is, when the reliability is low, the imaging diagnostic apparatus 2 makes the area on the corresponding gradation bar 6 a low reliability area and blank. Thereby, the certainty of the calculated result of the plaque burden can be presented to the user.

[0038] In this way, the imaging diagnostic apparatus 2 displays the gradation bar 6 together with the tomographic image 5 and quantitatively presents the state of the blood vessel at each position. Further, as shown in FIG. 5, the imaging diagnostic apparatus 2 displays the maximum value, minimum value, and average value of the plaque burden. Thereby, the user (medical staff) can grasp the state of the blood vessel and determine the treatment range to be treated with a treatment device such as a stent.

[0039] FIG. 6 is an explanatory diagram showing another example of the display of plaque burden. In the display example of FIG. 5, it was described that warm colors are displayed for areas where the plaque burden is equal to or greater than the threshold value. However, gradation display may also be performed for areas where the plaque burden is less than the threshold value, that is, areas with less plaque.

[0040] For example, the image diagnostic apparatus 2 sets the display color of areas where the plaque burden is equal to or greater than the threshold value to warm colors, and sets the display color of areas where the plaque burden is less than the threshold value to cool colors. In FIG. 6, areas where the plaque burden is equal to or greater than the threshold value and areas where the plaque burden is less than the threshold value are illustrated with different hatchings. Similar to the areas where the plaque burden is equal to or greater than the threshold value, the image diagnostic apparatus 2 changes the display color (for example, increases the brightness) as the plaque burden decreases, and performs gradation display. In this way, the image diagnostic apparatus 2 may present areas with a large amount of plaque that may require treatment, as well as areas with a small amount of plaque where the need for treatment is low.

[0041] In the example of FIG. 6, the number of areas for gradation display is two. However, further thresholds may be provided and gradation display may be performed by dividing into three or more areas.

[0042] Also, when performing the above-described gradation display, the user may be able to set the threshold value used as the display criterion. Specifically, the image diagnostic apparatus 2 accepts a setting input from the user to change the threshold value from the default value to an arbitrary value, determines areas where the plaque burden is equal to or greater than the threshold value and areas where the plaque burden is less than the threshold value according to the set threshold value, and changes the display color. Thereby, the user can visualize the area of interest by himself / herself, and can support the user more suitably.

[0043] FIG. 7 is an explanatory diagram regarding the determination process of the treatment range of blood vessels. In the present embodiment, further, based on the calculation result of the plaque burden, the treatment range to be treated with a predetermined treatment device is presented to the user.

[0044] Specifically, the imaging diagnostic apparatus 2 determines the positions of both ends of the stent to be placed in the blood vessel. When performing vascular treatment such as PCI, it is preferable to place the stent so that the lesion can be covered without leaving any residue. However, in reality, since plaques exist continuously, it is difficult to cover the lesion without leaving any residue. For example, basically, the stent is placed so that both ends of the stent are located at normal sites where there are no plaques. If this is difficult, the stent is placed so that both ends of the stent are located at sites where the plaque burden is below a predetermined value (e.g., 50%). If this is also difficult, the stent is placed so that both ends of the stent are located at the site with the least plaque burden.

[0045] Currently, as described above, the plaque burden of the entire frame is not calculated. Instead, the stenosis part is visually identified from the image, and the plaque burden for several frames is measured as needed. Therefore, in the present embodiment, based on the plaque burden calculated by the calculation model 51, the range where both ends of the stent should be located is determined.

[0046] In the present embodiment, a stent is cited as an example of the treatment device. However, the treatment device is not limited to the stent. For example, the imaging diagnostic apparatus 2 may determine the positions of both ends of the expansion range to be expanded by a balloon used for expanding the blood vessel. Further, for example, the imaging diagnostic apparatus 2 may determine the positions of both ends of the cutting range to be cut by a rotabrator used for cutting a hard lesion (e.g., calcified tissue). Thus, the imaging diagnostic apparatus 2 only needs to be able to determine the positions of both ends of the treatment range to be treated by the treatment device inserted into the blood vessel, and the treatment device is not limited to the stent.

[0047] First, the imaging diagnostic apparatus 2 divides the longitudinal sectional image 5 (longitudinal section) of the blood vessel into a first region where the plaque burden is equal to or greater than a predetermined threshold value and a second region where the plaque burden is less than the threshold value. The threshold value is, for example, the same value as the threshold value used as the criterion for the above-described gradation display, but may be a different value. The imaging diagnostic apparatus 2 divides the longitudinal sectional image 5 into first regions L1, L2, L3... (L: Lesion) where the plaque burden is equal to or greater than the threshold value and second regions H1, H2, H3... (H: Healthy) where the plaque burden is less than the threshold value based on the plaque burden in each frame calculated by the calculation model 51.

[0048] In the present embodiment, the imaging diagnostic apparatus 2 identifies a first region LX (X = 1, 2, 3...) including the position (frame) where the plaque burden takes the maximum value as the lesion region LX where the stent should be placed. Then, the imaging diagnostic apparatus 2 determines two second regions HY, HZ (Y, Z = 1, 2, 3...) located on both sides of the identified lesion region LX as the regions where both ends of the stent should be located.

[0049] FIG. 7 illustrates a longitudinal sectional image 5 in which there is one first region L1 and two second regions HP1, HD1 (P: Proximal, D: Distal). When the plaque burden takes the maximum value in the first region L1, the imaging diagnostic apparatus 2 determines the second regions HP1, HD1 sandwiching the first region L1 as the regions where both ends of the stent should be located.

[0050] Note that, for example, when there is a low-reliability region where the reliability of the calculation result of the plaque burden is low in the imaging diagnostic apparatus 2, it is preferable to group a plurality of first regions or second regions straddling the low-reliability region as the same region. For example, in FIG. 7, there are two low-reliability regions in the first region L1. In this case, the imaging diagnostic apparatus 2 determines whether the length of each low-reliability region in the axial direction of the blood vessel is equal to or less than a predetermined length (for example, 1 mm). When the low-reliability region is equal to or less than the predetermined length, the imaging diagnostic apparatus 2 divides a plurality of first regions straddling the low-reliability region as the same region. In the example of FIG. 7, since both of the two low-reliability regions are equal to or less than the predetermined length, the imaging diagnostic apparatus 2 divides the three regions straddling the low-reliability region into the same first region L1.

[0051] Similarly for the second region, when there are a plurality of second regions straddling a low-reliability region with a length equal to or less than a predetermined length, the image diagnostic apparatus 2 divides the plurality of second regions as the same region. In this way, when the length of the low-reliability region in the axial direction is equal to or less than the predetermined length, the image diagnostic apparatus 2 divides a plurality of first regions or a plurality of second regions straddling the low-reliability region as the same region. Thereby, while ensuring reliability, the stent placement position can be preferably determined.

[0052] Note that the above-mentioned predetermined length may be variable by the user. For example, the image diagnostic apparatus 2 accepts a setting input for changing the default value of the predetermined length to an arbitrary value (for example, a length at a predetermined interval such as 0.6 mm, 0.8 mm, 1.2 mm...). The image diagnostic apparatus 2 divides each region by determining whether it is equal to or less than the set predetermined length. Thereby, the user can more preferably determine the stent placement position.

[0053] Also, similar to the case of gradation display, the image diagnostic apparatus 2 may make the threshold value as a criterion for region division variable by the user. In this case, the image diagnostic apparatus 2 may be able to set different threshold values on the proximal side and the distal side of the blood vessel. For example, when 50% is set as the threshold value on the proximal side and 60% is set as the threshold value on the distal side, the image diagnostic apparatus 2 uses the position where the plaque burden becomes 50% or more as the starting point of the first region and the position where the plaque burden becomes less than 60% as the end point of the first region and divides them when viewed from the left side of FIG. 7. Thereby, the user can finely adjust the first region and the second region by himself / herself.

[0054] FIG. 8 is an explanatory diagram showing an example of the display of the lesion length. As described above, the image diagnostic apparatus 2 divides the tomographic image 5 into first regions L1, L2, L3,... and second regions H1, H2, H3,... and identifies the lesion region L1. When the lesion region L1 is identified, the image diagnostic apparatus 2 measures the length of the lesion region L1 in the axial direction of the blood vessel, that is, the lesion length, and displays it corresponding to the lesion region L1 in the tomographic image 5. Specifically, as shown in FIG. 8, the image diagnostic apparatus 2 displays the portion corresponding to the lesion region L1 on the gradation bar 6 with an arrow and also displays the lesion length. Thereby, the length of the lesion site can be visualized.

[0055] FIG. 9 is an explanatory diagram showing an example of the superimposed display of the stent. In the present embodiment, further, the image diagnostic apparatus 2 presents to the user the treatment range to be treated with the treatment device, that is, the range where the stent is to be placed. Specifically, the image diagnostic apparatus 2 superimposes and displays a stent region 7 (second object) representing the size of the stent (treatment device) on the tomographic image 5.

[0056] The stent region 7 is an object simulating a stent to be placed in a blood vessel, and is a rectangular object representing the placement range of the stent. Note that the stent region 7 only needs to have at least the length in the axial direction coincide with the length of the actual stent, and the length in the radial direction (the length in the vertical direction in FIG. 8) does not need to coincide.

[0057] Also, in the present embodiment, since the treatment device is a stent, the shape of the object is rectangular. However, for example, when the treatment device is a balloon, the shape of the object may be sandbag-like. Thus, the image diagnostic apparatus 2 only needs to be able to display an object having a shape corresponding to the treatment device.

[0058] For example, the imaging diagnostic apparatus 2 displays the stent region 7 having a length equal to or greater than the lesion length measured above such that both ends of the stent region 7 are positioned in the second regions HP1 and HD1 determined as the regions where both ends should be positioned above. For example, the imaging diagnostic apparatus 2 stores information on each stent product with a predetermined length increment (e.g., 9 mm, 12 mm, 15 mm,...) that can be used in the procedure in the auxiliary storage unit 27. The imaging diagnostic apparatus 2 selects a stent that exceeds the lesion length and has the minimum length from these stents. Then, the imaging diagnostic apparatus 2 superimposes and displays the stent region 7 having the length of the selected stent such that both ends are positioned in the second regions HP1 and HD1. Note that the selection of the stent, the determination of the superimposed position of the stent region 7, etc. may be manually performed by the user.

[0059] For example, the imaging diagnostic apparatus 2 superimposes and displays the stent region 7 such that the central position (midpoint) of the lesion region L1 in the axial direction of the blood vessel coincides with the central position of the stent region 7. In this case, as shown in FIG. 9, downward arrows are displayed at positions corresponding to both ends of the stent, and two-way arrows are displayed in portions exceeding the first region L1. The imaging diagnostic apparatus 2 moves the stent region 7 according to an operation input (e.g., mouse operation) from the user, and allows the user to determine the optimal placement position of the stent.

[0060] In this case, it is preferable that the imaging diagnostic apparatus 2 displays tomographic images at positions corresponding to both ends of the stent region 7 as shown in FIG. 9. Thereby, the user can search for the optimal placement position of the stent while moving the stent region 7.

[0061] Note that in the above, the stent region 7 is superimposed and displayed on the tomographic image 5, but the imaging diagnostic apparatus 2 may display the stent region 7 at a location other than the tomographic image 5 (e.g., above the gradation bar 6 or within the gradation bar 6). Thus, the imaging diagnostic apparatus 2 only needs to be able to display an object simulating the treatment device, and the display position is not limited to above the tomographic image 5.

[0062] FIG. 10 is a flowchart showing the procedure of the generation process of the calculation model 51. Based on FIG. 10, the processing content when generating the calculation model 51 by machine learning will be described. The control unit 11 of the server 1 acquires training data (second training data) for generating the calculation model 51 (step S11). The training data is data in which the correct value of the plaque burden is associated with a tomographic image of a blood vessel for training.

[0063] Based on the training data, the control unit 11 generates a calculation model 51 that calculates the plaque burden when a tomographic image of a blood vessel is input (step S12). Specifically, as described above, the control unit 11 generates a neural network such as a CNN as the calculation model 51. The control unit 11 inputs the tomographic image of the blood vessel for training into the calculation model 51 to calculate the plaque burden and compares it with the correct value. The control unit 11 optimizes parameters such as the weights between neurons so that the calculated plaque burden approximates the correct value, and generates the calculation model 51. The control unit 11 ends a series of processes.

[0064] FIG. 11 is a flowchart showing the procedure of the plaque burden calculation process. Based on FIG. 11, the processing content executed by the image diagnostic apparatus 2 will be described. The control unit 21 of the image diagnostic apparatus 2 acquires tomographic images of a plurality of frames obtained by imaging a patient's blood vessel (step S31). The control unit 21 inputs the tomographic image of each frame into the calculation model 51 and calculates the plaque burden in each frame (step S32). Specifically, the control unit 21 acquires the plaque burden in each frame and the reliability of the calculated value from the calculation model 51.

[0065] The control unit 21 identifies the blood vessel position (frame) where the plaque burden takes the maximum value (step S33). In addition, the control unit 21 identifies a low-reliability region where the reliability of the plaque burden is below a predetermined value in the axial direction of the blood vessel (step S34). In addition, the control unit 21 receives a threshold setting input (step S35). Note that the control unit 21 may skip step S35 and set a default threshold.

[0066] Based on the cross-sectional images of multiple frames acquired in step S31, the control unit 21 generates (reconstructs) a longitudinal sectional image of the blood vessel (step S36). The control unit 21 displays the generated longitudinal sectional image and an object (first object) representing the magnitude of the plaque burden at each position of the longitudinal sectional image along the axial direction of the blood vessel (step S37). Specifically, as described above, the control unit 21 displays a gradation bar whose display color gradually changes according to the magnitude of the plaque burden. In this case, the control unit 21 displays a gradation bar with the low-reliability region hidden.

[0067] Based on the plaque burden at each position (frame) of the blood vessel, the control unit 21 divides the longitudinal sectional image (longitudinal section) of the blood vessel into a first region where the plaque burden is equal to or greater than the threshold value and a second region where the plaque burden is less than the threshold value (step S38). Specifically, the control unit 21 divides the longitudinal sectional image into a plurality of regions according to whether the plaque burden is equal to or greater than the threshold value, and when the length of the low-reliability region is equal to or less than a predetermined length, divides a plurality of first regions or second regions straddling the low-reliability region into the same region.

[0068] The control unit 21 identifies the first region including the maximum value of the plaque burden as the lesion region (step S39). Then, the control unit 21 determines the regions where both ends of the treatment range to be treated with the treatment device should be located according to the identified lesion region (step S40). Specifically, the control unit 21 determines the second regions located on both sides of the lesion region as the regions where both ends of the stent to be placed in the blood vessel should be located.

[0069] The control unit 21 measures the length in the axial direction of the lesion area specified in step S39, that is, the lesion length, and displays it corresponding to the lesion area in the longitudinal sectional image (step S41). Further, the control unit 21 displays an object (second object) representing the size of the treatment device corresponding to the longitudinal sectional image (step S42). For example, as described above, the control unit 21 superimposes and displays a stent area representing the length of the stent in the axial direction of the blood vessel on the longitudinal sectional image so that both ends are located in the area determined in step S40. The control unit 21 ends a series of processes.

[0070] As described above, according to the first embodiment, the plaque burden at each position of the blood vessel is calculated from a plurality of frames of cross-sectional images of the patient's blood vessel, and an object such as a gradation bar representing the magnitude of the calculated plaque burden is displayed corresponding to the longitudinal sectional image. Thereby, it is possible to suitably support image diagnosis related to blood vessel treatment, such as determination of the stent placement position.

[0071] Further, according to the first embodiment, by making the low-reliability region where the reliability of the calculation result of the plaque burden is low non-displayed, it is possible to present the certainty of the calculation result of the plaque burden to the user.

[0072] Further, according to the first embodiment, by presenting also the length of the lesion area (lesion length), the placement range of the stent (stent area), etc., it is possible to more suitably support image diagnosis.

[0073] Further, according to the first embodiment, the longitudinal section (longitudinal sectional image) of the blood vessel is divided into a plurality of regions according to the plaque burden, and the regions where both ends of the treatment range of the blood vessel by a treatment device such as a stent should be located are determined. Thereby, it is possible to more suitably support image diagnosis related to blood vessel treatment, such as determination of the stent placement position.

[0074] Further, according to the first embodiment, when the length of the low-reliability region is equal to or less than a predetermined length, a plurality of regions (the first region or the second region) straddling the low-reliability region are regarded as the same region and divided. Thereby, while ensuring reliability, the placement position of the stent and the like can be suitably determined.

[0075] (Modification 1) In the first embodiment, the first region where the plaque burden takes the maximum value is specified as the lesion region. However, the lesion region does not necessarily have to be the region where the plaque burden takes the maximum value.

[0076] FIG. 12 is an explanatory diagram regarding the specific processing of the lesion region according to Modification 1. In the first embodiment, the lesion region was specified first, and then the regions where both ends of the stent should be located were determined. However, the regions where both ends of the stent should be located may be determined first, and then the lesion region may be specified.

[0077] Specifically, after the tomographic image 5 is divided into the first regions L1, L2, L3... and the second regions H1, H2, H3... by the imaging diagnostic apparatus 2, the imaging diagnostic apparatus 2 searches for two second regions Hy, Hz sandwiching the first region Lx. When two second regions Hy, Hz sandwiching the first region Lx are searched, the imaging diagnostic apparatus 2 specifies the first region Lx where the two second regions Hy, Hz are located on both sides as the lesion region Lx. Then, the imaging diagnostic apparatus 2 measures the length (lesion length) of the lesion region Lx and superimposingly displays the stent region 7.

[0078] In this way, the imaging diagnostic apparatus 2 may determine the region where the stent should be placed from the positional relationship between the first region and the second region. The determination method described in the first embodiment is an example, and various methods are assumed.

[0079] (Second Embodiment) In the first embodiment, the form in which the calculation model 51 directly calculates the plaque burden has been described. In the present embodiment, a form in which a plaque region is identified from a vascular tomographic image (cross-sectional image) using another machine learning model and the plaque burden is indirectly calculated will be described. Note that the same reference numerals are given to the contents overlapping with those in the first embodiment, and the description thereof is omitted.

[0080] FIG. 13 is a block diagram showing a configuration example of the image diagnostic apparatus 2 according to the second embodiment. The auxiliary storage unit 27 of the image diagnostic apparatus 2 according to the present embodiment stores the identification model 52. The identification model 52 is a machine learning model generated by learning predetermined training data, similar to the calculation model 51 according to the first embodiment, and is a learned model that takes a tomographic image of a blood vessel as an input and identifies a plaque region in the tomographic image. The identification model 52 is assumed to be used as a program module constituting a part of artificial intelligence software.

[0081] FIG. 14 is an explanatory diagram regarding the calculation process of the plaque burden according to the second embodiment. In FIG. 14, an image region corresponding to a plaque and a tunica media (hereinafter referred to as "plaque region") is identified from a cross-sectional image of a blood vessel using the identification model 52, and the state of calculating the plaque burden from the area of the identified plaque region is conceptually illustrated. Based on FIG. 14, the outline of the present embodiment will be described.

[0082] The identification model 52 is a machine learning model that has learned predetermined training data, and is, for example, a neural network generated by deep learning. In the present embodiment, the identification model 52 will be described as being a semantic segmentation model which is a type of CNN.

[0083] The semantic segmentation model is a neural network that identifies objects in an image in pixel units, and includes a convolutional layer (encoder) that convolves the input image and a transposed convolutional layer (decoder) that maps the convolved feature amounts to the original image size. In the transposed convolutional layer, based on the feature amounts extracted by the convolutional layer, it is identified which object exists at which position in the image, and data in which each pixel corresponds to which object is binarized is generated.

[0084] Server 1 generates an identification model 52 using training data (first training data) in which data indicating the correct plaque region is associated with the tomographic images for training. Server 1 inputs the tomographic images for training into the identification model 52 to identify the plaque region, and optimizes parameters such as the weights between neurons so that the identified plaque region approximates the correct plaque region. The image diagnostic apparatus 2 identifies the plaque region using the identification model 52 generated by Server 1.

[0085] In this embodiment, the identification model 52 will be described as a semantic segmentation model, but it may be a neural network other than the semantic segmentation model, a GAN (Generative Adversarial Network), or a model based on other learning algorithms.

[0086] Further, Server 1 may generate an identification model 52 capable of identifying regions other than the plaque region by learning training data provided with correct data for regions other than the plaque region (for example, the EEM region, the lumen region, etc.). That is, the identification model 52 only needs to be able to identify at least the plaque region, and may also be able to identify other image regions.

[0087] When imaging a patient's blood vessel, the image diagnostic apparatus 2 sequentially inputs the tomographic images of each frame into the identification model 52 to identify the plaque region in each frame. Then, the image diagnostic apparatus 2 calculates the plaque burden based on the identification result of the plaque region. That is, the image diagnostic apparatus 2 calculates the area of the plaque region and the area of the entire blood vessel including the plaque region, and calculates the plaque burden.

[0088] Thus, a configuration in which the machine learning model directly calculates the plaque burden is not essential, and the plaque burden may be calculated indirectly from the prediction result of the machine learning model.

[0089] FIG. 15 is a flowchart showing the procedure of the generation process of the identification model 52. Based on FIG. 15, the processing content when generating the identification model 52 by machine learning will be described. The control unit 11 of the server 1 acquires training data (first training data) for generating the identification model 52 (step S201). The training data is data in which data indicating the correct plaque region is associated with the cross-sectional images of blood vessels for training.

[0090] Based on the training data, the control unit 11 generates an identification model 52 that identifies the plaque region when a cross-sectional image of a blood vessel is input (step S202). Specifically, as described above, the control unit 11 generates a CNN related to semantic segmentation as the identification model 52. The control unit 11 inputs the training cross-sectional image into the identification model 52 to identify the plaque region, and compares it with the correct plaque region. The control unit 11 optimizes parameters such as the weights between neurons so that the two approximate, and generates the identification model 52. The control unit 11 ends a series of processes.

[0091] FIG. 16 is a flowchart showing the procedure of the plaque burden calculation process according to Embodiment 2. After acquiring cross-sectional images of a plurality of frames obtained by imaging a patient's blood vessel (step S31), the imaging diagnosis apparatus 2 executes the following processes. The control unit 21 of the imaging diagnosis apparatus 2 inputs the cross-sectional image of each frame into the identification model 52, and identifies the plaque region in each frame (step S221). Then, based on the identification result, the control unit 21 calculates the plaque burden in each frame (step S222). The control unit 21 shifts the process to step S31.

[0092] As described above, according to the second embodiment, the plaque burden can also be calculated using the identification result of the identification model 52 that identifies the plaque region from the cross-sectional image of the blood vessel.

[0093] (Modification 2) In Embodiment 2, the identification model 52 capable of identifying plaque regions in tomographic images was described. On the other hand, parameters such as weights obtained by learning the identification model 52 may be applied to the calculation model 51, and transfer learning may be performed to generate the calculation model 51 from the learning results of the identification model 52.

[0094] FIG. 17 is an explanatory diagram regarding the generation process of the calculation model 51 according to Modification 2. In FIG. 17, a part of the network structure and parameters of the identification model 52 (semantic segmentation model) are applied to the calculation model 51 (CNN), and the state of performing transfer learning is conceptually illustrated. In this modification, the server 1 is described as having already generated the identification model 52.

[0095] The server 1 performs learning by setting, as initial values, parameters such as weights obtained by generating (learning) the identification model 52 for a calculation model 51 having a network structure similar to a part of the network structure of the identification model 52. For example, the calculation model 51 has a structure in which an output layer for calculating plaque burden is connected to an input layer similar to the identification model 52 and a part of an intermediate layer (convolutional layer) corresponding to the decoder part of the identification model 52. The server 1 sets the parameters obtained by learning the identification model 52 as the initial values of the parameters of the intermediate layer, gives training data (first training data), and performs learning. The server 1 inputs a tomographic image for training, calculates the plaque burden, compares it with the correct value, and optimizes the parameters so that the two are approximated. By performing transfer learning in this way, the amount of data and learning time for which the calculation model 51 learns can be suppressed.

[0096] FIG. 18 is a flowchart showing the procedure of the generation process of the calculation model 51 according to Modification 2. After acquiring training data (first training data) for generating the calculation model 51 (step S11), the server 1 executes the following processing. The server 1 is described as having already generated the identification model 52.

[0097] The control unit 11 of server 1 generates a calculation model 51 based on the parameters obtained by generating the identification model 52 and the training data (step S301). Specifically, as described above, the control unit 11 sets the parameters such as the weights obtained by the learning of the identification model 52 as the initial values of the parameters of the calculation model 51 and then performs learning. The control unit 11 inputs the tomographic image for training into the calculation model 51 to calculate the plaque burden, compares it with the correct value, and optimizes the parameters so that the two are approximated. The control unit 11 ends a series of processes.

[0098] As described above, according to the first modification example, by diverting the learning result of the identification model 52, it is possible to suppress the amount of data and the learning time that the calculation model 51 learns.

[0099] The embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above meaning but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.

Explanation of Signs

[0100] 1 Server 11 Control unit 12 Main memory unit 13 Communication unit 14 Auxiliary storage unit P1 Program 2 Image diagnostic apparatus 21 Control unit 22 Main memory unit 23 Communication unit 24 Display unit 25 Input unit 26 Image processing unit 27 Auxiliary storage unit P2 Program 51 Calculation model 52 Identification model 5 Sagittal tomographic image 6 Gradation bar 7 Stent region

Claims

1. An acquisition unit that acquires cross-sectional images of a plurality of frames obtained by imaging a patient's blood vessel; A calculation unit that inputs the cross-sectional images of each acquired frame into a learned model that calculates a plaque burden when the cross-sectional images are input, and calculates the plaque burden for each frame; A display unit that displays a longitudinal cross-sectional image based on the cross-sectional images of the plurality of frames, and an object that is displayed corresponding to the longitudinal cross-sectional image, the first object representing the magnitude of the plaque burden at each position of the longitudinal cross-sectional image along the axial direction of the blood vessel; An information processing apparatus comprising the above.

2. The first object is a gradient bar in which the display color gradually changes according to the magnitude of the plaque burden. The information processing apparatus according to claim 1.

3. The display unit displays the gradient bar in which the display color is different between a region where the plaque burden is equal to or greater than a predetermined threshold value and a region where the threshold value is less than the threshold value. The information processing apparatus according to claim 2.

4. It includes a reception unit that receives a setting input for changing the threshold value, The gradient bar in which the display color is changed according to the set threshold value is displayed. The information processing apparatus according to claim 3.

5. The calculation unit calculates the plaque burden for each frame and a reliability indicating the certainty of the calculation result of the plaque burden for each frame, In the longitudinal cross-sectional image, it includes a specifying unit that specifies a low-reliability region where the reliability is equal to or less than a predetermined value, The display unit makes a part of the first object corresponding to the low-reliability region non-displayed. The information processing apparatus according to any one of claims 1 to 4.

6. It includes a length measuring unit that measures the length in the axial direction of the region where the plaque burden is equal to or greater than a predetermined threshold value. The display unit displays the measured length in association with the region equal to or greater than the threshold value in the longitudinal sectional image. The information processing apparatus according to any one of claims 1 to 5.

7. The display unit displays a second object representing the size of a predetermined treatment device corresponding to the longitudinal sectional image. The information processing apparatus according to any one of claims 1 to 6.

8. Obtain a plurality of frames of cross-sectional images of a patient's blood vessel. Input the obtained cross-sectional images of each frame into a learned model that calculates the plaque burden when the cross-sectional image is input, and calculate the plaque burden in each frame. Display on the display unit a longitudinal sectional image based on the plurality of frames of cross-sectional images and an object corresponding to the longitudinal sectional image, the first object representing the magnitude of the plaque burden at each position of the longitudinal sectional image along the axial direction of the blood vessel. An information processing method in which a computer executes processing.

9. Obtain a plurality of frames of cross-sectional images of a patient's blood vessel. Input the obtained cross-sectional images of each frame into a learned model that calculates the plaque burden when the cross-sectional image is input, and calculate the plaque burden in each frame. Display on the display unit a longitudinal sectional image based on the plurality of frames of cross-sectional images and an object corresponding to the longitudinal sectional image, the first object representing the magnitude of the plaque burden at each position of the longitudinal sectional image along the axial direction of the blood vessel. A program for causing a computer to execute processing.

Images