Image display method, image display program, and information processing apparatus

Abstract

To display an image in which an abnormal part of a blood vessel detected by artificial intelligence (AI), without disturbing visual diagnosis by a medical doctor.SOLUTION: An image display method (S1) comprises: an image acquisition process (S10) which acquires a first image including as a subject a blood vessel to be examined; a specification process (S11) which inputs the first image into a learned model and thereby specifies an abnormal part of the blood vessel; an image generation process (S13) which generates a second image in which a mark expressing the specified abnormal part is provided to the outside of the blood vessel; and a display control process (S14) which displays to a user the second image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image display method, an image display program, and an information processing device. [Background technology]

[0002] Ischemic heart diseases such as myocardial infarction and angina pectoris are diseases caused by the accumulation of fatty plaque on the inner wall of a coronary artery, which causes the coronary artery to become blocked or narrowed. In general, doctors visually diagnose whether an abnormality such as blockage or narrowing has occurred in a coronary artery based on CT (Computed Tomography) images or MRI (Magnetic Resonance Imaging) images that include the coronary artery as a subject.

[0003] In recent years, with the development of image analysis technology using artificial intelligence (AI), a technology for detecting abnormal parts of blood vessels using AI has been developed. Such a technology includes, for example, a technology described in Non-Patent Document 1 or Non-Patent Document 2, in which a contrast image of a coronary artery is input to AI and a stenosis part of the coronary artery is detected from the image. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] [online], Automated Detection of Coronary Artery Stenosis in X-ray Angiography using Deep Neural Networks[Retrieved June 1, 2023], Internet〈URL:https: / / arxiv.org / abs / 2103.02969〉 [Non-Patent Document 2] [online], healthcare in europe.com [searched on June 1, 2020], Internet <URL: https: / / healthcare-in-europe.com / en / news / ai-to-measure-coronary-plaque-in-six-seconds.html> Summary of the Invention [Problem to be solved by the invention]

[0005] The above technologies are still in development, and the accuracy of AI in detecting stenosis in coronary arteries is not high enough to replace a doctor's diagnosis. Therefore, in practice, doctors make visual diagnoses while referring to the AI ​​detection results.

[0006] However, the AI ​​detection results that doctors refer to are, for example, shown as an image of a coronary artery contrast image with a bounding box added to include the detected stenosis, or an image of the detected stenosis colored, which makes the original coronary artery contrast image difficult to see, hindering doctors' visual diagnosis.

[0007] One aspect of the present invention has been made in consideration of the above problems, and its purpose is to display an image in which abnormal parts of blood vessels detected by AI are marked in a manner that does not interfere with a doctor's visual diagnosis when diagnosing abnormal parts of blood vessels using AI in combination. [Means for solving the problem]

[0008] In order to solve the above problems, an image display method according to one embodiment of the present invention includes an image acquisition step of acquiring a first image including a blood vessel to be inspected as a subject; an identification step of identifying an abnormal part of the blood vessel to be inspected included as a subject in the first image by inputting the image including the blood vessel as input information into a trained model trained to detect abnormal parts of the blood vessel; an image generation step of generating a second image in which a mark representing the abnormal part identified in the identification step is added to the first image; and a display control step of displaying the second image to a user, wherein in the image generation step, the mark is added to the outside of the blood vessel near the abnormal part.

[0009] In order to solve the above problems, an image display device according to one embodiment of the present invention includes an image acquisition unit that acquires a first image including a blood vessel to be inspected as a subject, an identification unit that identifies an abnormal part of the blood vessel to be inspected included as a subject in the image by inputting the first image including the blood vessel as input information into a trained model that has been trained to detect abnormal parts of the blood vessel, an image generation unit that generates a second image in which a mark representing the abnormal part identified by the identification unit is added to the first image, and a display control unit that displays the second image to a user, wherein the image generation unit adds the mark to the outside of the blood vessel near the abnormal part.

[0010] The image display device according to each aspect of the present invention may be realized by a computer. In this case, the control program of the image display device that causes the computer to operate as each part (software element) of the image display device to realize the image display device, and the computer-readable recording medium on which it is recorded, also fall within the scope of the present invention. Effect of the Invention

[0011] According to one embodiment of the present invention, when diagnosing vascular abnormalities using AI, it is possible to display an image in which the abnormalities detected by the AI ​​are marked in a way that does not interfere with the doctor's visual diagnosis. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram showing a conventional technique. [Diagram 2] 1 is a block diagram showing a configuration of an image display system according to an embodiment of the present invention. [Diagram 3] 1 is a flowchart showing the flow of an image display method implemented by an image display system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a schematic diagram showing an example of the image generating process shown in FIG. 3. [Diagram 5] 3 is a schematic diagram showing an example of a display image displayed on a display unit of the image display device shown in FIG. 2. FIG. [Figure 6] 3 is a schematic diagram showing another example of a display image displayed on the display unit of the image display device shown in FIG. 2. [Figure 7] 3 is a block diagram showing an example of a configuration in which each unit of the information processing device shown in FIG. 2 is configured using a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [Embodiment 1] Hereinafter, one embodiment of the present invention will be described in detail with reference to FIGS.

[0014] (Overview of image display system 100) An overview of an image display system 100 according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an example of an image displayed by a conventional image display system.

[0015] The image display system 100 is an image display system including an information processing device 2 that identifies an abnormal part of a blood vessel to be examined from an image that includes the blood vessel as a subject, and displays to a user (e.g., a doctor) an image in which a mark representing the identified abnormal part is placed outside the blood vessel near the abnormal part.

[0016] In this embodiment, the "blood vessel to be inspected" refers to a coronary artery distributed on the surface of the heart. The "abnormal part" refers to a narrowed part of a blood vessel caused by unstable plaque, which causes ischemic heart disease. That is, the image display system 100 is a system that identifies a narrowed part of a coronary artery caused by unstable plaque and displays an image with a mark representing the narrowed part to a user. However, the image display system according to one aspect of the present disclosure is not limited to this. For example, in order to diagnose diseases such as aneurysm, thrombosis, arteriosclerosis, cerebral infarction, cerebral hemorrhage, cerebral arterial malformation, vascular trauma, peripheral arterial disease, vasculitis, renal artery stenosis, and aortic dissection, an image display system that identifies an abnormal part of a blood vessel corresponding to each disease and displays an image with a mark representing the identified abnormal part to a user is also within the scope of this embodiment.

[0017] In conventional image display systems, when AI detects an abnormal part of blood vessel X from an image (original image in Fig. 1) of blood vessel X to be examined, the system displays to the user an image in which the abnormal part of blood vessel X in the original image is colored and marked (lower left image in Fig. 1) or an image in which a bounding box is added to include the abnormal part of blood vessel X in the original image and the abnormal part is marked (lower right image in Fig. 1). This causes a problem of impeding the doctor's visual diagnosis.

[0018] In response to this, the image display system 100 identifies abnormal parts of the blood vessels to be examined from an image that contains the blood vessels as the subject of the examination, and places a mark representing the identified abnormal parts outside the blood vessels near the abnormal parts, thereby realizing an image display system that can display images that do not interfere with a doctor's visual diagnosis.

[0019] (Configuration of image display system 100) A schematic configuration of the image display system 100 will be described with reference to Fig. 2. Fig. 2 is a schematic diagram showing an example of the configuration of the image display system.

[0020] 2, the image display system 100 includes an imaging device 1, an information processing device 2, and an image display device 3. The information processing device 2 is configured to be able to communicate with the imaging device 1 and the image display device 3. The information processing device 2, the imaging device 1, and the image display device 3 only need to be able to transmit and receive data, and the communication format is not particularly limited. In this embodiment, it is assumed that the imaging device 1, the information processing device 2, and the image display device 3 are arranged in a facility (e.g., a medical institution) that uses the image display system 100, so that short-distance wireless communication such as WiFi (registered trademark) or Bluetooth (registered trademark) or wired communication is adopted as the communication format.

[0021] In addition, when the imaging device 1 and the image display device 3 are placed within a facility where the image display system 100 is used, and the information processing device 2 is placed outside the facility where the image display system 100 is used, the information processing device 2 is configured to be able to communicate with the imaging device 1 and the image display device 3 via long-distance wireless communication such as wireless LAN.

[0022] [Imaging device] The imaging device 1 is a device for imaging blood vessels of an inspection target. As an example, the imaging device 1 images a coronary artery. The imaging device 1 transmits imaging data of the blood vessels of the inspection target to the information processing device 2. In this embodiment, a known CT device is adopted as the imaging device 1.

[0023] A CT device is a device that rotates an X-ray tube and an X-ray detector around a subject (e.g., a patient), detects X-rays that have passed through the subject, and generates a cross-sectional image of the subject's living body based on the detected X-ray dose, etc. When a known CT device is used as the imaging device 1, data representing the detection results of X-rays that have passed through the subject, which are used to generate a cross-sectional image of the living body, or image data representing a cross-sectional image of the living body of the subject, is transmitted as imaging data to the information processing device 2. Here, the cross-sectional image of the living body is an image that includes the blood vessels of the subject as a subject.

[0024] In this embodiment, a known CT device is used as the imaging device 1, but the present invention is not limited to this. For example, a known MRI device may be used. In the following, a case where the imaging device 1 is a CT device will be described as an example.

[0025] [Information processing device] The information processing device 2 is a device for implementing an image display method according to one embodiment of the present invention. The flow of the image display method implemented by the information processing device 2 will be described later with reference to different drawings.

[0026] As shown in FIG. 2, the information processing device 2 includes a control unit 20, a communication unit 21, and a memory 22.

[0027] <Control Unit> The control unit 20 controls each of the components included in the information processing device 2. The control unit 20 also includes an image acquisition unit 200, a specification unit 201, an image generation unit 202, and a display control unit 203.

[0028] The image acquiring unit 200 acquires a first image including the blood vessel of the inspection target as a subject. As an example, the image acquiring unit 200 acquires the first image including the blood vessel of the inspection target as a subject by generating the first image based on the imaging data received from the imaging device 1.

[0029] The identification unit 201 identifies an abnormal part of the blood vessel to be inspected that is included as a subject in the first image acquired by the image acquisition unit 200. An example of an identification method implemented by the identification unit 201 is an identification method using a trained model. Specifically, the identification unit 201 first inputs the first image to a trained model that has been trained to detect an abnormal part of the blood vessel, using an image including a blood vessel as a subject as input information. Then, the identification unit 201 identifies the abnormal part of the blood vessel to be inspected by referring to the detection result by the trained model.

[0030] The identification unit 201 may have a function of the trained model, or a device other than the information processing device 2 may have the function of the trained model, and the identification unit 201 may output the first image to the device and obtain the detection result from the device. Details of the trained model will be described later.

[0031] The image generating unit 202 generates an image for displaying to a user the abnormal part of the blood vessel to be inspected identified by the identifying unit 201. As an example, the image generating unit 202 generates a second image in which a mark representing the abnormal part of the blood vessel identified by the identifying unit 201 is added to the outside of the blood vessel near the abnormal part of the blood vessel in a first image including the blood vessel to be inspected as a subject.

[0032] The display control unit 203 displays various images to the user. As an example, the display control unit 203 displays the second image generated in the image generation unit 202 to the user. One example of a method of displaying the second image to the user is a method of displaying the second image on a display unit 31 included in the image display device 3 owned by the user. Specifically, the display control unit 203 transmits image data representing the second image generated by the image generation unit 202 to the image display device 3 via the communication unit 21 of the information processing device 2, and outputs the second image to the display unit 31 included in the image display device 3. In this embodiment, a configuration is adopted in which the second image is displayed on the display unit 31 of the image display device 3, which is an external device of the information processing device 2, but the display control unit 203 may display the second image on a display unit (not shown) built into the information processing device 2.

[0033] The control unit 20 of the information processing device 2 may include at least one processor, and may have a configuration in which the processor reads and executes an information processing program recorded in the memory 22 to function as the image acquisition unit 200, the identification unit 201, the image generation unit 202, and the display control unit 203. Furthermore, the program for functioning as the image acquisition unit 200, the program for functioning as the identification unit 201, the program for functioning as the image generation unit 202, and the program for functioning as the display control unit 203 may each be separate programs, and may be recorded in the memory of separate devices. Such a configuration will be described later.

[0034] <Communications Department> The communication unit 21 communicates with the imaging device 1 and the image display device 3. As an example, the communication unit 21 receives imaging data from the imaging device 1 and transmits a second image to the image display device 3. In this embodiment, the communication unit 21 is a wireless communication interface such as WiFi or Bluetooth.

[0035] <Memory> Various types of information are recorded in the memory 22. As an example, the memory 22 records various programs executed by the control unit 20, a first image acquired by the image acquisition unit 200, and a second image generated by the image generation unit 202.

[0036] [Image display device] The image display device 3 is a device for displaying various images to a user. Devices that can be used as the image display device 3 include a desktop PC (Personal Computer), a tablet PC, a tablet terminal, a smartphone, and the like. In this embodiment, a tablet terminal carried by a user is assumed as the image display device 3. The image display device 3 also includes a communication unit 30 and a display unit 31, as shown in FIG. 2.

[0037] <Communications Department> The communication unit 30 communicates with the information processing device 2. As an example, the communication unit 30 receives image data representing a second image from the information processing device 2. The communication unit 30 may also accept a request to view the second image from a user at an input unit (not shown) of the image display device 3, and transmit a signal indicating the request to the information processing device 2.

[0038] <Display section> The display unit 31 displays various images. As an example, the display unit 31 displays a second image received from the information processing device 2. A specific example of the second image displayed on the display unit 31 will be described later.

[0039] (Pre-trained model) In this embodiment, the trained model used to identify an abnormal part of a blood vessel to be inspected will be described in detail. The trained model is a model trained to recognize an abnormal part of a blood vessel. The trained model's teacher data includes one or more pairs of an image including a blood vessel as a subject and an image in which an abnormal part in the image is labeled. For example, in the case of a trained model to recognize a stenosis caused by an unstable plaque in a coronary artery, the teacher data includes one or more pairs of an image including a coronary artery as a subject and an image in which a stenosis caused by an unstable plaque in the image is labeled. In this way, the trained model can be used to determine various diseases by adjusting the teacher data according to the abnormal part to be recognized.

[0040] The trained model uses an image including a blood vessel as an object as input information, and recognizes an abnormal part included in the input image. Then, the trained model outputs, as an example of a detection result, an image in which a bounding box is attached to a rectangular area including the recognized abnormal part (hereinafter, referred to as an "abnormal part area"). Note that the trained model may be a model generated by deep learning, or may be a model generated by machine learning other than deep learning.

[0041] (Flow of image display method) The image display method S1 executed by the information processing device 2 will be described with reference to Fig. 3. Fig. 3 is a flow diagram showing the flow of the image display method S1.

[0042] 3, the image display method S1 includes an image acquisition step S10, a specification step S11, a determination step S12, an image generation step S13, and a display control step S14. The timing at which the information processing device 2 performs the image display method S1 can be set arbitrarily. For example, the information processing device 2 may perform the image display method S1 every time it acquires imaging data from the imaging device 1.

[0043] [Image acquisition process] The image acquisition step S10 is a step for acquiring a first image including the blood vessel of the inspection target as a subject. In the image acquisition step S10, the image acquisition unit 200 generates an image including the blood vessel of the inspection target as a subject based on the imaging data captured by the imaging device 1, and acquires the image as the first image, for example. Examples of the image generated here include a CPR (Curve Planar Reconstruction) image and an MPR (Multi Planar Reconstruction) image. A CPR image is an image reconstructed as a curved surface along an arbitrary curve in a three-dimensional space, and is suitable for observing a long, meandering tubular structure such as a blood vessel or a site along a curved surface. An MPR image is an image cut out from an arbitrary cross section, and allows the positional relationship and shape of a blood vessel or an organ, which are difficult to grasp only from a cross-sectional image, to be viewed from an arbitrary direction. In this embodiment, a CPR image of the blood vessel of the inspection target is acquired as the first image, but is not limited thereto. The first image only needs to include the blood vessel of the inspection target as a subject and to include information about a center line indicating the center of the blood vessel. Here, the information about the center line may be information derived from a width line that is a straight line connecting opposing inner walls or outer walls of a blood vessel and perpendicular to the center line. In this case, it is sufficient that the first image includes information about the width line.

[0044] In this embodiment, a configuration is adopted in which an image including the blood vessels of the inspection target as a subject is generated based on imaging data received from the imaging device 1 and acquired as the first image, but a configuration may also be adopted in which the first image is generated from the imaging device 1 and the image acquisition unit 200 acquires the first image from the imaging device 1. Also, a configuration may be adopted in which a device other than the imaging device 1 has a function of generating the first image and the image acquisition unit 200 acquires the first image from the device.

[0045] [Specific process] The identification step S11 is a step for identifying an abnormal part of the blood vessel to be inspected in the first image acquired in the image acquisition step S10. As an example, in the identification step S11, the identification unit 201 of the information processing device 2 inputs the first image acquired in the image acquisition step S10 to a trained model, and acquires an output image in which a bounding box is added to the abnormal part area recognized by the trained model as a detection result. The identification unit 201 identifies the part in the output image to which the bounding box is added as an abnormal part of the blood vessel to be inspected.

[0046] [Judgment process] The determination step S12 is a step for determining whether or not an abnormal part is included in the blood vessel of the inspection target included as a subject in the first image. As an example, in the determination step S12, the identification unit 201 determines whether or not an abnormal part is included in the blood vessel of the inspection target by referring to the detection result of the trained model acquired in the identification step S11. For example, the identification unit 201 makes a determination based on whether or not a bounding box is attached to the output image acquired as the detection result. If a bounding box is attached to the output image, it is determined that "an abnormal part is present," and if a bounding box is not attached to the output image, it is determined that "no abnormal part is present."

[0047] In the determination step S12, when it is determined that the blood vessel to be inspected contained as a subject in the first image does not have an abnormal portion (NO in the determination step S12), the information processing device 2 performs the display control step S14 shown in FIG.

[0048] [Image generation process] In the determination step S12, when it is determined that the blood vessel to be inspected, which is included as a subject in the first image, has an "abnormal part" (YES in the determination step S12), the information processing device 2 carries out the image generating step S13.

[0049] The image generating step S13 is a step for generating a second image used by the image generating unit 202 to display to the user an abnormal part of the blood vessel of the inspection target included as a subject in the first image. As an example, in the image generating step S13, the image generating unit 202 of the information processing device 2 generates a second image in which a mark representing the abnormal part identified by the identifying unit 201 in the identifying step S11 is added to the outside of the blood vessel near the abnormal part in the first image. For example, the image generating unit 202 generates the second image based on an output image in which a bounding box is added to the abnormal part region of the blood vessel of the inspection target output as a detection result by the trained model in the identifying step S11.

[0050] In the image generating step S13, the process performed by the image generating unit 202 to generate the second image will be described with reference to Fig. 4. Fig. 4 is a schematic diagram for explaining the process performed to generate the second image.

[0051] The image generating unit 202 generates the second image by executing the following processes. (1) The process of projecting the vertices of a bounding box onto the center line. (2) The process of drawing perpendicular lines to the center line so that they pass through the center of each projection point. (3) The process of drawing lines connecting one end of each perpendicular line to the other end of each perpendicular line. (4) The process of deleting all lines except for those in (3).

[0052] FIG. 4(1) is a first image in which a bounding box (solid line frame) is added to the abnormal area of ​​the blood vessel to be inspected, which is output as a detection result by the trained model. Deposits P deposited on the inner wall of blood vessel X indicate unstable plaque that forms a stenosis, which is an abnormal part of blood vessel X, and the dashed line inside blood vessel X indicates the center line of blood vessel X. First, as shown in FIG. 4(2), the image generating unit 202 projects the vertices of the bounding box onto the center line of blood vessel X by drawing perpendicular lines from the vertices (black dots) of the bounding box to the center line (process (1)). Here, the vertices of the bounding box projected onto the center line are designated as projection points (white dots).

[0053] Next, the image generating unit 202 draws perpendicular lines (solid lines) that pass through the centers of the projection points (process (2)), as shown in Fig. 4(3). As an example, the perpendicular lines are all made the same length, and the midpoints of the perpendicular lines overlap the centers of the projection points.

[0054] Next, the image generating unit 202 draws a line (solid line) connecting one end of each perpendicular line to the other end of each perpendicular line, as shown in FIG. 4(4) (process (3)). The solid line drawn in process (3) is a marking representing an abnormal part of the blood vessel X. The marking is approximately parallel to the center line of the blood vessel X, and is therefore placed along the blood vessel even if it is elongated and meandering in a complex manner. This makes it easy to visually understand the correspondence between the marking representing the abnormal part and the position on the blood vessel X that the marking indicates. Note that one end of each perpendicular line is the end of the perpendicular line located on the blood vessel wall side where deposits P are deposited in the blood vessel X, and the other end is the blood vessel wall side where deposits P are not deposited in the blood vessel X.

[0055] Next, the image generating unit 202 deletes everything other than the line drawn in process (3) (process (4)), as shown in Fig. 4 (5). This allows a mark representing an abnormal part of the blood vessel X to be placed near the abnormal part and at a position that does not overlap with the blood vessel X. In addition, by placing a mark at a position that does not overlap with the blood vessel X, it has the effect of not interfering with the doctor's visual diagnosis.

[0056] In Fig. 4, in order to avoid complicating the drawing, the blood vessel X is shown as a straight line, and one deposit P and one bounding box are shown, but in reality, the blood vessel X is meandering, and multiple bounding boxes are attached to the deposit P (see the lower right diagram in Fig. 1). Even in such a case, the above-described processes (1) to (4) are executed for each bounding box, respectively, to generate the second image.

[0057] In this embodiment, the marking representing the abnormal part of the blood vessel to be inspected (blood vessel X in FIG. 4) is a solid line, but is not limited to this. For example, the marking may be various lines such as a dashed line or a dotted line, or any mark such as an arrow or a star may be arranged.

[0058] In the image generating step S13 of the present embodiment, the second image is generated by adding a marking representing an abnormal part of the blood vessel to be inspected, which is added along the blood vessel outside the blood vessel near the abnormal part, but the present invention is not limited to this. For example, the second image may be generated by further adding a marking representing an abnormal part of the blood vessel to be inspected, at least a part of which is located on the blood vessel. An example of such a marking is an arrow. As an example, the arrow is added so that at least a part of the arrow is located on the blood vessel to indicate the abnormal part corresponding to the marking added on the outside of the blood vessel, and the tip of the arrow indicates the apex of the abnormal part. This makes it possible to clearly show the user the correspondence between the marking added on the outside of the blood vessel and the abnormal part of the blood vessel X represented by the marking.

[0059] [Display control process] The display control step S14 is a step in which the display control unit 203 displays various images to the user. As an example, in the display control step S14, the display control unit 203 of the information processing device 2 displays the second image generated in the image generation step S13 on the display unit 31 of the image display device 3. Furthermore, when the blood vessel of the inspection target included as a subject in the first image is determined to have "no abnormality" in the determination step S12, the display control unit 203 similarly displays the image output by the trained model in the identification step S11 (i.e., the first image without a bounding box).

[0060] (Display example 1) A specific example of the second image displayed on the display unit 31 of the image display device 3 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the second image displayed on the display unit 31 of the image display device 3.

[0061] As shown in FIG. 5, the second image is marked with a mark indicating an abnormal part of the blood vessel to be inspected. The mark may further indicate the confidence level of the abnormal part identified in the identification step S11. The "confidence level of the abnormal part" is a score (numerical value) of the reliability (likelihood) of the abnormal part identified by the identification unit 201. The higher the numerical value, the higher the probability and reliability that the part is abnormal. As an example, when the output image of the trained model includes a bounding box, the identification unit 201 calculates a score that is the confidence level of the abnormal part for each bounding box based on the type of blood vessel to be inspected, the part to which the bounding box is attached, and any value set for any parameter such as medical information of the subject (e.g., patient), and a weighting coefficient. Here, the medical information of the subject includes the subject's age, medical history, lifestyle, allergies, blood pressure, weight, obesity, abdominal circumference, data such as protein, sugar, and occult blood obtained from urine or stool tests, data such as red blood cell count and white blood cell count obtained from general blood tests, data such as GPT, uric acid, blood sugar, neutral fat, and cholesterol obtained from blood biochemistry tests, data obtained from electrocardiograms or X-ray tests, and data obtained from X-ray or endoscopic tests of organs such as the stomach. The weighting coefficients can also be set arbitrarily and can be changed as appropriate. For example, if the maximum score is 1, any value between 0.1 and 0.9 is set for each parameter and weighting coefficient as follows: Parameters The output of the trained model is 0.9 0.6 as a type of blood vessel 0.5 for vascular area 0.4 for subject's medical information Weighting factor 0.7 for the output of the trained model 0.3 for each type of blood vessel 0.2 for vascular sites 0.1 for the subject's medical information ·Confidence level=0.9×0.7+0.6×0.3+0.5×0.2+0.4×0.1=0.86

[0062] In the above example, in the calculation of the score representing the confidence level of the identified abnormal part, 0.9 is set as the parameter when the trained model recognizes the abnormal part from the first image, but a constant of 0.9 may always be set, or an arbitrary variable between 0.1 and 0.9 may be set in consideration of the trained model's own recognition confidence level. In this case, as an example, the trained model outputs an image in which a bounding box corresponding to the recognition confidence level is attached to the recognized abnormal part as a detection result. A weighting coefficient corresponding to the recognition confidence level is further set, and a score is calculated for each bounding box based on the weighting coefficient and the weighting coefficient set in the arbitrary parameter described above. By reflecting the confidence level of the trained model in the detection of the abnormal part in the calculation of the score, the identification unit 201 can identify the abnormal part with higher accuracy.

[0063] In FIG. 5, the confidence level of the abnormal part represented by the score calculated as described above is expressed by changing the shade of the marking according to the score. A darker marking indicates a higher confidence level (higher score), and a lighter marking indicates a lower confidence level (lower score). When a user refers to the confidence level of the abnormal part recognized by the trained model, an output image to which a bounding box according to the confidence level of recognition is added is displayed on the display unit 31. Examples of the bounding box according to the confidence level of recognition include a bounding box whose shade of color differs depending on the confidence level, a bounding box whose color differs depending on the confidence level, and the like.

[0064] (Display example 2) Another specific example of the second image displayed on the display unit 31 of the image display device 3 will be described with reference to Fig. 6. Fig. 6 is a diagram showing another example of the second image displayed on the display unit 31 of the image display device 3.

[0065] As shown in FIG. 6, the second image is marked with a mark representing an abnormal part of the blood vessel to be inspected, and the mark may further represent the confidence level of the abnormal part identified in the identification step S11 and the risk level of the abnormal part. Here, the "confidence level of the identified abnormal part" is the same as in the display example 1. The "risk level of the abnormal part" is an evaluation of the necessity of medical intervention. For example, when the abnormal part is a stenosis caused by the accumulation of unstable plaque, the type of unstable plaque is determined, and the risk level of the stenosis is evaluated in three stages, middle, high, and very high, based on the combination of the types. In this case, middle indicates that follow-up observation is required, high indicates that medical intervention is required, and very high indicates that immediate medical intervention is required. The method and stage of evaluation can be arbitrarily set by the user. The risk level of such an abnormal part can be identified by a trained model that is trained to detect an abnormal part of the blood vessel using an image including a blood vessel as an object as input information, and detect the risk level of the abnormal part.

[0066] In Figure 6, the confidence level of the abnormality identified by the trained model is represented by the shade of the marking representing the abnormality in the blood vessel being examined, and the risk level of the abnormality (very high, high, medium) is represented by changing the color of the marking representing the abnormality in the blood vessel being examined.

[0067] In the display example 1 and the display example 2 of the present embodiment, the confidence level of the abnormal part identified by the trained model and the risk level of the abnormal part are expressed by the color or shade of the marking representing the abnormal part of the blood vessel to be inspected, but are not limited thereto. For example, the color or shade of the marking and the type of marking (solid line, arrow, star mark, line of different thickness, etc.) may be appropriately combined for expression. By using these combinations, it is possible to express multiple levels of risk in a limited space, so that the details of the risk level can be shown without interfering with the doctor's visual diagnosis.

[0068] (Effects of image display system 100) As described above, the image display system 100 performs the image display method S1 to display to the user a second image in which a mark representing an abnormal part of a blood vessel is placed outside the blood vessel near the abnormal part. This allows the user to perform a visual diagnosis based on the original image (first image) while referring to the detection result of the abnormal part by the trained model. In other words, the abnormal part in the blood vessel to be inspected can be diagnosed more accurately, and the necessary measures can be taken quickly. In addition, since the second image can be displayed as an intuitively easy-to-understand second image by changing the color, shade, and type of the marking, it can be used not only for diagnosis but also for other purposes such as displaying it when a doctor explains the condition to a patient or a patient's family.

[0069] (Additional features) The image display system 100 according to this embodiment may have a function of controlling the display of the marking added to the second image in response to an input operation by the user. A specific example of this function is shown below.

[0070] [Example 1] A specific example of the function of controlling the display of the markings attached to the second image in response to the user's input operation is a function of adjusting the ON / OFF of the markings in the second image. Here, a state where the user can confirm the markings attached to the second image is set to ON, and a state where the user cannot confirm the markings attached to the second image is set to OFF. For example, when the user wants to confirm the tissues at the position overlapping with the markings attached to the second image, the markings attached to the second image are turned OFF so that the user can confirm the tissues at the position overlapping with the markings. This function can be realized by increasing the transparency of the markings or deleting data related to the markings from the image data representing the second image when the image display device 3 receives an input operation from the user to turn OFF the markings in the second image.

[0071] This allows the user to diagnose an abnormal part of a blood vessel to be inspected, by taking into consideration the state of the tissue in the vicinity of the abnormal part, etc.

[0072] [Example 2] Another specific example of the function of controlling the display of the markings attached to the second image in response to the user's input operation is a function of selectively displaying the markings in the second image in response to the confidence level or risk level of the abnormal part represented by the markings. For example, only the markings representing the abnormal part having the confidence level with a predetermined score designated by the user or the risk level designated by the user are selectively displayed. This function can be realized by increasing the transparency of the markings that do not meet the conditions (score range or predetermined risk level) when the image display device 3 receives an input of the conditions of the markings to be displayed from the user (score range or predetermined risk level), or by deleting data related to the markings that do not meet the conditions from the image data representing the second image.

[0073] This allows the user to selectively display only the markings that the user desires in the second image, even if many markings indicating abnormal areas are displayed in the second image, thereby providing a second image that does not interfere with the user's visual diagnosis.

[0074] [Hardware configuration and software implementation examples] The information processing device 2 (mainly each unit included in the control unit 20) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit). In the latter case, the information processing device 2 may be configured using a computer (electronic calculator) as shown in FIG.

[0075] FIG. 7 is a block diagram showing an example of the configuration of a computer 4 that can be used as the information processing device 2. The computer 4 includes a calculation device 40, a main storage device 41, an auxiliary storage device 42, an input / output interface 43, and a communication interface 44, which are connected to each other via a bus. The calculation device 40, the main storage device 41, and the auxiliary storage device 42 may each be, for example, a CPU, a RAM (Random Access Memory), a solid-state drive, or a hard disk drive. The input / output interface 43 is connected to an input device 45 for a user to input various information to the computer 4, and an output device 46 for the computer 4 to output various information to the user. The input device 45 and the output device 46 may be built into the computer 4, or may be connected (externally) to the computer 4. For example, the input device 45 may be a button, a keyboard, a mouse, a touch sensor, or the like, and the output device 46 may be a lamp, a display, a printer, a speaker, or the like. In addition, a device having the functions of both the input device 45 and the output device 46, such as a touch panel in which a touch sensor and a display are integrated, may be applied. The communication interface 44 is an interface for the computer 4 to communicate with external devices.

[0076] The auxiliary storage device 42 stores an information processing program for operating the computer 4 as the information processing device 2. The arithmetic device 40 deploys the information processing program stored in the auxiliary storage device 42 on the main storage device 41 and executes instructions included in the information processing program, thereby causing the computer 4 to function as each unit of the information processing device 2. Note that the recording medium used by the auxiliary storage device 42 to record information such as the information processing program may be any computer-readable "non-transient tangible medium," and may be, for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like.

[0077] Also, a configuration may be adopted in which the computer 4 is made to function using a program recorded on a recording medium external to the computer 4, or a program supplied to the computer 4 via any transmission medium (such as a communication network or broadcast waves).The present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0078] 〔summary〕 The image display method according to aspect 1 of this embodiment includes an image acquisition step of acquiring a first image including a blood vessel to be inspected as a subject, an identification step of identifying an abnormal portion of the blood vessel to be inspected included as a subject in the first image by inputting the image including the blood vessel as input information into a trained model trained to detect abnormal portions of the blood vessel, an image generation step of generating a second image in which a mark representing the abnormal portion identified in the identification step is added to the first image, and a display control step of displaying the second image to a user, wherein in the image generation step, the mark is added to the outside of the blood vessel near the abnormal portion.

[0079] According to the above configuration, abnormalities in the blood vessels being inspected are diagnosed while referring to the abnormalities identified based on the detection results of the trained model, thereby enabling a more accurate and efficient diagnosis.

[0080] In the image display method according to Aspect 2 of the present embodiment, in the above-mentioned Aspect 1, the marking is provided along the blood vessel of the inspection object.

[0081] According to the above configuration, by marking along the blood vessel to be inspected, which has an elongated shape or a complexly meandering shape, abnormal parts of the blood vessel can be more clearly indicated to the user.

[0082] In the image display method according to aspect 3 of the present embodiment, in aspect 1 or 2 above, the image generation process generates the second image based on information indicating the center line of the blood vessel to be inspected and information indicating an abnormality region including the abnormality identified in the identification process.

[0083] According to the above configuration, the marking representing the abnormal part of the blood vessel to be inspected can be placed outside the blood vessel and along the center line of the blood vessel, thereby making it possible to mark the abnormal part of the blood vessel to be inspected without interfering with the doctor's visual diagnosis.

[0084] An image display method according to aspect 4 of the present embodiment is such that, in aspect 3 above, the abnormal area is a rectangular area including the abnormal area, and the image generation process projects each vertex of the rectangular area onto the center line, and generates the marking so as to connect each of the perpendicular lines passing through the projection points onto which each vertex is projected.

[0085] According to the above configuration, a mark representing an abnormal part of a blood vessel to be inspected can be placed outside the blood vessel and along the center line of the blood vessel, thereby making it possible to mark the abnormal part of the blood vessel to be inspected without interfering with a doctor's visual diagnosis.

[0086] An image display method according to aspect 5 of this embodiment is any one of aspects 1 to 4 above, in which the image generation process further adds to the first image a mark representing the abnormal area identified in the identification process, at least a portion of which is located on the blood vessel of the subject of inspection.

[0087] According to the above configuration, abnormal parts of the blood vessels to be inspected can be marked accurately without interfering with the user's visual diagnosis.

[0088] The image display method according to a sixth aspect of the present embodiment is any one of the first to fifth aspects, wherein the marking further indicates a degree of confidence of the abnormal part identified in the identifying step.

[0089] With the above configuration, the user can make an efficient diagnosis by utilizing the confidence level of the identified abnormal part.

[0090] The image display method according to aspect 7 of the present embodiment is any one of aspects 1 to 6 above, wherein the trained model is generated by machine learning using an image in which abnormal parts of blood vessels are labeled in an image that includes blood vessels as subjects.

[0091] With the above configuration, the trained model can effectively detect abnormal parts of blood vessels.

[0092] The image display method according to an eighth aspect of the present embodiment is any one of the first to seventh aspects, wherein the abnormal part is a stenosis.

[0093] According to the above configuration, it is possible to identify the stenosis in the blood vessel to be inspected, thereby enabling the user to administer necessary treatment to the patient to eliminate the stenosis in the blood vessel.

[0094] The image display program according to aspect 9 of this embodiment is a program for causing at least one processor to execute an image acquisition process for acquiring a first image including a blood vessel to be inspected as a subject; a specific process for identifying an abnormal part of the blood vessel to be inspected that is included as a subject in an image including the blood vessel as a subject by inputting the first image into a trained model that has been trained to detect abnormal parts of the blood vessel using the image including the blood vessel as input information; an image generation process for generating a second image in which a mark representing the abnormal part identified in the identification process is added to the first image; and a display control process for displaying the second image to a user, wherein in the image generation process, the mark is added to the outside of the blood vessel near the abnormal part.

[0095] According to the above configuration, abnormalities in the blood vessels being inspected are diagnosed while referring to the abnormalities identified based on the detection results of the trained model, thereby enabling a more accurate and efficient diagnosis.

[0096] An information processing device according to aspect 10 of this embodiment includes an image acquisition unit that acquires a first image including a blood vessel to be inspected as a subject, an identification unit that identifies an abnormal portion of the blood vessel to be inspected that is included as a subject in the image by inputting the first image including the blood vessel as input information into a trained model that has been trained to detect abnormal portions of the blood vessel, an image generation unit that generates a second image in which a mark representing the abnormal portion identified by the identification unit is added to the first image, and a display control unit that displays the second image to a user, wherein the image generation unit adds the mark to the outside of the blood vessel near the abnormal portion.

[0097] According to the above configuration, abnormalities in the blood vessels being inspected are diagnosed while referring to the abnormalities identified based on the detection results of the trained model, thereby enabling a more accurate and efficient diagnosis.

[0098] In the information processing device according to Aspect 11 of this embodiment, in Aspect 10 above, the display control unit outputs the second image generated in the image generation unit to an image display device.

[0099] According to the above configuration, the information processing device can output the second image to an image display device, which is an external device, allowing a user to use the second image for purposes such as explaining the condition of a patient.

[0100] The image display method according to aspect 12 of this embodiment includes an image acquisition step of acquiring a first image including a blood vessel to be inspected as a subject, an identification step of identifying an abnormal part of the blood vessel to be inspected included as a subject in the first image including the blood vessel as a subject by using an artificial intelligence algorithm to detect an abnormal part of the blood vessel, an image generation step of generating a second image in which a mark representing the abnormal part identified in the identification step is added to the first image, and a display control step of displaying the second image to a user, wherein in the image generation step, the mark is added to the outside of the blood vessel near the abnormal part.

[0101] According to the above configuration, the same effects as those of the first aspect are achieved.

[0102] [Additional Notes] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0103] 100 Image display system 1. Imaging device 2. Information processing equipment 3 Image display device 4. Computer

Claims

1. an image acquiring step of acquiring a first image including a blood vessel to be inspected as an object; a step of inputting the first image into a trained model trained to detect an abnormal part of a blood vessel using an image including the blood vessel as an object as input information, thereby identifying an abnormal part of the blood vessel of the inspection target included in the image as an object; an image generating step of generating a second image by adding a mark representing the abnormal part identified in the identifying step to the first image; a display control step of displaying the second image to a user; In the image generating step, the marking is added to the outside of the blood vessel near the abnormal part.

13. An image display method comprising:

2. The markings are applied along the blood vessel of the subject.

2. The image display method according to claim 1.

3. The image generating step generates the second image based on information indicating a center line of the blood vessel of the inspection target and information indicating an abnormal part region including the abnormal part identified in the identifying step.

3. The image display method according to claim 1 or 2.

4. the abnormality region is a rectangular region including the abnormality, The image generating step projects each vertex of the rectangular region onto the center line; generating the markings so as to connect each of the perpendicular lines to the center line, the perpendicular lines passing through the projection points of the vertices; 4. The image display method according to claim 3.

5. The image generating step further includes attaching to the first image a mark representing the abnormal part identified in the identifying step, the mark being at least partially located on the blood vessel of the inspection target.

2. The image display method according to claim 1.

6. The indication further indicates a degree of confidence of the abnormality identified in the identification step.

2. The image display method according to claim 1.

7. The trained model is generated by machine learning using an image in which an abnormal part of a blood vessel is labeled in an image including the blood vessel as a subject.

2. The image display method according to claim 1.

8. The abnormal portion is a stenosis portion.

2. The image display method according to claim 1.

9. At least one processor, an image acquisition process for acquiring a first image including a blood vessel to be inspected as an object; A process of inputting the first image into a trained model trained to detect an abnormal part of a blood vessel using an image including a blood vessel as an object as input information, thereby identifying an abnormal part of the blood vessel of the inspection target included as an object in the image; an image generating process for generating a second image by adding a mark representing the abnormal part identified in the identification process to the first image; a display control process for displaying the second image to a user, In the image generation process, the marking is added to the outside of the blood vessel near the abnormal part.

13. An image display program comprising:

10. an image acquisition unit that acquires a first image including a blood vessel to be inspected as a subject; an identification unit that inputs the first image into a trained model that is trained to detect an abnormal part of a blood vessel using an image including the blood vessel as an object as input information, thereby identifying an abnormal part of the blood vessel of the inspection target that is included as an object in the image; an image generating unit that generates a second image by adding a mark representing the abnormal part identified by the identifying unit to the first image; a display control unit that displays the second image to a user; The image generating unit places the mark outside the blood vessel near the abnormal part.

23. An information processing apparatus comprising:

11. The display control unit outputs the second image generated by the image generation unit to an image display device.

11. The information processing apparatus according to claim 10,

12. an image acquiring step of acquiring a first image including a blood vessel to be inspected as an object; a step of identifying an abnormal part of the blood vessel of the inspection target included as a subject in the first image by an artificial intelligence algorithm for detecting an abnormal part of the blood vessel from an image including the blood vessel as a subject; an image generating step of generating a second image by adding a mark representing the abnormal part identified in the identifying step to the first image; a display control step of displaying the second image to a user; In the image generating step, the marking is added to the outside of the blood vessel near the abnormal part.

13. An image display method comprising:

Images