Image processing device, image processing method, and program

Abstract

Selectively extracting a portion of a structure that has a branched shape. [Solution] The image processing apparatus according to the embodiment includes: a first acquisition unit that acquires volume data of a structure having a branched shape; a first extraction unit that extracts the core wires of the structure based on the volume data; a setting unit that sets a section for at least a part of the core wires; a second acquisition unit that acquires a cross-sectional image or a three-dimensional image including at least a part of the section based on the volume data; a likelihood information holding unit that holds likelihood information relating to the likelihood of existence in the cross-sectional image or the three-dimensional image; and a second extraction unit that extracts an extraction target region of the structure corresponding to the section based on the cross-sectional image or the three-dimensional image and the likelihood information.

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] An extraction process of anatomical structures may be performed on medical images. Here, the object of the extraction process may be not the whole of a specific structure but a part thereof. Also, although the structures to be extracted are various, some have a branched shape, such as blood vessels. In this case, it may be required to selectively extract only a part of the branch end of the structure.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to selectively extract a part of a structure having a branched shape. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. The problems corresponding to the respective effects of the respective configurations shown in the embodiments described later can also be regarded as other problems.

Means for Solving the Problems

[0005] The image processing apparatus according to the embodiment includes: a first acquisition unit that acquires volume data of a structure having a branched shape; a first extraction unit that extracts the core wires of the structure based on the volume data; a setting unit that sets a section for at least a part of the core wires; a second acquisition unit that acquires a cross-sectional image or a three-dimensional image including at least a part of the section based on the volume data; a likelihood information holding unit that holds likelihood information relating to the likelihood of existence in the cross-sectional image or the three-dimensional image; and a second extraction unit that extracts an extraction target region of the structure corresponding to the section based on the cross-sectional image or the three-dimensional image and the likelihood information. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a diagram showing the configuration of an image processing apparatus according to the first embodiment. [Figure 2] Figure 2 is a flowchart showing the processing procedure of the image processing apparatus according to the first embodiment. [Figure 3] Figure 3 illustrates a medical image, which is the input to the image processing apparatus according to the first embodiment. [Figure 4] Figure 4 is a diagram showing an example of the first region according to the first embodiment. [Figure 5] Figure 5 shows the core wire according to the first embodiment. [Figure 6] Figure 6 shows an example of a section set for the core wire according to the first embodiment. [Figure 7] Figure 7 shows an example of a cross-sectional image set for a section according to the first embodiment. [Figure 8] Figure 8 shows an example of a three-dimensional image set for a section according to the first embodiment. [Figure 9] Figure 9 shows an example of a cross-sectional image set for a section according to the first embodiment. [Figure 10] Figure 10 shows an example of likelihood information according to the first embodiment. [Figure 11]Figure 11 shows an example of the extraction results of the target region according to the first embodiment. [Figure 12] Figure 12 shows an example of the extraction results of the target region according to the first embodiment. [Figure 13] Figure 13 shows an example of the extraction results of the target region according to the first embodiment. [Figure 14] Figure 14 shows an example of a cross-sectional image set for a section according to the second embodiment. [Figure 15] Figure 15 shows an example of the extraction results of the target region according to the second embodiment. [Figure 16] Figure 16 shows an example of a method for setting intervals according to the third embodiment. [Figure 17] Figure 17 is a flowchart showing the processing procedure of an image processing apparatus according to the third embodiment. [Modes for carrying out the invention]

[0007] Embodiments of the present invention will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing are denoted by the same reference numerals, and redundant explanations are omitted as appropriate. Furthermore, some components, members, and processes are omitted in each drawing.

[0008] The following explanation will use CT images acquired by an X-ray computed tomography (CT) scanner as an example. However, the embodiments are not limited to this, and the same can be applied to medical images of different types than CT images. For example, the embodiments described below can also be applied to medical images acquired by modalities other than X-ray CT scanners, such as magnetic resonance imaging (MRI) scanners, positron emission tomography (PET) scanners, SPECT (Single Photon Emission Computed Tomography) scanners, ultrasound scanners, and optical coherence tomography (OCT) scanners.

[0009] Furthermore, the coronary artery will be described below as an example of an anatomical structure. However, the embodiments are not limited to this, and can also be applied to blood vessels other than coronary arteries, or to structures other than blood vessels. In other words, the embodiments described below can be applied to any structure having a branched shape. Anatomical structures will also be referred to simply as "structures" or "tissues."

[0010] <First Embodiment> Referring to Figure 1, the configuration of the image processing device 100 according to the first embodiment will be described. The image processing device 100 is a device for extracting coronary arteries depicted in a CT image. In addition to the image processing device 100, there is a data server 170 for storing data that is input to the image processing device 100 and data that is output by the image processing device 100. The data server 170 is an example of a computer storage medium, and is a large-capacity information storage device such as a hard disk drive (HDD) or solid state drive (SSD). The data server 170 may be held within the image processing device 100, or it may be provided separately outside the image processing device 100 and configured to be communicated via a network.

[0011] Referring to FIG. 1, the configuration of the image processing apparatus 100 will be described in more detail. The image processing apparatus 100 includes a first acquisition unit 110, a first extraction unit 120, a setting unit 130, a second acquisition unit 140, a likelihood information holding unit 150, and a second extraction unit 160, and is connected to a data server 170.

[0012] The first acquisition unit 110, the first extraction unit 120, the setting unit 130, the second acquisition unit 140, the likelihood information holding unit 150, and the second extraction unit 160 are realized by a processing circuit included in the image processing apparatus 100 executing the code included in the program.

[0013] For example, the image processing apparatus 100 includes a storage unit (not shown), and each processing function corresponding to the first acquisition unit 110, the first extraction unit 120, the setting unit 130, the second acquisition unit 140, the likelihood information holding unit 150, and the second extraction unit 160 is stored in the storage unit in the form of a program executable by a computer. The processing circuit included in the image processing apparatus 100 is a processor that realizes the functions corresponding to each program by reading and executing the program from the storage unit. In other words, the processing circuit in the state of having read the program has the functions corresponding to the read program. The first acquisition unit 110, the first extraction unit 120, the setting unit 130, the second acquisition unit 140, the likelihood information holding unit 150, and the second extraction unit 160 may be realized by being appropriately distributed or integrated into a single or a plurality of processing circuits.

[0014] The first acquisition unit 110 acquires volume data of a structure having a branched shape. The volume data is a three-dimensional medical image. In the following description, the volume data is also simply referred to as a "medical image". In the present embodiment, a CT image will be described as an example of the volume data. The CT image may be the image information itself, or may be a data set of the image information and various additional information. Further, the first acquisition unit 110 may acquire the CT image from the data server 170, or may directly acquire the CT image from an X-ray CT apparatus that has collected the CT image.

[0015] Here, a structure with a branched shape is, for example, a coronary artery. The coronary artery has a shape that originates at the point where it connects to the aorta and branches out to run to multiple peripheral parts. Structures like the coronary artery, which run from a certain origin to multiple peripheral parts via branches, are also referred to as tree structures. In other words, the first acquisition unit 110 acquires medical images of tree-structured structures.

[0016] An example of a medical image is shown in Figure 3. Image 300 in Figure 3 is an example of a medical image. Medical image 300 is a CT image taken near the heart. For illustrative purposes, a single axial cross-sectional image is shown in this figure, but the medical image in this embodiment is a three-dimensional image that includes volume data.

[0017] Medical image 300 depicts the heart 301, the aorta 302, and the coronary arteries 303. The coronary arteries 303 branch into two at the positions indicated by arrows 304 and 305. That is, the coronary arteries 303 shown in Figure 3 run from their origin to multiple peripheral parts via multiple branching points indicated by arrows 304 and 305. Structures that repeatedly branch in this manner are the target of extraction processing by the image processing device 100.

[0018] Although Figure 3 also shows the aorta 302, the medical image 300 does not necessarily have to include the aorta 302. Also, although Figure 3 shows the entire coronary artery 303 from its origin to its periphery, the medical image 300 may include only a portion of the coronary artery 303. The medical image 300 includes at least one bifurcation in the coronary artery 303.

[0019] The first acquisition unit 110 also acquires parameters from the data server 170 for operating the deep learning network. Details of the deep learning network and parameters will be described in the description of the second extraction unit 160.

[0020] The first extraction unit 120 extracts the core lines of the coronary arteries. For example, the first extraction unit 120 first extracts the coronary arteries from the medical image. Hereafter, the region extracted by the first extraction unit 120 will be referred to as the first region or structural image. Any known region extraction method can be used to extract the first region. For example, the graph cut method may be used, or a deep learning-based method such as U-Net may be used.

[0021] An example of the first region extracted by the first extraction unit 120 is shown in Figure 4. Image 400 in Figure 4 is the image output by the first extraction unit 120 when the medical image 300 in Figure 3 is input to the first extraction unit 120, and region 401 in image 400 is the first region extracted by the first extraction unit 120. For example, image 400 is a binary image in which the pixel values ​​other than region 401 are set to "0". As can be seen from the comparison between image 300 and image 400, the first region 401 corresponds to the coronary artery 303, and it can be seen that the first extraction unit 120 has extracted the coronary artery 303 fairly well.

[0022] Next, the first extraction unit 120 extracts the core line of the coronary artery based on the first region 401. That is, the first extraction unit 120 extracts a line representing the first region 401 as the core line of the coronary artery. In the following explanation, when viewed from a certain branching point in the tree structure, the origin side will be referred to as the root side and the peripheral side as the leaf side.

[0023] For example, the core line acquired by the first extraction unit 120 is a line that travels along the direction of travel of the structure that the image processing device 100 is extracting, and is a line that connects one end on the root side of the structure with one end on the leaf side of the structure. To explain it in other words, the core line is a line that travels along the direction of travel of the first region, and is a line that connects one end on the root side of the first region with one end on the leaf side of the first region. In this embodiment, the structure that the image processing device 100 is extracting is a coronary artery, so the end on the root side of the structure is the end on which the coronary artery branches off from the aorta, and the end on the leaf side of the structure is the end on which it is away from the aorta.

[0024] Figure 5 shows an example of core wires extracted by the first extraction unit 120. The solid lines in Figure 5 (core wires 501, 502, 503, 504, and 505) are examples of core wires obtained by the first extraction unit 120 from the first region 401 in Figure 4. Endpoint 511 is one end on the root side of the first region 401, and endpoints 513, 514, and 515 are one end on the leaf side of the first region 401. As can be seen, core wires 501, 502, 503, 504, and 505 run along the direction of travel of the first region 401 and connect one end on the root side of the first region to one end on the leaf side of the first region.

[0025] The method for obtaining the core wire performed by the first extraction unit 120 can be any method that yields a core wire that satisfies the above-mentioned requirements for the core wire. An example of a method for obtaining the core wire will be explained in step S1030 of the processing procedure described later.

[0026] The setting unit 130 sets a section for the core wires obtained by the first extraction unit 120. The section will be explained with reference to Figure 6. In Figure 6(a), core wires 501, 502, and 504 obtained by the first extraction unit 120 are shown as solid lines, and core wires 503 and 505 are shown as dashed lines. In Figure 6(a), core wires 501, 502, and 504 shown as solid lines are the sections set by the setting unit 130. As in this example, the setting unit 130 sets a section for part or all of the core wires obtained by the first extraction unit 120. However, the setting unit 130 sets the section so that it does not simultaneously include multiple core wires that branch off from the source core wire. For example, core wires 502 and 503 are not included in the section at the same time. Also, core wires 504 and 505 are not included in the section at the same time.

[0027] In other words, the setting unit 130 sets a section of the core line of a structure having a branching section, passing through the branching section from the root side to the leaf side, and forming a single line without branching. The set section may be a curve or a broken line, but it must be a line that can be drawn in one continuous stroke (one-stroke).

[0028] There are various ways to set the interval using the setting unit 130. For example, the setting unit 130 may set the interval from one end on the root side of a core wire to one end on the leaf side, as shown for core wires 501, 502, and 504 in Figure 6(a). Alternatively, the setting unit 130 may set the interval from one end on the root side of a core wire to the branch point of the core wire, as shown for core wire 501. Furthermore, the setting unit 130 may set the interval from the branch point of a core wire to one end on the leaf side, as shown for core wire 504. Also, the setting unit 130 may set the interval from one branch point to the next branch point, as shown for core wire 502. Finally, the setting unit 130 may set the interval to include the branch point of the core wire, as shown in Figure 6(b). In Figure 6(b), core wires 612 and 615, shown by solid lines, are set as intervals.

[0029] The setting unit 130 may set intervals based on the individual structures that make up the structure. For example, the interval may be set from the point immediately after the branching off from the aorta to the end of the coronary artery. Alternatively, the setting unit 130 may set intervals in units of coronary artery branches as defined by the AHA classification set by the American Heart Association (AHA). Furthermore, the setting unit 130 may change the way intervals are set for the left coronary artery and the right coronary artery.

[0030] The above describes the sections that the setting unit 130 sets for the core wires. For example, the setting unit 130 sets sections for the core wires according to the section setting method selected by the operator (not shown). Alternatively, for example, the setting unit 130 sets sections for the core wires according to the section setting method recorded in the data server 170.

[0031] In this embodiment, the setting unit 130 selectively sets one of the multiple sections divided by the method described above. Here, the selection of a section may be made by accepting a selection from the operator, or it may be made based on a predetermined rule. Structures belonging to the selected section are the structures to be extracted in this embodiment, and structures that do not belong to the selected section are structures of the same type as the structures to be extracted but are not structures to be extracted.

[0032] In other words, in structures with branched shapes, such as coronary arteries, only a portion of the branched areas may be the area of ​​interest. The area of ​​interest refers to the object of observation or image processing by a physician. For example, in the case of coronary arteries, only one specific vessel on the lobar side that can be reached from the aorta via multiple branches may be the area of ​​observation or treatment. Specifically, only vessels with lesions (calcification, plaque, etc.) or vessels that are narrowed due to lesions may be the area of ​​observation or treatment by a physician. Whether automatic or manual, the section set by the setting unit 130 relative to the core wire is set according to the area of ​​interest.

[0033] The second acquisition unit 140 sets at least one partial image for at least a portion of the section set by the setting unit 130. The partial image may be a two-dimensional image (cross-sectional image) or a three-dimensional image.

[0034] For example, the second acquisition unit 140 generates a cross-sectional image including the core line for at least a portion of the section set by the setting unit 130. Such a cross-sectional image is an example of a two-dimensional partial image. The cross-sectional image may be a two-dimensional image of a cross-section intersecting the core line, or a two-dimensional image of a cross-section along the core line. In other words, the cross-sectional image can be any image as long as it includes the core line or intersects with the core line.

[0035] Figures 7(a) and 7(b) show examples of two-dimensional cross-sectional images. The two-dimensional cross-sectional image 701 is a cross-sectional image generated from the medical image 300, and is set to intersect with the core line section 702 set by the setting unit 130.

[0036] For example, the setting unit 130 sets a cross-sectional image 701 that is approximately perpendicular to the core line. In Figure 7(a), the coronary artery 303 included in the medical image 300 is shown divided into three regions: region 703, region 704, and region 705. Region 703 corresponds to the core lines 501, 502, and 503 of the coronary artery 303 shown in Figure 5. Region 704 corresponds to the core line 504 of the coronary artery 303 shown in Figure 5. Region 705 corresponds to the core line 505 of the coronary artery 303 shown in Figure 5. Regions 703, 704, and 705 are extracted as the first region by the first extraction unit 120. Cross-sectional image 701 intersects with region 704 at position 706. Cross-sectional image 701 also intersects with region 705 at position 707.

[0037] Figure 7(b) shows a front view of the cross-sectional image 701. In Figure 7(b), the cross-sectional region of the cross-sectional image 701 and region 704 at position 706 (cross-sectional region 715), and the cross-sectional region of the cross-sectional image 701 and region 705 at position 707 (cross-sectional region 716) are shown on the cross-sectional image 701. That is, if regions 704 and 705 were extracted as the first region, these first regions would be projected onto the cross-sectional image 701 as cross-sectional regions 715 and 716. Alternatively, if coronary arteries are extracted using a known region extraction method on the cross-sectional image 701, cross-sectional regions 715 and 716 would be extracted.

[0038] Both cross-sectional regions 715 and 716 are part of the same structure (a coronary artery in this embodiment), and these cross-sectional regions have similar image characteristics. Therefore, with conventional image processing methods, it is difficult to extract only the desired cross-sectional region 715 and not extract cross-sectional region 716 (or vice versa). However, the image processing apparatus 100 according to this embodiment is capable of correctly extracting only the cross-sectional region 715.

[0039] Note that the cross-sectional regions 715 and 716 in Figure 7(b) are illustrations of the extraction results when coronary arteries are extracted using a known region extraction method, for illustrative purposes. The extraction process shown in Figure 7(b) may be omitted. In other words, it is not necessary for cross-sectional regions 715 and 716 to be extracted in the cross-sectional image 701.

[0040] The second acquisition unit 140 may generate a three-dimensional image as a partial image. Figure 8 shows an example of a three-dimensional partial image. The three-dimensional image 801 is a partial image generated from the medical image 300, and is set to include the core line section 702 set by the setting unit 130. In Figure 8, the three-dimensional image 801 is set to include the entire section 702, but the three-dimensional image 801 may include only a part of the section 702. Also in Figure 8, the coronary artery 303 included in the medical image 300 is shown divided into three regions: region 803, region 804, and region 805. Region 803 is the region of the coronary artery 303 corresponding to core lines 501, 502, and 504 shown in Figure 5. Region 804 is the region of the coronary artery 303 corresponding to core line 503 shown in Figure 5. Region 805 is the region of the coronary artery 303 corresponding to core line 505 shown in Figure 5. Regions 803, 804, and 805 are extracted as the first region by the first extraction unit 120. In Figure 8, region 803 is the structure to be extracted (coronary artery in this embodiment), while regions 804 and 805 are not structures to be extracted. The image processing device 100 according to this embodiment correctly extracts only region 803.

[0041] The second acquisition unit 140 may generate CPR images (curved planar reconstruction, or curved planar reformation) or SPR images (stretched CPR, or straightened CPR) as cross-sectional images. Figure 9 shows an example of an SPR image. The SPR image 900 is a cross-sectional image generated from the medical image 300 and is generated to include the core line section 702 set by the setting unit 130. In this figure, the cross-sectional region 902 is the structure to be extracted (coronary artery in this embodiment), while the cross-sectional regions 903 and 904 are not the structures to be extracted. The image processing device 100 according to this embodiment correctly extracts only the cross-sectional region 902. The SPR image in Figure 9 is an example of a two-dimensional image of a cross-section along the core line within the section. Note that SPR images can also be generated for cross-sections that intersect with the core line within the section.

[0042] The second acquisition unit 140 may generate partial images with a higher or lower spatial resolution than the medical image 300. For example, the second acquisition unit 140 generates a cross-sectional image or a three-dimensional image with a higher spatial resolution than the volume data acquired by the first acquisition unit 110. When the second acquisition unit 140 generates partial images with a higher spatial resolution than the medical image 300, the second extraction unit 160 (described later) can extract the target structure (coronary arteries in this embodiment) with high accuracy. Conversely, when the partial images are generated with a lower spatial resolution than the medical image 300, the second extraction unit 160 can extract the target structure with fewer computing resources (computation time, computer memory usage, etc.).

[0043] The likelihood information storage unit 150 generates and stores likelihood information relating to the likelihood of existence in a partial image. Likelihood information is information indicating the location (location-related information) and size (size-related information) of the structure (coronary artery in this embodiment) that the image processing device 100 is trying to extract in the partial image acquired by the second acquisition unit 140. The likelihood information is expressed, for example, in image format, with the pixel values ​​of the image representing the location-related information and the size-related information.

[0044] The likelihood information will be further explained with reference to Figure 10. Figure 10(a) is a diagram illustrating the likelihood information when the partial image is a cross-sectional image intersecting the core line. When the second acquisition unit 140 acquires a two-dimensional cross-sectional image such as the cross-sectional image 701 in Figure 7, the likelihood information holding unit 150 generates likelihood information 1001 for the cross-sectional image 701. The set of pixels with high pixel values ​​in the likelihood information 1001 (the set of pixels indicating existence 1002) indicates the position and size of the structure (cross-sectional region 715 in Figure 7) that the image processing device 100 is trying to extract.

[0045] The set of pixels 1002 indicating existence is set based on the position of the core line section 702 set by the setting unit 130. There are various ways to assign values. For example, a constant pixel value may be assigned to pixels within a predetermined distance range from the core line section 702. Alternatively, as with the set of pixels 1002 indicating existence, pixel values ​​may be assigned based on the distance from the core line section 702. That is, a high pixel value may be assigned to pixels close to the core line section 702, and a low pixel value may be assigned to pixels far from the core line section 702. When a two-dimensional image such as the cross-sectional image 701 is acquired as a partial image, the likelihood information also becomes two-dimensional cross-sectional information, as shown in the likelihood information 1001 in Figure 10(a).

[0046] Figure 10(b) is a diagram illustrating likelihood information when the cross-sectional image is a three-dimensional image. When the second acquisition unit 140 acquires the three-dimensional image 801 in Figure 8, the likelihood information holding unit 150 generates likelihood information 1003 for the three-dimensional image 801. In the likelihood information 1003, the set of pixels indicating existence 1004 indicates the position and size of the structure to be extracted (region 803 in Figure 8) in the three-dimensional image 801. Here, the set of pixels indicating existence 1004 is set based on the position of the core line section 702 set by the setting unit 130. For example, a constant pixel value may be assigned to pixels within a predetermined distance (here, distance in three-dimensional space) from the core line section 702, or multiple pixel values ​​may be switched and set based on the distance from the position of the core line section 702. Alternatively, the pixel value may be continuously changed and assigned based on the distance from the position of the core line section 702. When a three-dimensional image like the three-dimensional image 801 is acquired as a partial image, the likelihood information also becomes three-dimensional, as shown in the likelihood information 1003 in Figure 10(b).

[0047] Figure 10(c) is a diagram illustrating likelihood information when the cross-sectional image is an SPR image. When the second acquisition unit 140 acquires a two-dimensional cross-sectional image such as the cross-sectional image 900 in Figure 9, the likelihood information holding unit 150 generates likelihood information 1005 for the cross-sectional image 900. In the likelihood information 1005, the set of pixels indicating existence 1006 indicates the position and size of the structure to be extracted in the cross-sectional image (cross-sectional region 902 in Figure 9). Here, the set of pixels indicating existence 1006 is set based on the position of the core line section 702 set by the setting unit 130. For example, a constant pixel value may be assigned to pixels within a predetermined distance (here, distance in three-dimensional space) from the core line section 702, or multiple pixel values ​​may be switched and set based on the distance from the position of the core line section 702. Alternatively, the pixel value may be continuously changed and assigned based on the distance from the position of the core line section 702. When a two-dimensional image such as the cross-sectional image 900 is obtained as a partial image, the likelihood information will also be two-dimensional cross-sectional information, as shown in the likelihood information 1005 in Figure 10(c).

[0048] The likelihood information storage unit 150 may determine the set of pixels indicating existence based on prior knowledge about the structure to be extracted. The set of pixels indicating existence may be set to form a circle centered on the core line section, as shown in the sets of pixels indicating existence 1002, 1004, and 1006 in Figure 10. This shape is a shape commonly found in structures that the image processing device 100 extracts. On the other hand, if the structure to be extracted has a distinctive shape, the extent of the set of pixels indicating existence may be made to resemble the shape of that structure.

[0049] The likelihood information storage unit 150 may determine the size-related information of the likelihood information based on prior knowledge about the structure to be extracted. Specifically, the likelihood information storage unit 150 determines the size-related information of the likelihood information from the size of the structure. For example, the size of a structure often falls within a predetermined range. The likelihood information storage unit 150 can determine the size-related information based on such characteristics of the structure. Here, size refers to the distance from the core line section (section 702 in Figures 7 to 9) set in the medical image or partial image to the boundary of the structure to be extracted. Therefore, a suitable value is set as the size-related information according to the structure to be extracted. The extent of the sets of pixels indicating existence 1002, 1004, and 1006 may be made wider or narrower in proportion to the size obtained by this means.

[0050] The likelihood information holding unit 150 may determine information relating to the size of the likelihood information based on the first region obtained by the first extraction unit 120. That is, the likelihood information holding unit may determine information relating to the size from the structural image. For example, the likelihood information holding unit 150 obtains an approximate value of the size of the first region (first region 401 in Figure 4). Here, size refers to the distance from the core line interval set in each medical image or partial image (for example, the core line interval set for the core line of the first region 401) to the boundary of the first region. The extent of the sets of pixels indicating existence 1002, 1004, and 1006 may be made wider or narrower in proportion to the size obtained by this means.

[0051] The likelihood information storage unit 150 may acquire information relating to the size of the likelihood information from the partial image. That is, the likelihood information storage unit may determine the size-related information from the cross-sectional image or three-dimensional image acquired by the second acquisition unit 140. For example, the likelihood information storage unit 150 applies a known image processing method such as a Laplacian of Gaussian filter to the partial image (cross-sectional image 701 in Figure 7, three-dimensional image 801 in Figure 8, and cross-sectional image 900 in Figure 9). This makes it possible to obtain an approximate value of the size of the target structure (cross-sectional region 715 in Figure 7, region 803 in Figure 8, and cross-sectional region 902 in Figure 9) without actually extracting the target structure. Here, size refers to the distance from the core line section set in each cross-sectional image (section 702 in Figures 7 to 9) to the boundary of the target structure. The extent of the sets of pixels indicating existence 1002, 1004, and 1006 may be made wider or narrower in proportion to the size obtained by this method.

[0052] The likelihood information storage unit 150 may change the information relating to the magnitude of the likelihood information according to the distance from the endpoint of the core wire section. Generally, structures (coronary arteries in this embodiment) tend to be larger at one end on the root side and smaller at the other end on the leaf side. For example, the likelihood information storage unit 150 sets a large value as the magnitude information at the end of the core wire section closer to the root side, and a small value as the magnitude information at the end of the core wire section closer to the leaf side. The likelihood information storage unit 150 also sets a value proportional to the distance from the end of the core wire section closer to the root side or leaf side to that location as the magnitude information for other locations in the core wire section. The extent of the sets of pixels indicating existence 1002, 1004, and 1006 may be widened or narrowed in proportion to the magnitude set by this means.

[0053] The second extraction unit 160 uses the partial image generated by the second acquisition unit 140 and the likelihood information generated by the likelihood information holding unit 150 to extract structures (coronary arteries in this embodiment) depicted in the partial image. Hereafter, the region extracted by the second extraction unit 160 from among the structures depicted in the partial image will be referred to as the extraction target region. That is, the second extraction unit 160 extracts a part of the coronary artery having a branched shape as the extraction target region. A deep learning network is used to extract the extraction target region. Any type of deep learning network that performs region extraction is acceptable. An example of a deep learning network suitable for the second extraction unit 160 is U-Net.

[0054] For example, the second extraction unit 160 stores the partial image and likelihood information in different channels of the deep learning network's input data and inputs them to the deep learning network. For example, if the second acquisition unit 140 generates a cross-sectional image of a cross-section intersecting the core line, the two-dimensional cross-sectional image is stored in channel "0" of the deep learning network's input data, and the likelihood information related to existence is stored in channel "1". The same applies if the partial image generated by the second acquisition unit 140 is a three-dimensional image, a CPR image, or an SPR image. The second extraction unit 160 inputs the deep learning network's input data generated in this way and performs calculations on the deep learning network to extract the target region.

[0055] Here, we will explain the parameters such as the weights of the deep learning network. The deep learning network used by the second extraction unit 160 must be pre-trained. Since the deep learning network can be trained according to a known training procedure, we will focus on the aspects relevant to the embodiment here. First, for each of the multiple medical images used for training, the processing performed by the first acquisition unit 110, the first extraction unit 120, the setting unit 130, and the second acquisition unit 140 (the processing in steps S1010, S1020, S1030, S1040, and S1050 of the processing procedure described later) is applied to acquire a partial image. Next, for each acquired partial image, a correct answer indicating the extraction target area is assigned. The partial image may depict structures that are the target of extraction and structures of the same type as the target but that are not the target of extraction. Of these structures, the correct answer is assigned to the structures that are the target of extraction. Specifically, correct answers are assigned to structures related to the core line section set by the setting unit 130 (step S1040 of the processing procedure described later) among the structures depicted in the cross-sectional image. Any method of assignment is acceptable, such as manual assignment or semi-automatic assignment using known image processing techniques. By performing such assignment for all partial images, correct answers for the partial images can be created. Finally, a known learning algorithm is executed to train the deep learning network used by the second extraction unit 160. After the execution of the learning algorithm is complete, parameters such as the weights of the deep learning network output by the learning algorithm are saved to the data server 170. The second extraction unit 160 uses the parameters of the deep learning network obtained by the above method to extract the target region in the partial image.

[0056] Figures 11, 12, and 13 show examples of extraction results from the second extraction unit 160. Image 1100 in Figure 11 is the processing result of a two-dimensional cross-sectional image (cross-sectional image 701 in Figure 7), and the extracted region 1101 is the region extracted by the second extraction unit 160. As can be seen by comparing image 701 and image 1100, the cross-sectional region 715, which is the target of extraction, has been correctly extracted as the extracted region 1101. Also, the cross-sectional region 716, which is not the target of extraction, has not been extracted. Image 1200 in Figure 12 is the processing result of a three-dimensional cross-sectional image (three-dimensional image 801 in Figure 8), and the extracted region 1201 is the region extracted by the second extraction unit. As can be seen by comparing three-dimensional image 801 and image 1200, the region 803, which is the target of extraction, has been correctly extracted as the extracted region 1201. Also, the regions 804 and 805, which are not the target of extraction, have not been extracted. Image 1300 in Figure 13 shows the processing result when the cross-sectional image is an SPR image (cross-sectional image 900 in Figure 9), and the extracted region 1301 is the region extracted by the second extraction unit. As can be seen by comparing image 900 and image 1300, the cross-sectional region 902, which is the target of extraction, is correctly extracted as the extracted region 1301. Regions 903 and 904, which are not the target of extraction, are not extracted.

[0057] While the specific form of likelihood information can be modified in various ways as described above, generally, it is generated such that the smaller the distance from the selected interval corresponding to the area of ​​interest, the higher the likelihood of existence. In other words, likelihood information indicates the degree of likelihood of the area of ​​interest existing. By performing region extraction processing while considering likelihood information, the area of ​​interest appearing on the partial image can be appropriately extracted as the region to be extracted.

[0058] The likelihood information may be a map (image) showing the distribution of likelihood levels for the area of ​​interest, or it may be coordinate data indicating the location of the area of ​​interest with a high likelihood. For example, the likelihood information storage unit 150 may acquire the position coordinates of the intersection point between the core line and the partial image within the section as likelihood information.

[0059] Furthermore, an example was described in which the partial image and likelihood information are input to each channel of a deep learning network as a method for extracting target regions based on partial images and likelihood information. However, the embodiments are not limited to this.

[0060] For example, the second extraction unit 160 first inputs only the partial image into the deep learning network. In this case, the region extraction result output from the deep learning network includes regions that are not the target of extraction. Hereafter, the result of region extraction using only the partial image will also be referred to as the first extraction result.

[0061] Next, the second extraction unit 160 multiplies the first extraction result with the likelihood information. For example, the likelihood information holding unit 150 generates an image in which pixel values ​​are set according to the likelihood level, as shown in Figure 10. Then, the second extraction unit 160 generates an integrated image by accumulating the pixel values ​​of the corresponding pixels using the first extraction result and the likelihood information. In the integrated image, even if a region was extracted in the first extraction result, if the likelihood in the likelihood information is low, it is replaced with a smaller pixel value. In this way, in the integrated image, regions that were extracted in the first extraction result and have a high likelihood, i.e., the extraction target region, are emphasized. The second extraction unit 160 may perform thresholding on the integrated image to further emphasize the extraction target region.

[0062] Furthermore, although we have described an example in which only a partial image is input to the deep learning network to obtain the first extraction result, the method for obtaining the first extraction result is not limited to this. That is, the second extraction unit 160 may obtain the first extraction result without relying on machine learning methods, such as the graph cut method. Alternatively, the second extraction unit 160 may use the first region extracted by the first extraction unit 120 as the first extraction result.

[0063] The second extraction unit 160 transmits the extraction result of the desired structure (coronary artery in this embodiment) depicted in the cross-sectional image to the data server 170.

[0064] The second extraction unit 160 may transmit the extraction result of a desired structure (in this embodiment, a coronary artery) depicted in the cross-sectional image to an image processing device (not shown). This image processing device calculates all information useful for diagnosing the structure. For example, in this embodiment, the image processing device (not shown) may use the acquired coronary artery region to calculate the diameter of the coronary artery or to identify abnormal areas within the coronary artery. The processing performed by this image processing device (not shown) is generally called Computer Aided (or Assisted) Detection (CADe) or Computer Aided (or Assisted) Diagnosis (CADx).

[0065] The second extraction unit 160 may output the extraction result of a desired structure (coronary artery in this embodiment) depicted in the cross-sectional image to a display unit not shown in Figure 1. An example of a display device included in the display unit is a display. The display unit not shown may display only the extraction result. The display unit not shown may also superimpose the extraction result on the cross-sectional image, or display them side by side simultaneously. The display unit not shown may also superimpose the extraction result on the medical image, or display them side by side simultaneously.

[0066] Next, the processing procedure of the image processing device 100 will be explained with reference to Figure 2.

[0067] (S1010) In step S1010, the first acquisition unit 110 acquires a medical image. The acquired medical image is then transmitted to the first extraction unit 120 and the second acquisition unit 140.

[0068] Next, the first acquisition unit 110 acquires the parameters of the deep learning network stored in the data server 170. Then, it transmits the acquired deep learning network parameters to the second extraction unit 160.

[0069] (S1020) In step S1020, the first extraction unit 120 acquires a medical image from the first acquisition unit 110. Then, it applies a region extraction method to the medical image to extract structures (coronary arteries in this embodiment) in the medical image. The region extraction method performed by the first extraction unit 120 in this step can be any known region extraction method. For example, a graph cut method may be used, or a deep learning network such as U-Net may be used.

[0070] (S1030) In step S1030, the first extraction unit 120 obtains the core line based on the first region obtained in step S1020. First, the first extraction unit 120 applies a thinning method to the first region to convert it into a region (line figure) with a line width of "1". The resulting line figure runs near the center of the first region and moves along the direction of travel of the first region. Next, the first extraction unit 120 obtains the endpoints of the line figure obtained by thinning. Since the line figure obtained by thinning has a line width of "1", the endpoints can be easily obtained. Finally, the first extraction unit 120 classifies the obtained endpoints into endpoints corresponding to one end on the root side of the first region and endpoints corresponding to one end on the leaf side of the first region.

[0071] For example, the first extraction unit 120 classifies each endpoint of the line figure into either the root end or the leaf end of the first region based on prior information about the structure. Here, prior information about the structure refers to the position of the structure in the medical image and the positional relationship between the structure and other structures in the vicinity. In this embodiment, the root end of the structure is the end on which the coronary artery branches off from the aorta. That is, the root end is the origin. Therefore, after extracting the aorta using a method such as thresholding, the endpoint of the line figure that is closest in distance to the extracted aorta is designated as the root end of the first region. Since the structure to be extracted is a tree structure, there is only one endpoint corresponding to the root end of the first region. Therefore, all endpoints of the line figure that were not classified as the root end of the first region are designated as the leaf ends of the structure.

[0072] (S1040) In step S1040, the setting unit 130 obtains the core wires of the first region from the first extraction unit 120. Next, the setting unit 130 sets a section for the core wires of the first region that it has received. The types of sections and the setting method have already been explained, so the explanation will be omitted.

[0073] (S1050) In step S1050, the second acquisition unit 140 receives a medical image from the first acquisition unit 110. The second acquisition unit 140 also receives information about the core wire section from the setting unit 130. Next, the second acquisition unit 140 generates a partial image using the received medical image and the core wire section information. The types and methods of generating partial images have already been explained, so the explanation will be omitted here.

[0074] (S1060) In step S1060, the likelihood information holding unit 150 receives information about the core wire section from the setting unit 130. The likelihood information holding unit 150 also receives a cross-sectional image from the second acquisition unit 140. Next, the likelihood information holding unit 150 uses the received cross-sectional image and the core wire section information to generate likelihood information related to existence likelihood. The types of likelihood information and the generation method have already been explained, so the explanation will be omitted.

[0075] (S1070) In step S1070, the second extraction unit 160 receives parameters of the deep learning network from the first acquisition unit 110. The second extraction unit 160 also receives a partial image from the second acquisition unit 140. The second extraction unit 160 also receives likelihood information from the likelihood information storage unit 150. Next, the second extraction unit 160 uses the received deep learning network parameters, partial image, and likelihood information to extract the target region of the structure depicted in the partial image. The method for extracting the structure has already been explained, so the explanation will be omitted.

[0076] (S1080) In step S1080, the second extraction unit 160 transmits the extraction results of structures (coronary arteries in this embodiment) depicted in the cross-sectional image to the data server 170. As also described in the explanation of the functional configuration of the image processing device 100, the second extraction unit 160 may output the extraction results to a display unit equipped with a display or the like. Alternatively, the second extraction unit 160 may transmit the extraction results to an image processing device or information processing device that performs Computer Aided (or Assisted) Detection (CADe) or Computer Aided (or Assisted) Diagnosis (CADx), etc. Note that the extraction results may be transmitted only to the data server 170 and not output to the display unit, or output only to the display unit and not transmitted to the data server 170. Furthermore, the extraction results may be transmitted to an analysis unit (not shown) that performs analysis processing (for example, measurement processing of the diameter of the coronary arteries or detection processing of stenosis), and the extraction results may not be output, saved, or displayed externally.

[0077] The second extraction unit 160 may output the extraction target region as is, or it may output it after further processing based on the extraction target region. A specific method for outputting the extraction target region as is could be to display the images shown in Figures 11 to 13.

[0078] Furthermore, as a process based on the extraction target region, for example, extraction of a three-dimensional region corresponding to the area of ​​interest can be considered. That is, the coronary artery is a three-dimensional structure, and the area of ​​interest in the coronary artery is also a three-dimensional region. In contrast, for example, image 1100 shown in Figure 11 and image 1300 shown in Figure 13 show two-dimensional regions. Image 1200 shown in Figure 12 shows a three-dimensional region, but it does not necessarily show the entire area of ​​interest. Therefore, the second extraction unit 160 may use the extraction results of the extraction target regions such as image 1100, image 1200, and image 1300 as constraints to extract a three-dimensional region corresponding to the area of ​​interest from the medical image acquired in step S1010 and output the said three-dimensional region.

[0079] As described above, the image processing apparatus 100 according to the first embodiment comprises a first acquisition unit 110, a first extraction unit 120, a setting unit 130, a second acquisition unit 140, a likelihood information holding unit 150, and a second extraction unit 160. The first acquisition unit 110 acquires CT images of the coronary arteries. The first extraction unit 120 extracts the core lines of the coronary arteries based on the CT images. The setting unit 130 sets a section for at least a portion of the core lines. The second acquisition unit 140 acquires a partial image based on the CT images, including at least a portion of the section set by the setting unit 130. The likelihood information holding unit 150 holds likelihood information related to the likelihood of existence in the partial image. The second extraction unit 160 extracts a target region based on the partial image and the likelihood information. With this configuration, the image processing apparatus 100 can selectively extract a portion of the coronary arteries.

[0080] Specifically, not limited to the coronary arteries, in medical image analysis, the first step is to acquire anatomical structures depicted in the image using a region extraction technique, which is a type of image processing method. The structures to be analyzed include not only massive structures like the heart, but also tree-like structures that branch out at multiple points, such as blood vessels. When the target of medical image analysis is a tree-like structure, depending on the type of analysis, the region extraction technique may need to selectively extract only a portion of the structure depicted in the image, for example, just one branch. However, conventional methods (such as the technology disclosed in Patent Document 1) have the problem of mistakenly extracting structures that are not the target of extraction.

[0081] For example, the left coronary artery, which extends from the aorta, first branches into the left anterior descending coronary artery (LAD) and the left circumflex coronary artery (LCX). If, for example, only the LAD is the area of ​​interest, it is preferable to extract the LAD from the CT image and then perform various image processing and display operations. However, both the LAD and LCX are coronary arteries and are depicted with similar pixel values ​​on the CT image. That is, even if you want to extract only the LAD, if you simply apply known region extraction methods to the CT image, both the LAD and LCX will be extracted simultaneously.

[0082] In contrast, the image processing device 100 according to the first embodiment can selectively extract LADs based on a partial image and likelihood information. That is, the image processing device 100 can selectively extract LADs by performing region extraction that takes into account the likelihood of the presence of LADs in a partial image that includes LADs and LCXs.

[0083] Although an example has been described in which a first region 401 is acquired from a medical image 300 and core lines are extracted based on the first region 401, the embodiments are not limited to this. That is, the acquisition of the first region 401 may be omitted, and the core lines of the coronary arteries may be extracted directly from the medical image 300. For example, the first extraction unit 120 can extract the core lines of the coronary arteries from the medical image 300 using a trained model that is functionalized to identify the core lines of blood vessels in the input image.

[0084] <Second Embodiment> In Figure 7, only one partial image is shown (cross-sectional image 701). Similarly, in Figures 8 and 9, one partial image is shown at a time. In the second embodiment, an example of generating multiple partial images will be described.

[0085] The image processing apparatus 100 according to the second embodiment has the same configuration as the image processing apparatus 100 according to the first embodiment shown in Figure 1, with some differences in the processing performed by the second acquisition unit 140, the likelihood information holding unit 150, and the second extraction unit 160. Hereafter, the same reference numerals are used for parts described in the first embodiment, and their descriptions are omitted.

[0086] The second acquisition unit 140 according to the second embodiment acquires multiple partial images that include a portion of the section set by the setting unit 130. For example, the second acquisition unit 140 generates multiple cross-sectional images as multiple partial images, corresponding to multiple cross-sections that intersect with and are spaced apart from the core line within the section.

[0087] Referring to Figure 14, an example of multiple cross-sectional images will be explained. Figure 14 is an example of generating three two-dimensional cross-sectional images corresponding to three cross-sections that intersect the core line and are spaced apart from each other. The medical image 300 in Figure 14(a) includes the coronary artery 303, as in the case of Figure 7, and the section 702 of the core line is set. The three cross-sectional images (cross-sectional image 1504, cross-sectional image 1505, and cross-sectional image 1506) are two-dimensional cross-sectional images acquired by the second acquisition unit 140. As can be seen from cross-sectional images 1504, 1505, and 1506, the cross-sectional images generated by the second acquisition unit 140 are located at positions that intersect the core line and are spaced apart from each other. For example, the setting unit 130 sets each of the cross-sectional images 1504, 1505, and 1506 to be images of cross-sections that are approximately perpendicular to the core line at their respective positions. Note that the method of setting the cross-section is not limited to this. For example, for cross-section image 1505, an image of a cross-section approximately perpendicular to the center line may be set, and for cross-section images 1504 and 1506, images of a cross-section approximately parallel to cross-section image 1505 may be set.

[0088] The following describes the case where coronary artery 303 is extracted as the first region. Figure 14(b) shows cross-sectional images 1504, 1505, and 1506 from the front. As shown in Figure 14(a), cross-sectional image 1504 intersects with coronary artery 303 at position 1507. The cross-section of the first region at position 1507 is region 1521 in cross-sectional image 1504 of Figure 14(b). Also, as shown in Figure 14(a), cross-sectional image 1505 intersects with the first region at positions 1508 and 1509. The cross-sections of the first region at positions 1508 and 1509 are regions 1531 and 1532 in cross-sectional image 1505 of Figure 14(b). Furthermore, as shown in Figure 14(a), the cross-sectional image 1506 intersects the first region at positions 1510 and 1511. The cross-sections of the first region at positions 1510 and 1511 correspond to regions 1541 and 1542 in the cross-sectional image 1506, respectively.

[0089] Of regions 1521, 1531, 1532, 1541, and 1542, regions 1521, 1531, and 1541 correspond to the regions to be extracted (cross-sectional regions 1521, 1531, and 1541, respectively). On the other hand, regions 1532 and 1542 are parts of the same structure (coronary artery in this embodiment) as cross-sectional regions 1521, 1531, and 1541, but are not regions to be extracted. The image processing device 100 according to the second embodiment correctly extracts the part of the structure to be extracted even when multiple parts of the same structure are captured in the cross-sectional image.

[0090] Note that regions 1521, 1531, 1532, 1541, and 1542 in Figure 14(b) are illustrations of the extraction results when coronary arteries are extracted using a known region extraction method, for illustrative purposes. The extraction process shown in Figure 14(b) may be omitted. That is, regions 1521, 1531, 1532, 1541, and 1542 do not need to be extracted in cross-sectional images 1504, 1505, and 1506.

[0091] The likelihood information storage unit 150 generates and stores likelihood information relating to the likelihood of existence in the cross-section of each cross-section image for each of the multiple cross-sectional images generated by the second acquisition unit 140. The likelihood information generation process itself is the same as in the first embodiment, so a detailed explanation is omitted. That is, the likelihood information storage unit 150 performs the same likelihood information generation process as in the first embodiment for each of the multiple cross-sectional images. The likelihood information storage unit 150 may generate likelihood information relating to the likelihood of existence for each of the multiple cross-sectional images generated by the second acquisition unit 140, or it may generate likelihood information relating to the likelihood of existence only for the cross-sectional image of interest among the multiple cross-sectional images.

[0092] The second extraction unit 160 uses the multiple cross-sectional images generated by the second acquisition unit 140 and the likelihood information related to one or more existence likelihoods generated by the likelihood information holding unit 150 to extract structures (coronary arteries in this embodiment) depicted in the cross-sectional images. The differences between the extraction process performed by the second extraction unit 160 and that of the first embodiment will be described below.

[0093] There are several types of inputs to the deep learning network generated by the second extraction unit 160. Let D be the number of cross-sectional images (image size WxH) generated by the second acquisition unit 140, and M be the number of likelihood information related to existence likelihood generated by the likelihood information holding unit 150 (where M <= D). In this case, the input to the deep learning network may be four-dimensional data (WxHxDxM). Alternatively, it may be three-dimensional data (WxHxC, where C = D + M). The choice of input should be determined by considering the type of structure to be extracted and the computing resources used by the image processing device 100.

[0094] The second extraction unit 160 creates input data for the deep learning network as described above, inputs it into the deep learning network, and performs calculations to extract the cross-sectional region (extraction target region) of the structure of interest depicted in the cross-sectional image.

[0095] Figure 15 shows an example of the extraction results of the second extraction unit 160. Images 1600, 1610, and 1620 in Figure 15 are the processing results of two-dimensional cross-sectional images (cross-sectional images 1504, 1505, and 1506 in Figure 15), and the extracted regions 1601, 1611, and 1621 are the regions extracted by the second extraction unit 160. Comparing image 1504 and image 1600 in Figure 14(b), it can be seen that the cross-sectional region 1521, which is the target of extraction, has been correctly extracted as the extracted region 1601. Comparing image 1505 and image 1610 in Figure 14(b), it can be seen that the cross-sectional region 1531, which is the target of extraction, has been correctly extracted as the extracted region 1611. Furthermore, as can be seen by comparing image 1506 and image 1620 in Figure 14(b), the cross-sectional region 1541, which is the target of extraction, has been correctly extracted as the extracted region 1621. On the other hand, regions 1532 and 1542, which were not targeted for extraction, were not extracted.

[0096] Next, the processing procedure of the image processing apparatus 100 according to the second embodiment will be described.

[0097] (S2010) In step S2010, the first acquisition unit 110 acquires medical images and parameters of the deep learning network, similar to step S1010 shown in Figure 2.

[0098] (S2020) In step S2020, the first extraction unit 120 acquires a medical image from the first acquisition unit 110, similar to step S1020 shown in Figure 2. Then, a region extraction method is applied to the medical image to extract structures (coronary arteries in this embodiment) within the medical image.

[0099] (S2030) In step S2030, the first extraction unit 120 acquires the core wire of the first region obtained in step S2020, similar to step S1030 shown in Figure 2.

[0100] (S2040) In step S2040, the setting unit 130 sets a section for the core wire extracted in step S2030, similar to step S1040 shown in Figure 2.

[0101] (S2050) In step S2050, the second acquisition unit 140 receives a medical image from the first acquisition unit 110. The second acquisition unit 140 also receives information about the core wire section from the setting unit 130. Next, the second acquisition unit 140 uses the received medical image and the core wire section information to generate multiple cross-sectional images corresponding to multiple cross-sections that intersect and separate from each other within the section.

[0102] (S2060) In step S2060, the likelihood information holding unit 150 receives information about the core wire section from the setting unit 130. The likelihood information holding unit 150 also receives a cross-sectional image from the second acquisition unit 140. Next, the likelihood information holding unit 150 generates likelihood information related to the likelihood of existence using the received cross-sectional image and the information about the core wire section. Here, the likelihood information holding unit 150 generates likelihood information for each of the multiple cross-sectional images acquired by the second acquisition unit 140.

[0103] (S2070) In step S2070, the second extraction unit 160 receives parameters of the deep learning network from the first acquisition unit 110. The second extraction unit 160 also receives a cross-sectional image from the second acquisition unit 140. The second extraction unit 160 also receives likelihood information related to the likelihood of existence from the likelihood information holding unit 150. Next, the second extraction unit 160 uses the received parameters of the deep learning network, the cross-sectional image, and the likelihood information related to the likelihood of existence to extract the target region of the structure depicted in the cross-sectional image.

[0104] (S2080) In step S2080, the second extraction unit 160 transmits the extraction result of the target region of the structure (coronary artery in this embodiment) depicted in the cross-sectional image to the data server 170. However, as explained in the first embodiment, various modifications are possible regarding the specific manner of outputting the extraction result. Furthermore, when performing extraction processing of a three-dimensional region corresponding to the area of ​​interest using the extraction result of the target region as a constraint condition, the accuracy of the extraction processing of the three-dimensional region corresponding to the area of ​​interest is improved in the second embodiment because more constraint conditions can be used.

[0105] To date, as an example of multiple partial images, we have described multiple cross-sectional images corresponding to multiple cross-sections that intersect and separate from the core wire within the section set by the setting unit 130. However, the embodiments are not limited to these.

[0106] For example, the second acquisition unit 140 may acquire two-dimensional images of cross-sections along the core lines, such as SPR images and CPR images, as multiple partial images. For example, the second acquisition unit 140 may extract multiple non-overlapping core lines from the core lines within the interval set by the setting unit 130, and generate an SPR image for each extracted core line. In this case, the likelihood information holding unit 150 generates likelihood information for each of the multiple SPR images, and the second extraction unit 160 can extract the target region based on the multiple SPR images and the likelihood information.

[0107] Furthermore, for example, the second acquisition unit 140 may acquire a three-dimensional image as multiple partial images. For example, the second acquisition unit 140 may extract multiple non-overlapping core wires from core wires within a section set by the setting unit 130, and generate a three-dimensional image for each extracted core wire. For example, the second acquisition unit 140 may identify a three-dimensional region having a predetermined shape and inscribed by the extracted core wires, and generate a three-dimensional image for that three-dimensional region. In this case, the likelihood information holding unit 150 generates likelihood information for each of the multiple three-dimensional images, and the second extraction unit 160 can extract the target region based on the multiple three-dimensional images and the likelihood information.

[0108] <Third Embodiment> In the first and second embodiments, an example was described in which one section is set for the core wire extracted by the first extraction unit 120. In the third embodiment, an example will be described in which multiple sections are set for the core wire extracted by the first extraction unit 120.

[0109] The image processing apparatus 100 according to the third embodiment has the same configuration as the image processing apparatus 100 according to the first embodiment shown in Figure 1, with some differences in the processing performed by the setting unit 130, the second acquisition unit 140, the likelihood information holding unit 150, and the second extraction unit 160. Hereafter, the same reference numerals are used for parts described in the first and second embodiments, and their descriptions are omitted.

[0110] In the third embodiment, the setting unit 130 sets multiple different sections for the core wires extracted by the first extraction unit 120. The operation of the setting unit 130 will be described below with reference to Figure 16.

[0111] Figure 16(a) shows image 1800 as the result of core line extraction by the first extraction unit 120. For example, the first extraction unit 120 acquires a first region 1801 corresponding to a coronary artery by applying a known region extraction method to the medical image acquired by the first acquisition unit 110. Furthermore, the first extraction unit 120 acquires a line representing the first region 1801 as the core line of the coronary artery.

[0112] In Figure 16(a), the core wire extracted by the first extraction unit 120 is shown divided into three core wires: core wire 1802, core wire 1803, and core wire 1804. Core wire 1802 is a core wire that extends from the origin (one end on the root side) of the coronary artery to the distal end (one end on the lobe side) of the coronary artery. Core wire 1803 is a core wire that extends from a branching point in the middle of core wire 1802 to a different distal end than core wire 1802. Core wire 1804 is a core wire that extends from a branching point in core wire 1802 at a different location than the endpoint of core wire 1803 to a different distal end than core wires 1802 and 1803. The setting unit 130 sets multiple sections for these multiple core wires.

[0113] Figures 16(b), 16(c), and 16(d) illustrate images 1810, 1820, and 1830, respectively, as the result of setting the sections by the setting unit 130. Specifically, the setting unit 130 sets section 1813, which includes core wire 1802, as shown in Figure 16(b); section 1823, which includes core wire 1803, as shown in Figure 16(c); and section 1833, which includes core wire 1804, as shown in Figure 16(d). The setting unit 130 sets these multiple sections by sequentially switching between them. The procedure for this sequential switching will be described in detail in the processing procedure described later.

[0114] In the above description, the setting unit 130 was described as setting multiple sections 1813, 1823, and 1833 without any overlap or omissions in the entire first region 1801. However, the implementation of the present invention is not limited to this, and only a part of the above multiple sections may be set. Furthermore, the sections may be set so that a part of each section to be set overlaps.

[0115] Next, with reference to Figure 17, the processing procedure of the image processing apparatus 100 according to the third embodiment will be described.

[0116] (From S3010 to S3030) The processing from steps S3010 to S3030 is the same as the processing from steps S1010 to S1030 in the first embodiment. A detailed explanation is omitted.

[0117] (S3040) In step S3040, the setting unit 130 receives the core wires extracted by the first extraction unit 120. Next, the setting unit 130 separates the received core wires into multiple sections and sets a section for any one of them. However, when the process returns to this processing step via step S3080 (described later), the setting unit 130 sets a section from among the multiple sections that has not been set as a section in this processing step. In other words, the same section is not set again. To achieve this, the history of sections set in this processing step is recorded in the data server 170 or a storage medium (not shown), and by referring to this, it is possible to distinguish between sections that have already been set and sections that have not.

[0118] (From S3050 to S3070) The processing from step S3050 to step S3070 is the same as the processing from step S1050 to step S1070 in the first embodiment. A detailed explanation is omitted.

[0119] (S3080) In step S3080, the setting unit 130 determines whether there are any sections among the multiple sections obtained when step S3040 was first executed that have not yet been set as targets for processing in steps S3050 to S3070. If there are sections that have not yet been set, the process returns to step S3040; otherwise, the process proceeds to step S3090.

[0120] (S3090) In step S3090, the second extraction unit 160 transmits the extraction result of the extraction target area to the data server 170, similar to step S3080 of the first embodiment. That is, the second extraction unit 160 may output the extraction target area as is, or it may output it after further processing based on the extraction target area.

[0121] The second extraction unit 160 may perform a process to reconstruct the multiple extraction target regions extracted by repeatedly executing step S3070, and then output the results. Specifically, the second extraction unit 160 performs a process to integrate the structure extraction results acquired for each of the multiple sections into a common space. Here, the common space can be the space of the same coordinate system as the medical image acquired in step S3010. Then, the structure extraction results can be integrated based on the position and range of each section in this coordinate system of the medical image. This makes it possible to integrate the structure extraction results acquired for each of the multiple sections.

[0122] The processing of this embodiment is performed by following the above procedure. According to this, it is possible to extract the target region of the coronary artery depicted in the partial image of each section while sequentially switching between multiple sections of the coronary artery having a branched structure. Therefore, it has the effect of being able to extract a wider range of coronary arteries. For example, according to the image processing apparatus 100 of the third embodiment, the entire coronary artery can be extracted with high accuracy.

[0123] In the above explanation, we used the example of repeatedly executing the processes from step S3040 to step S3080, but it is also possible to execute the processes from step S3050 to step S3070 in parallel for each of the multiple intervals obtained in step S3040.

[0124] In the above explanation, the term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). Alternatively, instead of storing the program in memory, the program may be directly embedded within the processor's circuitry. In this case, the processor functions by reading and executing the program embedded within the circuitry.

[0125] Each component of the apparatus according to the above embodiment is a functional concept and does not necessarily have to be physically configured as shown in the illustration. That is, the specific form of distribution and integration of each apparatus is not limited to that shown in the illustration, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each apparatus can be implemented in whole or in any part by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0126] Furthermore, the image processing method described in the above-mentioned embodiments can be implemented by executing a pre-prepared program on a computer such as a personal computer or workstation. This program can be distributed via a network such as the Internet. Alternatively, this program can be recorded on a computer-readable non-transient recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO, or DVD, and executed by reading it from the recording medium by a computer.

[0127] According to at least one embodiment described above, a portion of a structure having a branched shape can be selectively extracted.

[0128] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0129] 100: Image processing device 110: First acquisition section 120: First extraction section 130: Settings Section 140: Second acquisition section 150: Likelihood Information Storage Unit 160: Second extraction section 170: Data Server

Claims

1. A first acquisition unit that acquires volume data of a structure having a branched shape, A first extraction unit extracts the core wire of the structure based on the volume data, A setting unit for setting a section for at least a portion of the core wire, A second acquisition unit acquires a cross-sectional image or a three-dimensional image including at least a portion of the section based on the volume data, A likelihood information holding unit that holds likelihood information relating to the likelihood of existence in the cross-sectional image or the three-dimensional image, A second extraction unit extracts the target region of the structure corresponding to the section based on the cross-sectional image or the three-dimensional image and the likelihood information. An image processing device equipped with the following features.

2. The second acquisition unit acquires multiple cross-sectional images or three-dimensional images, The image processing apparatus according to claim 1, wherein the second extraction unit extracts the target region based on a plurality of cross-sectional images or a plurality of three-dimensional images and the likelihood information.

3. The image processing apparatus according to claim 2, wherein the second acquisition unit acquires a plurality of cross-sectional images corresponding to a plurality of cross-sections that intersect with the core wire within the section and are spaced apart from each other.

4. The setting unit sets multiple sections for the core wire, The second acquisition unit acquires the cross-sectional image or the three-dimensional image for each of the plurality of sections. The image processing apparatus according to any one of claims 1 to 3, wherein the second extraction unit extracts the target extraction region for each of the plurality of cross-sectional images or the plurality of three-dimensional images.

5. The image processing apparatus according to claim 4, wherein the second extraction unit further performs a process of reconstructing the plurality of extraction target regions extracted from each of the plurality of cross-sectional images or the plurality of three-dimensional images.

6. The image processing apparatus according to claim 1, wherein the first extraction unit acquires a structural image obtained by extracting the structure from the volume data, and extracts lines representing the structural image as the core lines.

7. The image processing apparatus according to claim 1, wherein the second acquisition unit acquires a two-dimensional image of a cross-section intersecting the core line within the section as the cross-sectional image.

8. The image processing apparatus according to claim 1, wherein the second acquisition unit acquires a two-dimensional image of a cross-section along the core line within the section as the cross-sectional image.

9. The image processing apparatus according to claim 1, wherein the likelihood information holding unit generates and holds likelihood information relating to the likelihood of existence based on the position of the core line in the cross-sectional image or the three-dimensional image.

10. The image processing apparatus according to claim 1, wherein the second extraction unit extracts the target region by inputting the cross-sectional image or the three-dimensional image and the likelihood information into a deep learning network.

11. The image processing apparatus according to claim 1, wherein the likelihood information holding unit determines the size-related information from the likelihood information based on the size of the structure.

12. The image processing apparatus according to claim 1, wherein the likelihood information holding unit determines the size information among the likelihood information relating to the likelihood of existence based on the characteristics of the structure.

13. The image processing apparatus according to claim 6, wherein the likelihood information holding unit determines information relating to size from the likelihood information relating to existence from the structure image.

14. The image processing apparatus according to claim 1, wherein the likelihood information holding unit determines information relating to size from the likelihood information relating to existence from the cross-sectional image or the three-dimensional image.

15. The image processing apparatus according to claim 1, wherein the likelihood information holding unit determines the likelihood information relating to the likelihood of existence based on the distance from the core wire.

16. The image processing apparatus according to claim 1, wherein the likelihood information holding unit determines the magnitude information of the likelihood information relating to the likelihood of existence based on the distance from the endpoint of the core line section.

17. The image processing apparatus according to claim 1, wherein the second acquisition unit generates the cross-sectional image or the three-dimensional image with a spatial resolution higher than that of the volume data.

18. The image processing apparatus according to claim 1, wherein the setting unit sets the interval according to the area of ​​interest.

19. The aforementioned structure has a structure that runs from the starting point through branching points to multiple peripheral points, The volume data is an image that includes at least one branch of the structure. The image processing apparatus according to claim 1, wherein the setting unit sets the section such that it passes through the branching portion and becomes a single line without branching, moving from the root side near the origin to the leaf side near the periphery.

20. We obtain volume data of a structure with a branched shape, Based on the volume data, the core wire of the structure is extracted. A section is set for at least a portion of the core wire, Based on the volume data, a cross-sectional image or a three-dimensional image including at least a portion of the section is obtained. The system retains likelihood information relating to the likelihood of existence in the cross-sectional image or the three-dimensional image. The extraction target region corresponding to the section of the aforementioned structure is extracted based on the cross-sectional image or the three-dimensional image and the likelihood information. An image processing method that includes the following.

21. We obtain volume data of a structure with a branched shape, Based on the volume data, the core wire of the structure is extracted. A section is set for at least a portion of the core wire, Based on the volume data, a cross-sectional image or a three-dimensional image including at least a portion of the section is obtained. The system retains likelihood information relating to the likelihood of existence in the cross-sectional image or the three-dimensional image. The extraction target region corresponding to the section of the aforementioned structure is extracted based on the cross-sectional image or the three-dimensional image and the likelihood information. A program that instructs a computer to perform various processes.

Images