Image processing device, method and program

Abstract

To enable the precise labeling of a plurality of structures such as vertebrae included in an image, in an image processing device, method and program.SOLUTION: A processor extracts a key point about each of a plurality of structures in association with a label of a first layer from an image including the plurality of structures which spatially and continuously exist with each corresponding label having a hierarchy, derives a graph structure obtained by associating the label of the first layer with a node by using the key point as the node, and associates the node with a label of a second layer lower than the first layer by analyzing the graph structure.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, a method, and a program. [Background technology]

[0002] The spinal cord is an extremely important part of the body, as it plays a role in transmitting messages between the brain and various parts of the body. For this reason, the spinal cord is protected by the spine, which is made up of multiple vertebrae. The presence or absence of lesions, such as damage and cancer metastasis, in the vertebrae that make up the spine is confirmed by interpreting tomographic images of a subject. When interpreting the images, it is necessary to identify each vertebra in order to identify the damaged or diseased vertebra. For this reason, various image processing algorithms have been proposed that acquire tomographic images of a subject, separate and recognize each of the multiple vertebrae based on the acquired tomographic images, and assign labels to each vertebra.

[0003] For example, Non-Patent Document 1 proposes a method of extracting and separating the spine in a three-dimensional image such as a CT (Computed Tomography) image or an MRI (Magnetic Resonance Imaging) image into cervical, thoracic, and lumbar vertebrae, and then using a convolutional neural network with a mechanism for recursively extracting vertebrae by reflecting the output back into the input, to extract and label each individual vertebra contained in the cervical, thoracic, and lumbar vertebrae. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Naoto Masuzawa, et. al, Automatic Segmentation, Localization, and Identification of Vertebrae in 3D CT Images Using Cascaded Convolutional Neural Networks, MICCAI 2020, pp. 681-690 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method described in Non-Patent Document 1, the individual vertebrae contained in the cervical, thoracic, and lumbar vertebrae are extracted recursively one by one. Therefore, if an error occurs in the extraction process, the error affects subsequent extraction processes, making it difficult to accurately label the vertebrae.

[0006] The present disclosure has been made in consideration of the above circumstances, and aims to enable accurate labeling of multiple structures, such as vertebrae, contained in an image. [Means for solving the problem]

[0007] An image processing device according to the present disclosure includes at least one processor, The processor extracts key points for each of the structures from an image including a plurality of structures that are spatially continuous and whose corresponding labels have a hierarchical structure, by associating the key points with a first-level label; Using keypoints as nodes, a graph structure is derived that associates first-level labels with nodes. By analyzing the graph structure, nodes are associated with labels in the second layer, which is lower than the first layer.

[0008] In the image processing device according to the present disclosure, the plurality of structures includes a plurality of vertebrae, The first-level label is at least one of the cervical, thoracic, and lumbar vertebrae. The second tier labels may be labels that identify individual bones in at least one of the cervical, thoracic, and lumbar vertebrae.

[0009] In the image processing device according to the present disclosure, the plurality of structures includes a plurality of ribs, The first level label is rib, The second layer of labels may be labels that identify individual ribs.

[0010] In this case, the second layer labels may be labels that identify the first to twelfth left ribs and the first to twelfth right ribs.

[0011] In addition, in the image processing device according to the present disclosure, the processor may analyze the graph structure to determine links to be cut in the graph structure using a graph cut method and determine second-level labels to be associated with each node.

[0012] In the image processing device according to the present disclosure, the processor may extract a plurality of structures associated with second-layer labels.

[0013] In the image processing device according to the present disclosure, the processor may display second-level labels.

[0014] In the image processing device according to the present disclosure, the processor may display second-layer labels in association with the extracted structures.

[0015] The image processing method according to the present disclosure includes extracting key points for each of a plurality of structures from an image including a plurality of structures that are spatially continuous and whose corresponding labels have a hierarchical structure, by associating the key points with a first-level label; Using keypoints as nodes, a graph structure is derived that associates first-level labels with nodes. By analyzing the graph structure, nodes are associated with labels in the second layer, which is lower than the first layer.

[0016] The image processing program according to the present disclosure includes the steps of extracting key points for each of a plurality of structures from an image including a plurality of structures that exist spatially consecutively and whose corresponding labels have a hierarchical structure, by associating the key points with a first-level label; A procedure for deriving a graph structure that associates first-level labels with nodes using keypoints as nodes; By analyzing the graph structure, the computer executes a procedure for associating nodes with labels in the second layer, which is lower than the first layer. [Effects of the Invention]

[0017] According to the present disclosure, multiple structures included in an image can be labeled with high accuracy. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a medical information system to which an image processing apparatus according to an embodiment of the present disclosure is applied; [Figure 2] FIG. 1 is a diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention; [Figure 3] Functional configuration diagram of an image processing apparatus according to this embodiment [Figure 4] A diagram showing an example of training data for constructing the first SS model. [Figure 5] A diagram showing the key point map [Figure 6] A perspective view of key points for explaining the derivation of the graph structure [Figure 7] Diagram for explaining graph generation [Figure 8] Diagram to explain weight settings [Figure 9] Diagram to explain weight settings [Figure 10] Diagram to explain the link to the ribs [Figure 11] A diagram showing an example of training data for building the second SS model. [Figure 12] A diagram showing the labeling results for vertebrae. [Figure 13] A diagram showing the display screen of the labeling results for ribs. [Figure 14] A flowchart showing the processing performed in this embodiment [Figure 15]FIG. 10 is a diagram showing another example of a display screen of the labeling results for vertebrae. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. First, the configuration of a medical information system to which an image processing device according to this embodiment is applied will be described. FIG. 1 is a diagram showing a schematic configuration of the medical information system. In the medical information system shown in FIG. 1, a computer 1 incorporating an image processing device according to this embodiment, an imaging device 2, and an image storage server 3 are connected in a communicable state via a network 4.

[0020] The computer 1 includes an image processing device according to this embodiment and has the image processing program of this embodiment installed. The computer 1 may be a workstation or personal computer operated directly by a doctor making a diagnosis, or may be a server computer connected to either of these via a network. The image processing program is stored in a storage device of the server computer connected to the network or in network storage in an externally accessible state, and is downloaded and installed into the computer 1 used by the doctor upon request. Alternatively, the program may be recorded on a recording medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed into the computer 1 from the recording medium.

[0021] The imaging device 2 is a device that captures an image of a diagnostic target region of a subject to generate a three-dimensional image representing the region, and is specifically a CT device, an MRI device, a PET (Positron Emission Tomography) device, or the like. The three-dimensional image composed of a plurality of tomographic images generated by the imaging device 2 is transmitted to and stored in the image storage server 3. In this embodiment, the diagnostic target region of the subject is the spine and ribs, and the imaging device 2 is a CT device that generates a CT image including the subject's spine and ribs as a three-dimensional image.

[0022] The image storage server 3 is a computer that stores and manages various data, and is equipped with a large-capacity external storage device and database management software. The image storage server 3 communicates with other devices via a wired or wireless network 4, sending and receiving image data and the like. Specifically, it acquires various data, including image data of 3D images generated by the imaging device 2, via the network, and stores and manages them on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between devices via the network 4 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0023] Next, an image processing device according to this embodiment will be described. Fig. 2 is a diagram showing the hardware configuration of the image processing device according to this embodiment. As shown in Fig. 2, the image processing device 20 includes a CPU (Central Processing Unit) 11, non-volatile storage 13, and memory 16 as a temporary storage area. The image processing device 20 also includes a display 14 such as a liquid crystal display, input devices 15 such as a keyboard and a mouse, and a network I / F (Interface) 17 connected to a network 4. The CPU 11, storage 13, display 14, input devices 15, memory 16, and network I / F 17 are connected to a bus 18. The CPU 11 is an example of a processor in the present disclosure.

[0024] The storage 13 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage 13 as a storage medium stores an image processing program 12. The CPU 11 reads the image processing program 12 from the storage 13, loads it into the memory 16, and executes the loaded image processing program 12.

[0025] Next, the functional configuration of the image processing device according to this embodiment will be described. Fig. 3 is a diagram showing the functional configuration of the image processing device according to this embodiment. As shown in Fig. 3, the image processing device 20 includes an image acquisition unit 21, a first extraction unit 22, a graph structure derivation unit 23, an analysis unit 24, a second extraction unit 25, and a display control unit 26. When the CPU 11 executes the image processing program 12, the CPU 11 functions as the image acquisition unit 21, the first extraction unit 22, the graph structure derivation unit 23, the analysis unit 24, the second extraction unit 25, and the display control unit 26.

[0026] The image acquisition unit 21 acquires a target image G0 to be processed from the image storage server 3 in response to an instruction from an operator via the input device 15. In this embodiment, the target image G0 is a CT image including the spine and ribs of a human body, as described above. The target image G0 is an example of an image disclosed herein.

[0027] Preprocessing may be performed on the target image G0. Examples of preprocessing include normalizing the spatial spacing, which differs for each target image, normalizing the brightness values, and reducing noise using a smoothing filter. The spatial spacing normalization process is a process of matching the image size per pixel and pixel spacing, which differ for each target image, between the images to be processed. The brightness value normalization process is a process of matching the brightness values, which differ for each target image, between the images. Specifically, a process of matching the average brightness values ​​may be performed. The preprocessing may be performed by the image acquisition unit 21 or the first extraction unit 22.

[0028] The spine includes multiple vertebrae. Each vertebra is anatomically labeled. Here, the spine is made up of four parts: the cervical vertebrae, the thoracic vertebrae, the lumbar vertebrae, and the sacrum. The cervical vertebrae consist of seven vertebrae, anatomically labeled C1 to C7. The thoracic vertebrae consist of twelve vertebrae, anatomically labeled T1 to T12. The lumbar vertebrae consist of five vertebrae, anatomically labeled L1 to L5. The sacrum consists of only one bone, anatomically labeled S1. In the following description, the vertebrae targeted by this embodiment are the cervical vertebrae, thoracic vertebrae, and lumbar vertebrae, excluding the sacrum.

[0029] Ribs are arch-shaped bones that extend from the thoracic vertebrae to the left and right. There are 12 ribs on each side, corresponding to the 12 vertebrae in the thoracic vertebrae. Therefore, the ribs can be anatomically labeled as the right 1st rib R1 to the right 12th rib R12 and the left 1st rib R1 to the left 12th rib R12.

[0030] Vertebrae can be labeled as cervical, thoracic, and lumbar vertebrae. The cervical, thoracic, and lumbar vertebrae can be associated with labels C1 through C7, T1 through T12, and L1 through L5, respectively. Ribs can be associated with rib labels, and can be associated with labels R1 through R12 (right rib R1) and R1 through R12 (left rib R1). Therefore, vertebrae and ribs are examples of structures that exist spatially contiguously in the present disclosure, and whose corresponding labels are hierarchical. The cervical, thoracic, lumbar, and ribs are examples of first-level labels. In addition, the first cervical vertebra C1 to the seventh cervical vertebra C7, the first thoracic vertebra T1 to the twelfth thoracic vertebra T12, the first lumbar vertebra L1 to the fifth lumbar vertebra L5, the first right rib R1 to the twelfth right rib R12, and the first left rib R1 to the twelfth left rib R12 are examples of second-level labels.

[0031] The image processing device according to this embodiment performs labeling by associating the labels of seven cervical vertebrae, twelve thoracic vertebrae, and five lumbar vertebrae with the vertebrae included in the target image G0, and also performs labeling by associating the labels of the first right rib R1 to the twelfth right rib R12 and the first left rib R1 to the twelfth left rib R12 with the ribs.

[0032] The first extraction unit 22 extracts key points for each of the vertebrae and ribs from the target image G0, associating them with labels in the first layer. Specifically, the first extraction unit 22 extracts key points for the vertebrae in association with labels for the cervical vertebrae, thoracic vertebrae, and lumbar vertebrae, and extracts key points for the ribs in association with labels for the ribs. To this end, the first extraction unit 22 has a semantic segmentation model for point extraction (hereinafter referred to as a first SS model) 30 and an identification unit 31.

[0033] In this embodiment, the first extraction unit 22 extracts points within each vertebra and center points within each rib included in the target image G0 as key points. Here, the center of gravity of the vertebral bodies constituting the vertebra is extracted as the center point within the vertebra, and multiple points on the center line of the rib are extracted as center points within the rib. The center line of the rib refers to a line connecting the centers (centers of gravity) of multiple cross sections that intersect with the longitudinal direction of the rib. Note that, since a line is a continuous line of points, in the following description, points on the center line of the rib will be referred to as the center point of the rib.

[0034] The first extraction unit 22 uses the first SS model 30 to output a probability distribution map representing the probability of existence of the center points of the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and ribs in the target image G0. The probability distribution map has elements that correspond one-to-one to each pixel in the target image G0, and the element values ​​of each element are data registered as combinations of the probability of existence of the center point of the cervical vertebrae, the probability of existence of the center point of the thoracic vertebrae, the probability of existence of the center point of the lumbar vertebrae, the probability of existence of the center point of the ribs, and the probability of non-existence. The non-existence probability is the probability that the center points of the cervical vertebrae, the thoracic vertebrae, the lumbar vertebrae, and the ribs do not exist. For example, an element value of (1.0,0,0,0,0) represents a 100% probability of existence of the center point of the cervical vertebrae, a 0% probability of existence of the center point of the thoracic vertebrae, a 0% probability of existence of the center point of the lumbar vertebrae, a 0% probability of existence of the center point of the ribs, and a 0% probability of non-existence.

[0035] The first SS model 30 is constructed by a convolutional neural network (hereinafter abbreviated as CNN). Examples of CNN include U-Net and Residual Network (ResNet). Specifically, the first SS model 30 is constructed by machine learning a CNN so that when a three-dimensional CT image including vertebrae and ribs is input, the first SS model 30 outputs a probability distribution map indicating the presence probability of extracting the center points of the cervical vertebrae, the thoracic vertebrae, the lumbar vertebrae, and the rib centers included in the target image G0.

[0036] The first SS model 30 is constructed by training a CNN using training data. FIG. 4 is a diagram showing an example of training data for constructing the first SS model. As shown in FIG. 4, the training data 40 includes training CT images 41 including ribs and vertebrae and ground truth data 42. The training CT images 41 are three-dimensional images, and FIG. 4 shows a tomographic image of a sagittal plane. Note that ribs are not included in the sagittal plane including the vertebrae, but the ribs are shown in FIG. 4 for the sake of explanation.

[0037] The correct answer data 42 is an image in which the center points of the vertebral bodies constituting the vertebrae and the center points of the ribs in the training CT image 41 are identified along with their labels. For example, in the correct answer data 42, a pixel corresponding to the center point of the vertebral bodies constituting the cervical vertebrae is assigned a label of 1, a pixel corresponding to the center point of the vertebral bodies constituting the thoracic vertebrae is assigned a label of 2, a pixel corresponding to the center point of the vertebral bodies constituting the lumbar vertebrae is assigned a label of 3, and a pixel corresponding to the center point of the ribs is assigned a label of 4. In FIG. 4, the label 1 is indicated by a white circle, the label 2 by a white triangle, the label 3 by a white square, and the label 4 by a black circle. Note that ribs exist on the left and right sides of the human body, but the training CT image 41 in FIG. 4 only shows the ribs on the left side of the human body, while the correct answer data 42 only shows the center point of the left rib on the left side of the human body.

[0038] During learning, a learning CT image 41 is input to the first SS model 30. The first SS model 30 outputs a learning probability distribution map for the learning CT image 41. Then, a loss calculation is performed for the first SS model 30 based on this learning probability distribution map and the supervised data 42. Then, various coefficients of the first SS model 30 are updated according to the results of the loss calculation, and the first SS model 30 is updated according to the update settings.

[0039] During training of the first SS model 30, a series of processes, including input of training CT images 41 to the first SS model 30, output of the training probability distribution map from the first SS model 30, loss calculation, update setting, and update of the first SS model 30, are repeated while training data 40 is exchanged. The repetition of the series of processes is terminated when the prediction accuracy of the training probability distribution map for the ground truth data 42 reaches a predetermined set level. The first SS model 30 whose prediction accuracy has thus reached the set level is stored in the storage 13 and used by the first extraction unit 22.

[0040] The identification unit 31 extracts elements in the probability distribution map output by the first SS model 30, whose existence probability of the center point of the cervical vertebrae is equal to or greater than a predetermined threshold (e.g., 0.9), as the center point of the cervical vertebrae, i.e., key points of the cervical vertebrae, and identifies the position coordinates of the center point of the cervical vertebrae in the target image G0. The identification unit 31 also extracts elements in the probability distribution map output by the first SS model 30, whose existence probability of the center point of the thoracic vertebrae is equal to or greater than a predetermined threshold (e.g., 0.9), as the center point of the thoracic vertebrae, i.e., key points of the thoracic vertebrae, and identifies the position coordinates of the center point of the thoracic vertebrae in the target image G0. The identification unit 31 also extracts elements in the probability distribution map output by the first SS model 30, whose existence probability of the center point of the lumbar vertebrae is equal to or greater than a predetermined threshold (e.g., 0.9), as the center point of the lumbar vertebrae, i.e., key points of the lumbar vertebrae, and identifies the position coordinates of the center point of the lumbar vertebrae in the target image G0. The identification unit 31 also extracts elements in the probability distribution map output by the first SS model 30 where the probability of existence of a rib center point is equal to or greater than a predetermined threshold (e.g., 0.9) as rib center points, i.e., rib key points, and identifies the position coordinates of the rib center points in the target image G0. Note that since the rib center points exist adjacent to each other in the target image G0, they may also be identified as lines.

[0041] Fig. 5 is a diagram showing a keypoint map that is the result of extracting keypoints from the target image G0. In the keypoint map 45 shown in Fig. 5, the center points of the cervical vertebrae extracted from the target image G0 are indicated by white circles, the center points of the thoracic vertebrae by white triangles, the center points of the lumbar vertebrae by white squares, and the center points of the ribs by black circles.

[0042] Each of the key points extracted by the first extraction unit 22 has a feature amount. The feature amount includes information indicating that the key point is one of the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and ribs, and the position coordinates of the key point in the target image G0. For example, the information indicating that the key point is a cervical vertebrae, the information indicating that the key point is a thoracic vertebrae, the information indicating that the key point is a lumbar vertebra, and the information indicating that the key point is a rib are (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1), respectively.

[0043] The graph structure derivation unit 23 uses the keypoints extracted from the target image G0 by the first extraction unit 22 as nodes to derive a graph structure that associates the labels of the keypoints with the nodes. Here, the vertebrae are arranged in the order of cervical, thoracic, and lumbar vertebrae, starting from the head. The graph structure derivation unit 23 links keypoints labeled as cervical, thoracic, and lumbar vertebrae only to keypoints within a predetermined distance in the body axis direction of the human body. As a result, the graph structure derivation unit 23 derives a graph structure by regarding the keypoints labeled as cervical, thoracic, and lumbar vertebrae as multiple connected elements. Meanwhile, for the ribs, the graph structure derivation unit 23 derives a graph structure by linking only the keypoints closest to the thoracic vertebrae in the anterior-posterior direction of the human body from the keypoints of the thoracic vertebrae. As a result, graph structures for the left ribs and the right ribs are derived.

[0044] FIG. 6 is a perspective view showing a graph structure. In the graph structure 46 shown in FIG. 6, the key points of the vertebrae are linked only to adjacent key points, but may also be linked to the vertebrae one level above depending on how a predetermined distance is selected. Also, in the graph structure 46, the key points of the ribs are linked only to key points in a direction intersecting the body axis of the human body, but may also be linked to key points of ribs aligned in the body axis direction. Also, some nodes and links are omitted in FIG. 6.

[0045] The analysis unit 24 analyzes the graph structure derived by the graph structure derivation unit 23, thereby associating nodes in the graph structure with labels in a second layer lower than the first layer. To this end, the analysis unit 24 analyzes the graph structure 46 by using a graph cut method to determine links to be cut in the graph structure 46 and to determine second layer labels to be associated with each of the nodes in the graph structure 46. That is, the analysis unit 24 analyzes the graph structure 46 so as to associate the labels of the first cervical vertebra C1 to the seventh cervical vertebra C7 with nodes corresponding to key points of the cervical vertebrae, the labels of the first thoracic vertebra T1 to the twelfth thoracic vertebra T12 with nodes corresponding to key points of the thoracic vertebrae, the labels of the first lumbar vertebra L1 to the fifth lumbar vertebra L5 with nodes corresponding to key points of the lumbar vertebrae, and the labels of the first right rib R1 to the twelfth right rib R12 and the first left rib R1 to the twelfth left rib R12 with nodes corresponding to key points of the ribs. This results in the extracted keypoints being labeled with second-level labels.

[0046] In this embodiment, the analysis unit 24 determines links to be cut in the graph structure using, for example, the method described in Japanese Patent Application Laid-Open No. 2017-187824. The method described in Japanese Patent Application Laid-Open No. 2017-187824 is a method for classifying N elements (N>2) arranged in a specific order from 0 to N-1 in a predetermined direction into one of N labels in a specific order.

[0047] The analysis unit 24 applies the method described in Japanese Patent Application Laid-Open Publication No. 2017-187824 to place reference nodes s and t in the graph structure 46, place a node w corresponding to a keypoint between nodes s and t, and generate a graph linking each node. FIG. 7 is a diagram for explaining graph generation. For simplicity's sake, assume that there are two cervical vertebrae, four thoracic vertebrae, and two lumbar vertebrae. As shown in FIG. 7, the analysis unit 24 defines, between the reference node s and node t, nodes w0 and w1 corresponding to two cervical vertebrae, nodes w2 to w5 corresponding to four thoracic vertebrae, and nodes w6 and w7 corresponding to two lumbar vertebrae. An s-link SL is defined between node s and node w1, and a t-link TL is defined between node w7 and node t. N-links NL1 to NL7 are defined between nodes w0 to w7. Nodes w0 to w7 correspond to n-links NL1 to NL7 and t-link TL, respectively. In this way, the analysis unit 24 generates the graph 47 .

[0048] The analysis unit 24 then assigns weights to the s links, t links, and n links in the direction from head to tail along the body axis (hereinafter referred to as the first direction), with the weights decreasing as the link corresponds to the node with the highest score at each node. Furthermore, the analysis unit 24 assigns weights that restrict cutting, as will be described later. Then, by performing graph cut processing on the graph for which the weights have been assigned, the analysis unit 24 determines which of the s links, t links, and n links to cut, and associates labels with multiple nodes, i.e., key points.

[0049] The setting of weights will be explained below. In this embodiment, weights are set for multiple links in a graph structure. Specifically, weights are set by adding the scores for all nodes to the weights of links other than the link corresponding to that node. FIG. 8 is a diagram for explaining the setting of weights. Note that the setting of weights will be explained here using the graph generated as shown in FIG. 7.

[0050] In this embodiment, the information that each keypoint, i.e., node, has, representing a cervical vertebra, information representing a thoracic vertebra, or information representing a lumbar vertebra, is used as a score. That is, a keypoint that has information representing a cervical vertebra as a feature will have a score of 1 for being a cervical vertebra, and a score of 0 for being a thoracic vertebra and a lumbar vertebra. A keypoint that has information representing a thoracic vertebra as a feature will have a score of 0 for being a cervical vertebra, a score of 1 for being a thoracic vertebra, and a score of 0 for being a lumbar vertebra. A keypoint that has information representing a lumbar vertebra as a feature will have a score of 0 for being a cervical vertebra and a thoracic vertebra, and a score of 1 for being a lumbar vertebra.

[0051] First, the setting of link weights for the cervical vertebrae will be explained. Fig. 8 is a diagram for explaining the setting of link weights for the cervical vertebrae. Here, as mentioned above, the information indicating that the key point is the cervical vertebrae is (1,0,0,0) for the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and ribs. Therefore, the scores of nodes w0 to w7 are (1,1,0,0,0,0,0).

[0052] Since the score of the first node w0 is 1, 1 is added to the weight of s link SL, the score of node w0 is not added to the weight of n link NL1 corresponding to node w0, and a score of 1 is added to each of the weights of n links NL2 to NL7 corresponding to nodes w1 to w6 and the weight of t link TL corresponding to node w7. As a result, the weights of s link SL, n links NL1 to NL7 and t link TL become (1,0,1,1,1,1,1,1,1).

[0053] Next, because the score of node w1 is 1, the score of node w1, 1, is added to the weight of s link SL, and the score of node w1, 1, is added to the weight of n link NL1 corresponding to node w0. The score of node w1 is not added to the weight of n link NL2 corresponding to node w1, and a score of 1 is added to each of the weights of n links NL3 to NL7 corresponding to nodes w2 to w6, and the weight of t link TL corresponding to node w7. As a result, the weights of s link SL, n links NL1 to NL7, and t link TL become (2,1,1,2,2,2,2,2,2,2).

[0054] Next, since the score of node w2 is 0, no weights are added to s link SL, n links NL1 to NL7, and t link TL. As a result, the weights of s link SL, n links NL1 to NL7, and t link TL become (2,1,1,2,2,2,2,2,2).

[0055] Similarly, by adding the scores for all nodes to the weights of links other than the corresponding link, the weights of s link SL, n links NL1 to NL7, and t link TL become (2,1,1,2,2,2,2,2,2). Therefore, the weights of n links NL1 and NL2 are the smallest.

[0056] Next, we will explain how to set the link weights for the thoracic vertebrae. Figure 9 is a diagram for explaining how to set the link weights for the cervical vertebrae. Here, as mentioned above, the information indicating that the key point is the thoracic vertebrae is (0,1,0,0) for the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and ribs. Therefore, the scores for nodes w0 to w7 are (0,0,1,1,1,1,0,0).

[0057] Since the score of the first node w0 is 0, no weights are added to the s link SL, the n links NL1 to NL7, and the t link TL. As a result, the weights of the s link SL, the n links NL1 to NL7, and the t link TL become (0,0,0,0,0,0,0,0,0).

[0058] Since the score of the second node w1 is also 0, no weights are added to s link SL, n links NL1 to NL7, and t link TL. As a result, the weights of s link SL, n links NL1 to NL7, and t link TL become (0,0,0,0,0,0,0,0,0).

[0059] Since the score of node w2 is 1, 1 is added to the weight of link SL, and the score of node w2, 1, is added to the weights of n links NL1 and NL2 corresponding to nodes w0 and w1. The score of node w2 is not added to the weight of n link NL3 corresponding to node w2, and a score of 1 is added to the weights of n links NL4 to NL7 corresponding to nodes w3 to w6, and the weight of t link TL corresponding to node w7. As a result, the weights of s link SL, n links NL1 to NL7, and t link TL become (1,1,1,0,1,1,1,1,1).

[0060] Similarly, by adding the scores for all nodes to the weights of links other than the corresponding link, the weights of s link SL, n links NL1 to NL7, and t link TL become (4, 4, 4, 3, 3, 3, 3, 4, 4). Therefore, the weights of n links NL3 to NL6 are the smallest.

[0061] Similarly, when the weights of s link SL, n links NL1 to NL7, and t link TL are set for the lumbar vertebrae, the weights of s link SL, n links NL1 to NL7, and t link TL are (2,2,2,2,2,2,2,1,1). Therefore, the weights of n link NL7 and t link TL are the smallest.

[0062] In this embodiment, the analysis unit 24 applies the method described in Japanese Patent Application Publication No. 2017-187824 to set weights that restrict cutting so that the same label does not appear consecutively. Ideally, the keypoints extracted by the first extraction unit 22 correspond to one vertebra, so ideally, weights that restrict cutting can be set so that the same label does not appear twice or more consecutively. However, in cases where a lesion such as a compression fracture is present, keypoints may not be extracted in an ideal manner, and multiple keypoints corresponding to a single vertebra may be extracted. Taking such cases into consideration, weights that restrict cutting can be set to allow two or three consecutive identical labels. In any case, by applying the method described in Japanese Patent Application Publication No. 2017-187824, cutting can be controlled so that the same label does not appear consecutively more than necessary. By determining globally optimal cutting by combining the links between keypoints of each vertebra and the links between keypoints of the ribs and thoracic vertebrae, it is possible to prevent local keypoint extraction errors. For the sake of simplicity, it is assumed below that the analysis unit 24 sets weights to restrict cutting of the graph 47 so that the same label does not appear twice or more consecutively.

[0063] By setting the weights in this manner, the analysis unit 24 can first cut links at n links NL1 and NL2 for the cervical vertebrae shown in Fig. 7. Furthermore, it can cut links at n links NL3, NL4, NL5, and NL6 for the thoracic vertebrae shown in Fig. 7. Furthermore, it can cut links at n link NL7 and t link TL for the lumbar vertebrae shown in Fig. 7.

[0064] Therefore, when applied to the actual vertebral link structure shown in Fig. 6, the analysis unit 24 can cut the links between the key points of the first to seventh cervical vertebrae, the link between the key point of the seventh cervical vertebra and the key point of the first thoracic vertebra, the link between the key points of the first to twelfth thoracic vertebrae, the link between the key point of the twelfth thoracic vertebra and the key point of the first lumbar vertebra, and the link between the key points of the first to fifth lumbar vertebrae, for the cervical vertebrae. As a result, all links between adjacent key points in the body axis direction for the vertebrae are cut.

[0065] The analysis unit 24 also associates a label with each keypoint according to the position of the severed link. For example, in this embodiment, since the link between the keypoint of the seventh cervical vertebra and the keypoint of the first thoracic vertebra is severed, the analysis unit 24 associates the label of the seventh cervical vertebra C7 with the keypoint above the severed link in the body axis direction. Furthermore, the analysis unit 24 associates the labels of the sixth cervical vertebra C6, the fifth cervical vertebra C5, the fourth cervical vertebra C4, the third cervical vertebra C3, the second cervical vertebra C2, and the first cervical vertebra C1 with the keypoints above the seventh cervical vertebra C7 in the body axis direction. Furthermore, the analysis unit 24 associates the label of the first thoracic vertebra T1 with the keypoint below the severed link in the body axis direction. The analysis unit 24 associates the labels of the second thoracic vertebra T2 to the twelfth thoracic vertebra T12 with the keypoints below the first thoracic vertebra T1 in the body axis direction.

[0066] In this embodiment, the link between the key point of the twelfth thoracic vertebra T12 and the key point of the first lumbar vertebra L1 is severed, so the analysis unit 24 associates the label of the first lumbar vertebra L1 with the key point below the cutting position in the body axis direction, and then associates the labels of the second lumbar vertebra L1 to the fifth lumbar vertebra L5 with the key points below the first lumbar vertebra L1 in the body axis direction.

[0067] As a result, the vertebrae are labeled as follows: the first cervical vertebra C1 to the seventh cervical vertebra C7, the first thoracic vertebra T1 to the twelfth thoracic vertebra T12, and the first lumbar vertebra L1 to the fifth lumbar vertebra L5.

[0068] On the other hand, for the ribs, a large weight is first set for the link between the left rib and the adjacent thoracic vertebra for each of the right and left ribs. Furthermore, a large weight is set for the link in the direction intersecting the body axis, and a small weight is set for the link in the direction of the body axis. FIG. 10 is a diagram for explaining the links for the ribs. Note that FIG. 10 shows the links for three ribs and the sternum nodes to which the ribs correspond. As shown in FIG. 10, the key point of the top rib is linked by links D1 to D5 and is linked to the thoracic vertebra by link D6. Furthermore, the top rib is linked to the second rib by links D7 to D9, and the second rib is linked to the third rib by links D10 and D11. The analysis unit 24 sets a large weight for the links D1 to D6 and a small weight for the links D7 to D11.

[0069] As a result, the links between the left and right ribs are cut in the direction of the body axis, but not in the direction intersecting the body axis. Furthermore, the links between the key points of the corresponding thoracic vertebrae are also not cut. Therefore, the left and right ribs can be separated into 12 pieces each. Furthermore, since each rib is associated with a thoracic vertebra, the labels of the first left rib R1 to the twelfth left rib R12 and the first right rib R1 to the twelfth right rib R12 can be associated by assigning the same numbers as the thoracic vertebrae.

[0070] The second extraction unit 25 extracts the labeled vertebrae and ribs from the target image G0. To this end, the second extraction unit 25 has a semantic segmentation model for region identification (hereinafter referred to as the second SS model) 32. When the target image G0 and the keypoint map derived by the first extraction unit 22 are input, the second SS model 32 outputs an identified image in which the individual vertebrae, left ribs, and right ribs are identified.

[0071] The second SS model 32 is constructed by a CNN. Examples of the CNN include U-Net and ResNet. Specifically, when the target image G0 and the keypoint map are input, the second SS model 32 is constructed by machine learning the CNN so that the CNN identifies the regions of individual vertebrae included in the target image G0.

[0072] The second SS model 32 is trained by being given training data in the training phase. FIG. 11 is a diagram showing an example of training data for constructing the second SS model. As shown in FIG. 11, the training data 50 includes a training CT image 51, a training keypoint map 52, and supervised data 53. The training CT image 51 is an image similar to the training CT image 41 used to train the first SS model 30. The training keypoint map 52 is an image from which keypoints of the vertebrae and ribs in the training CT image 51 have been extracted. The supervised data 53 is an image from which the regions of the vertebrae and ribs in the training CT image 51 have been identified.

[0073] During learning, a learning CT image 51 and a learning keypoint map 52 are input to the second SS model 32. The second SS model 32 outputs a learning probability distribution map for the learning CT image 51. Then, a loss calculation is performed for the second SS model 32 based on this learning probability distribution map and the correct answer data 53. Then, update settings for various coefficients of the second SS model 32 are performed according to the results of the loss calculation, and the second SS model 32 is updated according to the update settings.

[0074] During training of the second SS model 32, a series of processes, including input of training CT images 51 to the second SS model 32, output of the training probability distribution map from the second SS model 32, loss calculation, update setting, and update of the second SS model 32, are repeatedly performed while the training data 50 is exchanged. The repetition of the series of processes is terminated when the prediction accuracy of the training probability distribution map for the ground truth data 53 reaches a predetermined set level. The second SS model 32 whose prediction accuracy has thus reached the set level is stored in the storage 13 and used by the second extraction unit 25.

[0075] The identification image extracted by the second extraction unit 25 when the target image G0 is input will be described later.

[0076] The display control unit 26 displays the labeling results together with the identification images. FIG. 12 is a diagram showing a display screen of the labeling results for the vertebrae. As shown in FIG. 12, an identification image G1 for the vertebrae is displayed on the display screen 60. The identification image G1 is a tomographic image of a sagittal section of the human body, and the labeling results are displayed in association with each of the vertebrae extracted by the second extraction unit 25. Note that the identification image G1 shown in FIG. 12 includes the sixth cervical vertebra C6, the seventh cervical vertebra C7, the first to twelfth thoracic vertebrae T1 to T12, the first lumbar vertebra L1, and the second lumbar vertebra L2, and therefore the labels assigned to each vertebra are displayed. Note that the colors of the labels may be different. In this case, the vertebrae may be assigned the same color as the label. Furthermore, the keypoints detected in the vertebrae may be displayed in the same color as the label.

[0077] FIG. 13 is a diagram showing a display screen of the labeling results for the ribs. As shown in FIG. 13, a discrimination image G2 is displayed on the display screen 61. The discrimination image G2 is a tomographic image of an axial section of the human body, and the labeling results are displayed in association with each of the ribs extracted by the second extraction unit 25. The discrimination image G2 shown in FIG. 13 displays the seventh thoracic vertebra T7 and the seventh rib R7 to the second rib R2 on the left and right. In the discrimination image G2 for the ribs, the colors of the labels may also be different, or the same colors as the labels may be assigned to the vertebrae and ribs. Furthermore, the key points detected in the vertebrae and ribs may be displayed in the same colors as the labels.

[0078] Next, the processing performed in this embodiment will be described. FIG. 14 is a flowchart showing the processing performed in this embodiment. It is assumed that the target image G0 is acquired from the image storage server 3 and stored in the storage 13. First, the first extraction unit 22 extracts keypoints from the target image G0 by associating them with labels in the first layer (first extraction; step ST1). Next, the first extraction unit 22 derives a graph structure in which the labels of the keypoints are associated with the nodes using the keypoints extracted from the target image G0 as nodes (step ST2). Next, the analysis unit 24 analyzes the graph structure derived by the graph structure derivation unit 23 to associate the nodes in the graph structure with labels in the second layer, which is lower than the first layer (analysis processing; step ST3). Next, the second extraction unit 25 extracts vertebrae and ribs associated with the labels from the target image G0 (second extraction; step ST4). Then, the display control unit 26 displays the labeling results (step ST5), and the processing ends.

[0079] In this manner, in this embodiment, key points of the vertebrae and ribs contained in the target image G0 are extracted in association with labels in the first layer, the key points are used as nodes to derive a graph structure in which the labels in the first layer and the nodes are associated, and by analyzing the graph structure, the nodes are associated with labels in the second layer, which is lower than the first layer. This allows for accurate labeling of multiple structures such as the vertebrae and ribs contained in the target image G0.

[0080] In the above embodiment, when labeling each vertebra included in the cervical, thoracic, and lumbar vertebrae, the analysis unit 24 may set scores that reflect the relative relationship with other vertebrae at the boundaries between the cervical, thoracic, and lumbar vertebrae. For example, when labeling the thoracic vertebrae, at the boundary between the cervical and thoracic vertebrae, the score of the keypoint of the cervical vertebra adjacent to the thoracic vertebra may be set to 1, and the score of the keypoint of the thoracic vertebra adjacent to the cervical vertebra may be set to 10. In addition to this, any known graph cut method may be applied, such as setting scores taking into account the features surrounding each keypoint.

[0081] In the above embodiment, the analysis unit 24 determines the links to be cut in the graph structure by the graph cut method, but the present invention is not limited to this. A graph convolutional neural network (GCN) for analyzing the graph structure may be used to predict the class to which the node in the graph structure belongs, thereby associating the graph structure with a label in a second layer lower than the first layer.

[0082] In the above embodiment, the display control unit 26 displays the identified images G1 and G2 and the labeling results after the second extraction unit 25 extracts the vertebrae and rib regions, but this is not limited to this. Once the analysis unit 24 associates the nodes in the graph structure with the second-level labels, a keypoint map may be superimposed on the target image G0, and the labeling results may be added to the superimposed image. A display screen of the labeling results for the vertebrae displayed in this case is shown in FIG. 15. In FIG. 15, the target image G0 with the keypoints superimposed is displayed on the display screen 62.

[0083] Furthermore, although the above embodiment includes the second extraction unit 25, the present invention is not limited to this. The second extraction unit 25 may not be included. In this case, the labeling results may be displayed on the target image G0 with the key points superimposed thereon, as shown in FIG. 15, without extracting the vertebrae and ribs from the target image G0.

[0084] In the above embodiment, the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and ribs are labeled, but this is not limiting. Labeling may be performed only for each of the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and ribs. Alternatively, labeling may be performed only for a combination of two or more of the cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and ribs.

[0085] Furthermore, in the above embodiment, the following various processors can be used as the hardware structure of the processing units that perform various processes, such as the image acquisition unit 21, the first extraction unit 22, the graph structure derivation unit 23, the analysis unit 24, the second extraction unit 25, and the display control unit 26. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD), a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0086] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0087] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0088] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0089] 1. Computer 2. Imaging equipment 3. Image storage server 4 Network 11 CPU 12 Image Processing Programs 13. Storage 14 Display 15 Input Devices 16 memory 17 Network I / F 18 Bus 20 Image processing device 21 Image acquisition unit 22 1st extraction part 23 Graph structure derivation part 24 Analysis Department 25 Second extraction part 26 Display control unit 30 1st SS model 31 Specific section 32 2nd SS model 40,50 training data 41,51 Training CT images 52 Learning Keypoint Map 42,53 Correct data 45 Keypoint Map 46 Graph Structure 60,61,62 display screen D1~D11,NL1~NL7,SL,TL Link G0 Target image G1, G2 identification image ss node tt node w0~w7 nodes

Claims

1. at least one processor; The processor: extracting key points for each of a plurality of structures, including a plurality of vertebrae, from an image including the plurality of structures that are spatially continuous and have hierarchical labels corresponding to the plurality of structures, in association with a first-level label that is at least one of cervical vertebrae, thoracic vertebrae, and lumbar vertebrae; deriving a graph structure in which the first layer labels are associated with the nodes using the key points as nodes; An image processing device that analyzes the graph structure to associate the nodes with labels in a second hierarchy that are lower than the first hierarchy and identify individual bones in at least one of the cervical vertebrae, thoracic vertebrae, and lumbar vertebrae.

2. at least one processor; The processor: extracting key points for each of the plurality of structures, including a plurality of ribs, from an image including the plurality of structures that are spatially continuous and have hierarchical labels corresponding to the plurality of structures, by associating the key points with a first-level label that is a rib; deriving a graph structure in which the first layer labels are associated with the nodes using the key points as nodes; An image processing device that analyzes the graph structure to associate the nodes with labels in a second layer that is lower than the first layer and that identifies individual ribs.

3. The image processing device according to claim 2 , wherein the second layer of labels are labels for identifying the first to twelfth left ribs and the first to twelfth right ribs.

4. 4. The image processing device according to claim 1, wherein the processor analyzes the graph structure to determine links to be cut in the graph structure using a graph cut method and to determine the second-level labels to be associated with each of the nodes.

5. The image processing device according to claim 1 , wherein the processor extracts the plurality of structures associated with the second layer label.

6. The image processing device according to claim 1 , wherein the processor displays the second layer of labels.

7. The image processing device according to claim 5 , wherein the processor displays the extracted structures in association with the second layer labels.

8. A computer extracts key points for each of a plurality of structures, including a plurality of vertebrae, from an image containing the structures, which are spatially continuous and have hierarchical labels corresponding to each of the structures, by associating the key points with a first-level label, which is at least one of cervical vertebrae, thoracic vertebrae, and lumbar vertebrae; deriving a graph structure in which the first layer labels are associated with the nodes using the key points as nodes; An image processing method that analyzes the graph structure to associate the nodes with labels in a second hierarchy that are lower than the first hierarchy and identify individual bones in at least one of the cervical vertebrae, thoracic vertebrae, and lumbar vertebrae.

9. A computer extracts key points for each of a plurality of structures, including a plurality of ribs, from an image that includes a plurality of structures that exist spatially consecutively and whose corresponding labels have a hierarchy, by associating the key points with a first-level label that is a rib; deriving a graph structure in which the first layer labels are associated with the nodes using the key points as nodes; An image processing method for associating the nodes with labels at a second layer, which is lower than the first layer and identifies individual ribs, by analyzing the graph structure.

10. extracting key points for each of a plurality of structures, including a plurality of vertebrae, from an image including the plurality of structures that are spatially continuous and have hierarchical labels corresponding to the plurality of structures, in association with a first-level label that is at least one of cervical vertebrae, thoracic vertebrae, and lumbar vertebrae; a step of deriving a graph structure in which the first layer labels are associated with the nodes by using the key points as nodes; and a procedure for associating the nodes with labels in a second hierarchy, which are labels that identify individual bones in at least one of the cervical vertebrae, thoracic vertebrae, and lumbar vertebrae, lower than the first hierarchy, by analyzing the graph structure.

11. extracting key points for each of a plurality of structures, including a plurality of ribs, from an image containing the plurality of structures, which are spatially continuous and have hierarchical labels, by associating the key points with a first-level label, which is a rib; a step of deriving a graph structure in which the first layer labels are associated with the nodes by using the key points as nodes; and a procedure for associating the nodes with labels at a second layer, which is a lower layer than the first layer and identifies individual ribs, by analyzing the graph structure.

Images