Image processing device, image processing system, image processing method, and program

Abstract

To enable an accurate and stable positioning between images.SOLUTION: An image processing device includes: an acquiring part for acquiring at least one of first identification information for identifying a first plurality of bones depicted in a first image on which a subject is imaged and second identification information for identifying a second plurality of bones depicted in a second image on which a subject is imaged; a classification part for classifying the first plurality of bones and the second plurality of bones, using at least one of pieces of the identification information, into a first bone group including bones that move following a movement of a first part of the subject and a second bone group including bones that move following a movement of a second part which is different from the first part; and a positioning part which conducts first positioning of the first plurality of bones and the second plurality of bones classified into the first bone group, and second positioning of the first plurality of bones and the second plurality of bones classified into the second bone group.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The disclosed technology relates to an image processing device, an image processing system, an image processing method, and a program. [Background technology]

[0002] In the medical field, attempts have been made to visualize changes over time in lesions, etc. by presenting a user with a difference image generated from two images taken at different times using various modalities.

[0003] Non-Patent Document 1 discloses a technique for performing alignment focusing on bones between two 3D images obtained by imaging using a CT device and generating a difference image between the images. Patent Document 1 also discloses a technique for recognizing multiple bone sites contained in each of the two 3D images and performing alignment processing between images of the corresponding bone sites. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Ryo Sakamoto, et al. "Temporal Subtraction of Serial CT Images with Large Deformation Diffeomorphic Metric Mapping in the Identification of Bone Metastases", Radiology,November 2017. [Patent documents]

[0005] [Patent Document 1] JP 2017-63936 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the alignment technique described in Non-Patent Document 1, when the images to be aligned contain multiple bone parts whose movements differ significantly or multiple bone parts whose boundaries experience discontinuous deformation, it may be difficult to accurately align each bone part.

[0007] On the other hand, the alignment technology of Patent Document 1 requires that individual bone parts be recognized and matched. For example, in cases where part of a bone has been removed during surgery, there may be deviations in the alignment, and it may be difficult to stably align the recognized bones.

[0008] In view of the above-mentioned problems, the disclosed technology has an object to provide an image processing technology that enables accurate and stable alignment between images. [Means for solving the problem]

[0009] An image processing device according to one aspect of the disclosed technology has the following configuration: That is, the image processing device includes an acquisition means for acquiring at least one of first identification information for identifying a first plurality of bones depicted in a first image of a subject and second identification information for identifying a second plurality of bones depicted in a second image of the subject; a classification means for classifying the first and second bones into a first bone group including bones that move in association with a movement of a first part of the subject and a second bone group including bones that move in association with a movement of a second part different from the first part, using the at least one of the identification information; and an alignment means for performing a first alignment of the first plurality of bones classified into the first bone group with the second plurality of bones, and a second alignment of the first plurality of bones classified into the second bone group with the second plurality of bones.

[0010] An image processing device according to another aspect of the disclosed technique includes the following configuration: the image processing device includes an acquisition unit that acquires identification information for identifying a plurality of bones depicted in a first image of a subject and a second image of the subject; a classification means for classifying the plurality of bones into a first bone group including bones that move in association with a movement of a first part of the subject, using the identification information; and a registration means for performing a first registration between a plurality of bones depicted in the first image and a plurality of bones depicted in the second image, the plurality of bones being classified into the first bone group. Effect of the Invention

[0011] The disclosed technology enables accurate and stable alignment between images. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing the arrangement of an image processing system according to a first embodiment. [Diagram 2] FIG. 2 is a flowchart showing an overall processing procedure in the first embodiment. [Diagram 3] 4A and 4B are diagrams showing bone identification information in a first medical image and a second medical image. [Figure 4] 1A-1C are diagrams illustrating a group of bones in a first medical image and a second medical image; [Diagram 5] FIG. 13 is a diagram showing the arrangement of an image processing system according to a second embodiment. [Figure 6] FIG. 11 is a flowchart showing an overall processing procedure according to the second embodiment. [Figure 7] FIG. 11 is a flowchart showing an overall processing procedure according to a third embodiment. [Figure 8] FIG. 13 is a diagram showing the arrangement of an image processing system according to a fourth embodiment. [Figure 9] FIG. 13 is a flowchart showing an overall processing procedure according to the fourth embodiment. [Figure 10] FIG. 13 is a diagram showing the arrangement of an image processing system according to a fifth embodiment. [Figure 11] FIG. 13 is a flowchart showing an overall processing procedure in the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] <First embodiment> The image processing device according to this embodiment is a device that performs registration between two images captured at different times and generates a difference image, etc. Specifically, a plurality of bones are identified from the two images, and the bones are classified into a plurality of groups of bones that move in association with a plurality of different body parts of a subject, and then registration between the images is performed for each bone group.

[0015] 1 is a diagram showing a configuration of an image processing system 10 according to the first embodiment. The image processing system 10 has an image processing device 100, which is provided with a communication interface 107 (communication I / F) for connection to a data server 130 via a communication network 120. The image processing device 100 according to the first embodiment is a device that performs registration between two images (a first medical image and a second medical image) captured at different times, and generates a difference image between the first medical image and the second medical image, etc., based on the registration result.

[0016] The data server 130 holds a plurality of medical images. The data server 130 represents, for example, a PACS (Picture Archiving and Communication Systems) that receives medical image data captured by a modality and stores and manages it through a network. In the following description, it is assumed that the data server 130 holds a plurality of three-dimensional tomographic images obtained by capturing images of a subject in advance under different conditions (different modalities, imaging modes, imaging dates and times, body positions, etc.) as a first medical image and a second medical image. In this embodiment, it is assumed that the first medical image and the second medical image are three-dimensional tomographic images (three-dimensional medical images) captured by an X-ray CT apparatus as examples of medical images.

[0017] In this specification, the axis representing the direction from the right hand to the left hand of the subject is defined as the X-axis, the axis representing the direction from the front to the back of the subject is defined as the Y-axis, and the axis representing the direction from the head to the feet of the subject is defined as the Z-axis. The XY section is defined as the axial plane, the YZ section as the sagittal plane, and the ZX section as the coronal plane. That is, the X-axis direction is a direction perpendicular to the sagittal plane (hereinafter, "sagittal direction"). The Y-axis direction is a direction perpendicular to the coronal plane (hereinafter, "coronal direction"). The Z-axis direction is a direction perpendicular to the axial plane (hereinafter, "axial direction"). In this case, in the case of a CT image, which is a three-dimensional image formed as a collection of two-dimensional tomographic images (slices), the slice plane of the image represents the axial plane, and the direction perpendicular to the slice plane (hereinafter, "slice direction") represents the axial direction. The method of taking the coordinate system is just an example, and it is possible to set the coordinate system by other definitions.

[0018] The modality for capturing the three-dimensional tomographic image may be an MRI device, a three-dimensional ultrasound imaging device, a photoacoustic tomography device, a PET / SPECT, an OCT device, or the like. Furthermore, the first medical image and the second medical image may be any images as long as they are three-dimensional tomographic images to be aligned. For example, they may be images captured at the same time using different modalities or different imaging modes. Furthermore, they may be images captured of the same patient using the same modality and in the same position on different dates and times for follow-up observation. The first medical image and the second medical image are three-dimensional medical images (three-dimensional tomographic images) configured as a collection of two-dimensional tomographic images. The positions and orientations of the two-dimensional tomographic images are converted into a reference coordinate system (a coordinate system in space based on the subject) and then stored in the data server 130. At this time, the first medical image and the second medical image expressed in the reference coordinate system are input to the image processing device 100 in response to an instruction from a user who operates the instruction unit 140. The instruction unit 140 includes various input devices that accept various commands from a user, such as a mouse, a keyboard, a trackball, a touch panel, and the like, and includes various devices that allow the user to input various information and processing requests.

[0019] The image processing device 100 is a device that receives a processing request from a user from the instruction unit 140, performs image processing, and outputs the result of the image processing to the display unit 150, and functions as a terminal device for interpretation operated by a user such as a doctor. Specifically, based on an instruction from the user via the instruction unit 140, a first medical image and a second medical image to be subjected to image processing are acquired from the data server 130 as a pair of images (image pair) to be subjected to image processing. Then, the image processing device 100 performs registration processing of the acquired first medical image and second medical image, and generates a difference image between the first medical image and the second medical image based on the registration result, and outputs it to the display unit 150.

[0020] The image processing device 100 is composed of the components described below. The functions of the following components are realized, for example, by one or more central processing units (CPUs) functioning as control units of the image processing device 100 executing programs. The components of the image processing device 100 may be composed of integrated circuits or the like as long as they perform similar functions.

[0021] The image acquisition unit 101 acquires information on the first medical image and the second medical image input to the image processing device 100. The identification information acquisition unit 102 acquires identification information for identifying each of the multiple bones from the input medical image. The classification unit 103 classifies the multiple bones into multiple bone groups associated with the movements of multiple different parts (body parts) of the subject based on the identification information of the multiple bones acquired from the identification information acquisition unit 102. The alignment unit 104 performs alignment processing between the first medical image and the second medical image for each bone group, and calculates a displacement field between the images for each bone group. The image generation unit 105 generates a difference image between the deformed images of the first medical image and the second medical image as a result image based on the acquired multiple displacement fields. The display control unit 106 performs display control to display the generated difference image, deformed image, etc. on the display unit 150.

[0022] The display unit 150 is composed of any device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a plasma display, an organic EL panel, etc., and displays medical images and the like for interpretation by a doctor. Specifically, it displays cross-sectional images of the first medical image and the second medical image acquired from the image processing device 100. It also displays cross-sectional images of the deformed image and the differential image generated by the image processing device 100.

[0023] In this embodiment, the registration between images refers to a process of calculating deformation information for displacing each pixel position of one image to the corresponding pixel position of the other image. For example, when there are two images captured at different times, the displacement field from one image to the other image is calculated as deformation information by estimating the pixel positions of the other image corresponding to each pixel position of the one image used as a reference. Here, the displacement field between three-dimensional tomographic images is a three-dimensional vector field that stores the displacement in each of the X, Y, and Z directions at each position in the image. In addition, one image (fixed image) used as the reference for registration is called a reference image, and the other image (image to be deformed) is called a floating image. In this embodiment, the first medical image is treated as a reference image, and the second medical image is treated as a floating image.

[0024] (Processing procedure of image processing device 100) FIG. 2 is a flowchart showing the overall processing procedure performed by the image processing device 100.

[0025] (S1010: Acquire the first medical image and the second medical image) In step S1010, the image acquisition unit 101 acquires the first medical image and the second medical image designated by the user through the instruction unit 140 from the data server 130. Then, the image acquisition unit 101 outputs the first medical image and the second medical image to the identification information acquisition unit 102, the positioning unit 104, and the display control unit 106. Note that the acquisition of the images is not limited to that based on the user's instruction, and may be performed by any other method. For example, when a captured image is stored in the data server 130, a pair of the image (first medical image) and an image to be compared (second medical image) may be automatically acquired based on a predetermined rule. An example of the image to be compared is automatic acquisition of a past image (e.g., the image with the latest shooting date and time) of the same patient as the captured image. As an example of the automatic acquisition process, for example, the first medical image can be automatically acquired as the most recent examination image of the subject to be examined, and the second medical image can be automatically acquired as the second most recent examination image of the same subject. The method of automatic acquisition is not limited to this, and for example, the second medical image may be automatically acquired as the oldest examination image of the same subject. Also, a configuration may be adopted in which only the first medical image is designated by the user, and the second medical image is automatically acquired based on a predetermined rule in the same manner as above.

[0026] (S1020: Identifying the first and second multiple bones from two images, respectively) In step S1020, the identification information acquisition unit 102 identifies a plurality of bones from each of the first medical image and the second medical image. Here, the plurality of bones depicted in the first medical image are referred to as the first plurality of bones, and the plurality of bones depicted in the second medical image are referred to as the second plurality of bones. Also, information for identifying each of the first plurality of bones from the first medical image is referred to as the first identification information, and information for identifying each of the second plurality of bones from the second medical image is referred to as the second identification information.

[0027] The identification information acquiring unit 102 acquires at least one of first identification information for identifying each of a first plurality of bones depicted in a first medical image obtained by capturing an image of a subject, and second identification information for identifying each of a second plurality of bones depicted in a second medical image different from the first medical image. Then, the identification information acquiring unit 102 outputs at least one of the acquired first identification information and second identification information to the classification unit 103.

[0028] 3A and 3B are diagrams showing identification information of a plurality of bones in a first medical image and a second medical image. FIG. 3(a) shows the first identification information in the first medical image, and FIG. 3(b) shows the second identification information in the second medical image. In this embodiment, the bone identification information represents a label image having a different label value for each bone. Here, BL1 to BL14 in FIGS. 3(a) and 3(b) respectively represent the location of each bone.

[0029] Specifically, BL1 indicates the skull, BL2 indicates the right upper humerus, BL3 indicates the left upper humerus, BL4 indicates the right clavicle, and BL5 indicates the left clavicle. BL6 indicates the right scapula, BL7 indicates the left scapula, BL8 indicates the right rib, and BL9 indicates the left rib. BL10 indicates the spine, BL11 indicates the ilium, BL12 indicates the sacrum, BL13 indicates the right femur, and BL14 indicates the left femur. The label image in which different label values ​​are assigned to the regions of these bone sites on the medical image is the bone identification information.

[0030] In this embodiment, the same identification information is set for bones belonging to the same part of the subject. For example, each vertebra in a spine consisting of multiple vertebrae is considered to belong to the same spine, and a label value is assigned to each vertebra as a part of the same part, such as the label BL10 indicating the spine in FIG. 3.

[0031] Similarly, each rib on the right side of the subject (the left side of the paper in FIG. 3) is assigned the label BL8 as a bone constituting the right rib and is assigned the label BL9 as a bone constituting the left rib and is assigned the label BL9 as a bone constituting the left rib. Note that the classification shown above is merely an example, and classification may be performed in different divisions such as cervical vertebrae, thoracic vertebrae, and lumbar vertebrae instead of the spine. Also, bones belonging to the same region do not necessarily need to be given the same label, and bones may be identified in units that can be considered to be roughly rigid bodies, such as by assigning different labels to each individual vertebra.

[0032] Here, a known method can be used to identify bones on an image. For example, bones may be identified by a known area extraction method using machine learning. In this case, a trained inference model that has been trained with the label areas of bones from a large number of cases may be constructed in advance, and the target medical image may be input to the inference model to extract the label area of ​​the bone. A known area extraction method using machine learning may be, for example, a method such as a convolutional neural network (CNN). Note that the area extraction method using machine learning is not limited to the convolutional network (CNN) and may be other known techniques. For example, a method may be used in which the entire bone area is extracted from the target medical image using a known extraction method, and then a statistical shape model of the bone having identification information for each part of the bone is applied to the bone area extracted from the medical image. In addition, there are a deep neural network (DNN), a recurrent neural network (RNN), a generative adversarial network (GAN), and the like, and any of these methods may be applied.

[0033] The bone identification information may be acquired by any other method. For example, the identification information acquiring unit 102 may acquire the bone identification information stored in the data server 130 in advance in association with the first medical image and the second medical image, without performing the bone identification process.

[0034] (S1030: Classifying the first and second bones into groups of bones) In step S1030, the classification unit 103 classifies the first and second bones into a group of bones that move in association with the movement of each body part that performs a different movement based on the first identification information and the second identification information. Based on at least one of the first and second identification information, the classification unit 103 classifies the first and second bones into a first bone group including bones that move in association with the movement of a first part (body part) of the subject and a second bone group including bones that move in association with the movement of a second part (body part) different from the first part (body part). In addition, the classification unit 103 classifies the first and second bones into a third bone group including bones that move in association with the movement of a third part (body part) different from the first and second parts (body parts) using at least one of the identification information. Then, the classification unit 103 outputs information on the classified groups of the multiple bones to the alignment unit 104. For example, the first part is the trunk of the subject, the second part is any one of the limbs of the subject, and the third part is any one of the limbs of the subject different from the second part. The second part and the third part are physically continuous with the first part, and the second part and the third part are parts that can operate independently of each other.

[0035] Here, body parts that perform different movements refer to body parts that move and rotate differently when the subject moves his or her body. For example, like the relationship between the trunk and the limbs, the movement of the trunk is mainly caused by bending the spine back and forth and left and right, whereas the movement of the limbs is caused by rotating the arms and legs in various directions around the axis of the position where they are connected to the trunk. Since the movement and rotation of these body parts are different, they are referred to here as body parts that perform different movements.

[0036] On the other hand, body parts that perform the same movement are those that have the same relationship in the way they move and rotate when the subject moves his or her body. For example, body parts that perform the same movement are those that perform the same movement, such as the relationship between the chest and abdomen, where the movements of the chest and abdomen are caused by bending the spine back and forth and side to side.

[0037] In addition, bones that move in conjunction with the movement of a body part refer to bones that move with the movement of the body part when it is moved. For example, the bones that move in conjunction with the movement of the right arm among the limbs are mainly the arm bones and the scapula. Just as the scapula is defined as a bone that moves in conjunction with the movement of the arm, if a bone is included in a body part but its movement is mainly accompanied by the movement of another body part, it is defined as a bone that moves in conjunction with the movement of another body part.

[0038] FIG. 4 is a diagram showing a first group of bones in a first medical image and a second group of bones in a second medical image. FIG. 4(a) shows a group of bones that move in association with the movement of the torso on the first medical image. FIG. 4(b) shows a group of bones that move in association with the movement of the torso on the second medical image. FIG. 4(c) shows four groups of bones that move in association with the movement of the limbs on the first medical image. FIG. 4(d) shows four groups of bones that move in association with the movement of the limbs on the second medical image. BL1 to BL14 in FIG. 4 represent the locations of each bone, similar to FIG. 3.

[0039] In this embodiment, the first and second bones are classified into a group G1 (e.g., the first bone group) of bones that move in association with the movement of the trunk, which are different body parts, and groups G2, G3, G4, and G5 (e.g., the second bone group) of bones that move in association with the movement of each of the limbs. However, the method of classifying the bones is not limited to this, and other classification results may be obtained. For example, the right clavicle BL4 may be classified into the group G2 of bones of the right arm, and the left clavicle BL5 may be classified into the group G3 of bones of the left arm.

[0040] At this time, as a result of the classification, the classification unit 103 adopts a group of bones that exist in common in the first medical image and the second medical image as a final classification result, and does not adopt a group of bones that exist only in one of the images as a final classification result. The classification unit 103 identifies bones that are commonly depicted between the first medical image and the second medical image using at least one of the first identification information on each of the first plurality of bones and the second identification information on each of the second plurality of bones depicted in the second medical image. Then, the classification unit 103 adopts a common group of bones in which a commonly depicted bone exists as a final bone group (first bone group, second bone group). The classification unit 103 classifies the first plurality of bones into the first bone group, and classifies the second plurality of bones into the second bone group.

[0041] In the example of FIG. 4, bone group G1 includes the skull BL1, right clavicle BL4, left clavicle BL5, sternum BL6, right rib BL8, left rib BL9, spine BL10, ilium BL11, and sacrum BL12, which move in conjunction with the movement of the torso.

[0042] The bone group G2 includes the right upper humerus BL2 and the right scapula BL6, which move in association with the movement of the right arm. The bone group G3 includes the left upper humerus BL3 and the left scapula BL7, which move in association with the movement of the left arm. The bone group G4 includes the right femur BL13, which moves in association with the movement of the right leg. The bone group G5 includes the left femur BL14, which moves in association with the movement of the left leg. Therefore, the entire trunk region including a part of the head to a part of the foot is common between the first medical image and the second medical image. In other words, since all of the bone groups G1 to G5 are common, the classification unit 103 adopts the bone groups G1 to G5 as the final classification result as they are.

[0043] On the other hand, for example, if the imaging range of the first medical image is the entire torso region as described above, and the imaging range of the second medical image is only the upper body, the first plurality of bones will be five groups G1 to G5 on the first medical image, but the second plurality of bones will only be three groups G1, G2, and G3 on the second medical image. That is, since there are only three common groups G1, G2, and G3, the classification unit 103 adopts these three groups G1, G2, and G3 as the final classification results, and does not adopt G4 and G5, which are only present in the first plurality of bones, as the final classification results.

[0044] In addition, when the imaging range of the first medical image is the entire torso region and the imaging range of the second medical image is only the lower body, the first plurality of bones will have five groups G1 to G5 on the first medical image, but the second plurality of bones will have only three groups G1, G4, and G5 on the second medical image. That is, since the first and second plurality of bones have only three groups G1, G4, and G5 in common, the classification unit 103 adopts these three groups G1, G4, and G5 as the final classification results, and does not adopt G2 and G3, which are only present in the first plurality of bones, as the final classification results.

[0045] Therefore, the common group in which the bones depicted in common between the first and second medical images exist differs depending on the degree of overlap of the imaging range between the first and second medical images. In this manner, in this embodiment, among the multiple bones identified in step S1030, only the groups common between the first and second medical images are classified, so that all bones depicted in each medical image are not necessarily classified.

[0046] However, the method of classifying bone groups in this embodiment is not limited to the above classification method. For example, bones with the same label may be associated in advance between the first and second bones, and only the associated pairs of bones may be classified into each bone group. In this case, for example, when the imaging range of the first medical image is the entire torso region and the imaging range of the second medical image is only the upper body, the following differences occur.

[0047] That is, in the case of the above-mentioned group-based correspondence, the bones BL1, BL4, BL5, BL6, BL8, BL9, BL10, BL11, and BL12 on the first medical image are classified into group G1 of bones associated with trunk movement.

[0048] Furthermore, since the upper body does not include the ilium BL11 and the sacrum BL12 on the second medical image, only the bones BL1, BL4, BL5, BL6, BL8, BL9, and BL10 are classified.

[0049] On the other hand, when matching bones with the same label, only BL1, BL4, BL5, BL6, BL8, BL9, and BL10, which are common to both the first and second medical images, are classified into bone group G1.

[0050] In this embodiment, the first and second bones are classified into a first group of bones associated with the movement of the trunk and a second group of bones to a fifth group of bones associated with the movement of the limbs, but the classification is not limited to this. For example, if there is little difference between the movements of the trunk and the limbs among multiple medical images, it is not necessary to distinguish between bones associated with the movement of the limbs and bones associated with the movement of the trunk.

[0051] For example, a group G1 of bones associated with the movement of the torso is roughly aligned between multiple medical images using a known method, and the images are aligned with respect to the group G1. After that, if there is no significant difference in position and orientation between multiple medical images in any of groups G2 to G5, that group may be included in group G1.

[0052] In this case, a method for roughly aligning group G1 includes a method of sampling the surface points of the bones included in each group and then performing rigid body alignment using a method such as ICP (Iterative Closest Points). In addition, the average distance of the surface points of the bones included in each group can be used to calculate the difference in position and orientation between the images in groups G2 to G5. This makes it possible to avoid classifying the bones into different groups even though the movements between the torso and the limbs do not change between the multiple medical images.

[0053] (S1040: Aligning between a first plurality of bones and a second plurality of bones for each group of bones) In step S1040, the registration unit 104 performs registration between the first plurality of bones and the second plurality of bones for each group of bones, and outputs the generated displacement field to the image generation unit 105.

[0054] In this embodiment, the registration unit 104 extracts images of bone regions belonging to each bone group from each of the first and second medical images based on the information on the bone groups acquired from the classification unit 103. Specifically, when the first medical image is I1 and the second medical image is I2, the registration unit 104 extracts images I1 of bone regions belonging to the bone group G1 associated with the movement of the trunk from each of the first medical image I1 and the second medical image I2. G1 , I2 G1 The registration unit 104 also obtains (cuts out) an image I1 of a bone region belonging to groups G2 to G5 of bones associated with the movement of the limbs from each of the first medical image I1 and the second medical image I2. G2 ~I1 G5 , I2 G2 ~I2 G5 Obtain (extract).

[0055] At this time, image I1 G1 It is not necessary to extract the bone region belonging to the bone group G1 from the first medical image. For example, an image in which pixel values ​​of bone regions belonging to the bone groups G2 to G5 associated with the movement of the limbs are filled with a predetermined pixel value (for example, a pixel value close to the surrounding tissue outside the bone) is taken as image I1. G1 It is also possible to obtain the image I2 G1 can also be obtained in a similar manner.

[0056] Then, the positioning unit 104 aligns the image I1 cut out from the first medical image I1 G1 ~I1 G5 and an image I2 cut out from the second medical image I2 G1 ~I2 G5 The position adjustment unit 104 performs position adjustment between the image I2, which is a floating image in this embodiment, for each corresponding group. G1 ~I2 G5 Transform the image I1, which is the reference image. G1 ~I1 G5 Displacement field D to approximately match G1 ~D G5 Generate.

[0057] For example, image I1 G1 and image I2 G1 In the alignment, the alignment unit 104 calculates a displacement amount between a position in a region corresponding to the first bone group in the first medical image and a position in a region corresponding to the first bone group in the second medical image as a first displacement field D G1 Then, the registration unit 104 acquires an image I1 of a bone region belonging to the first bone group in the first medical image. G1 and an image I2 of a bone region belonging to the first bone group in the second medical image. G1 The first displacement field D to match G1 A first alignment is performed based on the above.

[0058] Also, image I1 G2 ~I1 G5 and image I2 G2 ~I2 G5 In the alignment, the alignment unit 104 calculates a displacement amount between a position in a region corresponding to the second bone group in the first medical image and a position in a region corresponding to the second bone group in the second medical image as a second displacement field D G2 ~D G5 Then, the registration unit 104 acquires an image I1 of a bone region belonging to the second bone group in the first medical image. G2 ~I1 G5 and an image I2 of a bone region belonging to a second bone group in the second medical image. G2 ~I2 G5 A second displacement field D to match G2 ~D G5 A second alignment is performed based on the above.

[0059] The registration unit 104 can perform registration by a known image processing method that uniformly evaluates the entire image. For example, registration can be performed by deforming one of multiple medical images to obtain a displacement field so as to increase the image similarity between the images.

[0060] As the image similarity, a commonly used known method such as the Sum of Squared Difference (SSD), mutual information, cross-correlation coefficient, etc., can be used. As the image deformation model, a known deformation model such as affine transformation, Free Form Deformation (FFD), Demons algorithm, Large Deformation Diffeomorphic Metric Mapping (LDDMM), etc. can be used.

[0061] In this way, by performing independent registration for each group of bones associated with the movements of body parts performing different actions, the overall registration can be decomposed into registrations between groups of bones that move in a similar way, and we do not need to directly deal with discontinuous deformations that may occur at the boundaries between the groups of bones.

[0062] More specifically, by classifying multiple bones as group G2 of bones of the right arm, separately from group G1 of bones of the torso to which they are originally physically connected, it is possible to perform alignment focusing on the right arm, which moves differently from the torso, and to prevent a decrease in alignment accuracy.

[0063] In addition, by independently aligning the ribs belonging to the torso bone group G1 and the scapula belonging to the right arm bone group G2, it is possible to avoid directly dealing with discontinuous deformations that may occur between the two, thereby preventing a decrease in alignment accuracy due to this.

[0064] In other words, it is possible to prevent a decrease in alignment accuracy when the movements of multiple bones belonging to different body parts differ significantly between multiple medical images, or when discontinuous deformation occurs at the boundaries between the bones.

[0065] Furthermore, by classifying a plurality of bones as a group G3 of bones of the left arm, separately from the group G1 of bones of the torso to which they are originally physically connected, it is possible to perform alignment focusing on the left arm, which moves differently from the torso. It is possible to perform alignment focusing on each of the right arm and the left arm, which can move independently of each other with respect to the torso, and independently of each other.

[0066] In addition, by performing registration between multiple medical images by associating groups of bones, even if an abnormality exists in part of one bone group, the first multiple bones and the second multiple bones are registered so that the entire multiple bones included in the group match between the multiple medical images. Therefore, since registration is performed so that bones with normal structures around the abnormality coincide, stable registration is possible even if an abnormality exists.

[0067] More specifically, even if some of the vertebrae of the spine in the second medical image are crushed due to a fracture or the like, the multiple bones associated with the movement of the torso, including the spine, are aligned between the multiple medical images. Therefore, normal vertebrae and ribs surrounding the vertebrae crushed by the fracture are aligned to match those in the first medical image. This makes it possible to suppress a decrease in the accuracy of the overall alignment caused by local differences in individual bones between the first and second medical images.

[0068] (S1050: Integrate the alignment results to generate the result image) In step S1050, the image generating unit 105 generates an image (result image) that integrates the alignment results based on the displacement field acquired by the processing of step S1040. That is, the image generating unit 105 generates a deformed image by deforming the second medical image so as to match it with the first medical image based on the displacement field acquired by the alignment unit 104. The image generating unit 105 also generates a difference image by subtracting the deformed image from the first medical image. Then, the image generating unit 105 outputs the generated deformed image and difference image to the display control unit 106 as a result image.

[0069] In this embodiment, the image generating unit 105 generates the plurality of acquired displacement fields D G1 ~D G5 The integrated displacement field D total A more specific method is described below. G1 ~D G5 are stored displacements from each position in the region corresponding to the bone groups G1 to G5 on the first medical image to each position in the region of the corresponding bone groups G1 to G5 on the second medical image. Therefore, the displacement fields D G1 ~D G5 Each of the displacement fields D G1 ~D G5 Since each of the data stores only a partial displacement amount, it is necessary to integrate these data in order to displace the positions of the entire regions of a plurality of bones in an image.

[0070] Therefore, the image generating unit 105 first sets an empty displacement field region in which the displacement amount at each position having the same size as the first medical image is all 0. Then, the image generating unit 105 generates a plurality of displacement fields D G1 ~D G5 By storing each of the displacement amounts in the corresponding groups G1 to G5 in the empty displacement field, the integrated displacement field D total Generate.

[0071] For example, the image generator 105 generates a first displacement field D G1 The displacement amount of is stored in the region of the first bone group among the set regions, and the second displacement field D G2 ~D G5 The displacement amount is stored in the region of the second bone group among the set regions. Then, the image generating unit 105 generates different displacement fields D G1 ~D G5 The integrated displacement field D total Generate.

[0072] Then, the image generating unit 105 calculates the integrated displacement field Dtotal A deformed image is generated by deforming the second medical image using the above, and a difference image is generated by subtracting the deformed image from the first medical image. Note that if the purpose is to output a difference image (the deformed image is not necessary), the deformed image does not necessarily have to be generated as an image. For example, the integrated deformation field may be used to calculate pixel information (e.g., pixel value) of each pixel of the deformed image (but without saving it as an image), and the difference value between the pixel information of the corresponding pixel of the first medical image and the pixel information of each pixel of the deformed image may be calculated to generate only the difference image.

[0073] Alternatively, the image generating unit 105 may generate a plurality of acquired displacement fields D G1 ~D G5 A medical image I2 cut out from the second medical image generated in step S1040 using G1 ~I2 G5 Deformed image DI2 G1 ~DI2 G5 These deformed images DI2 G1 ~DI2 G5 is a medical image I1 cut out from the first medical image G1 ~I1 G5 Therefore, the deformed image DI2 G1 ~DI2 G5 Each of these stores pixel information (e.g., pixel values) for each position within an area on the second medical image that corresponds to bone groups G1 to G5 on the first medical image, using the first medical image as a reference.

[0074] In addition, these deformed images DI2 G1 ~DI2 G5 Since each of the first and second medical images contains only partial pixel information, it is necessary to integrate these images in order to generate a deformed image of the entire region of multiple bones in the image. Therefore, the image generating unit 105 first generates a blank image in which the pixel values ​​of all the pixels having the same size as the first medical image are set to 0. The image generating unit 105 then generates a deformed image DI2 G1 ~DI2 G5By storing each pixel information of the blank image, the integrated deformed image DI2 in which pixel information is stored at different positions is obtained. total Then, the image generating unit 105 generates an integrated deformed image DI2 from the first medical image I1. total A difference image is generated by subtracting

[0075] Alternatively, the image generating unit 105 may generate a plurality of displacement fields D G1 ~D G5 Using this, the deformed image DI2 G1 ~DI2 G5 After generating the medical image I1 cut out from the first medical image, G1 ~I1 G5 Deformed image DI2 from G1 ~DI2 G5 The difference image SI obtained by subtracting G1 ~SI G5 Generate the difference image SI G1 ~SI G5 The area in which the difference values ​​are stored is medical image I1 G1 ~I1 G5 Therefore, the difference image SI G1 ~SI G5 Each of these stores a difference value between each position in the area corresponding to the bone groups G1 to G5 on the first medical image and each position in the corresponding area on the second medical image, with the first medical image being used as a reference. G1 ~SI G5 Since each of the first and second medical images is data in which only partial difference values ​​are stored, it is necessary to integrate these images in order to generate a difference image of the entire region of multiple bones in the image. Therefore, the image generating unit 105 first generates a blank image in which the values ​​of all pixels having the same size as the first medical image are set to 0. The image generating unit 105 then generates the difference image SI G1 ~SI G5 By storing the difference values ​​of the above in a blank image, an integrated difference image SI is generated in which the difference values ​​are stored in different positions.

[0076] Alternatively, the image generating unit 105 may generate the first displacement field D G1A deformed image DI21 obtained by deforming the second medical image using the displacement amount of the second medical image, and a second displacement field D G2 ~D G5 The image generating unit 105 generates deformed images DI22 to DI25 by deforming the second medical image using the displacement amount. Then, the image generating unit 105 generates difference images SI1 to SI5 by subtracting the deformed images DI21 and the deformed images DI22 to DI25, respectively, from the first medical image. Each of the difference images SI1 to SI5 stores a difference value between the corresponding positions on the second medical image and not only the bone regions of each bone group but also all positions on the first medical image as a result of each alignment.

[0077] The image generating unit 105 first generates a blank image in which the values ​​of all pixels having the same size as the first medical image are set to 0. Then, at each position in the blank image region corresponding to the bone groups G1 to G5 on the first medical image, the image generating unit 105 stores the difference values ​​in the difference images of the target bone groups among the difference images SI1 to SI5 to generate an integrated difference image SI. For example, at a position in the blank image region corresponding to the bone group G2 on the first medical image, the image generating unit 105 generates a displacement field D G2 The difference value of the difference image SI2 generated based on the amount of displacement is stored.

[0078] Also, for any one of the difference images SI1 to SI5, the difference value may be stored at each of all other positions of the sky image that do not correspond to the bone group regions to generate an integrated difference image SI. For example, the difference values ​​of each of the difference images SI2 to SI5 may be stored at each position in the sky image regions that correspond to the bone groups G2 to G5 on the first medical image, and the difference value of the difference image SI1 may be stored at each of all positions of the sky image other than the regions that correspond to the bone groups G2 to G5 on the first medical image to generate an integrated difference image SI.

[0079] That is, the difference value of one of the difference images SI1 to SI5 is stored as the difference value at each position on the integrated difference image SI. This allows the difference values ​​to be stored without missing any of the bone regions on the integrated difference image SI even if there is a bone region that is not classified into any of the bone groups G1 to G5 due to insufficient extraction of the bone region.

[0080] However, when storing the difference value of any one of the difference images SI1 to SI5 at each position on the integrated difference image SI, inconsistency may occur in the alignment results for each bone group near the boundary between the bone groups on the integrated difference image SI. For example, in a region other than the bone region between the right femur and the ilium on the first medical image, a part of the region of the right femur that is not included in the bone group G1 on the deformed image DI21 and a part of the region of the ilium that is not included in the bone group G4 on the deformed image DI24 may overlap in the image space. In this overlapping region, since both the difference images SI1 and SI4 generated from the respective deformed images are differences between the region other than the bone region and the bone region on the first medical image, when one of the difference values ​​is set as the difference value of the integrated difference image SI, a difference value may occur between the right femur and the ilium regardless of which difference value is stored.

[0081] Even in cases where overlapping of regions occurs in the image space, in this embodiment, in the deformed images DI21-DI25, regions that do not belong to each of the bone groups G1-G5 are set as invalid regions, and when invalid regions of multiple deformed images overlap, the alignment result is determined to be invalid and a predetermined difference value (for example, 0) is stored in a blank image to generate an integrated difference image SI. This makes it possible to eliminate inconsistencies in the alignment result that may occur in the integrated difference image SI.

[0082] In this embodiment, the image generating unit 105 stores the generated resultant image (deformed image, difference image) in a storage unit (not shown). As a result, when it is desired to obtain the resultant image again after the processing of the image processing device 100 is completed, the image generating unit 105 can easily obtain the resultant image by reading the resultant image stored in the storage unit. However, it is not necessary to store the generated resultant image in a storage unit (not shown).

[0083] Moreover, it is not necessary to generate both the deformed image and the difference image of the second medical image as the resultant image. Only one of the deformed image and the difference image of the second medical image may be generated. Alternatively, instead of generating the deformed image or the difference image, the integrated deformation field may be stored in a storage unit (not shown) as information for generating them. The generation of the deformed image and the difference image from the integrated deformation field may be performed by another device. (S1060: Show image) In step S1060, the display control unit 106 controls the display unit 150 to display the cross-sectional image of the resultant image acquired from the image generating unit 105. The display control unit 106 also controls the display unit 150 to display the cross-sectional images of the first medical image and the second medical image. The display unit 150 is composed of any device such as an LCD or a CRT, and displays medical images and the like for a doctor to interpret. The display unit 150 has a GUI for acquiring instructions from a user such as a doctor. The user can freely switch the first medical image and the second medical image being displayed to a difference image or a deformation image by using the GUI. The display control unit 106 can also control the display unit 150 to combine the images and display them in combination based on an instruction from the GUI.

[0084] It should be noted that when the resultant image is to be stored or output externally, it is not necessarily required to display the resultant image. Furthermore, when the purpose is image processing using the resultant image, it is not necessary to store or display the resultant image. For example, it may be configured to detect areas where pixel information (e.g., pixel values) in the difference image is large, i.e., areas where there is a large change in pixel information between medical images, and output them as abnormal area candidates, or to input the resultant image (deformed image, difference image) to a classifier to determine the presence or absence of an abnormality. The processing of the image processing device 100 is performed as described above.

[0085] According to this embodiment, the following effects can be obtained by independently aligning each group of bones associated with the movements of body parts performing different actions. That is, when the movements of multiple bones belonging to different body parts between images are significantly different, or when discontinuous deformation occurs at the boundary between the bones, a decrease in alignment accuracy can be suppressed. Furthermore, by aligning images by corresponding each group of bones, stable alignment can be achieved even when an abnormal part exists in a part of one bone group. That is, according to this embodiment, accurate and stable alignment between images can be achieved.

[0086] (Variation 1) In the first embodiment, a configuration has been described in which, in step S1020, multiple bones are identified from each of the first medical image and the second medical image, and then, in step S1030, the bones are classified into multiple bone groups.

[0087] However, the identification of a plurality of bones and the classification of the plurality of bones into groups do not necessarily have to be performed in this manner. For example, in step S1020, bones may be identified in the same units as the bone groups classified in step S1030. More specifically, the identification information acquisition unit 102 may identify BL1, BL4, BL5, BL8, BL9, BL10, BL11, and BL12 included in the bone group G1 in FIG. 4 as identification information having the same label value. In addition, the identification information acquisition unit 102 may identify BL2 and BL6 included in the bone group G2, BL3 and BL7 included in the bone group G3, BL13 included in the bone group G4, and BL14 included in the bone group G5 as identification information having the same label value. In this way, by first identifying bones in units of bone groups associated with the movements of body parts performing different actions, the classification process in step S1030 can be skipped.

[0088] <Second embodiment> In the first embodiment, a configuration was described in which multiple bones are identified from a first medical image and a second medical image, the bones are classified into multiple bone groups, and the bones are aligned for each group. In the present embodiment, a configuration will be described in which multiple bones are identified from one of the first medical image and the second medical image, and the bones are classified into multiple bone groups.

[0089] The identification information acquisition unit 102 acquires identification information of a plurality of bones identified from either the first medical image or the second medical image. Then, the classification unit 103 classifies the plurality of bones identified from one of the images into groups of bones that move in association with different movements for each part of the subject performing different movements based on the identification information.

[0090] The registration unit 104 then performs initial registration of one medical image with the other medical image, and then obtains information on the groups of bones on the multiple medical images by associating the groups of bones on one medical image with the bone regions on the other medical image. Finally, registration is performed for each group of bones associated in both images.

[0091] Fig. 5 is a diagram showing the configuration of an image processing system 20 according to the second embodiment. An image processing device 200 according to the second embodiment has a correspondence unit 201. The configuration other than the correspondence unit 201 is the same as that in Fig. 1, and therefore description thereof will be omitted. The correspondence unit 201 performs processing to correspond information of a group of multiple bones set in one of two input medical images, a first medical image and a second medical image, to the other image.

[0092] The processing of this embodiment will be described below with reference to the flowchart in Fig. 6. Note that steps S2010 and S2060 to S2080 in the flowchart showing the overall processing procedure performed by image processing device 200 are similar to steps S1010 and S1040 to S1060 in the first embodiment, and therefore will not be described.

[0093] Hereinafter, only the parts in the flowchart of FIG. 6 that are different from the first embodiment will be described.

[0094] (S2020: Identifying multiple bones from only the first medical image) In step S2020, the identification information acquisition unit 102 identifies multiple bones from either the first medical image or the second medical image. Then, the identification information acquisition unit 102 outputs information on the identified multiple bones as identification information to the classification unit 103. In this embodiment, for example, the identification information acquisition unit 102 identifies only the first multiple bones depicted in the first medical image and acquires only the first identification information that is the identification information thereof. Here, the process of identifying multiple bones from one medical image is the same as that in step S1020, and therefore a description thereof will be omitted.

[0095] (S2030: Classify only the first bones into a group of bones) In step S2030, the classification unit 103 classifies the multiple bones identified from the first medical image and the second medical image processed in step S2020 based on the identification information into groups of multiple bones that move in association with each of the subject's body parts performing different movements.

[0096] Then, information on the classified groups of bones is output to the association unit 201. In this embodiment, the first bones are classified into groups of bones based on the first identification information. Here, the process of classifying the bones into groups of bones is the same as that in step S1030, and therefore a description thereof will be omitted.

[0097] (S2040: Initial alignment of the first medical image and the second medical image) In step S2040, the alignment unit 104 performs initial alignment between the first medical image and the second medical image, and generates a displacement field that deforms the second medical image into the first medical image. Then, the alignment unit 104 outputs the generated displacement field to the image generation unit 105. Next, the image generation unit 105 generates a deformed image by deforming the second medical image based on the displacement field acquired from the alignment unit 104. Then, the image generation unit 105 outputs the generated deformed image to the association unit 201.

[0098] Here, the initial alignment can be performed by a known image processing method that uniformly evaluates the entire image as shown in step S1040. However, in this embodiment, unlike step S1040, since the second medical image does not have bone group information, the alignment unit 104 does not generate an image in which the bone region corresponding to each bone group is cut out. Therefore, when performing alignment, the alignment unit 104 performs processing such as extracting the bone region from the first medical image and the second medical image in advance so that the bone regions between the first medical image and the second medical image approximately match, and then performs alignment so that the bone regions between the first medical image and the second medical image approximately match. For example, in a CT image, the bone region is depicted with a higher brightness than other organ regions, so that the high brightness region can be extracted as the bone region by threshold processing. In addition, as a method for aligning bone regions so that they approximately coincide with each other, a method for aligning so as to minimize the distance between surface point groups of the bones, or a method for generating a distance field image from the contour of the bone region and aligning the distance field images can be applied.

[0099] (S2050: Associating a group of bones in a first medical image with a deformed image) In step S2050, the association unit 201 associates information of a group of bones set in one of the first and second medical images with a deformed image of the other medical image. The association unit 201 associates a region of a group of bones in one medical image, classified based on the identification information of the one medical image, with a region of the other medical image. In this embodiment, the association unit 201 will exemplarily explain a process of associating information of a group of bones on the first medical image with a bone region on the deformed image of the second medical image. A more specific method will be explained below.

[0100] The matching unit 201 generates a deformed image by deforming the other medical image using the displacement field generated in the registration, and matches each of the regions of the multiple bone groups in one medical image to the corresponding region in the deformed image. First, the matching unit 201 identifies regions on the deformed image that correspond to each of the regions of the multiple bone groups on the first medical image, and calculates the difference in image feature information between each region. Specifically, the matching unit 201 matches each of the regions of the multiple bone groups G1 to G5 on the first medical image in FIG. 4 with region R1. G1 ~R1 G5 Then, the association unit 201 assigns the region R1 G1 ~R1 G5 Region R2 on the deformed image corresponding to G1 ~R2 G5 Identify the region R1 G1 ~R1 G5 and region R2 G1 ~R2 G5 The correspondence between can be determined using the displacement field generated as a result of the initial registration performed in step S2040.

[0101] Next, the association unit 201 associates the region R1 G1 ~R1 G5 and region R2 G1 ~R2 G5 The difference in feature information between the regions, S G1 ~S G5In this case, the difference in feature information between images can be, for example, SSD (Sum of Squared Difference), which is used for the similarity of alignment. However, the difference in feature information between regions is not limited to this, and a known method such as SAD (Sum of Absolute Difference) can be used.

[0102] The matching unit 201 matches the regions of a group of multiple bones in one medical image as bone regions in the deformed image using the difference in feature information. The matching unit 201 matches the regions of a group of multiple bones in one medical image (first medical image) to bone regions in the deformed image in order of bone groups in an area with the smallest difference in feature information between the images. For example, the matching unit 201 matches the regions of a group of multiple bones in one medical image (first medical image) to bone regions in the deformed image in order of the smallest difference in feature information between the images. G1 G4 G5 G2 G3 If so, the association unit 201 first associates the bone group G1 on the first medical image with the bone region on the second medical image. Here, the association unit 201 extracts the bone region on the second medical image, for example, by the method described in S2040, and sets it as the bone region BR2. Specifically, in the bone region BR2, the association unit 201 associates the bone group G1 on the first medical image with the bone region R2 on the deformed image. G1 The area overlapping with the above is set as the area of ​​bone group G1. As a result, the area of ​​the bone group associated with the movement of the trunk is associated with the bone areas on the deformed image.

[0103] Next, the region R2, which already has a bone group region associated with it, G1 In the bone region BR2 excluding the bone region R2 on the deformed image, the matching unit 201 G4 ​​​​The area overlapping with the right femur is set as the area of ​​bone group G4. As a result, the areas of the bone groups associated with the movement of the right femur among the bone areas on the deformed image are associated. Similarly, the association unit 201 associates the bone groups G5, G2, and G3 in order of decreasing difference in feature information between the images, thereby making it possible to associate the bone groups in order of decreasing difference in feature information between the images, that is, in order of increasing reliability of association.

[0104] At this time, if the difference in feature information between the first medical image and the deformed image is greater than a predetermined value, the above matching process does not necessarily match all of the bone region BR2. For example, if there is a large difference in the arm posture between the images, the bone group G2 and the bone group G3 may not be aligned in the initial alignment. In such a case, the bone region BR2 may not be aligned in the region R2. G1 ~R2 G5 Among the regions in which the bone region BR2 has not yet been associated with the bone region BR2, the bone group of the region closest to the position of the region can be associated with the bone region BR2. In this manner, the processing of the image processing device 200 is performed.

[0105] According to this embodiment, by simply identifying multiple bones and classifying them into bone groups in either the first or second medical image, it is possible to match the bone groups to the other image and align each bone group.

[0106] <Third embodiment> In the first embodiment, a configuration has been described in which a plurality of bones are identified from a first medical image and a second medical image, the bones are classified into a plurality of bone groups, and the bone groups are aligned. In the present embodiment, a configuration will be described in which, after the alignment for each bone group, alignment is further performed for each individual bone between the first plurality of bones and the second plurality of bones.

[0107] The configuration of the image processing system according to this embodiment is the same as that shown in FIG. 1, and therefore a duplicated description will be omitted. The registration unit 104 associates bone regions identified as the same bone region between the first and second multiple bones based on the first and second identification information. The registration unit 104 aligns the first displacement field D G1 and the second displacement field D G2 ~D G5 as an initial value (initial displacement field), rigid registration is performed between the bone region in the first medical image and the bone region in the second medical image that are associated with each other. Then, the registration unit 104 performs deformation registration that allows the degree of freedom of deformation in the bone region to be set between the bone region in the first medical image and the bone region in the second medical image that have been rigidly registered.

[0108] The processing of this embodiment will be described below with reference to the flowchart in Fig. 7. Note that steps S3010 and S3050 to S3070 in the flowchart showing the overall processing procedure performed by the image processing device 100 are similar to steps S1010 and S1040 to S1060 in the first embodiment, and therefore will not be described.

[0109] Hereinafter, only the parts in the flowchart of FIG. 7 that are different from the first embodiment will be described.

[0110] (S3050: Aligning a first plurality of bones with a second plurality of bones for each bone) In step S3050, the registration unit 104 aligns the displacement field (e.g., D G1 ~D G5 ) as an initial value (referred to as an initial displacement field), alignment is performed between the first and second bones for each individual bone. Then, the alignment unit 104 outputs the generated displacement field (referred to as a final displacement field) to the image generation unit 105. The specific processing will be described below.

[0111] First, the registration unit 104 associates bone regions of the same label between the first and second bones based on the first and second identification information. At this time, the individual bone units to be associated are individually identified as units that can be regarded as roughly rigid bodies. For example, in the case of the spine, each of the vertebrae that constitutes the spine is identified by a different label. This allows the individual bones to be aligned in the following process by regarding them as roughly rigid bodies.

[0112] Secondly, the registration unit 104 performs rigid registration between the associated individual bones using the initial displacement field. For example, the registration unit 104 can perform rigid registration using the initial displacement field in the following manner. When the bone on the first medical image to be subjected to rigid registration is designated as BN1 and the bone on the second medical image is designated as BN2, the registration unit 104 first samples a point group P1 at a predetermined interval from within the region of BN1. Next, the registration unit 104 generates an initial displacement field D corresponding to the group of bones to which BN1 belongs. Init Using the above, the point group P1 is displaced to a position on the second medical image to generate a point group P2. The registration unit 104 then uses the point groups P1 and P2 as corresponding points to obtain a rigid transformation that approximates the displacement between the corresponding points using a known rigid registration method. This allows rigid registration between the first medical image and the second medical image based on the initial displacement field. The same process is performed for all bones that have been associated.

[0113] Thirdly, the registration unit 104 performs deformation registration between the associated individual bones using the rigid transformation calculated above as an initial value. Here, the registration unit 104 performs deformation registration using a known image processing method as the registration method, similar to step S3040. However, while the bone group unit registration is performed in step S3040, in this step, the object of registration is an individual bone region that can be regarded as a rigid body, so it is desirable to adopt deformation parameters with a smaller degree of freedom of deformation. By performing deformation registration using deformation parameters with a smaller degree of freedom of deformation, it is possible to prevent the bone region from being deformed more than necessary and stabilize the registration.

[0114] For example, in alignment using a FFD (Free Form Deformation) model, which is a common deformation method, a grid of control points is arranged on an image, and the positions of the control points are moved to displace an existing area or position in the image. If the interval at which the control points are arranged is large, the degree of freedom of deformation is small, and if the interval is small, the degree of freedom of deformation is large. In this embodiment, for example, the alignment unit 104 sets a parameter p that controls the interval at which the control points are arranged to p times (2≦p) more than in step S3040, thereby making it possible to perform deformation alignment with a small degree of freedom of deformation. Note that the setting of the parameter p is merely exemplary, and it is also possible to perform deformation alignment with a large degree of freedom of deformation.

[0115] Thus, in this step, the registration unit 104 first performs rigid registration between individual bones between images based on the initial displacement field for each bone group, and then performs deformation registration.

[0116] On the other hand, if the degree of freedom of deformation is set high in order to properly express the deformation of a bone group in the registration in step S3050, a bone region containing a lesion may be forcibly deformed.

[0117] By performing the alignment in this step, such deformation can be appropriately aligned as a whole bone in a bone group, and then aligned while maintaining the shape of the bone without forcibly deforming the bone region containing the lesion. In this manner, the processing of the image processing device 100 is performed.

[0118] According to this embodiment, after aligning groups of bones, further alignment is performed for each individual bone between the first and second plurality of bones, thereby enabling more accurate alignment of bones between images.

[0119] <Fourth embodiment> In the first embodiment, a configuration was described in which a plurality of bones are identified from a first medical image and a second medical image, classified into a plurality of bone groups, and alignment of each bone group is performed. In the present embodiment, a configuration will be described in which it is determined whether or not a group of bones of the limbs is likely to be classified correctly, and a group of bones of the limbs determined to be likely to be classified incorrectly is reclassified into a group with bones of the trunk and then aligned.

[0120] In this embodiment, the judgment of whether or not the limb bone group is likely to be classified correctly is based on the judgment of whether or not a predetermined implant such as an artificial bone is present in or near the limb bone group, or whether or not there is a bone recognition region in which the bone extraction reliability does not satisfy a predetermined threshold. A bone recognition region in which the bone extraction reliability does not satisfy a predetermined threshold may be called a low-reliability bone region or a bone erroneous extraction region.

[0121] A group of limb bones that is not classified correctly may not be able to properly separate the bones of the trunk and the bones of the limbs, which may result in a decrease in the performance of the registration. According to this embodiment, a group of limb bones that is determined to be highly likely to be classified incorrectly is registered without being separated from the bones of the trunk, thereby suppressing a decrease in the registration accuracy caused by the separation of the bones of the limbs.

[0122] Fig. 8 is a diagram showing the configuration of an image processing system 40 according to the fourth embodiment. An image processing device 400 according to the fourth embodiment has a determination unit 401 as a component. The configuration other than the determination unit 401 and the classification unit 103 is similar to the configuration of the image processing device 100 described in Fig. 1, and therefore description thereof will be omitted.

[0123] The determination unit 401 acquires determination information indicating whether or not the groups of bones of the limbs classified by the classification unit 103 are likely to be classified correctly. Then, the classification unit 103 reclassifies the groups of bones of the limbs into groups of bones of the trunk, based on the determination information acquired by the determination unit 401.

[0124] The determination unit 401 determines whether or not at least one of a transplant and a bone recognition region whose bone extraction reliability does not satisfy a predetermined threshold is included in the second bone group classified in at least one of the first medical image and the second medical image. When the result of the determination by the determination unit 401 indicates that at least one of a transplant and a bone recognition region whose bone extraction reliability does not satisfy a predetermined threshold is included, the classification unit 103 reclassifies the bones belonging to the second bone group into the first bone group.

[0125] The processing of this embodiment will be described below with reference to the flowchart in Fig. 9. Note that steps S4070 and S4080 in the flowchart showing the overall processing procedure performed by image processing device 400 are similar to steps S1050 and S1060 in the first embodiment, and therefore will not be described.

[0126] Hereinafter, only the parts in the flowchart of FIG. 9 that are different from the first embodiment will be described.

[0127] (S4010: Acquire the first medical image and the second medical image) In step S4010, the image acquisition unit 101 acquires a first medical image and a second medical image in the same manner as in the first embodiment. Then, the image acquisition unit 101 outputs the first medical image and the second medical image to the identification information acquisition unit 102, the determination unit 401, the position adjustment unit 104, and the display control unit 106.

[0128] (S4020: Identifying the first and second multiple bones from two images) In step S4020, the identification information acquisition unit 102 acquires at least one of the first identification information and the second identification information in the same manner as in the first embodiment. Then, the identification information acquisition unit 102 outputs at least one of the acquired first identification information and the second identification information to the classification unit 103 and the determination unit 401.

[0129] (S4030: Classifying a first plurality of bones and a second plurality of bones into a group of bones) In step S4030, the classification unit 103 classifies the first and second bones into a plurality of bone groups, as in the first embodiment. Then, the classification unit 103 outputs information on the classified bone groups to the determination unit 401 and the alignment unit 104.

[0130] (S4040: Determine whether a group of limb bones is likely to be classified correctly) In step S4040, the determination unit 401 acquires first determination information that determines whether or not it is highly likely that the second bone group in the first medical image is classified correctly based on the first medical image and information on the multiple bone groups. The determination unit 401 also acquires second determination information that determines whether or not it is highly likely that the second bone group in the second medical image is classified correctly based on the second medical image and information on the multiple bone groups. Then, the determination unit 401 outputs the acquired first determination information and second determination information to the classification unit 103.

[0131] In this embodiment, the first judgment information and the second judgment information are information indicating whether or not there is a high possibility that there is an abnormality in the information of the group of bones of the limbs. When a predetermined implant such as an artificial bone is present in the body of the subject, there is a possibility that the identification information acquired in step S4020 has not been acquired normally (a label different from that of the original bone is identified) due to a decrease in image quality around the implant due to the influence of metal artifacts, etc. In this embodiment, the judgment unit 401 judges the presence or absence of a predetermined implant in the group of bones of the limbs, and if there is a transplant, judges that there is a high possibility that there is an abnormality in the information of the group of bones of the limbs, and acquires judgment information indicating the presence of an abnormality.

[0132] An example of a specific processing method will be described below. The determination unit 401 determines whether or not a transplant is included in each of the bone groups G2 to G5 that move in association with the movement of each of the limbs in the first medical image, and obtains first determination information.

[0133] Specifically, when there are a predetermined volume or more of pixels with pixel values ​​equal to or greater than a predetermined threshold in a bone region on the first medical image identified as the limb bone groups G2 to G5 or in the surrounding region, the determination unit 401 determines that a transplant is present in the group. Since there is a possibility that the accuracy of the identification information will decrease if a transplant is present, it is preferable to make the determination using not only the pixels on the first medical image identified as the limb bone groups G2 to G5, but also the pixel values ​​of the surrounding pixels.

[0134] For example, when the right femur BL13 is classified into the bone group G4, the presence or absence of a transplant in the bone group G4 may be determined by using pixels in a rectangular area where the bounding box of the right femur BL13 overlaps with the bounding box of the adjacent ilium BL11. The presence or absence of a transplant may be determined in this manner by using only the vicinity of the boundary between the limb bone group G2 (or G3, G4, G5) and the first bone group G1 as the target area. Alternatively, the entire area including each of the limb bone groups G2 to G5 may be determined as the target area for determination. Here, the predetermined threshold value may be, for example, a CT value (for example, 6000 H.U.) corresponding to a metal artificial bone, or may be a CT value corresponding to another transplant such as a pacemaker present near the limb bone. The predetermined volume may be a value corresponding to the volume of a predetermined general transplant (for example, an artificial femur), or may be a predetermined constant (for example, one pixel).

[0135] For example, when the bone groups G2 and G3 in the first medical image are determined to have no graft, the determination unit 401 obtains determination information indicating no abnormality for the bone groups G2 and G3 as the first determination information. When the bone groups G4 and G5 in the first medical image are determined to have a graft, the determination unit 401 obtains determination information indicating abnormality for the bone groups G4 and G5, and obtains the determination information for each bone group as the first determination information.

[0136] In this embodiment, the process of determining the presence or absence of a transplant is not limited to the process using a pixel value threshold. The presence or absence of a transplant may be determined by a known method. For example, the transplant may be identified by a known area extraction method using machine learning. In this case, a trained inference model that has learned the label areas of transplants in many cases in advance may be constructed, and the target medical image may be input to the inference model to extract the label area of ​​the transplant. For example, a method such as a convolutional network (Convolutional Neural Network (CNN)) may be cited as a known area extraction method using machine learning. Note that the inference model may not necessarily extract the area of ​​the transplant, but may infer the presence or absence of the transplant within a predetermined range on the image. In addition, it is not necessarily required to analyze the first medical image and the second medical image to determine the presence or absence of the transplant, and information regarding the presence or absence of the transplant in the first medical image and the second medical image (for example, information on the surgical history related to the artificial joint included in the medical record information, etc.) may be obtained from the data server 130.

[0137] The determination unit 401 may acquire information other than the above-mentioned information on the presence or absence of a transplant as the first determination information. For example, the determination unit 401 may determine the presence or absence and the degree of misidentification in the first identification information acquired by the identification information acquisition unit 102, and acquire this as the first determination information. Specifically, for example, the degree of discrepancy between the shape of the label region of the bone of the limb acquired as the first identification information and the statistical shape model of the bone may be measured, and it may be determined that there is an abnormality in the first identification information when the degree of discrepancy is greater than a predetermined threshold. The degree of discrepancy may be determined by acquiring the shape of the label region of the bone of the limb acquired as the first identification information and the statistical shape model of the bone aligned with the shape as binary images, and using a known index indicating the degree of agreement between the two regions, such as the DICE coefficient. That is, when the calculated degree of agreement is equal to or less than a threshold, it is possible to determine that there is a large degree of discrepancy and an abnormality (misidentification).

[0138] The above-mentioned deviation degree can be discussed by replacing it with the extraction reliability obtained by normalizing the deviation degree and taking the difference from 1. The determination unit 401 may determine that the first identification information has an abnormality based on a determination result that the extraction reliability does not satisfy (is less than) a predetermined threshold value for the extraction reliability.

[0139] In addition, the determination unit 401 may be configured to determine that there is an abnormality in the first identification information based on whether or not at least one of a first determination result that a transplant is included in the second bone group in at least one of the first image and the second image, and a second determination result that a bone recognition region that does not have a predetermined extraction reliability is included in the second bone group.

[0140] Alternatively, the determination information may be whether or not an anatomically abnormal shape has occurred, such as when a cavity exists in the label area of ​​a bone of a limb acquired as the first identification information, or when one bone is separated into two label areas. Also, the positional relationship of the label areas of each bone may be used to determine whether or not there is an abnormality. For example, when the center of gravity of the label area of ​​the right femur BL13 is located on the cranial side of the center of gravity of the label area of ​​the spine BL10, it can be determined that there is an abnormality in the information of the bone group to which the right femur belongs.

[0141] The second determination information, which is the determination information regarding the second bone group in the second medical image, can also be acquired in a similar manner to the first determination information.

[0142] (S4050: Reclassification of limb bones identified as abnormal to trunk bones) In step S4050, the classification unit 103 reclassifies the first and second bones into groups of bones based on at least one of the first and second determination information acquired in step S4040. Then, the classification unit 103 outputs information on the reclassified groups of bones to the alignment unit 104.

[0143] The classification unit 103 uses the identification information acquired by the identification information acquisition unit 102 and information on the result of the judgment by the judgment unit 401 to reclassify the bones that belong to the second bone group and that are judged to have an abnormality into the first bone group. When it is judged that there is an abnormality in any of the limb bone groups G2 to G5, the classification unit 103 reclassifies the group to which the limb judged to have an abnormality belongs so as to include it in the trunk bone group G1. At this time, the group of the limb having an abnormality may be acquired based on one of the first judgment information and the second judgment information, or may be acquired based on both. When acquiring based on both, for example, the classification unit 103 reclassifies the group of the limb bones that are judged to have an abnormality or a transplant based on at least one of the first judgment information and the second judgment information so as to include it in the trunk bone group G1.

[0144] For example, when images of the same subject captured at different times are processed as the first medical image and the second medical image, it is rare that an artificial bone is present in the medical image on the past side in the course of time and is not present in the medical image on the future side in the course of time. In this way, as a form that takes into consideration a use case regarding the history of the introduction of a graft including an artificial bone to a subject of interest, a form in which the classification unit 103 performs an abnormality determination of a group of limbs using only the determination information of the medical image on the past side, which is imaged earlier, is also a modified form of this embodiment. Note that, when an abnormality is determined based on only one of the determination information (for example, the first determination information), the process of the determination unit 401 in S4040 may obtain only the determination information to be used (for example, the first determination information) and not obtain the determination information not to be used (for example, the second determination information). The determination unit 401 may adopt an adaptive implementation form that performs the above-mentioned abnormality determination determination operation depending on whether or not a graft is included in the second bone group in the first image and the second image.

[0145] When there are multiple groups of limb bones, in addition to the above-described configuration of reclassifying only the limb bone group determined to have an abnormality into group G1, it is also possible to reclassify the other groups into group G1 as well. As an example of the latter, since artifacts caused by transplants may occur over a wide area, if there is an abnormality in the group into which the right femur BL13 is classified, in addition to that group, the group into which the left femur BL14 is classified may also be reclassified into group G1 of trunk bones.

[0146] (S4060: Alignment between a first plurality of bones and a second plurality of bones for each group of bones) In step S4060, the registration unit 104 performs registration between the first plurality of bones and the second plurality of bones for each group of bones, and outputs the generated displacement field to the image generation unit 105.

[0147] In this embodiment, the image generating unit 105 generates an image in which pixel values ​​of bone regions of a group of bones of a limb that are determined to have no abnormality from the first medical image are filled with predetermined pixel values ​​(for example, pixel values ​​close to those of surrounding tissues outside the bones), and generates the image I1. G1 This makes it possible to generate images of each group of bones of the limbs that are determined to have an abnormality without extracting the group, thereby preventing a failure in the extraction process due to an abnormality in the information of the bone group. Furthermore, it is possible to suppress a decrease in the accuracy of the registration compared to a case where an image that has been unsuccessfully extracted is used for registration.

[0148] According to this embodiment, when there is an abnormality in the identification information of a body part performing a different movement, the abnormal part is aligned without being separated from the torso, and therefore, compared to the first embodiment, it is possible to suppress a decrease in the alignment accuracy of the abnormal part. Also, for parts without abnormalities, it is possible to suppress a decrease in the alignment accuracy when the movements of multiple bones belonging to different body parts between images are significantly different or discontinuous deformation occurs at the boundary between them, as in the first embodiment. That is, according to this embodiment, it is possible to accurately and stably align images.

[0149] (Variation 1) In the fourth embodiment, in step S4040, the determination process is performed on the group of bones of the limbs, but the determination process may be performed on the group of bones of the trunk. For example, in step S4030, when the individual bones belonging to the trunk are classified into different groups, the determination unit 401 may determine the bone recognition area in which the extraction reliability of the graft and the bone does not satisfy a predetermined threshold for each individual bone group. For example, in step S4040, when one or more vertebrae in the spine are classified into different groups, the determination unit 401 performs the determination process on each vertebra group. Then, in step S4050, the classification unit 103 reclassifies the group of vertebrae determined to have an abnormality into the group G1 of bones of the trunk that is not subjected to the segmentation process. This makes it possible to suppress the deterioration of the alignment caused by an abnormality in the information of the group of bones of the trunk, as in the case of the bones of the limbs.

[0150] (Variation 2) In the fourth embodiment, in step S4040, the determination unit 401 acquires the determination information as information on the result of the determination for the group of bones of the limbs, but the target for acquiring the determination information does not necessarily have to be a group of bones. For example, the determination unit 401 may acquire the first determination information and the second determination information using the first identification information and the second identification information acquired in step S4020. The determination unit 401 may be configured to determine whether or not there is a transplant or whether or not there is a bone recognition area in which the bone extraction reliability does not satisfy a predetermined threshold value, as in step S4040, for the label area of ​​the bones of the limbs in the first identification information and the second identification information, and acquire the first determination information and the second determination information.

[0151] In this case, the process of step S4030 may be skipped, and in step S4050, the classification unit 103 may classify the multiple bone groups based on the first identification information, the second identification information, the first judgment information, and the second judgment information. The classification unit 103 reclassifies the bones that are determined to have an abnormality and belong to the second bone group into the first bone group using the identification information (the first identification information and the second identification information) acquired by the identification information acquisition unit 102 and information on the result of the judgment (the first judgment information and the second judgment information). That is, the classification unit 103 classifies the bones of the abnormal limbs so that they are included in the bone group of the trunk, and classifies the bones of the other limbs into the second bone group as in the first embodiment. This allows the process of step S4030 to be omitted, and therefore the process can be executed efficiently.

[0152] (Variation 3) In the fourth embodiment, in step S4020, the identification information acquisition unit 102 may acquire a label region of the transplant as the first identification information and the second identification information. For example, in addition to identifying the bone, the transplant may also be identified by a known area extraction method using machine learning. Then, in step S4040, the determination unit 401 may determine whether the transplant is included in the group of bones of the limb based on the first identification information and the second identification information including the label region of the transplant acquired in step S4020.

[0153] <Fifth embodiment> In the first embodiment, a configuration has been described in which a plurality of bones are identified from a first medical image and a second medical image, classified into a plurality of bone groups, and alignment is performed for each bone group. In the present embodiment, a configuration will be described in which, before alignment for each bone group, initial alignment is performed for each individual bone (referred to as a partial bone) in the bone group for the bone group of the limbs, and each initial alignment result is integrated.

[0154] Fig. 10 is a diagram showing the configuration of an image processing system 50 according to the fifth embodiment. An image processing device 500 according to the fifth embodiment has a corresponding point cloud generating unit 501 as a component. The configuration other than the corresponding point cloud generating unit 501 is the same as that in Fig. 1, so a description thereof will be omitted. The corresponding point cloud generating unit 501 performs processing to generate a corresponding point cloud based on the initial alignment result for each of a plurality of input partial bones.

[0155] The processing of this embodiment will be described below with reference to the flowchart in Fig. 11. Note that steps S5010 to S5030 and S5090 to S5100 in the flowchart showing the overall processing procedure performed by image processing device 500 are similar to steps S1010 to S1030 and S1050 to S1060 in the first embodiment, and therefore description thereof will be omitted.

[0156] Hereinafter, only the parts of the flowchart in FIG. 11 that are different from the first embodiment will be described.

[0157] (S5040: Acquire identification information of partial bones included in a group of limb bones) In step S5040, the alignment unit 104 acquires identification information of partial bones included in a second group of bones (hereinafter also referred to as a group of bones of limbs) that move in association with the movement of the limbs, based on the information of the bone group acquired from the classification unit 103. The alignment unit 104 acquires identification information of partial bones belonging to a plurality of bones classified into a group of bones of limbs on the first medical image, and identification information of partial bones belonging to a plurality of bones classified into a group of bones of limbs on the second medical image. That is, based on the information acquired from the classification unit 103, the alignment unit 104 acquires first identification information of a first partial bone that identifies a first partial bone in the first plurality of bones that is included in the first plurality of bones depicted in the first medical image and classified into the second group of bones, and first identification information of a second partial bone that identifies a second partial bone different from the first partial bone. The alignment unit 104 also acquires second identification information of the first partial bone for identifying the first partial bone that is included in the second plurality of bones depicted in the second medical image and that is classified into the second bone group, and second identification information of the second partial bone for identifying the second partial bone. Specific processing will be described below.

[0158] In this step, the alignment unit 104 acquires identification information of partial bones belonging to each of the limb bone groups G2, G3, G4, and G5. A case where there are multiple partial bones in a group will be described with reference to FIG.

[0159] 4, the group G2 of bones of the right arm includes the right upper humerus BL2 and the right scapula BL6, which move in association with the movement of the right arm. At this time, the alignment unit 104 acquires identification information of the right upper humerus BL2 as the first partial bone and identification information of the right scapula BL6 as the second partial bone. Similarly, for the group G3 of the left arm, the alignment unit 104 acquires identification information of the left upper humerus BL3 as the first partial bone and identification information of the left scapula BL7 as the second partial bone.

[0160] As described in the first embodiment, when the classification unit 103 classifies the right clavicle BL4 into group G2, the alignment unit 104 further acquires identification information of the right clavicle BL4 as a third partial bone in group G2. Similarly, when the left clavicle BL5 is classified into group G3, the alignment unit 104 further acquires identification information of the left clavicle BL5 as a third partial bone in group G3.

[0161] Also, when there is only one partial bone in a group, the alignment unit 104 acquires that partial bone as the first partial bone. For example, in the case of the group G4 of the bones of the right leg in Fig. 4, the alignment unit 104 acquires identification information of the right femur BL13 as the first partial bone, and in the case of the group G5 of the bones of the left leg, the alignment unit 104 acquires identification information of the left femur BL14 as the first partial bone.

[0162] (S5050: Initial alignment between a first plurality of bones and a second plurality of bones for each partial bone) In step S5050, the alignment unit 104 performs initial alignment between the first and second bones for each partial bone based on the identification information of the partial bone included in the first bones on the first medical image acquired in S5040 and the identification information of the partial bone included in the second bones on the second medical image. The alignment unit 104 performs initial alignment of the first partial bone between the first medical image and the second medical image based on the first identification information of the first partial bone and the second identification information of the first partial bone. Also, the alignment unit 104 performs initial alignment of the second partial bone based on the first identification information of the second partial bone and the second identification information of the second partial bone. Then, the alignment unit 104 outputs the initial conversion information generated by performing the initial alignment to the corresponding point group generation unit 501. A specific process will be described below.

[0163] In this step, the alignment unit 104 extracts a surface point cloud of each partial bone based on the first identification information representing the partial bone belonging to the first plurality of bones. More specifically, the alignment unit 104 extracts the surface point cloud of each partial bone from the label area on the label image, which is the first identification information, using a known point cloud sampling method. The extracted surface point cloud is referred to as a first surface point cloud SP1. The alignment unit 104 performs the process of extracting a surface point cloud for each partial bone in each limb bone group. As a result, for example, for the right arm bone group G2, the first surface point cloud SP1 of the first partial bone (right upper humerus BL2) is extracted. BL2 and the first surface point cloud SP1 of the second partial bone (right scapula BL6). BL6 and

[0164] Similarly, the registration unit 104 obtains a second surface point cloud SP2 for each partial bone in each group of bones of each limb by sampling the surface point cloud from the label region representing the partial bone belonging to the second plurality of bones. As a result, for example, for the group G2 of bones of the right arm, the second surface point cloud SP2 of the first partial bone (right upper humerus BL2) is obtained. BL2 and the second surface point cloud SP2 of the second partial bone (right scapula BL6). BL6 can be obtained respectively.

[0165] Next, the alignment unit 104 performs initial alignment between corresponding partial bones using the first surface point group SP1 and the second surface point group SP2. As a method of initial alignment using the surface point groups, for example, an ICP algorithm that performs rigid alignment to minimize the distance between the surface point groups can be used. The alignment unit 104 performs rigid alignment for each partial bone in each group of bones of each limb, thereby acquiring each rigid transformation matrix for each partial bone as initial transformation information. As a result, for example, with respect to the group G2 of bones of the right arm, the alignment unit 104 obtains a rigid transformation matrix T that approximately matches the first partial bone (right upper humerus BL2) on the first image with the partial bone (right upper humerus BL2) on the second image. BL2Similarly, the positioning unit 104 obtains a rigid body transformation matrix T BL6 is obtained as the initial conversion information.

[0166] As another method for performing initial alignment, a known method may be used in which rigid body alignment is performed using indexes such as image similarity or area coincidence between a label image which is a first identification information representing a partial bone on a first image and a label image which is a second identification information representing a partial bone on a second image.

[0167] In addition, the initial transformation information obtained in the initial alignment does not necessarily have to be a rigid transformation matrix. For example, an affine transformation matrix that has a higher degree of freedom of deformation may be used. This can be realized by replacing the transformation information sequentially updated by the ICP algorithm in the above example from a rigid transformation matrix to an affine transformation matrix.

[0168] (S5060: Obtain corresponding points based on the initial alignment results for each partial bone) In step S5060, the corresponding point group generating unit 501 obtains corresponding point groups for each of the second bone groups (groups of bones of the limbs) between the first medical image and the second medical image based on the initial conversion information obtained by performing the initial alignment. Then, the corresponding point group generating unit 501 outputs the obtained information of the corresponding point group to the alignment unit 104. The specific processing will be described below.

[0169] In this step, first, the corresponding point cloud generating unit 501 samples the inside of the region of the partial bone on either the first medical image or the second medical image to obtain discrete position information within the region as a point cloud. In this embodiment, the point cloud obtained in this manner is called a sampled point cloud. When sampling within the region of the partial bone on the first medical image, for example, sampling is performed at equal intervals (e.g., 3 mm intervals) within the label region of the label image, which is identification information of the partial bone on the first medical image. Note that the sampling interval is not limited to equal intervals and may be set arbitrarily.

[0170] The corresponding points cloud generating unit 501 performs sampling on each partial bone in each group of bones of each limb to obtain a sampling point cloud P1. For example, the corresponding points cloud generating unit 501 obtains a sampling point cloud P1 sampled on a first partial bone (right upper humerus BL2) on the first image. BL2 and the sampling point group P1 sampled on the second partial bone (right scapula BL6) on the first image. BL6 The target for acquiring the sampling point group may be a partial bone on the second image.

[0171] Next, based on the initial conversion information acquired in step S5050, the corresponding point cloud generating unit 501 acquires a sampling point cloud by converting the coordinates of the sampling point cloud acquired at the partial bone on either the first medical image or the second medical image into the position of the partial bone on the other medical image. The corresponding point cloud generating unit 501 acquires a corresponding point cloud for the first partial bone and a corresponding point cloud for the second partial bone between the first medical image and the second medical image.

[0172] Here, the corresponding points cloud generating unit 501 obtains a sampling point cloud P2 by converting the coordinates of the sampling point cloud P1 of a partial bone on the first image into the position of the partial bone on the second image. The corresponding points cloud generating unit 501 then generates a corresponding points cloud CP by associating each point in the sampling point cloud P1 before conversion with each point in the sampling point cloud P2 after conversion. The corresponding points cloud generating unit 501 performs the same process on each partial bone in each limb bone group to obtain each corresponding point cloud. For example, the corresponding points cloud generating unit 501 obtains the corresponding points cloud P2 by converting the sampling point cloud P1 on the first partial bone (right upper humerus BL2) on the first image. BL2 The initial transformation information is the rigid body transformation matrix T BL2 The converted sampling point group P2 is converted to the coordinates of the partial bone (right upper humerus BL2) on the second image using BL2 The corresponding points group generating unit 501 also generates a sampling point group P1 on the second partial bone (right scapula BL6) on the first image. BL6 The initial transformation information is the rigid body transformation matrix T BL6 The converted sampling point group P2 is converted to the coordinates of the partial bone (right upper humerus BL2) on the second image using BL2 Generate.

[0173] Then, the corresponding point group generating unit 501 extracts the sampling point group P1 BL2 and the sampling point group P2 BL2 The corresponding points group CP BL2 In addition, the corresponding points generation unit 501 generates the sampling points P1 BL6 and the sampling point group P2 BL6 The corresponding points group CP BL6 The direction in which the sampling point group is transformed may be from the second medical image to the first medical image. In this case, the sampling point group coordinates acquired from the partial bone on the second medical image are transformed by the inverse transformation (inverse matrix in the case of a rigid transformation matrix) of the initial transformation information acquired in step S5050, thereby obtaining the sampling point group transformed into the coordinates of the partial bone on the first medical image.

[0174] Next, the corresponding point cloud generating unit 501 generates an integrated corresponding point cloud CP by integrating the corresponding point clouds of the partial bones belonging to each bone group of the limbs. I The corresponding point cloud generating unit 501 can integrate the corresponding point clouds by combining the corresponding point clouds of the respective partial bones into one. The corresponding point cloud generating unit 501 obtains an integrated corresponding point cloud by combining the corresponding point cloud for the first partial bone and the corresponding point cloud for the second partial bone. For example, in the group G2 of bones of the right arm, the corresponding point cloud generating unit 501 generates the corresponding point cloud CP BL2 (N BL2 pairs) and the corresponding point cloud CP of the second partial bone (right scapula BL6) BL6 (N BL6 The integrated corresponding point cloud CP I_G2 (N BL2 +N BL6 The corresponding point cloud generating unit 501 performs the same process on the other limb groups G3 to G5 to generate an integrated corresponding point cloud CP I_G3 ~CP I_G5 Get the.

[0175] Note that the acquisition of the corresponding points in this step does not necessarily have to be performed for a group of limb bones that includes only one partial bone (eg, G4 (right femur), G5 (left femur)).

[0176] (S5070: Alignment between a first and a second bone of a group of limb bones using a cloud of corresponding points as a constraint) In step S5070, the alignment unit 104 performs alignment between the first and second bones of the limb bone group using the integrated corresponding point group acquired in step S5060 as a constraint condition, and acquires a displacement field. Specifically, the alignment unit 104 acquires a deformation that makes the transformation of other positions smooth, using the constraint condition of making each corresponding point group constituting the integrated corresponding point group approximately coincident. The alignment unit 104 performs a deformation process using the corresponding point group related to the first partial bone and the corresponding point group related to the second partial bone as a constraint condition, and acquires integrated transformation information (hereinafter, integrated initial transformation information) that combines the initial transformation information generated by the initial alignment of the first partial bone and the initial transformation information generated by the initial alignment of the second partial bone. Then, the alignment unit 104 performs a second alignment between the first and second bones based on the integrated initial transformation information. That is, the positioning unit 104 integrates the initial transformation information of each partial bone in each limb bone group, and obtains integrated initial transformation information re-expressed as one displacement field for each limb bone group. Specific processing will be described below.

[0177] In this step, the registration unit 104 registers the integrated corresponding point groups CP I_G2 ~CP I_G5 The positioning unit 104 performs positioning by calculating the correspondence relationship in a continuous space using the correspondence relationship between discrete positions defined as above as a constraint condition. At this time, a known method using a group of corresponding points (landmarks) can be used as the positioning method. For example, as the image deformation model, a known deformation model such as the FFD (Free Form Deformation) model and the Demons algorithm described in step S1040, as well as TPS (Thin Plate Spline) can be used. As a result, the positioning unit 104 calculates the correspondence relationship between the continuous space using the integrated initial conversion information ID G2 ~ID G5 Get the.

[0178] The alignment in this step does not have to be performed for a group of limb bones that includes only one partial bone (e.g., G4 (right femur), G5 (left femur)). In other words, the alignment in this step may not be performed, and the initial transformation information acquired in step S5050 may be applied as is as integrated initial transformation information. In this case, the alignment unit 104 uses the rigid transformation matrix TB acquired as the initial transformation information L13 and the rigid body transformation matrix T BL14 Using the integrated initial conversion information ID G4 = Rigid body transformation matrix TB L13 , Integrated initial conversion information ID G5 = Rigid body transformation matrix T BL14 It can be obtained as follows.

[0179] (S5080: Aligning a first set of bones with a second set of bones for each group of bones) In step S5080, the alignment unit 104 aligns the first and second bones for each group of bones. At this time, for the group G1 of bones of the trunk, the same alignment as in step S1040 is performed, and therefore a description thereof will be omitted. On the other hand, for the groups G2 to G5 of bones of the limbs, the alignment unit 104 aligns the first and second bones using the integrated initial conversion information acquired by the process of S5070 as an initial value. A specific process will be described below.

[0180] The position adjustment unit 104 adjusts the position of the image I1 cut out from the first medical image I1. G2 ~I1 G5 and an image I2 cut out from the second medical image I2 G2 ~I2 G5 At this time, the position adjustment unit 104 first performs position adjustment between the image I2, which is a floating image in this embodiment, and the image I3, which is a floating image in this embodiment. G2 ~I2 G5 , the integrated initial conversion information ID G2 ~ID G5 Image initially deformed using (initial deformed image) I2 ID_G2 ~I2 ID_G5 Generate.

[0181] Next, the position adjustment unit 104 adjusts the initial deformed image I2 ID_G2 ~I2 ID_G5 is further transformed to the reference image I1 G2 ~I1 G5 Displacement field D to approximately match G2 ~D G5 This is the floating image I2 in step S1040. G2 ~I2 G5 The initial deformed image I2 ID_G2 ~I2 ID_G5 Since the process is the same as step S1040 except for replacing it with, the description will be omitted.

[0182] According to this embodiment, it is possible to obtain a transformation result that causes each partial bone in the same group to be approximately matched between images. In particular, when the posture of each partial bone differs between images, it is possible to obtain initial transformation information with higher accuracy than when the initial transformation of the group is expressed by a single rigid transformation or the like. By using the initial transformation information obtained in this way as an initial value to execute the processes from step S5080 onwards, it is possible to perform registration between images more stably and with higher accuracy.

[0183] (Variation 1) The processing from step S5050 to step S5070 in the fifth embodiment has been described as an example in which the processing is performed as alignment processing within the group of bones classified in step S5030. However, the disclosed technology is not limited to this. For example, it is also possible to perform alignment processing for a plurality of partial bones adjacent to each other, regardless of the grouping of bones. For example, as shown in FIG. 4, alignment processing may be performed for the right upper humerus BL2 and the right scapula BL6, which are adjacent to each other, or alignment processing may be performed for the spine BL10 and the rib BL8.

[0184] As a specific process, the processes of steps S5030 and S5040 are omitted, and instead of the omitted processes, any adjacent bones identified in step S5020 are treated as partial bones to be processed. Then, the registration unit 104 performs the processes of steps S5060 and S5070 on the partial bones to be processed, thereby acquiring a registration result (one displacement field) for the partial bones of the target.

[0185] Then, the registration unit 104 omits the process of step S5080, and performs a subtraction process between images based on the registration result (displacement field) obtained in step S5070 as the process of step S5090, thereby obtaining a subtraction image of the target bones. The above-mentioned processing target may be an anatomical structure other than bones. For example, instead of the multiple partial bones, the right atrium and left atrium or the right ventricle and left ventricle inside the heart, which are adjacent to each other and are considered not to cause discontinuous deformation at the boundary between them, may be applied to this process.

[0186] (Variation 2) In the fifth embodiment, in step S5070, the initial alignment of partial bones belonging to a group of bones in the limbs is expressed as a single displacement field, and then in step S5080, the alignment of the groups of bones in each limb is performed using the displacement field acquired in step S5070 as the initial value.

[0187] However, the processes of steps S5070 and S5080 may be performed simultaneously as one process. More specifically, the registration unit 104 may perform registration using two constraints: a match between the integrated corresponding point groups in step S5070, and an image similarity in step S5080.

[0188] This registration can be performed by a known method (FFD (Free Form Deformation) model, Demons algorithm, etc.) described in steps S5070 and S5080. This allows for registration that considers both the discrete position constraint condition by the integrated corresponding point group and the image similarity at the same time. This allows for registration that is closer to the integrated corresponding point group than when the registration is performed in two stages in the processing of steps S5070 and S5080.

[0189] In the process of step S5080, the registration unit 104 may use the displacement field obtained in step S5070 as an initial value and perform registration using both the match of the integrated corresponding point group and the image similarity as constraint conditions, as described above. According to this process, after the rough initial registration, detailed registration is performed using the corresponding point group and the image similarity, so that high-speed and high-precision registration can be performed.

[0190] The disclosure of this specification includes the following image processing device, image processing system, image processing method, and program.

[0191] (Item 1) An acquisition means for acquiring at least one of first identification information for identifying a first plurality of bones depicted in a first image of a subject and second identification information for identifying a second plurality of bones depicted in a second image of the subject; a classification means for classifying the first and second bones into a first bone group including bones that move in association with a movement of a first part of the subject and a second bone group including bones that move in association with a movement of a second part different from the first part, using the at least one of the identification information; an alignment means for performing a first alignment between the first plurality of bones classified into the first bone group and the second plurality of bones, and a second alignment between the first plurality of bones classified into the second bone group and the second plurality of bones.

[0192] (Item 2) The classification means identifies a bone that is commonly depicted between the first image and the second image using the at least one of the identification information, 2. The image processing device according to item 1, wherein a common group in which the commonly depicted bones exist is classified as the first bone group and the second bone group.

[0193] (Item 3) The image processing device according to item 2, wherein the common group differs depending on a degree of overlap of imaging ranges between the first image and the second image.

[0194] (Item 4) The classification means classifies the first plurality of bones and the second plurality of bones into a third bone group including bones that move in association with a movement of a third part different from the first part and the second part, using the at least one of the identification information; 3. The image processing device according to item 1 or 2, wherein the alignment means performs a third alignment between the first plurality of bones and the second plurality of bones classified into the third bone group.

[0195] (Item 5) The image processing device according to Item 4, wherein the second part and the third part are physically continuous with the first part, and the second part and the third part are parts capable of operating independently of each other.

[0196] (Item 6) An image processing device described in Item 4 or 5, wherein the first region is the torso of the subject, the second region is one of the subject's limbs, and the third region is one of the subject's limbs different from the second region.

[0197] (Item 7) An image processing device described in any one of Items 4 to 6, wherein the first group of bones includes the ribs of the subject, the second group of bones includes one of the left and right scapulas of the subject, and the third group of bones includes the other of the left and right scapulas of the subject.

[0198] (Item 8) The acquiring means acquires identification information of a plurality of bones identified from one of the first image and the second image, The classification means 8. The image processing device according to any one of items 1 to 7, wherein the plurality of bones identified from the one image are classified into groups of bones that move in association with different movements for each part of the subject performing different movements based on the identification information.

[0199] (Item 9) The image processing device described in Item 1 or 2, wherein the acquisition means acquires at least one of the first identification information identifying each of the first plurality of bones and the second identification information identifying each of the second plurality of bones.

[0200] (Item 10) The positioning means acquires bone regions from the first image and the second image based on image processing for extracting regions, 9. The image processing device according to item 8, which performs registration so as to match the bone region between the first image and the second image.

[0201] (Item 11) The method further includes a matching unit for matching a region of a group of a plurality of bones classified in one of the first image and the second image with a region of the other image, 9. The image processing device according to item 8, wherein the association means associates a region of a group of a plurality of bones in the one image, classified based on the identification information, with a region of the other image.

[0202] (Item 12) The matching means generates a deformed image by deforming the other image using the displacement field generated in the registration, 12. The image processing device according to item 11, wherein each of the regions of a group of a plurality of bones in the one image is associated with a corresponding region in the deformed image.

[0203] (Item 13) The matching means obtains a difference in feature information between a region of a group of a plurality of bones in the one image and a matched region in the deformed image, 13. The image processing device according to item 12, wherein a region of a group of multiple bones in the one image is associated as a bone region in the deformed image by using the difference in the feature information.

[0204] (Item 14) The image processing device according to Item 13, wherein the association means associates the regions of the plurality of bone groups with bone regions in the deformed image in ascending order of difference in the feature information.

[0205] (Item 15) An image processing device according to item 1 or 2, wherein the alignment means performs the first alignment based on a first displacement field for matching an image of a bone region belonging to the first bone group in the first image with an image of a bone region belonging to the first bone group in the second image.

[0206] (Item 16) The image processing device described in Item 15, wherein the alignment means acquires the amount of displacement between a position within a region corresponding to the first bone group in the first image and a position within a region corresponding to the first bone group in the second image as the first displacement field.

[0207] (Item 17) The image processing device described in Item 15, wherein the alignment means performs the second alignment based on a second displacement field for matching an image of a bone region belonging to the second bone group in the first image with an image of a bone region belonging to the second bone group in the second image.

[0208] (Item 18) The image processing device described in Item 17, wherein the alignment means acquires the amount of displacement between a position within an area corresponding to the second bone group in the first image and a position within an area corresponding to the second bone group in the second image as the second displacement field.

[0209] (Item 19) The apparatus further includes an image generating means, the image generating means being: Setting a region of a displacement field for the first image; storing the displacement amount of the first displacement field in a region of the first bone group among the set regions; storing the displacement amount of the second displacement field in a region of the second bone group among the set regions; Item 18. The image processing device according to item 17, which generates an integrated displacement field in which displacement amounts of different displacement fields are stored at different positions within the set region.

[0210] (Item 20) The image generating means generates a deformed image by deforming one of the first image and the second image so as to match the other medical image by using the integrated displacement field; 20. The image processing device according to item 19, wherein the difference image is generated by subtracting pixel information of the deformed image from pixel information of the other medical image.

[0211] (Item 21) The image processing device described in Item 17, wherein the alignment means matches bone regions identified as the same bone region between the first plurality of bones and the second plurality of bones based on the first identification information and the second identification information.

[0212] (Item 22) The positioning means is performing a rigid body registration between the associated bone regions in the first image and the associated bone regions in the second image using the first displacement field and the second displacement field as an initial displacement field; 22. The image processing device according to item 21, which performs deformation registration between the bone region in the first image and the bone region in the second image after the rigid body registration, capable of setting the degree of freedom of deformation in the bone region.

[0213] (Item 23) An image processing device according to Item 20, further comprising a display control means, the display control means performing display control to cause a display means to display at least one of the transformed image and the difference image generated by the image generation means.

[0214] (Item 24) An acquisition means for acquiring identification information for identifying a plurality of bones depicted in each of a first image obtained by photographing a subject and a second image obtained by photographing the subject; a classification means for classifying the plurality of bones into a first bone group including bones that move in association with a movement of a first part of the subject, using the identification information; a registration means for performing a first registration between a plurality of bones depicted in the first image, which are classified into the first bone group, and a plurality of bones depicted in the second image; An image processing device comprising:

[0215] (Item 25) The classification means classifies the plurality of bones into a second bone group including a bone that moves in association with a movement of a second part different from the first part, using the identification information; 25. The image processing device according to item 24, wherein the alignment means performs a second alignment between a plurality of bones depicted in the first image classified into the second bone group and a plurality of bones depicted in the second image.

[0216] (Item 26) An image processing device according to item 1 or 2, further comprising a determination means for determining whether the second bone group classified in at least one of the first image and the second image includes at least one of a transplant and a bone recognition area in which the bone extraction reliability does not satisfy a predetermined threshold value.

[0217] (Item 27) An image processing device according to item 1 or 2, further comprising a judgment means for determining whether at least one of a first judgment result that a transplant is included in the second bone group in at least one of the first image and the second image, and a second judgment result that the second bone group includes a bone recognition region that does not have a predetermined extraction reliability is satisfied.

[0218] (Item 28) The classification means reclassifies the bones belonging to the second bone group, which are identified as the erroneous extraction region, into the first bone group using the identification information acquired by the acquisition means and information related to the result of the determination, 28. The image processing device according to item 26 or 27, wherein bones that are not identified as the erroneous extraction region and belong to the second bone group are reclassified into the second bone group.

[0219] (Item 29) The positioning means, based on the information acquired from the classification means, Obtaining first identification information of a first partial bone that identifies a first partial bone in the first plurality of bones and that is classified into the second bone group, and first identification information of a second partial bone that identifies a second partial bone different from the first partial bone; The image processing device described in item 1 or 2 acquires second identification information of a first partial bone that identifies the first partial bone included in the second plurality of bones and classified into the second bone group, and second identification information of a second partial bone that identifies the second partial bone.

[0220] (Item 30) The positioning means is configured to: performing initial alignment of the first partial bone based on first identification information of the first partial bone and second identification information of the first partial bone; 30. The image processing device according to item 29, further comprising: an image processing device for performing initial position alignment of the second partial bone based on first identification information of the second partial bone and second identification information of the second partial bone.

[0221] (Item 31) The method further includes a corresponding point cloud acquisition means for acquiring a corresponding point cloud related to the second bone group between the first image and the second image based on initial transformation information acquired by performing the initial registration, The image processing device according to item 30, wherein the corresponding point cloud acquisition means acquires a corresponding point cloud relating to the first partial bone and a corresponding point cloud relating to the second partial bone between the first image and the second image.

[0222] (Item 32) The positioning means is performing a transformation process using the corresponding point group for the first partial bone and the corresponding point group for the second partial bone as constraint conditions to obtain integrated transformation information that combines initial transformation information generated by initial alignment of the first partial bone and initial transformation information generated by initial alignment of the second partial bone; Item 32. The image processing device according to item 31, wherein the second alignment is performed based on the integrated conversion information.

[0223] (Item 33) An image processing device according to item 1 or 2, comprising a communication interface communicatively connected to a data server via a communication network.

[0224] (Item 34) The data server connected to the communication interface so as to be able to communicate with the data server via the communication network; Item 34. An image processing system having the image processing device according to item 33.

[0225] (Item 35) A step of acquiring at least one of first identification information for identifying a first plurality of bones depicted in a first image of a subject and second identification information for identifying a second plurality of bones depicted in a second image of the subject; classifying the first and second bones into a first bone group including bones that move in association with a movement of a first part of the subject and a second bone group including bones that move in association with a movement of a second part different from the first part, using the at least one of the identification information; performing a first alignment between the first plurality of bones classified into the first bone group and the second plurality of bones, and a second alignment between the first plurality of bones classified into the second bone group and the second plurality of bones.

[0226] (Item 36) A step of acquiring identification information for identifying a plurality of bones depicted in each of a first image taken of a subject and a second image taken of the subject; classifying the plurality of bones into a first bone group including bones that move in association with a movement of a first part of the subject using the identification information; and performing a first alignment between a plurality of bones depicted in the first image that are classified into the first bone group and a plurality of bones depicted in the second image.

[0227] (Item 35) A program for causing a computer to execute the image processing method according to Item 35 or 36.

[0228] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0229] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0230] 10: image processing system, 100: image processing device, 101: image acquisition unit, 102: identification information acquisition unit, 103: classification unit, 104: alignment unit, 105: image generation unit, 106: display control unit, 120: network, 130: data server, 140: instruction unit, 150: display unit, 401: judgment unit, 501: corresponding point cloud generation unit

Claims

1. an acquiring means for acquiring at least one of first identification information for identifying a first plurality of bones depicted in a first image obtained by photographing a subject and second identification information for identifying a second plurality of bones depicted in a second image obtained by photographing the subject; a classification means for classifying the first and second bones into a first bone group including bones that move in association with a movement of a first part of the subject and a second bone group including bones that move in association with a movement of a second part different from the first part, using the at least one of the identification information; an alignment means for performing a first alignment between the first plurality of bones classified into the first bone group and the second plurality of bones, and a second alignment between the first plurality of bones classified into the second bone group and the second plurality of bones; An image processing device comprising:

2. The classification means identifies a bone commonly depicted between the first image and the second image using the at least one of the identification information, The image processing apparatus according to claim 1 , wherein common groups in which the commonly depicted bones exist are classified as the first bone group and the second bone group.

3. The image processing device according to claim 2 , wherein the common group differs depending on a degree of overlap of the imaging range between the first image and the second image.

4. the classification means classifies the first plurality of bones and the second plurality of bones into a third bone group including bones that move in association with a movement of a third part different from the first part and the second part, using the at least one of the identification information; The image processing apparatus according to claim 1 , wherein the alignment means performs a third alignment between the first plurality of bones and the second plurality of bones classified into the third bone group.

5. The image processing device according to claim 4 , wherein the second section and the third section are physically continuous with the first section, and the second section and the third section are sections capable of operating independently of each other.

6. 5. The image processing device according to claim 4, wherein the first region is a torso of the subject, the second region is one of the limbs of the subject, and the third region is one of the limbs of the subject that is different from the second region.

7. 5. The image processing device according to claim 4, wherein the first group of bones includes the ribs of the subject, the second group of bones includes one of the left and right scapulas of the subject, and the third group of bones includes the other of the left and right scapulas of the subject.

8. the acquiring means acquires identification information of a plurality of bones identified from one of the first image and the second image; The classification means The image processing device according to claim 1 , further comprising: a processor for processing the bones of the subject by a first image; a processor for processing the bones of the subject by a second image;

9. 2 . The image processing device according to claim 1 , wherein the acquiring means acquires at least one of the first identification information for identifying each of the first plurality of bones and the second identification information for identifying each of the second plurality of bones.

10. The positioning means acquires bone regions from the first image and the second image based on image processing for extracting regions; The image processing apparatus according to claim 8 , wherein registration is performed so as to match the bone region between the first image and the second image.

11. a matching unit for matching a region of a group of a plurality of bones classified in one of the first image and the second image with a region of the other image, The image processing apparatus according to claim 8 , wherein the associating means associates a region of a group of a plurality of bones in the one image, which has been classified based on the identification information, with a region in the other image.

12. The matching means generates a deformed image by deforming the other image using the displacement field generated in the registration, The image processing apparatus according to claim 11 , wherein each of the regions of a group of a plurality of bones in the one image is associated with a corresponding region in the deformed image.

13. the matching means obtains a difference in feature information between a region of a group of a plurality of bones in the one image and a matched region in the deformed image; The image processing apparatus according to claim 12 , wherein a region of a group of a plurality of bones in the one image is associated with a bone region in the deformed image by using the difference in the feature information.

14. The image processing apparatus according to claim 13 , wherein the association means associates the regions of the plurality of bone groups with bone regions in the deformed image in ascending order of difference in the feature information.

15. 2. The image processing device according to claim 1, wherein the alignment means performs the first alignment based on a first displacement field for matching an image of a bone region belonging to the first bone group in the first image with an image of a bone region belonging to the first bone group in the second image.

16. The image processing device according to claim 15, wherein the alignment means acquires, as the first displacement field, a displacement amount between a position within a region corresponding to the first bone group in the first image and a position within a region corresponding to the first bone group in the second image.

17. The image processing device according to claim 15, wherein the alignment means performs the second alignment based on a second displacement field for matching an image of a bone region belonging to the second bone group in the first image with an image of a bone region belonging to the second bone group in the second image.

18. The image processing device according to claim 17, wherein the alignment means acquires, as the second displacement field, a displacement amount between a position within a region corresponding to the second bone group in the first image and a position within a region corresponding to the second bone group in the second image.

19. Further comprising an image generating means, The image generating means includes: defining a region of a displacement field for the first image; storing the displacement amount of the first displacement field in a region of the first bone group among the set regions; storing the displacement amount of the second displacement field in a region of the second bone group among the set regions; The image processing device according to claim 17 , wherein an integrated displacement field is generated by storing displacement amounts of different displacement fields at different positions within the set region.

20. the image generating means generates a deformed image by deforming one of the first image and the second image by using the integrated displacement field so as to match the other medical image; The image processing apparatus according to claim 19 , wherein a difference image is generated by subtracting pixel information of the deformed image from pixel information of the other medical image.

21. The image processing device according to claim 17 , wherein the alignment means matches bone regions identified as the same bone region between the first plurality of bones and the second plurality of bones based on the first identification information and the second identification information.

22. The positioning means includes: performing a rigid body registration between the associated bone regions in the first image and the associated bone regions in the second image using the first displacement field and the second displacement field as an initial displacement field; The image processing device according to claim 21 , further comprising: a deformation registration that enables setting a degree of freedom of deformation in a bone region between the bone region in the first image and the bone region in the second image, after the rigid body registration has been performed.

23. 21. The image processing apparatus according to claim 20, further comprising a display control means for performing display control to cause at least one of the transformed image and the difference image generated by the image generation means to be displayed on a display means.

24. an acquiring means for acquiring identification information for identifying a plurality of bones depicted in each of a first image obtained by photographing a subject and a second image obtained by photographing the subject; a classification means for classifying the plurality of bones into a first bone group including bones that move in association with a movement of a first part of the subject, using the identification information; a registration means for performing a first registration between a plurality of bones depicted in the first image and a plurality of bones depicted in the second image, the plurality of bones being classified into the first bone group; An image processing device comprising:

25. The classification means classifies the plurality of bones into a second bone group including bones that move in association with a movement of a second part different from the first part, using the identification information; The image processing device according to claim 24 , wherein the alignment means performs a second alignment between a plurality of bones depicted in the first image and a plurality of bones depicted in the second image, the plurality of bones being classified into the second bone group.

26. 2. The image processing device according to claim 1, further comprising a determination means for determining whether or not the second bone group classified in at least one of the first image and the second image includes at least one of a transplant and a bone recognition area in which the bone extraction reliability does not satisfy a predetermined threshold value.

27. 2. The image processing device according to claim 1, further comprising a determination means for determining whether at least one of a first determination result that a transplant is included in the second bone group in at least one of the first image and the second image and a second determination result that a bone recognition region that does not have a predetermined extraction reliability is included in the second bone group is satisfied.

28. 28. The image processing device according to claim 26 or 27, wherein the classification means uses the identification information acquired by the acquisition means and information regarding the result of the judgment to reclassify a bone belonging to the second bone group that is determined to include at least one of the transplant and a bone recognition area in which the bone extraction reliability does not satisfy a predetermined threshold into the first bone group.

29. The alignment means, based on the information acquired from the classification means, obtaining first identification information of a first partial bone that identifies a first partial bone included in the first plurality of bones and classified into the second bone group, and obtaining first identification information of a second partial bone that identifies a second partial bone; The image processing device according to claim 1, further comprising: acquiring second identification information of a first partial bone that identifies the first partial bone that is included in the second plurality of bones and classified into the second bone group; and acquiring second identification information of a second partial bone that identifies the second partial bone.

30. The alignment means performs the alignment between the first image and the second image. performing initial alignment of the first partial bone based on first identification information of the first partial bone and second identification information of the first partial bone; The image processing apparatus according to claim 29 , further comprising: an image processing device for performing initial alignment of the second partial bone based on first identification information of the second partial bone and second identification information of the second partial bone.

31. a corresponding points cloud acquisition means for acquiring a corresponding points cloud relating to the second bone group between the first image and the second image based on initial transformation information acquired by performing the initial registration; 31. The image processing device according to claim 30, wherein the corresponding points acquisition means acquires a corresponding points group relating to the first partial bone and a corresponding points group relating to the second partial bone between the first image and the second image.

32. The positioning means includes: performing a transformation process using the corresponding point group related to the first partial bone and the corresponding point group related to the second partial bone as constraint conditions, thereby acquiring integrated transformation information that combines initial transformation information generated by initial alignment of the first partial bone and initial transformation information generated by initial alignment of the second partial bone; The image processing apparatus according to claim 31 , wherein the second alignment is performed based on the integrated conversion information.

33. 3. The image processing apparatus according to claim 1, further comprising a communication interface for communicatively connecting to a data server via a communication network.

34. the data server communicatively connected to the communication interface via the communication network; An image processing system comprising: the image processing device according to claim 33.

35. acquiring at least one of first identification information for identifying a first plurality of bones depicted in a first image of a subject and second identification information for identifying a second plurality of bones depicted in a second image of the subject; classifying the first and second bones into a first bone group including bones that move in association with a movement of a first part of the subject and a second bone group including bones that move in association with a movement of a second part different from the first part, using the at least one of the identification information; performing a first alignment of the first plurality of bones classified into the first group of bones with the second plurality of bones, and a second alignment of the first plurality of bones classified into the second group of bones with the second plurality of bones; An image processing method comprising the steps of:

36. acquiring identification information for identifying a plurality of bones depicted in each of a first image taken of a subject and a second image taken of the subject; classifying the plurality of bones into a first bone group including bones that move in association with a movement of a first part of the subject using the identification information; performing a first registration between a plurality of bones depicted in the first image and a plurality of bones depicted in the second image, the plurality of bones being classified into the first bone group; An image processing method comprising the steps of:

37. A program for causing a computer to execute the image processing method according to claim 35 or 36.

Images