Image processing device, image processing method and program

Abstract

To provide an image processing technology capable of executing positioning between medical image data with a high degree of precision.SOLUTION: An image processing device includes: an acquisition unit for acquiring first medical image data and second medical image data imaging a subject; an intermediate deformation information acquisition unit for acquiring intermediate deformation information, which is a result of applying positioning processing up to a predetermined stage, of first positioning processing composed of a plurality of stages to the acquired first medical image data and second medical image data; a determination unit for determining an abnormality of deformation for the acquired intermediate deformation information; and a deformation unit for executing second positioning processing different from the first positioning processing for the first medical image data and the second medical image data when the determination unit determines that there is an abnormality of deformation, and calculating deformation information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosure of this specification relates to an image processing device, an image processing method, and a program for performing registration between medical image data. [Background technology]

[0002] In the medical field, attempts have been made to visualize changes over time in lesions and other conditions by presenting users with subtraction image data generated from two sets of medical image data acquired at different times using various modalities.

[0003] Non-Patent Document 1 discloses a technique for generating subtraction image data by aligning two sets of three-dimensional medical image data obtained by imaging with a CT device. In addition, alignment between medical image data is utilized in various situations for purposes other than generating subtraction image data, such as superimposing, comparing, and analyzing multiple sets of medical image data. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Ryo Sakamoto, et al. “Temporal Subtraction of Serial CT Images with Large Deformation Diffeomorphic Metric Mapping in the Identification of Bone Metastases”, Radiology, November 2017. Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the alignment technique described in Non-Patent Document 1, when the position or posture of the subject differs significantly between medical image data, it can be difficult to accurately align corresponding anatomical structures between medical image data.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide an image processing technique that can perform registration between medical image data with high accuracy. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, an image processing apparatus according to an embodiment of the present invention includes: an acquisition unit that acquires first medical image data and second medical image data obtained by capturing an image of a subject; a first registration processing unit that performs a first registration process including a plurality of successive processing steps associated with intermediate deformation processing to perform registration between the first medical image data and the second medical image data, updating a displacement field at each processing step and passing the updated displacement field to a next processing step; a second registration processing unit that performs a second registration processing based on information about a predetermined local anatomical structure in the first medical image data and the second medical image data, which is different from the first registration processing; an intermediate deformation information acquisition unit that acquires intermediate deformation information based on the updated displacement field in the plurality of successive processing steps; a determination unit that determines whether or not there is an abnormality in the deformation associated with the alignment in the first alignment process based on the intermediate deformation information, When the determination unit determines that there is a deformation abnormality in the plurality of consecutive processing stages, the first alignment process is stopped and the second alignment process is started based on the first medical image data and the second medical image data, and when the determination unit determines that there is no deformation abnormality in the plurality of consecutive processing stages, the first alignment process is continued to the next processing stage until it reaches the end stage. Furthermore, the image processing method according to the embodiment of the present invention is a method for processing a first image of a subject. 1medical image data and 2 a first registration processing step of performing a first registration processing including a plurality of successive processing steps associated with an intermediate deformation processing to perform registration between the first medical image data and the second medical image data, updating a displacement field for each processing step and passing the updated displacement field to a next processing step; a second registration processing step of performing a second registration processing based on information relating to predetermined local anatomical structures in the first medical image data and the second medical image data, which is different from the first registration processing; and an intermediate deformation information acquisition step of acquiring intermediate deformation information based on the updated displacement field in the plurality of successive processing steps. and a judgment step for judging whether or not there is an abnormality in deformation associated with alignment in the first alignment process based on the intermediate deformation information, wherein if it is judged in the judgment step that there is an abnormality in deformation in the plurality of consecutive processing stages, the first alignment process is stopped and the second alignment process is started based on the first medical image data and the second medical image data, and if it is judged in the judgment step that there is no abnormality in deformation in the plurality of consecutive processing stages, the first alignment process is continued to the next processing stage until it reaches an end stage. [Effects of the Invention]

[0008] According to the disclosure of this specification, registration between medical image data can be performed with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the functional configuration of an image processing system according to a first embodiment. [Figure 2] FIG. 4 is a flowchart showing a procedure for alignment processing in the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of the positional relationship of ribs between images before alignment. [Figure 4]10A to 10C are diagrams illustrating examples of the positional relationship of ribs between images at a predetermined stage of registration. [Figure 5] FIG. 10 is a flowchart showing a registration process procedure according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. [Example]

[0011] First Embodiment The image processing device according to this embodiment is a device that performs registration between two medical image data captured at different times and generates difference image data, etc. Specifically, the device determines whether there is an abnormality in the intermediate deformation information that is the result of applying a first registration process consisting of multiple stages to the two medical image data up to a predetermined stage. If an abnormality is determined to exist, the device applies a second registration process that is different from the first registration process consisting of multiple stages and in which the abnormality is less likely to occur, and performs registration again on the two medical image data. Here, the registration process at a predetermined stage refers to a stage other than the final stage of the multiple stages of the registration process.

[0012] 1 is a diagram showing the configuration of an image processing system according to this embodiment. The image processing system has an image processing device 100, which is connected to a data server 130 via a network 120. The image processing device 100 according to this embodiment performs registration between two sets of medical image data (first medical image data and second medical image data) obtained by imaging a subject at different times. The image processing device 100 is a device that generates differential image data between the first medical image data and the second medical image data based on the registration result.

[0013] The data server 130 stores a plurality of medical image data. The data server 130 represents, for example, a PACS (Picture Archiving and Communication Systems) that receives medical image data captured by a modality and stores and manages it via a network. In the following description, it is assumed that the data server 130 stores a plurality of 3D tomographic image data obtained by capturing images of a subject under different conditions (different modalities, imaging modes, imaging dates and times, body positions, etc.) as first and second medical image data. In this embodiment, it is assumed that the first and second medical image data are 3D tomographic image data (3D medical image data) captured by an X-ray CT scanner.

[0014] In this specification, the axis representing the direction from the subject's right hand to the left hand is defined as the X-axis, the axis representing the direction from the subject's front to back as the Y-axis, and the axis representing the direction from the subject's head to feet as the Z-axis. The X-Y cross section is defined as the axial plane, the Y-Z cross section as the sagittal plane, and the Z-X cross section as the coronal plane. That is, the X-axis direction is the direction perpendicular to the sagittal plane (hereinafter referred to as the sagittal direction). The Y-axis direction is the direction perpendicular to the coronal plane (hereinafter referred to as the coronal direction). The Z-axis direction is the direction perpendicular to the axial plane (hereinafter referred to as the axial direction). In the case of CT image data configured as a collection of two-dimensional tomographic image data (slice images), the slice plane of the image represents the axial plane, and the direction perpendicular to the slice plane (hereinafter referred to as the slice direction) represents the axial direction. Note that the coordinate system is merely an example, and other definitions may be used.

[0015] The modality for capturing the 3D tomographic image may be an MRI apparatus, a 3D ultrasound imaging apparatus, a photoacoustic tomography apparatus, a PET / SPECT apparatus, an OCT apparatus, or the like. Furthermore, the first medical image data and the second medical image data may be any medical image data as long as they are 3D tomographic image data to be aligned. For example, they may be images captured simultaneously using different modalities or different imaging modes. Furthermore, they may be medical image data captured on the same subject in the same position using the same modality at different dates and times for follow-up observation. The first medical image data and the second medical image data are 3D medical image data (3D tomographic image data) configured as a collection of 2D tomographic image data. The position and orientation of each 2D tomographic image data are converted into a reference coordinate system (a coordinate system in space based on the subject) and stored in the data server 130. At this time, the first medical image data and the second medical image data expressed in the reference coordinate system are input to the image processing device 100 in response to an instruction from a user operating the instruction unit 140. The instruction unit 140 includes, for example, a mouse, a keyboard, and buttons.

[0016] The image processing device 100 is a device that receives a processing request from a user via the instruction unit 140, performs image processing, and outputs the processing results to the display unit 150, and functions as a terminal device for image interpretation operated by a user such as a doctor. Specifically, based on an instruction from the user via the instruction unit 140, the image processing device 100 acquires first and second medical image data to be subjected to image processing as a pair of medical image data from the data server 130. Then, the image processing device 100 performs registration processing on the pair of medical image data, and generates differential image data between the first and second medical image data based on the registration results and outputs the data to the display unit 150.

[0017] The image processing device 100 includes the components described below.

[0018] The acquisition unit 101 acquires information on the first medical image data and the second medical image data input to the image processing device 100.

[0019] The deformation unit 102 includes a first alignment unit 103 and a second alignment unit 104, and performs alignment processing on the medical image data acquired by the acquisition unit 101 to calculate deformation information. The first alignment unit 103 constituting the deformation unit 102 performs alignment processing between the first medical image data and the second medical image data using a first alignment method consisting of multiple stages. The first alignment unit 103 then transmits intermediate deformation information representing a predetermined stage of the alignment processing to the intermediate deformation information acquisition unit 105. The alignment processing at a predetermined stage refers to alignment processing that is not the final stage among the multiple stages.

[0020] The intermediate deformation information acquisition unit 105 transmits intermediate deformation information, which is the result of applying the alignment process up to a predetermined stage of the first alignment process consisting of multiple stages from the deformation unit 102, to the determination unit 106. Here, the intermediate deformation information is a displacement field that stores displacement vectors that associate pixels that make up the first medical image data and the second medical image data.

[0021] The determination unit 106 determines whether there is an abnormality in the deformation based on the acquired intermediate deformation information, and transmits the determination result to the deformation unit 102. For example, the determination unit 106 determines whether there is an abnormality in the alignment based on the displacement field corresponding to the intermediate deformation information, and transmits the determination result to the deformation unit 102.

[0022] If the determination unit 106 determines that there is no abnormality in the intermediate deformation information, the deformation unit 102 causes the first alignment unit 103 to perform alignment processing from a predetermined stage onward that constitutes the first alignment processing based on the intermediate deformation information. On the other hand, if the determination unit 106 determines that there is an abnormality in the intermediate deformation information, the deformation unit 102 causes the second alignment unit 104 to perform second alignment processing between the first medical image data and the second medical image data. Furthermore, the deformation unit 102 does not perform alignment processing from a predetermined stage onward in the first alignment unit 103.

[0023] The second alignment unit 104 performs alignment processing between the first medical image data and the second medical image data using a second alignment method, and calculates a displacement field between the processed medical image data. The second alignment processing in the second alignment unit 104 performs alignment processing without inputting intermediate deformation information from the first alignment processing.

[0024] The image data generation unit 107 generates deformed image data of the second medical image data by deforming the second medical image data to match the first medical image data based on the acquired displacement field. The image data generation unit 107 also generates differential image data between the first medical image data and the deformed image data of the second medical image data as resultant image data.

[0025] The display control unit 108 controls the output of the generated differential image data, transformed image data, etc. to the display unit 150.

[0026] The display unit 150 is configured with any display device such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube), and displays medical image data and the like for a doctor to interpret. Specifically, it displays cross-sectional image data of the first medical image data and the second medical image data acquired from the image processing device 100. It also displays cross-sectional image data of the deformed image data and cross-sectional image data of the differential image data generated by the image processing device 100. The medical image data displayed by the display unit 150 can be combined or changed as appropriate. The display unit 150 may also be configured as a device integrated with the image processing device 100.

[0027] In addition, in this embodiment, alignment between medical image data by the image processing device 100 refers to a process of calculating deformation information for displacing the pixel positions of pixels constituting one medical image data to the corresponding pixel positions of the other medical image data.

[0028] Here, we will explain an example in which there are two sets of medical image data captured at different times for a subject. Among the pixels constituting the second medical image data, the pixel positions corresponding to the pixel positions of the first pixels are estimated relative to the pixel positions of the pixels constituting the first medical image data, which serves as reference. The image processing device 100 then calculates the displacement field from one set of medical image data to the other set of medical image data as deformation information. Here, the displacement field between the three-dimensional tomographic image data is a three-dimensional vector field that stores displacement vectors in the X, Y, and Z directions at the pixel positions of the pixels constituting the medical image data. Furthermore, the medical image data used as the reference for registration (image data to be fixed) is referred to as reference image data, and the other medical image data (image data to be deformed) is referred to as floating image data. In this embodiment, the first medical image data is treated as reference image data, and the second medical image data is treated as floating image data.

[0029] 2 is a flowchart showing the processing procedure for aligning medical image data performed by the image processing device 100. Note that the steps described below may be realized as an image processing system consisting of multiple image processing devices. Alternatively, the image processing device 100 may perform a specific step and transmit the results of that step to another image processing device.

[0030] (S1010) (Acquire two medical image data) In step S1010, the acquisition unit 101 acquires the first medical image data and the second medical image data specified by the user through the instruction unit 140 from the data server 130. Then, the acquired first medical image data and the second medical image data are output to the first alignment unit 103 of the deformation unit 102 and the display control unit 108. Note that acquisition of the medical image data by the acquisition unit 101 is not limited to acquisition based on an instruction from the user, and may be performed by any other method.

[0031] (S1020) (The first alignment process is performed up to a predetermined stage) In step S1020, the first alignment unit 103 of the deformation unit 102 performs a first alignment process to align the first medical image data and the second medical image data so that pixels representing the same part of the image substantially coincide with each other. The first alignment unit 103 performs alignment up to a predetermined stage of the first alignment process, which is composed of multiple stages, and transmits the alignment result to the intermediate deformation information acquisition unit 105. The intermediate deformation information acquisition unit 105 acquires intermediate deformation information, which is deformation information at the predetermined stage, and transmits it to the determination unit 106. In this embodiment, the intermediate deformation information acquisition unit 105 acquires, as intermediate deformation information from the first alignment unit 103, a displacement field that associates the positions of pixels constituting both images at the predetermined stage of alignment. The predetermined stage alignment process refers to alignment that is not the final stage of the multiple stages of alignment process.

[0032] In this embodiment, the first registration process in the first registration unit 103 is performed using a known image processing method that uniformly evaluates the entire medical image data. For example, a method is used in which one image is deformed so as to increase the image similarity between the medical image data. As the image similarity, known methods such as the commonly used Sum of Squared Difference (SSD), mutual information, and cross-correlation coefficient can be used. Furthermore, as the image data deformation model, known models such as affine transformation, free form deformation (FFD), Demons algorithm, and large deformation diffeomorphic metric mapping (LDDMM) can be used.

[0033] In this embodiment, the first alignment performed by the first alignment unit 103 is a process consisting of multiple stages. For example, in order to increase the robustness of the alignment, the first alignment unit 103 performs a multi-resolution alignment process that starts with alignment between input image data with a coarse resolution and progresses to alignment between input image data with a finer resolution.

[0034] The multi-resolution alignment performed by the first alignment unit 103 involves calculating a displacement field resulting from the alignment process at each resolution and then applying the displacement field to the input image at the next resolution. This allows the first alignment unit 103 to output a displacement field for each resolution. The intermediate deformation information acquisition unit 105 can then use a predetermined resolution level in the multi-resolution alignment as the predetermined level of the first alignment and acquire the displacement field resulting from the alignment at the predetermined level of resolution as intermediate deformation information. For example, in the case of five-level multi-resolution alignment, the intermediate deformation information acquisition unit 105 can acquire the displacement field resulting from the alignment at the third resolution level as the predetermined level of resolution as intermediate deformation information.

[0035] The first registration process in the first registration processor 103 does not necessarily have to be a multi-step process of multi-resolution registration. The first registration process in the first registration processor 103 may be a process that sequentially combines different types of registration methods. For example, if the first registration processor 103 performs registration in the order of affine registration, the above-mentioned FFD, and LDDMM, and a predetermined stage is the second stage, the intermediate deformation information acquisition unit 105 can acquire the displacement field by FFD as intermediate deformation information. Furthermore, the first registration process in the first registration processor 103, which consists of multiple stages, may be an iterative calculation in an optimization process. Generally, a registration method that deforms medical image data to increase the image similarity between medical image data uses image similarity as an evaluation function and performs iterative calculations to minimize (or maximize) it. Since a displacement field can be generated at each iteration, the intermediate deformation information acquisition unit 105 can acquire the displacement field at a predetermined stage of the iterative calculation as intermediate deformation information. For example, the intermediate deformation information acquisition unit 105 can acquire the displacement field at the 50th step of the iterative calculation as the intermediate deformation information.

[0036] The intermediate deformation information calculated by the first registration unit 103 is not necessarily limited to a displacement field, but may be any information that allows for positional correspondence between medical image data. For example, the intermediate deformation information calculated by the first registration process in the first registration unit 103 may be a combination of a deformation model based on a B-Spline curve used in FFD and the positions and control amounts of control points arranged at predetermined intervals in the medical image data. In this case, the intermediate deformation information acquisition unit 105 can acquire the displacement of any position in the medical image data using a B-Spline curve based on the positions and control amounts of the control points.

[0037] (S1030) (Determination of deformation abnormality based on intermediate deformation information) In step S1030, the determination unit 106 detects regions in the intermediate deformation information where deformation is locally abnormal. That is, the determination unit 106 determines whether or not there is an abnormality in a local region in the displacement field. For example, if the intermediate deformation information is a displacement field, the determination unit 106 generates a Jacobian map that indicates volumetric changes in the displacement field and detects regions where the volume has changed significantly locally. Here, the Jacobian J(x,y,z) at a given position (x,y,z) in the Jacobian map of the displacement field can be calculated using the following equation, where Fx(x,y,z), Fy(x,y,z), and Fz(x,y,z) are the displacement fields in the X, Y, and Z directions, respectively. Note that in the following equations, the input coordinates (x,y,z) are omitted.

[0038]

number

[0039] At this time, Jacobian J takes a value of 1 if there is no local change in volume, a value smaller than 1 (a positive decimal) if there is a decrease in volume, and a value larger than 1 if there is an increase in volume. Therefore, in this step, the determination unit 106 determines an area where the deformation is abnormal by threshold determination based on the value of Jacobian J. Specifically, the determination unit 106 determines that the area is an area where the volume change is abnormal, that is, an area where the deformation is abnormal, if the condition of the following mathematical formula is satisfied. |log2J|>Th1(2)

[0040] Here, the determination unit 106 takes the logarithm of the Jacobian J to convert a volume that does not change to 0, a volume that decreases to a negative value, and a volume that increases to a positive value. Furthermore, the magnifications that are inversely related to a volume decrease and an increase can be treated as positive and negative values ​​of equal magnitude. Specifically, taking log2 of a volume decrease magnification of 0.5 and an increase magnification of 2.0, which are inversely related to each other, results in values ​​of −1 and +1, respectively, which are positive and negative values ​​of equal magnitude. Then, by taking the absolute value of these values, the magnifications of a volume decrease and an increase can be converted to values ​​of the same scale. Here, Th1 represents a predetermined threshold. For example, if Th1 = 0.5, then, according to equation (2), if the local volume change does not fall within the range of approximately 0.7 to approximately 1.4, the determination unit 106 determines that the volume change is abnormal. That is, the determination unit 106 determines whether or not there is an abnormality in the deformation based on the volume change information in the displacement field.

[0041] The following describes specific examples of target areas for registration on medical image data and deformation abnormalities that occur in intermediate deformation information, using the drawings. Figures 3 and 4 show examples of the positional relationship of ribs between medical image data before and at a specific stage of registration, respectively. In Figure 3, the shapes drawn with solid lines represent the shapes of ribs and spine on medical image data when the first medical image data, which is reference image data, is viewed from the side of the subject before registration. R1, R2, R3, R4, and R5 represent individual ribs, and RB represents the spine. Also, in Figure 3, the shapes drawn with dotted lines represent the shapes of ribs and spine when the second medical image data, which is floating image data, is viewed from the side before registration. F1, F2, F3, F4, and F5 represent individual ribs corresponding to the ribs R1, R2, R3, R4, and R5 on the first medical image data, respectively, and FB represents the spine. As can be seen from Figure 3, before alignment, at the tips of the ribs between the medical image data, instead of corresponding ribs, ribs that are shifted by one rib are located close to each other, such as R2 and F1, R3 and F2, R4 and F3, and R5 and F4. This is a phenomenon that occurs when the posture, breathing state, and physique of the subject change between the medical image data.

[0042] Next, in Figure 4, the shapes drawn with solid lines represent ribs on the first medical image data, which is reference image data, at a predetermined stage of the first registration process performed by the first registration unit 103, with R1 to R5 representing the same as those in Figure 3. Also, in Figure 4, the shapes drawn with dotted lines represent ribs on the second medical image data, which is floating image data, at a predetermined stage of the registration process, with F1 to F5 representing the same as those in Figure 3. Figure 4 shows the result of the first registration unit 103 attempting to align ribs that are close to each other during the registration process at a predetermined stage of the first registration. In this example, at the bases of the ribs near the spine RB and FB, corresponding ribs R1 and F1, R2 and F2, R3 and F3, and R4 and F4 are aligned to close positions. On the other hand, at the tip portions of the ribs far from the spine RB and FB, it can be seen that ribs R2 and F1, R3 and F2, R4 and F3, and R5 and F4, which are misaligned by one position, are incorrectly aligned close to each other. In other words, each rib in the floating image data has been transferred to a different rib. This reveals that localized irregular deformations have occurred in the areas E1, E2, E3, and E4 between the base and tip of ribs F1, F2, F3, and F4 in the second medical image data, which is floating image data. Areas with such localized irregular deformations result in localized abnormal volume changes. More specifically, in E1, E2, E3, and E4, the ribs have been locally elongated, resulting in abnormal volume changes in the increasing direction. Such areas can be detected using the above-mentioned formula (2). The determination unit 106 may also determine whether deformation is abnormal for the entire intermediate deformation information. Furthermore, the method by which the determination unit 106 detects regions where deformation in the intermediate deformation information is abnormal is not limited to the above-described method. For example, the determination unit 106 may detect, as deformation abnormalities, a location where the displacement in a predetermined direction in a local region on the displacement field changes drastically. A specific example is shown below. As shown in FIG. 4, each rib on the floating image data is transferred to a different corresponding rib in the vertical direction (Z direction, axial direction) of the subject at an intermediate stage. This reveals that the rib has stretched in the Z direction.Therefore, the determination unit 106 detects areas on the medical image data where the displacement in the Z direction changes drastically as areas where the deformation is abnormal. The determination unit 106 first sets a predetermined range (e.g., a radius of 10 mm) for a predetermined position (x, y, z) on the displacement field, and then determines the maximum value dz of the displacement in the Z direction within that range. max and the minimum value dz min (each unit is mm). Then, the determination unit 106 calculates the maximum value dz of the displacement in the Z direction. max and the minimum value dz min Specifically, when the condition of the following formula is satisfied, the area is determined to be an area where deformation is abnormal. dz max -dz min >Th2(3)

[0043] At this time, dz max and dz min By calculating the difference between these values, the determination unit 106 can detect changes in Z-direction displacement within a predetermined range. Th2 represents a predetermined threshold. For example, if Th2 = 20, then, according to equation (3), if the difference between the maximum and minimum Z-direction displacements in a local region exceeds 20 mm, the determination unit 106 determines that the Z-direction displacement has changed drastically. This allows regions where ribs are stretched in the Z direction, such as E1, E2, E3, and E4 in Figure 4, to be detected using equation (3). That is, the determination unit 106 determines whether or not there is an abnormality in deformation based on the difference between the maximum and minimum displacement values ​​in the displacement field. Note that this method used by the determination unit 106 is not limited to the Z direction; it can also be applied to displacements in other directions, or by combining displacements in multiple directions. By utilizing this characteristic that abnormal deformation is likely to occur in the Z direction when rib transfers occur between images, the determination unit 106 can perform abnormality determination specialized for rib transfers.

[0044] The method for detecting an abnormally deformed area in the intermediate deformation information by the determination unit 106 may be a combination of the above methods. For example, the determination unit 106 may detect a candidate area for the abnormally deformed area using equation (2), set a predetermined range around the candidate area, and perform a determination for the range using equation (3) to ultimately determine an abnormally deformed area.

[0045] Furthermore, the determination unit 106 may set different thresholds (Th1 and Th2 above) for determining abnormal deformation for each part to which the pixel of interest belongs. For example, the determination unit 106 can reduce false detections by setting a higher threshold for parts that are inherently more susceptible to deformation and a lower threshold for parts that are less susceptible to deformation. Parts can be classified into, for example, "bones and non-bone" or "bones, chest, and abdomen." Here, the parts can be recognized using known image recognition technology.

[0046] The method for detecting an abnormal deformation region in the intermediate deformation information in the determination unit 106 may be any known method for detecting abnormal deformation. In this way, the determination unit 106 can appropriately determine an abnormality for each part by changing the abnormality determination threshold in consideration of the characteristics of ease of deformation for each part.

[0047] (S1040) (No abnormalities?) In step S1040, the determination unit 106 determines whether the alignment is proceeding normally at a predetermined stage based on the result of step S1030. The determination unit 106 makes this determination, for example, based on the result of step S1030, based on whether or not there are any local regions with abnormal deformation throughout the entire medical image data. If the determination unit 106 determines that the alignment is proceeding normally, the process proceeds to step S1050. On the other hand, if the determination unit 106 determines that the alignment is not proceeding normally, the process proceeds to step S1060. The determination unit 106 may also determine whether the alignment is proceeding normally by providing a certain dead zone, such as determining that an abnormality exists when the volume of an abnormal local region is equal to or greater than a predetermined value. The determination unit 106 may also use a method other than determining the entire medical image data. The determination may be limited to a predetermined region of interest, or may be performed by masking an area to be excluded. For example, when only the alignment of bones between medical image data is of interest, regions other than bones may be excluded (masked) from the region for abnormality determination.

[0048] (S1050) (Restart the first alignment from a predetermined stage) In step S1050, if the determination unit 106 determines that there is no deformation abnormality, the deformation unit 102 resumes the alignment process from a predetermined stage onward, which is the result of the alignment process up to the predetermined stage of the first alignment process, which is composed of multiple stages, based on the intermediate deformation information generated in step S1030 by the first alignment unit 104. The first alignment unit 104 then acquires, as first deformation information, deformation information resulting from execution of the first alignment process up to the final step. The first alignment unit 104 in the deformation unit 102 then outputs the first deformation information to the image data generation unit 107 as final deformation information. In this embodiment, as in step S1030, a displacement field that associates positions between medical image data is acquired as the final deformation information.

[0049] Here, specific processing of this step will be described. However, the specific processing method of the first alignment in the first alignment unit 103 is the same as that in step S1020, so description thereof will be omitted. If the first alignment processing in the first alignment unit 103 is a five-level multi-resolution alignment processing and the first alignment unit 103 has executed up to the third level of resolution in step S1020, the following processing is performed. That is, the first alignment unit 103 uses the intermediate deformation information obtained at the third level of resolution as input and executes alignment processing for the remaining fourth and fifth level of resolution. Also, if the first alignment processing is a processing in which different alignment methods are combined in sequence (affine alignment, FFD, LDDMM) and has executed up to the second level of processing in step S1020, the following processing is performed. That is, the first alignment unit 103 uses the intermediate deformation information obtained in the second level of FFD as input and executes alignment processing for the remaining LDDMM.

[0050] (S1060) (Run the second alignment from the beginning) In step S1060, the second alignment unit 104 performs a second alignment process, which is different from the alignment process constituting the first alignment process, from the beginning, and acquires deformation information resulting from the second alignment process as second deformation information.The second deformation information is then output to the image data generation unit 107 as final deformation information, which is the final deformation information.In this embodiment, as in step S1030, a displacement field that associates pixel positions between medical image data is acquired as the final deformation information.

[0051] Here, the second registration performed by the second registration unit 104 is a method that is less likely to cause the abnormal deformation detected in step S1030 than the first registration. Specifically, the second registration process performed by the second registration unit 104 is a registration method based on information about specific local anatomical structures (hereinafter referred to as local anatomical structures) of the subject on the medical image data. The local anatomical structures are, for example, individual bones (such as the spine, ribs, skull, and pelvis). Note that anatomical structures other than bones may also be used. In this case, the information about the local anatomical structures of the subject on which the second registration process performed by the second registration unit 104 is based includes information about the local region in which the abnormal deformation was detected in step S1030. For example, if the local anatomical structures are ribs, the information includes the region in which the irregular deformation occurs due to rib relocation as shown in FIG. 4.

[0052] In this case, the second registration performed by the second registration unit 104 extracts and recognizes, from each medical image data, feature points of local anatomical structures, including the local region of the subject where abnormal deformation was detected in step S1030. The second registration unit 104 then performs registration processing so that the positions of the recognized local anatomical structures match between the medical image data. More specifically, when the determination unit 106 detects abnormal rib deformation as shown in FIG. 4, the second registration unit 104 extracts and recognizes feature points, such as the tip, center, and base, for each individual rib of the subject. Then, registration processing is performed so that the feature points of corresponding ribs match. Here, the extraction and identification of feature points of individual rib regions by the second registration unit 104 can be performed using known image processing-based, machine learning-based, or atlas-based region extraction or feature point extraction techniques. Furthermore, a registration method for matching corresponding feature points between medical image data is preferably a method for matching not only corresponding feature points but also the positions of the entire medical image data. As a method for this, a method that incorporates not only image similarity but also the degree of matching of feature points into the evaluation function of the above-mentioned FFD, Demons algorithm, or LDDMM can be applied. Alternatively, the second registration unit 104 may first perform registration to match feature points using a TPS (Thin Plate Spline) method or the like, and then perform registration based on the above-mentioned image similarity.

[0053] Furthermore, as in the above example, it is desirable that the second registration unit 104 use multiple pieces of information about local anatomical structures in its second registration process. This is because, as in the above example, by individually recognizing multiple ribs adjacent in the vertical direction of the subject, it is possible to register them correctly. Furthermore, the information about local anatomical structures used in the second registration process in the second registration unit 104 does not necessarily have to be feature points; information about the area or shape of the local anatomical structures may also be used. In this case, the second registration unit 104 may, for example, first sample points on the area or shape of the local anatomical structures and perform point cloud correspondence using a known point cloud correspondence technique. Then, similar to the method of matching corresponding feature points described above in the second registration unit 104, registration can be performed by incorporating image similarity and the degree of match between feature points into an evaluation function.

[0054] The second registration process performed by the second registration unit 104 is based on information about specific local anatomical structures of the subject. Therefore, it achieves higher registration performance for specific local anatomical structures than the first registration process, which uniformly treats all medical image data without using information about specific local anatomical structures. However, because the second registration process requires special processing based on the subject's local anatomical structures, it takes longer than the first registration process. Alternatively, it is highly likely that the second registration process cannot be applied to a wide variety of medical image data, including images of different subject regions. However, the second registration process performed by the second registration unit 104 is performed only when an abnormality in deformation is detected in step S1030. In this embodiment as a whole, the deformation unit 102 applies the first registration process to the end of processing in many cases of various variations in medical image data, and applies the second registration process to cases in which an abnormality in deformation is detected. This reduces the disadvantages of the second registration process, such as long processing time or limited versatility. Furthermore, the advantage of the second registration process, i.e., the high accuracy of registration including specific topographical structures, can be utilized to cover the failure of the first registration. If the abnormal deformation detection in step S1030 is specialized for detecting abnormal deformation in specific topographical structures, the second registration process in the second registration unit 104 can be set in advance to a registration process specialized for preventing abnormal deformation in the topographical structures. For example, if the abnormal deformation detection is specialized for detecting the above-mentioned rib transitions, the second registration process can be set to a registration process that uses only the feature points and shapes of the ribs, as described above, thereby making it difficult for abnormal deformation in the ribs to occur. On the other hand, if the abnormal deformation detection in step S1030 is performed appropriately for each region, a registration process that is less likely to cause abnormal deformation can be set according to the region (topographical structure). This allows the region in which an abnormality is detected to be identified and the registration process appropriate for the region to be selected.For example, if a deformation abnormality is identified in the ribs, the feature points and shape of the ribs are used to perform alignment, and if a deformation abnormality is identified in the spine, the feature points and shape of the spine are used to perform alignment. If abnormalities are identified in both, the feature points and shapes of both are used to perform alignment. In this way, appropriate alignment processing can be performed depending on the location where the abnormality is identified. In this case, the location where the abnormality is identified and the information on the location used for alignment do not necessarily have to correspond one-to-one, as long as the information on the location used for alignment includes at least the location where the abnormality is identified. For example, if an abnormality is identified in the ribs, alignment can be performed using the feature points and shapes of multiple locations including the ribs.

[0055] (S1070) (Generate result image data) In step S1070, the image data generation unit 107 generates resultant image data based on the final transformation information acquired by the processing of step S1050 or step S1060. That is, the image data generation unit 107 generates deformed image data of the second medical image by transforming the second medical image data so that it matches the first medical image data based on the acquired final transformation information. The image data generation unit 107 also generates difference image data by subtracting the deformed image data of the second medical image data from the first medical image data. The generated resultant image data is then output to the display control unit 106.

[0056] In this embodiment, the generated result image data is stored in a storage unit (not shown). As a result, when the result image data is to be acquired again after the processing of the image processing device 100 is completed, the result image data can be easily acquired by reading the stored result image data. However, it is not necessary to store the generated result image data in a storage unit (not shown).

[0057] Furthermore, it is not necessary to generate both the deformed image data and the differential image data of the second medical image data as the resultant image data, and it is also possible to generate only one of them.

[0058] (S1080) (display image data) In step S1080, the display control unit 106 controls the display of the cross-sectional image data of the resultant image data acquired from the image data generation unit 105 on the display unit 150. The display control unit 106 also controls the display of the cross-sectional image data of the first medical image data and the second medical image data on the display unit 150.

[0059] In this manner, processing is performed by the image processing device 100. That is, the image processing device 100 includes an intermediate deformation information acquisition unit 105 that acquires intermediate deformation information that is a result of applying the alignment process up to a predetermined stage of the first alignment process, which is composed of multiple stages, to the first medical image data and second medical image data acquired by the acquisition unit 101.

[0060] The image processing device 100 further includes a determination unit 106 that determines whether there is any deformation abnormality in the intermediate deformation information, and a transformation unit 102 that performs a second alignment process that is at least partially different from the first alignment process based on the determination result, and calculates deformation information. With this configuration of the image processing device 10, according to this embodiment, the determination unit 106 detects deformation abnormalities in the first alignment, and the transformation unit 102 performs alignment in the second alignment that is less likely to cause such abnormalities, thereby reducing alignment failures. Furthermore, the determination unit 106 detects an abnormality in the intermediate deformation information acquired by the intermediate deformation information acquisition unit 103, and the transformation unit 102 immediately performs the second alignment, thereby preventing unnecessary increases in processing time even when the determination unit 106 detects a deformation abnormality.

[0061] It is not necessary to display or generate deformed image data or differential image data based on the final deformation information; the acquired deformation information may simply be stored in a storage unit. Alternatively, the medical image data may be subjected to any processing or analysis using the correspondence between the coordinates of the first and second medical image data obtained from the acquired deformation information. In other words, any configuration may be used as long as the registration results of the first and second medical image data are used.

[0062] (Variation 1) In the first embodiment, in step S1060, the second registration is performed using clear position information, such as feature points, regions, and shapes, of specific local anatomical structures in which deformation abnormalities are detected in the medical image data. However, a method that clearly identifies the position of the local anatomical structure is not necessarily used. For example, assume that the local anatomical structure in which deformation abnormalities are detected is a rib, as shown in FIG. 4. In this case, when the subject is viewed from the front, the characteristic bones of the skull, clavicle, spine, and pelvis are arranged in order from top to bottom, making it relatively easy to register the spinal region between images. Therefore, assuming that the spinal region can be registered with high accuracy between medical image data, the second registration unit 104 initially performs registration between medical image data by focusing on the spinal region. Then, a method of gradually expanding the target region from the base of the rib toward the tip of the rib and registering the target region may be used as the second registration.

[0063] More specifically, when viewing the axial plane of the 3D medical image data, the second registration unit 104 radially divides the image region of the medical image data into multiple regions based on the center point of the subject. The second registration unit 104 then performs initial registration on only the divided region including the spine, and then adds adjacent divided regions to the region and performs registration again. This process of adding adjacent divided regions and performing registration again is repeated until the target region covers the entire medical image data. For example, if the medical image data is radially divided into 12 regions based on the center point of the subject when viewed on the axial plane, the second registration unit 104 selects the two divided regions including the spine as the initial target regions for registration. The second registration unit 104 then performs registration by simultaneously increasing the number of adjacent divided regions, one on each side, from the dorsal side to the ventral side, and finally aligns the entire image, including the two ventral divided regions. This allows the target region to be gradually expanded from the spine toward the tips of the ribs while being registered. Therefore, even if the correspondence between ribs before alignment is shifted one by one as shown in Figure 3, in the first half of the second alignment process, the correspondence between rib tips is ignored and alignment is performed sequentially from the base of the rib. This prevents alignment failures such as those shown in Figure 4, where a rib tip erroneously corresponds to (transfers to) another rib.

[0064] (Variation 2) In the first embodiment, the second alignment performed by the second alignment unit 104 in step S1060 involves re-executing alignment from the beginning. However, this is not necessarily a method of re-executing alignment from the beginning. For example, consider a case where the first alignment process is composed of a rough initial alignment process in the first stage and a fine alignment process in the second stage, and the rough initial alignment process is rigid alignment or affine alignment between image data. In this case, local deformation abnormalities do not generally occur in the rough initial alignment process. Therefore, in the second alignment process performed by the second alignment unit 104, the rough initial alignment process is considered to be a common process with the first alignment, and deformation information from the rough initial alignment stage in the first alignment is acquired as initial deformation information. Then, the second alignment unit 104 performs alignment using the method described in step S1060, using the initial deformation information as an initial value. This allows the second alignment unit 104 to start the second alignment when the medical image data are aligned to a certain extent, compared to when the second alignment is performed from scratch, and therefore allows the alignment to be completed in a shorter time.

[0065] <Second embodiment> In the first embodiment, the determination unit 106 determines whether there is an abnormality in the deformation in the first alignment processing once at a predetermined stage. On the other hand, in the present embodiment, the determination unit 106 determines whether there is an abnormality for each stage of the alignment processing in the first alignment processing. The processing of this embodiment will be described below. Note that the configuration of the image processing system according to this embodiment is the same as that shown in FIG. 1, and therefore a description thereof will be omitted. Furthermore, steps S2010, S2030, and S2060 to S2080 in the flowchart showing the overall processing procedure performed by the image processing device 100 are the same as steps S1010, S1030, and S1060 to S1080 in the first embodiment, and therefore a description thereof will be omitted. Only the differences from the first embodiment in the flowchart of FIG. 2 will be described below.

[0066] (S2020) (Perform one step of the first alignment) In step S2020, the first alignment unit 103 performs one stage of a first alignment process to align the first medical image data and the second medical image data so that pixels representing the same region substantially coincide with each other, and calculates intermediate deformation information. The intermediate deformation information acquisition unit 105 acquires the intermediate deformation information calculated by the first alignment unit 103 and transmits it to the determination unit 106. The first alignment unit 103 stores the intermediate deformation information obtained in this alignment process to use it as an initial value for alignment in the next alignment. In this embodiment, as in the first embodiment, a displacement field between the medical image data is calculated as intermediate deformation information. Furthermore, the first alignment performed by the first alignment unit 103 in this embodiment is the same technique as in the first embodiment, and therefore a description thereof will be omitted. Furthermore, as in the first embodiment, the first alignment performed by the first alignment unit 103 in this embodiment is a process consisting of multiple stages. Details of the process consisting of multiple stages are the same as in the first embodiment, and therefore a description thereof will be omitted. In the first embodiment, the intermediate deformation information acquisition unit 105 acquires, as intermediate deformation information, the displacement field obtained when the alignment process for a predetermined stage among the multiple stages is executed from the beginning to the predetermined stage. On the other hand, in the present embodiment, the intermediate deformation information acquisition unit 105 acquires, as intermediate deformation information, the displacement field obtained when the alignment process for one stage among the multiple stages is executed.

[0067] Here, when the first alignment processing in the first alignment processing unit 103 is a multi-resolution processing, one resolution or a predetermined number of resolutions (e.g., two levels of resolution) is considered to be one level of alignment processing. Also, when the first alignment processing is configured by a combination of different types of alignment methods (e.g., affine transformation, FFD, LDDMM), the unit of each individual alignment method is considered to be one level of processing. Also, when the first alignment processing is a process of repeated calculation in optimization processing, a predetermined number of steps (e.g., 10 steps) is considered to be one level.

[0068] In the first registration unit 103, when the first stage of the first registration process is currently being performed, the first stage of the registration process is executed. On the other hand, when at least one stage of the registration process constituting the first registration process has already been executed, the first registration unit 103 executes one stage of the first registration process using the stored displacement field obtained in the previous registration as an initial value.

[0069] (S2040) (No abnormalities?) In step S2040, the determination unit 106 determines, based on the result of step S2030, whether the alignment is progressing normally in the intermediate deformation information, which is the alignment processing result of a predetermined stage of the first alignment processing consisting of multiple stages. Here, the method of determination by the determination unit 106 is the same as in the first embodiment. Then, if it is determined that the alignment is progressing normally, the process proceeds to step S2050. On the other hand, if it is determined that the alignment is not progressing normally, the process proceeds to step S2060.

[0070] (S2050) (First alignment completed?) In step S2050, first alignment unit 103 determines whether to end the first alignment. If it determines that it should end, it outputs the currently stored intermediate deformation information to image data generation unit 107 as final deformation information, and proceeds to step S2070. If it determines that it should not end, it proceeds to step S2020.

[0071] In this way, the processing of the image processing device 100 is carried out.

[0072] According to this embodiment, deformation abnormalities are determined at each stage of the first alignment process, so the determination unit 106 can find deformation abnormalities at an earlier stage than in the first embodiment. Furthermore, this configuration allows recovery through the second alignment process by the second alignment unit 104 in the deformation unit 102. Therefore, the processing time when a deformation abnormality is found can be shorter than in the first embodiment.

[0073] <Other embodiments> Furthermore, the technology disclosed in this specification can be embodied as, for example, a system, a device, a method, a program, or a recording medium (storage medium), etc. Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or it may be applied to an apparatus consisting of a single device.

[0074] Needless to say, the object of the technology disclosed in this specification can be achieved by the following: A recording medium (or storage medium) on which software program code (computer program) that realizes the functions of the above-described embodiments is recorded is supplied to a system or device. The storage medium is, of course, a computer-readable storage medium. The computer (or CPU or MPU) of the system or device then reads and executes the program code stored on the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiments, and the recording medium on which the program code is recorded constitutes the technology disclosed in this specification. [Explanation of symbols]

[0075] 101 Acquisition Department 102 Deformed part 103 First alignment unit 104 Second alignment part 105 Intermediate deformation information acquisition unit 106 Judgment section

Claims

1. an acquisition unit that acquires first medical image data and second medical image data obtained by imaging a subject; a first registration processing unit that performs a first registration process including a plurality of successive processing steps associated with intermediate deformation processing to perform registration between the first medical image data and the second medical image data, updating a displacement field at each processing step and passing the updated displacement field to a next processing step; a second registration processing unit that performs a second registration processing based on information about a predetermined local anatomical structure in the first medical image data and the second medical image data, which is different from the first registration processing; an intermediate deformation information acquisition unit that acquires intermediate deformation information based on the updated displacement field in the plurality of successive processing steps; a determination unit that determines whether or not there is an abnormality in the deformation associated with the alignment in the first alignment process based on the intermediate deformation information, When the determination unit determines that there is an abnormality in deformation in the plurality of consecutive processing stages, the first registration process is stopped, and the second registration process is started based on the first medical image data and the second medical image data, and when the determination unit determines that there is no abnormality in deformation in the plurality of consecutive processing stages, the first registration process is continued to a next processing stage until it reaches an end stage. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the second alignment process is performed without using the intermediate deformation information from the first alignment process as an input.

3. The intermediate deformation information is a displacement field storing displacement vectors that correspond between pixels constituting the first medical image data and the second medical image data, The image processing apparatus according to claim 1 , wherein the determining unit determines whether the alignment is abnormal with respect to the displacement field.

4. The image processing device according to claim 3 , wherein the determining unit determines whether or not there is an abnormality in the deformation based on information about a volume change in the displacement field.

5. The image processing device according to claim 3 , wherein the determining unit determines whether or not there is an abnormality in the deformation based on a difference between a maximum value and a minimum value of the displacement in the displacement field.

6. 6. The image processing apparatus according to claim 3, wherein the determination unit determines the presence or absence of the abnormality by threshold determination.

7. The image processing device according to claim 3 , wherein the determining unit determines whether or not there is an abnormality in the deformation for a local region in the displacement field.

8. The image processing device according to claim 7 , wherein the determination unit determines that the deformation is abnormal when the volume of the local region determined to have the abnormality in the displacement field is equal to or greater than a predetermined value.

9. 9. The image processing device according to claim 7, wherein the second alignment process performs alignment based on information about the local anatomical structure included in the local region determined by the determination unit to have an abnormality.

10. The image processing apparatus according to claim 1 , wherein the determining unit determines whether the deformation is abnormal with respect to intermediate deformation information of the plurality of successive processing stages that constitute the first registration processing.

11. The image processing device according to claim 1 , wherein the determining unit determines whether or not there is an abnormality in the deformation based on information about a local volume change in the displacement field.

12. The image processing device according to claim 1 , wherein the determining unit determines whether or not the deformation is abnormal based on the logarithm of a Jacobian of the displacement field.

13. an acquiring step of acquiring first medical image data and second medical image data obtained by imaging a subject; a first registration processing step that includes a plurality of successive processing steps involving intermediate deformation processing to perform registration between the first medical image data and the second medical image data, and executes a first registration processing so as to update a displacement field at each processing step and pass the updated displacement field to a next processing step; a second registration processing step of performing a second registration processing based on information about predetermined local anatomical structures in the first medical image data and the second medical image data, which is different from the first registration processing; an intermediate deformation information acquisition step of acquiring intermediate deformation information based on the updated displacement field in the plurality of successive processing steps; a determination step of determining whether or not there is an abnormality in deformation associated with the alignment in the first alignment process based on the intermediate deformation information, When it is determined in the determination step that there is an abnormality in deformation in the plurality of consecutive processing stages, the first registration process is stopped, and the second registration process is started based on the first medical image data and the second medical image data, and when it is determined in the determination step that there is no abnormality in deformation in the plurality of consecutive processing stages, the first registration process is continued to a next processing stage until it reaches an end stage. An image processing method comprising:

14. A program for executing the steps of the image processing method according to claim 13 on a computer.

Images