Medical image processing device, medical image processing method and program

Abstract

To generate a data set of new medical images used for a deep learning model without requiring a large amount of existing data.SOLUTION: A medical image processing apparatus according to an embodiment comprises a key information point generation unit and a data set generation unit. The key information point generation unit extracts feature points related to a specific disease from medical images included in a data set of existing medical images, and displaces the feature points in accordance with the feature of the disease to generate key information points. The data set generation unit deforms, within a predetermined allowable range, the periphery of the key information points in the medical images to become similar to deformation due to the disease to generate a data set of new medical images from the data set of existing medical images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to a medical image processing apparatus, a medical image processing method, and a program. [Background technology]

[0002] In recent years, deep learning techniques have been increasingly applied to image acquisition and processing, such as medical images. In deep learning-based image acquisition and processing, a key challenge is how to acquire a large dataset of images to train a deep learning model.

[0003] Conventionally, in order to address the lack of a data set in deep learning-based image processing, a technique for increasing a data set for an existing image label is known, for example, using a data augmentation technique based on standard image transformation. The standard image transformations referred to here include, for example, rotation, Gaussian noise, jittering, scaling, Gaussian blur, shearing (affine transformation), and the like.

[0004] Recently, methods have also been proposed to assist data augmentation by using Generative Adversarial Networks (GANs) to generate new datasets through image simulation or synthesis.

[0005] However, the above-mentioned standard image transformation may not be able to sufficiently increase the label of an image. For example, in the field of medical images, standard image transformation can only generate many images with the same label based on existing labels (e.g., disease / lesion / health / pseudo), and cannot generate images with new types of labels. In addition, the amount of data set that can be generated by standard image transformation is also limited. On the other hand, when GAN is used for image simulation and synthesis, it can only simulate or synthesize abnormal images from existing labels, and although a certain degree of diversity can be obtained by adjusting the existing level, the GAN technology itself cannot generate new labels. The mechanism of the generator and discriminator in GAN requires training the model with a large amount of existing data, and its application is limited in the field of medical images where there is little existing data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 8,165,425 [Patent Document 2] Special Publication No. 2013-506478 [Patent Document 3] JP 2015-093192 A [Patent Document 4] Special Publication No. 2012-507331 Summary of the Invention [Problem to be solved by the invention]

[0007] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to generate a new data set of medical images to be used in deep learning models, etc., without requiring a large amount of existing data. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0008] A medical image processing apparatus according to an embodiment includes a key information point generating unit and a data set generating unit. The key information point generating unit extracts feature points related to a specific disease from a medical image included in an existing medical image data set, and generates key information points by displacing the feature points according to features of the disease. The data set generating unit generates a new medical image data set from the existing medical image data set by deforming the periphery of the key information point in the medical image within a predetermined allowable range so as to resemble deformation caused by the disease. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of a medical image processing apparatus according to the first embodiment. [Diagram 2] FIG. 2 is a flowchart showing a medical image processing method for image data augmentation according to the first embodiment. [Figure 3A] FIG. 3A is a schematic diagram showing an example of a method for automatically generating key information points in the scoliosis medical image processing according to the first embodiment. [Figure 3B] FIG. 3B is a schematic diagram showing an example of a method for manually generating key information points in the scoliosis medical image processing according to the first embodiment. [Figure 4] FIG. 4 is a schematic diagram showing a specific process of generating a deformation field and generating a new data set using the deformation field in the scoliosis medical image processing according to the first embodiment. [Figure 5A] FIG. 5A is a diagram showing the principle of a linearly interpolated displacement field according to a comparative example of the first embodiment. [Figure 5B] FIG. 5B is a schematic diagram illustrating the principle of a multi-point warping field applied to multiple key information points according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating the principle of obtaining deformation fields of different features with different parameters σ. [Figure 7] FIG. 7 is a diagram showing the principle of obtaining optimal parameters of a deformation field according to the first embodiment. [Figure 8] FIG. 8 is a schematic diagram showing a specific process of generating a new dataset by applying the generated key information points and deformation field to images of an existing dataset in the scoliosis medical image processing according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing a medical image processing method for data augmentation of an MSP displaced image according to the second embodiment. [Figure 10] FIG. 10 is a diagram showing five key information points for locating the MSP plane according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] This embodiment has been made to solve the problems of the conventional technology described above, and aims to provide a medical image processing device and a medical image processing method that can generate new medical image datasets to be used in deep learning models, etc., without requiring large amounts of existing data.

[0011] According to one aspect of the present embodiment, there is provided a medical image processing apparatus comprising: a key information point generation unit that extracts feature points relating to a specific disease from a medical image included in an existing medical image dataset, and generates key information points by displacing the feature points in accordance with the characteristics of the disease; and a dataset generation unit that generates a new medical image dataset from the existing medical image dataset by deforming the area around the key information point in the medical image within a predetermined tolerance range so as to resemble the deformation caused by the disease.

[0012] According to another aspect of the present embodiment, there is provided a medical image processing method including the steps of: extracting feature points relating to a specific disease from a medical image included in an existing medical image dataset, and generating key information points by displacing the feature points in accordance with the characteristics of the disease; and generating a new medical image dataset from the existing medical image dataset by deforming the area around the key information points in the medical image within a predetermined tolerance range so as to resemble the deformation caused by the disease.

[0013] According to another aspect of the present embodiment, there is provided a program that causes a computer to execute the steps of: extracting feature points relating to a specific disease from a medical image included in an existing medical image dataset, and generating key information points by displacing the feature points in accordance with the characteristics of the disease; and generating a new medical image dataset from the existing medical image dataset by deforming the area around the key information point in the medical image within a predetermined tolerance range so as to resemble the deformation caused by the disease.

[0014] According to this embodiment, it is possible to generate new medical image datasets for use in deep learning models, etc., without requiring large amounts of existing data.

[0015] This embodiment relates to a medical image processing device for data augmentation of an image. This medical image processing device may be realized by executing software having each function of the medical image processing device using an independent computer or other device having a CPU (Central Processing Unit), or may be realized in hardware as a circuit capable of executing each function of the medical image processing device. In addition, the medical image processing device of the present invention may be incorporated in advance into a medical image imaging device such as a Magnetic Resonance Imaging (MRI) device as a part of the medical image imaging device.

[0016] Hereinafter, the present embodiment will be described with reference to the drawings.

[0017] In the following, in different embodiments, the same components are denoted by the same reference symbols, and duplicated descriptions will be omitted as appropriate.

[0018] (First embodiment) FIG. 1 is a block diagram showing the configuration of a medical image processing apparatus according to the first embodiment.

[0019] As shown in Fig. 1, the medical image processing apparatus 100 according to the present embodiment generates a new medical image dataset from an existing medical image dataset. The medical image processing apparatus 100 includes a key information point generating unit 10 that extracts and displaces feature points representing the features of an existing medical image from the existing medical image dataset to generate key information points, a deformation field generating unit 20 that generates a deformation field representing deformation around the key information point due to the displacement of the feature point with respect to the key information point, and a dataset generating unit 30 that generates a new medical image dataset by deforming the existing medical image based on the generated key information points and deformation field.

[0020] For example, in this embodiment, the medical images included in the existing medical image dataset are normal medical images related to a disease, and the medical images included in the new medical image dataset are abnormal medical images related to the disease. The generated new medical image dataset has a new label that is different from the attributes of the medical images included in the existing medical image dataset. According to this embodiment, a new medical image dataset and a new label to be used in a deep learning model or the like can be generated from the existing medical image dataset.

[0021] Hereinafter, medical image processing by the medical image processing apparatus 100 will be described in detail with reference to FIGS.

[0022] FIG. 2 is a flowchart showing a medical image processing method for image data augmentation according to the first embodiment.

[0023] As shown in FIG. 2, the medical image processing method according to the first embodiment includes the following steps. In the acquisition step of S100, the key information point generating unit 10 acquires a data set of an existing medical image. The acquired data set of the existing medical image is, for example, a medical image showing normal tissues and organs related to a specific disease. In the key information point generating step of S101, the key information point generating unit 10 extracts and displaces a feature point representing the characteristics of the existing medical image from a medical image included in the acquired data set of the existing medical image, thereby generating a key information point. Here, the key information point (Landmark) is an anatomical feature point related to a disease that is displaced (displaced from a normal position) based on the characteristics of the disease, and may be, for example, a position in a medical image or a part of an image. In the deformation field generating step of S102, the deformation field generating unit 20 generates a deformation field representing the deformation around the key information point due to the displacement of the feature point for the key information point generated by the key information point generating unit 10 in step S101. The specific contents of the deformation field and the details of the method of generating the deformation field in step S102 will be described later. In the dataset generation step of S103, the dataset generation unit 30 generates a new dataset of the medical image by deforming the existing medical image based on the generated key information points and the deformation field.

[0024] The medical images included in the new medical image dataset are generated according to the deformation field. Therefore, if the deformation field can reflect the characteristics of the deformation of human body tissue due to a specific disease, the medical images included in the new medical image dataset to be generated will be the required disease images similar to the true medical images of the disease. In addition, by introducing the deformation field, the generated disease images have new labels that are different from the normal images included in the existing medical image dataset. As a result, in this embodiment, a new medical image dataset and new labels to be used in deep learning models, etc. can be generated without requiring a large amount of existing data.

[0025] The first embodiment will be described below by taking image processing of a medical image of a scoliosis as an example.

[0026] The incidence of scoliosis is approximately 2-4% in adolescents, for example. Recognizing a patient's vertebrae and intervertebral discs in medical images related to scoliosis is a very difficult task, and deep neural networks are increasingly being used to solve the problem of recognizing such complex images with an accuracy comparable to that of human doctors. However, since the incidence of scoliosis itself is not high, the amount of data on existing disease images available is very limited compared to normal images.

[0027] For example, the key information point generating unit 10 acquires an existing normal spine image, extracts and displaces feature points that represent the characteristics of the existing normal spine image, and generates key information points. The deformation field generating unit 20 generates a deformation field that reflects the characteristics of the deformation of the human spine and its surrounding tissues due to scoliosis for the key information points. The data set generating unit 30 generates a new data set by deforming the existing normal spine image based on the generated key information points and deformation field. This generates a medical image with a new label that is similar to a true scoliosis medical image, and makes it possible to generate a new medical image data set and new labels to be used in a deep learning learning model of scoliosis without requiring a large amount of existing data.

[0028] Hereinafter, specific processing when the scoliosis medical image processing according to this embodiment is applied will be described in detail.

[0029] 3A and 3B are schematic diagrams showing two methods of generating key information points in step S101 in the scoliosis medical image processing according to the first embodiment.

[0030] As shown in Figures 3A and 3B, the medical image processing method according to this embodiment includes at least two methods for generating key information points from an existing data set. One is a method for generating key information points by automatically extracting and displacing feature points through image alignment between a normal image and a disease image (Figure 3A). The other is a method for generating key information points by extracting feature points based on a user's manual operation and displacing the feature points through image alignment (Figure 3B).

[0031] Specifically, as shown in Fig. 3A, in this embodiment, the key information point generating unit 10 uses ten vertebrae (T1 to T10 in the figure, the reference numerals may be omitted in other figures) of the human spine as an example, and performs automatic image alignment between a normal spine image and an existing scoliosis image to identify changes in the positions and angles of T2 to T8 at T1 to T10. In addition, the key information point generating unit 10 extracts points (here, the end points on both sides of the vertebrae T2 to T8) that best reflect the characteristics of the above changes among T2 to T8 whose positions and angles have changed according to the shape and anatomical characteristics of the vertebrae as feature points, and generates key information points by displacing each feature point according to the disease (here, scoliosis).

[0032] Alternatively, as shown in FIG. 3B, in this embodiment, a user (e.g., a doctor) performs manual operations such as clicking and dragging on the image with a mouse via a human-machine interface based on only the normal spine image, and the key information point generating unit 10 generates a scoliosis image corresponding to the disease (scoliosis in this case) in response to the operation, extracts feature points, and further displaces the feature points by alignment to generate key information points. Specifically, for the normal spine image on the left side of the figure, the user first performs manual operations to establish feature points at T2, T8, and T5 in T1 to T10. The feature points here may be, for example, end points or midpoints of the spine part where scoliosis is most likely to occur, which the user judges from experience or disease characteristics, and may be changed according to the actual state of the specific tissue or disease of interest. Next, the user performs dragging to move the feature points with the mouse via the human-machine interface. The key information point generating unit 10 generates a scoliosis image having a desired spinal curvature form according to these operations. Furthermore, as in FIG. 3A, the key information point generating unit 10 performs image alignment between a normal spine image and an existing scoliosis image manually generated by the user to identify changes in the positions and angles of T2 to T8 at T1 to T10, extracts points on both sides of the thoracic vertebrae T2 to T8 as feature points, and generates key information points by generating the displacement shown in the figure.

[0033] The deformation field and the method for generating the deformation field in step S102 will now be described in detail.

[0034] FIG. 4 is a schematic diagram showing the specific processing of step S102 of generating a deformation field and step S103 of generating a new data set using the deformation field in the scoliosis medical image processing according to the first embodiment of the present invention.

[0035] In Fig. 4, the left side is a normal spine image of the existing data set, the center is the key information points generated by the key information point generating unit 10 in step S101 and the deformation field generated for the key information points, and the right side is a scoliosis image of a new data set generated by combining the two. Note that Fig. 4 shows an example in which the positions and angles of T2 to T9 in T1 to T10 have changed, and for clearer illustration, the displacement of the rectangles representing the vertebrae T2 to T9 is drawn larger than the actual deformation of the spine.

[0036] First, the principle of applying the deformation field to disease image generation in this embodiment will be described.

[0037] When a part of a living body is displaced due to a disease (for example, the spine of a scoliosis patient is displaced and rotated from its normal position), the surrounding tissue of the diseased part is also deformed accordingly. In deep learning of medical images, an important issue is how to make the generated training image closer to the deformation of the true diseased image. A true medical image including the vicinity of a diseased part displaced due to a disease of a living body has a feature that the surrounding tissue is deformed with the displacement of the diseased part, and the deformation is nonlinear, and the deformation becomes smaller as the distance from the displacement point increases, and the deformation becomes zero when a certain distance is exceeded. The inventor of the present application found that such deformation is consistent with the characteristics of a point warping field. Therefore, in this embodiment, a point warping field is used to model the deformation of the surrounding tissue of a key information point due to the displacement of the key information point.

[0038] A single point warping field has already been applied to image deformation, for example, when performing decompression processing on an image. In the past, research has been conducted (for example, Patent Document 1) to model the displacement of the decompression point and the displacement of the surrounding pixels when performing image decompression processing as a linear interpolated displacement field in the point warping field.

[0039] FIG. 5A is a diagram showing the principle of a linearly interpolated displacement field according to a comparative example of the first embodiment.

[0040] As shown in FIG. 5A, an image is divided into a number of blocks, for example 5×5=25, and a user clicks on one point (the block in the lower right corner of the figure, hereinafter referred to as the drag point) on the image via a human-machine interface, then drags the mouse and releases it at another point (hereinafter referred to as the release point). The vector connecting the release point and the drag point represents the reference displacement that the user intends to locally impart to the image. All voxels within a circle of radius R centered on the drag point are affected by the reference displacement, and as a result, as shown on the right side of the figure, an elongation deformation occurs in the entire image including the drag point. That is, all nodes within the circle of radius R are displaced by a distance according to a roll-off function in a direction parallel to the direction d of the reference displacement as the drag point moves. This roll-off function is a function of the distance x from the drag point, and the function value decreases as the distance x increases, and becomes zero when the distance x exceeds the radius R.

[0041] The inventors of the present application have discovered that by regarding a feature point in a medical image as the drag point, and regarding the displacement of the feature point from its normal position due to disease as the reference displacement due to the dragging action, it is possible to similarly model the deformation of the tissue surrounding the key information point due to the displacement of the feature point using a deformation field.

[0042] In this embodiment, there are multiple key information points (feature points). For example, in scoliosis, each vertebra has at least two key information points, one on the left and one on the right, and scoliosis is caused by rotation and movement of multiple vertebrae. In other words, in this embodiment, it is necessary to model the deformation of surrounding tissues caused by the displacement of multiple key information points.

[0043] FIG. 5B is a schematic diagram illustrating the principle of a multi-point warping field applied to multiple key information points according to the present embodiment.

[0044] For the sake of convenience, this figure shows only the key information points and the state of the key information points before they are displaced, and the corresponding expansion deformation is omitted from the illustration.

[0045] As shown in Fig. 5B, when there are multiple key information points (the right side of Fig. 5B), some of the points are affected by multiple point warping fields. In this case, the deformation by the multiple point warping fields is obtained by superimposing the point warping fields of each of the multiple key information points (two are shown in the figure).

[0046] According to the above principle, in the present embodiment, a mathematical model of the deformation area around the disease is constructed using a deformation field, and the key information points and the deformation field are applied to the normal medical image to obtain the disease medical image.

[0047] That is, the deformation field generating unit 20 generates a deformation field representing deformation around the key information point due to displacement of the feature point with respect to the key information point generated by the key information point generating unit 10. The data set generating unit 30 performs processing to deform the normal spine image on the left side of Fig. 4 based on the key information points generated by the key information point generating unit 10 and the deformation field generated by the deformation field generating unit 20, thereby generating a new data set showing the spine scoliosis image on the right side of Fig. 4.

[0048] A specific process for generating a deformation field will now be described.

[0049] In this embodiment, as described above, a roll-off function is used as a mathematical model of the deformation field. This roll-off function is a function of the distance x from a displacement point (key information point), and the function value decreases as the distance x increases, and becomes zero when the distance x exceeds a predetermined value (radius R).

[0050] When using a roll-off function as a mathematical model of the deformation field, it is important to select an appropriate parameter (deformation field parameter) of the roll-off function. Therefore, in this embodiment, when selecting the deformation field parameter, an image similarity metric that represents the degree of similarity between images is applied to the roll-off function, and the maximum value of the image similarity metric in parameter traversing is obtained to obtain the optimal parameter of the roll-off function, thereby obtaining an image that is sufficiently close to the true disease image with a reasonable amount of calculation.

[0051] Here, various types of functions can be used as the roll-off function, but below, a specific method of selecting the deformation field parameters according to this embodiment will be described using a Gaussian function as an example.

[0052] The equation for the Probability Density Function (PDF) of the multivariate Gaussian distribution is:

[0053]

number

[0054] Here, p is a probability density function, and of the three parameters of p, x is a d-dimensional column vector, μ is the mean column vector of x with d dimensions, and Σ is the d×d-dimensional covariance matrix of x.

[0055] FIG. 6 is a diagram illustrating the principle of obtaining deformation fields of different features with different parameters σ.

[0056] As shown in FIG. 6, when a simplified and normalized isotropic two-dimensional Gaussian function is considered, the two-dimensional shape of the Gaussian distribution shows the characteristics of the deformation field.

[0057] In this case, the two parameters μ and Σ of the Gaussian function are as follows:

[0058]

number

[0059] where σ is the variance of x.

[0060] As a result, as shown in FIG. 6, different deformation fields can be obtained by selecting different σ. When σ=1 is taken as the reference state, the larger σ (e.g., σ=2), the larger the deformation range is, but the smaller the deformation width is. Conversely, when σ is smaller (e.g., σ=0.6), the smaller the deformation range is, but the larger the deformation width is. By using the simplified and normalized isotropic two-dimensional Gaussian function, different deformation effects that match the characteristics of the deformation field can be obtained by one parameter σ, so that by selecting the optimal parameter σ, the optimal deformation field can be obtained, and the disease image closest to the true disease image can be obtained.

[0061] The Gaussian function is merely one example of the roll-off function used in this embodiment. It goes without saying that the present embodiment is not limited to the Gaussian function, and similar effects can be obtained by using, for example, the Simpson distribution function.

[0062] FIG. 7 is a diagram showing the principle of obtaining optimal parameters of a deformation field according to the first embodiment.

[0063] In this embodiment, in order to obtain the optimal parameter σ, we introduce the image similarity metric and parameter traversal, apply the image similarity metric to the roll-off function, and obtain the maximum value of the image similarity metric in the parameter traversal to obtain the optimal parameter of the roll-off function. That is, we obtain σ when the image similarity metric is maximum within the range from σ1 to σ2 by the following equation.

[0064]

number

[0065] Here, Ir is the true disease image, I'(σ) is the disease image obtained by the deformation field, S(Ir, I'(σ)) is a function that represents the image similarity metric between Ir and I'(σ), and argmax(S(Ir, I'(σ))) is a function that represents σ when S(Ir, I'(σ)) is the maximum value.

[0066] 7, I is an existing normal medical image in this embodiment, σ represents a deformation field, I'(σ) is a disease image obtained by applying the deformation field to the existing normal medical image, and Ir is a true disease image. In this embodiment, the optimal parameter σ is obtained by finding σ when the image similarity metric between Ir and I'(σ) is the maximum value Smax within the range from σ1 to σ2.

[0067] The function S representing the image similarity metric in this embodiment can use any image similarity metric known in the art, such as normalized cross correlation, image entropy, mutual information, gradient correlation, pattern intensity, gradient difference, etc.

[0068] The function representing the image similarity metric may also be implemented by a visualization score, for example by presenting disease images generated with different deformation field parameters to a user (e.g., a physician) and allowing the user to select the corresponding optimal parameters by assigning a score to each disease image.

[0069] The selected optimal parameter σ may be one value or multiple values. In fact, for example, scoliosis is divided into different disease levels, such as mild (10-25°), moderate (26-40°), and severe (40° or more), depending on the difference in the bending angle of the spine, and tissue deformation differs depending on the degree of the disease due to the mutual influence of each point warping field in the multi-point warping field. Therefore, by making multiple optimal parameters σ correspond to different disease levels, one σ can be selected from multiple σ depending on the degree of the disease. Specifically, the degree of the disease is determined based on the distribution of key information points from the characteristics of the disease obtained in advance, and the corresponding optimal parameter σ is selected from multiple optimal parameters depending on the degree of the disease. When the Gaussian function is used as the function of the deformation field, as described above, when σ=1 is taken as the reference state, the larger σ (e.g., σ=2) the larger the deformation range is, but the smaller the deformation width is, and conversely, the smaller σ (e.g., σ=0.6) the smaller the deformation range is, but the larger the deformation width is. Therefore, taking scoliosis as an example, if a moderate scoliosis corresponds to a standard optimal parameter σ, a smaller optimal parameter σ is suitable for a severe scoliosis, and a larger optimal parameter σ is suitable for a mild scoliosis.

[0070] Multiple values ​​of σ can be obtained, for example, by determining a single optimal parameter σ for moderate scoliosis, and then determining a discrete set of values ​​or a continuous range of values ​​using a predetermined value of σ+ and a predetermined value of σ-, respectively.

[0071] In this way, by multi-valuing σ, it is possible to obtain a disease image that is closer to a true image and corresponds to the severity of the disease, and it is possible to generate more labels.

[0072] Above, the deformation field and the method for generating the deformation field in step S102 have been described in detail.

[0073] FIG. 8 is a schematic diagram showing a specific process of generating a new dataset by applying the generated key information points and deformation field to images of an existing dataset in the scoliosis medical image processing according to the first embodiment.

[0074] As shown in FIG. 8, after the deformation field is obtained, in the dataset generation step S103, the generated key information points and deformation field are applied to images of multiple existing datasets, and multi-point warping is performed on normal images to generate disease images of a new dataset.

[0075] According to the first embodiment, deformation of the spine and its surrounding tissues in, for example, scoliosis is modeled by key information points and deformation fields for the key information points, and optimal deformation field parameters are selected by image similarity metrics and parameter traversing. Therefore, a large number of disease images that are very close to true disease images can be generated from only normal medical images with a reasonable amount of calculation, and more image labels can be obtained.

[0076] Moreover, by selecting different optimal deformation field parameters depending on the severity of the disease, it is possible to generate an image that corresponds to the severity of the disease and is closer to a true disease image.

[0077] Second Embodiment Hereinafter, the second embodiment will be described with reference to FIGS. 9 and 10, taking as an example image processing of a midsagittal plane-displaced image of a human brain.

[0078] Reliable recognition of the mid-sagittal plane (MSP) of the human brain is crucial for brain image alignment, symmetry analysis, and group studies. Clinically, the error of the desired MSP positioning estimation must be within about 1 degree (deg), making detection of such small MSP deviations a crucial task.

[0079] Examples of diseases that can lead to MSP misalignment include brain tumors, brain infections, metabolic disorders, brain injuries, perinatal brain disorders, etc. The incidence of these examples is low, so it is very difficult to obtain MSP misalignment images in different examples for training deep learning models.

[0080] According to the second embodiment, a deformation field reflecting the characteristics of deformation of human brain tissue due to MSP deviation is generated based on an existing normal MSP image and key information points in the existing normal MSP image, and a medical image having a different new label similar to a medical image with a true MSP deviation can be generated by deforming the existing normal MSP image based on the generated key information points and deformation field. This makes it possible to generate a new disease image dataset and new labels to be used in a deep learning model of MSP deviation without requiring a large amount of existing data.

[0081] FIG. 9 is a flowchart illustrating a medical image processing method for data augmentation of MSP displaced images according to the second embodiment of the present invention.

[0082] As shown in FIG. 9, the medical image processing method according to the second embodiment includes the following steps. In the acquisition step of S200, the key information point generation unit 10 acquires a data set of an existing medical image. The acquired data set of the existing medical image is, for example, a 3D medical image showing a normal brain. In the key information point generation step of S201, the key information point generation unit 10 extracts and displaces feature points in a three-dimensional space from a brain image included in the acquired data set of the existing medical image to generate three-dimensional key information points. Here, the key information points (Landmarks) refer to brain anatomical feature points. In the deformation field generation step of S202, the deformation field generation unit 20 generates a 3D deformation field for performing data augmentation of the MSP-displaced image according to this embodiment for the key information points generated by the key information point generation unit 10 in step S201. In the dataset generation step of S203, the dataset generation unit 30 generates a new medical image dataset by deforming the brain images included in the existing medical image dataset based on the generated key information points and deformation field.

[0083] Step S200 is similar to step S100 in FIG. 2 in the first embodiment except for the target to be acquired, and therefore a detailed description thereof will be omitted here.

[0084] FIG. 10 is a diagram showing five key information points for locating the MSP plane according to the second embodiment.

[0085] The key information point generation step of S201 is similar to step S101 in FIG. 2, with the difference being that in step S201, five feature points in 3D space are extracted to locate the MSP plane, as shown in FIG. 10.

[0086] The five characteristic points shown in FIG. 10 are explained as follows.

[0087] ACC: anterior cingulate cortex PCC: posterior cingulate cortex CG: Chicken crest (crista galli) STEM: Brain stem OB: Occipital bone

[0088] Many instances of disease causing MSP deviation show anatomical displacement of at least one of the above five landmarks, so by using the key information points where the above five landmarks are displaced, it is possible to mark disease images with MSP deviation.

[0089] Unlike the first embodiment in which the key information points are generated in a two-dimensional plane in step S101 in Fig. 2, in step S201, the key information points are generated in a three-dimensional space. Since the rest is the same as step S101, detailed description of step S201 is omitted.

[0090] The deformation field generation step S202 is also similar, and differs from step S102 in Fig. 2 in the first embodiment in that the deformation field is three-dimensional. That is, in the second embodiment, the three-dimensional deformation of the MSP plane is modeled using, for example, a three-dimensional Gaussian function.

[0091] In this case, when a normalized isotropic three-dimensional Gaussian function is considered, the two parameters μ and Σ of the Gaussian function are as follows:

[0092]

number

[0093] In this embodiment, as in the first embodiment, image similarity metrics and parameter traversing are introduced to obtain the optimal parameter σ. That is, in the section where σ traverses from σ1 to σ2, σ when the image similarity metric is at its maximum value is obtained by the following equation.

[0094]

number

[0095] Equation 2-3 is similar to equation 1-3, that is, the selection of optimal parameters in the second embodiment is similar to that in the first embodiment, and therefore a detailed description thereof will be omitted.

[0096] According to the second embodiment, both the existing data set and the new data set to be generated are data sets of 3D images. In this case, the key information point generating unit 10 extracts and displaces feature points in a three-dimensional space to generate three-dimensional key information points. The function for deformation field modeling (e.g., the above Gaussian function) is a three-dimensional function, and the deformation field generating unit 20 generates a three-dimensional deformation field for the three-dimensional key information points.

[0097] According to the second embodiment, in addition to the effects of the first embodiment, it can also be applied to processing for data augmentation of three-dimensional medical images.

[0098] (Other embodiments) The above-described embodiment is not limited to the contents described above, and various modifications are possible.

[0099] For example, in the above-mentioned embodiment, in the key information point generating step, multiple sets of key information points (each set may include multiple key information points) may be generated. In this case, for example, the user may set the movement range and movement width of the key information points in advance, and after the user generates and moves the first set of key information points by the same operation as in the above-mentioned embodiment, the corresponding displacement may be performed by automatically generating multiple sets of key information points using the movement range and movement width. Furthermore, multiple sets of disease images may be generated at once by the same operation as in the above-mentioned embodiment using the generated multiple sets of key information points. In this way, it becomes possible to generate a larger number of disease image data sets.

[0100] In addition, in the above-mentioned embodiment, the scoliosis and the brain MSP plane deviation are described as examples, but the embodiment is not limited to this, and it is also possible to perform image data augmentation on medical images of other organs and diseases, and it is possible to generate corresponding key information points and deformation fields according to anatomical and medical features.

[0101] The medical image processing apparatus according to the above-mentioned embodiments may be implemented in medical equipment as a circuit capable of realizing the functions described in each embodiment, or may be distributed as a computer-executable program stored in a recording medium such as a magnetic disk (flexible disk, hard disk, etc.), a compact disk (CD (Compact Disc)-ROM (Read Only Memory), DVD (Digital Versatile Disc), BD (Blu-ray (registered trademark) Disc), etc.), a magneto-optical (MagnetoOptic:MO) disk, or a semiconductor memory.

[0102] In addition, an operating system (OS), which operates a computer according to instructions of a program installed on the computer from a recording medium, middleware (MW) such as database management software and network software, may execute some of the processes for realizing the above-described embodiments.

[0103] Although some embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0104] 100 Medical image processing device 10. Key Information Point Generation Unit 20 Deformation field generation section 30 Dataset Generation Unit

Claims

1. a key information point generating unit that extracts feature points related to a specific disease from a medical image included in a data set of existing medical images and generates key information points by displacing the feature points according to the features of the disease; a dataset generation unit that generates a new medical image dataset from the existing medical image dataset by deforming the periphery of the key information point in the medical image within a predetermined allowable range so as to resemble deformation caused by the disease; A medical image processing device comprising:

2. the key information point generation unit extracts a plurality of feature points and generates a plurality of key information points by displacing each of the plurality of feature points; the dataset generation unit generates a new medical image dataset from the existing medical image dataset by deforming the periphery of each of the plurality of key information points in the medical image within the allowable range; The medical image processing device according to claim 1 .

3. A deformation field generating unit generates a deformation field representing a deformation around the key information point caused by a displacement of the feature point with respect to the key information point, The data set generation unit generates a data set of the new medical image by deforming a periphery of the key information point in the existing medical image based on the key information point and the deformation field. The medical image processing device according to claim 1 .

4. the deformation field generator generates the deformation field by applying an image similarity metric representing a degree of similarity between images to a mathematical model of the deformation field; The medical image processing device according to claim 3 .

5. The mathematical model is a mathematical model based on a roll-off function, which is a function of distance from the key information point, the function value of which decreases as the distance increases, and the function value of which becomes zero when the distance exceeds a predetermined value; the deformation field generation unit applies the image similarity metric to the roll-off function, and obtains parameters when the image similarity metric is at a maximum value as optimal parameters of the roll-off function, thereby generating the deformation field. The medical image processing device according to claim 4 .

6. The roll-off function is a normalized isotropic Gaussian probability density function. The medical image processing device according to claim 5 .

7. the optimal parameter is a plurality of optimal parameters, The plurality of optimal parameters each correspond to a degree of disease.

7. The medical image processing device according to claim 5 or 6.

8. The key information point generating unit generates the key information points by extracting and displacing the feature points through alignment between a medical image included in the data set of the existing medical images and an existing disease image.

6. The medical image processing apparatus according to claim 1.

9. The key information point generation unit extracts the feature points from the medical images included in the data set of the existing medical images based on a manual operation by a user, and generates the key information points by displacing the feature points through image alignment between the medical images and existing disease images.

6. The medical image processing apparatus according to claim 1.

10. the key information point generation unit extracts a plurality of feature points and generates a plurality of key information points by displacing each of the plurality of feature points; the deformation field generation unit generates the deformation field for each of the plurality of key information points; The data set generation unit generates a new data set of the medical image by deforming the periphery of each of the key information points in the medical image by superimposing the plurality of deformation fields. The medical image processing device according to any one of claims 3 to 7.

11. the existing medical image dataset and the new medical image dataset are three-dimensional image datasets; the Gaussian distribution probability density function is a three-dimensional Gaussian distribution probability density function; The deformation field generation unit generates a three-dimensional deformation field for the key information points. The medical image processing device according to claim 6 .

12. The key information point generation unit generates a plurality of sets of key information points; The data set generator generates a new medical image data set from the existing medical image data set by deforming the periphery of each of the sets of key information points in the medical image within the allowable range. The medical image processing device according to any one of claims 3 to 11.

13. The key information point is a position in the medical image or a part of the medical image. The medical image processing device according to any one of claims 1 to 12.

14. The dataset generation unit generates, as the new medical image dataset, a medical image dataset having a new label different from a label indicating an attribute of a medical image included in the existing medical image dataset. The medical image processing device according to any one of claims 1 to 13.

15. The medical images included in the existing medical image dataset are normal medical images related to a disease, and the medical images included in the new medical image dataset are abnormal medical images related to the disease. The medical image processing device according to any one of claims 1 to 14.

16. Extracting feature points related to a specific disease from medical images included in a data set of existing medical images, and generating key information points by displacing the feature points according to the characteristics of the disease; generating a new medical image dataset from the existing medical image dataset by deforming the periphery of the key information point in the medical image within a predetermined tolerance range so as to resemble the deformation caused by the disease; A medical image processing method comprising:

17. A step of extracting feature points related to a specific disease from medical images included in a data set of existing medical images, and generating key information points by displacing the feature points according to the characteristics of the disease; generating a new medical image data set from the existing medical image data set by deforming the periphery of the key information point in the medical image within a predetermined tolerance range so as to resemble the deformation caused by the disease; A program that causes a computer to execute the following.

Images