Image processing apparatus, method, and program

Abstract

A processor converts the size of a relevant image so as to derive a size-converted image, segments the size-converted image into the region of at least one class using a segmentation model having been constructed by training a neural network by machine learning so as to derive a plurality of class images in which the pixel value of each pixel represents class likelihood regarding the at least one class, converts the at least one class image to the size of the relevant image and derives at least one converted class image, and segments the relevant image on the basis of the pixel value in each pixel of the at least one converted class image.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, method, and program.

Background Art

[0002] In recent years, machine learning techniques using deep learning have attracted attention. In particular, various models for segmenting an object included in an image have been proposed by learning a convolutional neural network (hereinafter referred to as CNN (Convolutional Neural Network)), which is one of multi-layer neural networks in which a plurality of processing layers are hierarchically connected, by deep learning. For example, a method has been proposed in which CNN is branched and processed in the middle, and the processing results are further combined to perform segmentation with high accuracy (see Japanese Patent Application Laid-Open No. 2020-119568).

[0003] On the other hand, in the case of a three-dimensional medical image composed of a plurality of tomographic images acquired by a CT apparatus or an MRI apparatus, there is a concept of Pixel Spacing, and the length per pixel in the image is defined. In order to segment images with different Pixel Spacings, it becomes easier to perform segmentation and further learning of CNN by aligning the Pixel Spacings. In particular, in a three-dimensional medical image, the Pixel Spacing in the tomographic image plane (xy plane) is the same in both the x direction and the y direction, but the Pixel Spacing in the direction perpendicular to the tomographic image (z direction) is often different from the x direction and the y direction. Therefore, the size of the three-dimensional image is converted so that the Pixel Spacings in all directions of xyz match, and segmentation is performed on the three-dimensional image after the size conversion. In this case, by applying the segmentation result to the three-dimensional image of the original size, the three-dimensional image of the original size can be segmented.

[0004] Furthermore, to perform segmentation quickly, some methods involve resizing the image to reduce its size, inputting the resized image into a CNN for segmentation, and then applying the segmentation results to the image at its original size. [Overview of the project] [Problems that the invention aims to solve]

[0005] However, when applying the segmentation results from a resized image to an image of the original size, problems may arise at the boundaries of the segmented regions. In particular, when segmentation is performed after reducing the size, applying the segmentation results to an image of the original size can result in noticeable jagged edges at the boundaries of the segmented regions.

[0006] This disclosure is made in view of the above circumstances and aims to smooth the boundaries of the segmented regions even when the segmentation results after resizing are applied to the original image. [Means for solving the problem]

[0007] The image processing apparatus according to this disclosure comprises at least one processor, The processor derives a resized image by converting the size of the target image. By using a segmentation model constructed through machine learning of a neural network, the resized image is segmented into regions of at least one class, thereby deriving multiple class images in which the pixel value of each pixel represents the likelihood of belonging to at least one class. Convert at least one class image to the size of the target image to derive at least one converted class image, The target image is segmented based on the pixel value of each pixel in at least one transformed class image.

[0008] In addition, in the image processing apparatus according to this disclosure, the size conversion may be scaling, reduction, or normalization in at least one direction in which pixels are arranged in the target image.

[0009] Furthermore, in the image processing apparatus according to this disclosure, the pixel values ​​of the class image may be scores derived by a neural network that represent the probability of being in at least one class.

[0010] Furthermore, in the image processing apparatus according to this disclosure, the processor may convert a class image to the size of a target image by interpolation.

[0011] Furthermore, in the image processing apparatus according to this disclosure, the processor may segment the target image by deriving the argmax of the pixel value of the corresponding pixel in at least one transformed class image.

[0012] Furthermore, in the image processing apparatus according to this disclosure, the processor may sequentially derive the conversion class image and segment the target image for each class.

[0013] The image processing method disclosed herein derives a resized image by converting the size of the target image, By using a segmentation model constructed through machine learning of a neural network, the resized image is segmented into regions of at least one class, thereby deriving multiple class images in which the pixel value of each pixel represents the likelihood of belonging to at least one class. Convert at least one class image to the size of the target image to derive at least one converted class image, The target image is segmented based on the pixel value of each pixel in at least one transformed class image.

[0014] Furthermore, the image processing method described herein may be provided as a program for causing a computer to execute it.

Advantages of the Invention

[0015] According to the present disclosure, even when the segmentation result after size conversion is applied to the original image, the boundary of the segmented region can be smoothed.

Brief Description of the Drawings

[0016] [Figure 1] Diagram showing the schematic configuration of a diagnostic support system to which an image processing apparatus according to the first embodiment of the present disclosure is applied [Figure 2] Diagram showing the schematic configuration of an image processing apparatus according to the first embodiment [Figure 3] Functional configuration diagram of an image processing apparatus according to the first embodiment [Figure 4] Diagram for explaining the processing performed in the first embodiment [Figure 5] Diagram schematically showing the configuration of a segmentation model [Figure 6] [[ID=…]] Diagram showing the score in a class image [Figure 7] Diagram showing the segmentation result using a class image [Figure 8] Diagram showing a conversion class image [Figure 9] Diagram showing the segmentation result of a target image [Figure 10] Diagram showing the display screen of a segmented target image [Figure 11] Flowchart showing the processing performed in the first embodiment [Figure 12] Diagram for explaining the processing performed in the second embodiment [Figure 13] Diagram for explaining the processing performed in the second embodiment [Figure 14] Diagram for explaining the processing performed in the second embodiment

Modes for Carrying Out the Invention

[0017] It should be noted that there seems to be an incomplete tag "[[ID=…]]" in the original text. Please check and correct it if necessary.Embodiments of this disclosure will be described below with reference to the drawings. Figure 1 is a hardware configuration diagram showing an overview of a diagnostic support system to which the image processing apparatus according to the first embodiment of this disclosure is applied. As shown in Figure 1, in the diagnostic support system, a computer 1 containing the image processing apparatus according to this embodiment, an imaging device 2, and an image storage server 3 are connected via a network 4 in a manner that enables communication.

[0018] Computer 1 contains the image processing device according to this embodiment, and the image processing program of this embodiment is installed on it. Computer 1 may be a workstation or personal computer directly operated by the physician performing the diagnosis, or it may be a server computer connected to them via a network. The image processing program is stored in a storage device of the server computer connected to the network, or in network storage, in a state that is accessible from the outside, and is downloaded and installed on Computer 1 used by the physician as needed. Alternatively, it may be recorded on a recording medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on Computer 1 from that recording medium.

[0019] The imaging device 2 is a device that generates a three-dimensional image representing a part of the subject to be diagnosed by imaging that part of the subject. Specifically, it is a CT scanner, an MRI scanner, or a PET (Positron Emission Tomography) scanner. The three-dimensional image, consisting of multiple slice images, generated by this imaging device 2 is transmitted to the image storage server 3 and stored. In this embodiment, the imaging device 2 is a CT scanner, and for example, it generates a three-dimensional image from a CT scan of the patient's chest and abdomen.

[0020] The image storage server 3 is a computer that stores and manages various types of data, and is equipped with a large-capacity external storage device and database management software. The image storage server 3 communicates with other devices via a wired or wireless network 4 to send and receive image data, etc. Specifically, it acquires various types of data, including image data of medical images generated by the imaging device 2, via the network, and stores and manages them on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between each device via the network 4 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0021] Next, an image processing apparatus according to the first embodiment will be described. Figure 2 illustrates the hardware configuration of the image processing apparatus according to the first embodiment. As shown in Figure 2, the image processing apparatus 20 includes a CPU (Central Processing Unit) 11, non-volatile storage 13, and memory 16 as a temporary storage area. The image processing apparatus 20 also includes a display 14 such as a liquid crystal display, input devices 15 such as a keyboard and mouse, and a network I / F (Interface) 17 connected to a network 4. The CPU 11, storage 13, display 14, input devices 15, memory 16, and network I / F 17 are connected to a bus 18. Note that the CPU 11 is an example of a processor in this disclosure.

[0022] Storage 13 is implemented by HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. Image processing programs are stored in storage 13 as a storage medium. The CPU 11 reads the image processing program 12 from storage 13, expands it into memory 16, and executes the expanded image processing program 12.

[0023] Next, the functional configuration of the image processing apparatus according to the first embodiment will be described. Figure 3 is a diagram showing the functional configuration of the image processing apparatus according to the first embodiment. As shown in Figure 3, the image processing apparatus 20 includes an image acquisition unit 21, a first conversion unit 22, a first segmentation unit 23, a second conversion unit 24, a second segmentation unit 25, and a display control unit 26. The CPU 11 executes the image processing program 12, and the CPU 11 functions as the image acquisition unit 21, the first conversion unit 22, the first segmentation unit 23, the second conversion unit 24, the second segmentation unit 25, and the display control unit 26.

[0024] The image acquisition unit 21 acquires the target image G0 to be processed from the image storage server 3 based on instructions from the radiologist operator via the input device 15.

[0025] The processes performed by the first conversion unit 22, the first segmentation unit 23, the second conversion unit 24, and the second segmentation unit 25 will be described below with reference to Figure 4.

[0026] The first conversion unit 22 converts the size of the target image G0 to derive a size-converted image GS0. In this embodiment, the size of the target image G0 is converted by reducing its size, but it is not limited to this. The size-converted image GS0 may also be derived by enlarging the target image G0.

[0027] Furthermore, in the multiple tomographic images constituting the target image G0, the pixel spacing in the plane (xy plane) is the same in both the x and y directions, but the pixel spacing in the direction perpendicular to the tomographic image (z direction, i.e., the axial direction) may differ from that in the x and y directions. For this reason, the first transformation unit 22 may derive a resized image GS0 by transforming the size of the target image G0 so that the pixel spacing in all x, y, and z directions is the same. Alternatively, the target image G0 of any size may be normalized so that a resized image GS0 of a predetermined size is derived.

[0028] The first segmentation unit 23 derives a plurality of class images in which the pixel value of each pixel represents the classness of at least one class by segmenting the resized image GS0 into regions of at least one class. In this embodiment, the first segmentation unit 23 segments the resized image GS0 into regions of multiple classes. For this purpose, the first segmentation unit 23 has a segmentation model 23A that has been machine-trained to segment the resized image GS0 into regions of multiple classes.

[0029] In this embodiment, the segmentation model 23A is machine-learned to segment the resized image GS0 into, for example, three classes. For example, in this embodiment, the target image G0 is a CT image of the subject's chest and abdomen. Therefore, the segmentation model 23A is machine-learned to segment the CT image into three classes of regions: lung, liver, and another.

[0030] In this embodiment, the segmentation model 23A consists of a convolutional neural network that has been machine-learned using training data, such as deep learning, to segment each pixel of a medical image into three objects.

[0031] Figure 5 is a schematic diagram showing the configuration of the segmentation model 23A. As shown in Figure 5, the convolutional neural network 30 that constitutes the segmentation model 23A has an encoder 31 and a decoder 32.

[0032] Here, the convolutional neural network consists of multiple processing layers. Each processing layer performs convolution on the input image using various kernels and outputs a feature map consisting of feature data obtained through the convolution process. The kernel has an n x n pixel size (for example, n=3) and each element is assigned a weight. Specifically, weights such as differential filters that emphasize the edges of the input image are assigned. The convolutional layer of the encoder 31 applies the kernel to the entire input image or the feature map output from the previous processing layer, while shifting the pixel of interest of the kernel. Furthermore, the convolutional layer applies an activation function such as a sigmoid function or a softmax function to the convolved values ​​and outputs a feature map.

[0033] The decoder 32 segments the input image into multiple classes (three classes in this embodiment) based on the feature map derived by the encoder 31. The decoder 32 also has multiple processing layers. The processing layers of the decoder 32 perform the same processing as the processing layers of the encoder 31, but upsample the input feature map and apply a kernel for deconvolution to the feature map.

[0034] Here, the decoder 32, in the processing layer 34 before the output layer 33, derives three feature maps of the same size as the input size-converted image GS0 as class images C0 to C2. For each pixel of the input image, the class images C0 to C2 are defined as a score representing the probability of belonging to one of the three classes 0 to 2. For example, class 0 represents regions other than the lungs and liver, class 1 represents the lung region, and class 2 represents the liver region.

[0035] Here, the processing layer 34 outputs a logit (regression value) as the score, but is not limited to this. It may also be a value obtained by applying the softmax function to the logit. On the other hand, if there is only one class to segment, the score may also be a value obtained by applying the sigmoid function to the logit.

[0036] Figure 6 shows the scores for class images C0 to C2. In Figure 6, class images C0 to C2 are assumed to consist of 4x5 pixels. Also, in Figure 6, for explanatory purposes, the scores are assumed to take values ​​from 0 to 4. As shown in Figure 6, the pixel values ​​of class image C0 are the scores for class 0, class image C1 are the scores for class 1, and class image C2 are the scores for class 2.

[0037] Furthermore, the output layer 33 of the decoder 32 outputs the segmentation result of the size-transformed image GS0 by deriving the argmax of the pixel value for the corresponding pixels in the class images C0 to C2. By deriving the argmax, a pixel is classified into the class of the pixel that takes the maximum value among the pixel values ​​of the class images C0 to C2.

[0038] Here, if the class images C0 to C2 output by the processing layer 34 are as shown in Figure 6, the segmentation result output by the output layer 33 will be as shown in Figure 7. In Figure 7, the numbers at the pixel positions represent the values ​​for classes 0 to 2. Also, in Figure 7, pixels with the same score in different classes are given priority in the order of class 0, class 1, and class 2, and segmented accordingly.

[0039] The second transformation unit 24 converts the class images C0 to C2 to the size of the target image G0 to derive the converted class images CG0 to CG2. In this process, the second transformation unit 24 converts the size of the class images C0 to C2 using an interpolation operation other than the nearest neighbor method. In addition to linear interpolation, any method such as cubic interpolation, spline interpolation, and B-spline interpolation can be used for the interpolation operation.

[0040] Figure 8 shows the converted scores in the converted class images. As shown in Figure 8, in the converted class images CG0 to CG2, the logit value changes smoothly between pixels, ranging from 0 to 4.

[0041] The second segmentation unit 25 segments the target image G0 based on the pixel value (i.e., logit) of each pixel in the converted class images CG0 to CG2 and derives the segmentation result R0. The second segmentation unit 25 derives the argmax of the converted class images CG0 to CG2 and derives the segmentation result of the target image G0. Figure 9 shows the segmentation result of the target image G0. In the segmentation result R0 shown in Figure 9, each pixel of the target image G0 is segmented into class 0, class 1, and class 2, respectively, by deriving the argmax of the pixel value for the corresponding pixels in the converted class images CG0 to CG2. Note that for pixels with the same pixel value between the converted class images CG0 to CG2, priority may be given in the order of class 0, class 1, and class 2, but this is not the only option.

[0042] The display control unit 26 displays the segmentation results of the target image G0 on the display 14. Figure 10 is a diagram showing the display screen of the segmentation results of the target image G0. As shown in Figure 10, the target image G0 is displayed on the display screen 40. For example, the target image G0 is a CT image of the chest and abdomen of a human body, with a mask 41 applied to the lungs, a mask 42 applied to the liver, and no masks applied to areas other than the lungs and liver. As a result, the target image G0 is segmented into three classes: lungs, liver, and other areas. Although difficult to illustrate, the boundaries between the lungs, liver, and other areas change smoothly.

[0043] Next, the processing performed in the first embodiment will be described. Figure 11 is a flowchart showing the processing performed in the first embodiment. The target image G0 to be processed is assumed to be acquired by the image acquisition unit 21 and stored in the storage 13. First, the first conversion unit 22 converts the size of the target image G0 to derive a size-converted image GS0 (step ST1). Next, the first segmentation unit 23 segments the size-converted image GS0 into regions of at least one class to derive a plurality of class images in which the pixel value of each pixel represents the classness for at least one class (step ST2).

[0044] Next, the second conversion unit 24 converts the class images C0 to C2 to the size of the target image G0 using interpolation calculations other than the nearest neighbor method to derive the converted class images CG0 to CG2 (step ST3). Then, the second segmentation unit 25 segments the target image G0 based on the pixel values ​​of each pixel in the converted class images CG0 to CG2 (step ST4). Finally, the display control unit 26 displays the segmented target image G0 on the display 14 (step ST5) and terminates the process.

[0045] Thus, in this embodiment, the class images C0 to C2 derived by segmenting the size-converted image GS0 are converted to the size of the target image G0 using an interpolation operation other than the nearest neighbor method to derive the converted class images CG0 to CG2. Therefore, by segmenting the target image G0 based on the pixel values ​​of the converted class images CG0 to CG2, it is possible to prevent jagged edges from being noticeable at the boundaries of the segmented regions. Consequently, the boundaries of the segmented regions can be made smooth.

[0046] In the first embodiment described above, the conversion class images CG0 to CG2 are derived for all classes before segmenting the target image G0, but this is not the only way. The derivation of the conversion class images CG0 to CG2 and the segmentation of the target image G0 may be performed sequentially for each class. This will be described below as the second embodiment.

[0047] Figures 12 to 14 illustrate the processing performed in the second embodiment. In Figures 12 to 14, the maximum value buffer 51, which stores the maximum value of the conversion class images CG0 to CG2 used in the processing, and the segmentation result 52 are schematically shown. Both the maximum value buffer 51 and the segmentation result 52 are data areas within the memory 16.

[0048] First, as shown in Figure 12, the second conversion unit 24 and the second segmentation unit 25 convert the class image C0 of class 0 to the size of the target image G0 and store the maximum value of the converted class image CG0 in the maximum value buffer 51. The conversion to the size of the target image G0 can be performed by interpolation other than the nearest neighbor method, as in the first embodiment described above. At this point, only the converted class image CG0 of class 0 has been derived. Therefore, each pixel in the maximum value buffer 51 stores the pixel value of each pixel of the converted class image CG0. Consequently, the segmentation result of the target image G0 at this point is that the entire region of the target image G0 is class 0.

[0049] Next, as shown in Figure 13, the second conversion unit 24 and the second segmentation unit 25 convert the class image C1 of class 1 to the size of the target image G0 and store the maximum value of the converted class image CG1 in the maximum value buffer 51. At this point, the derived values ​​are the converted class image CG0 of class 0 and the converted class image CG1 of class 1. Therefore, the pixel value of each pixel in the maximum value buffer 51 is updated with the larger of the two pixel values ​​of the converted class image CG0 and the converted class image CG1. At this point, the target image G0 is segmented into two regions: class 0 and class 1.

[0050] Furthermore, as shown in Figure 14, the second conversion unit 24 and the second segmentation unit 25 convert the class image C2 of class 2 to the size of the target image G0 and store the maximum value of the converted class image CG2 in the maximum value buffer 51. At this point, the converted class images CG0 to CG2 of three classes, classes 0 to 2, have been derived. Therefore, the pixel value of each pixel in the maximum value buffer 51 is updated with the largest pixel value among the converted class images CG0 to CG2. By referring to the maximum value buffer 51 at this point, the target image G0 can be segmented into three regions of classes 0 to 2.

[0051] As in the second embodiment, by sequentially deriving the conversion class images CG0 to CG2 and segmenting the target image G0 for each class, only memory equivalent to two target image G0s is needed during processing, eliminating the need to prepare memory the same size as the target image G0 for all classes. Therefore, the amount of memory used during processing can be reduced.

[0052] In the embodiments described above, the target image G0 is segmented into three classes, but the invention is not limited to this. The technology of this disclosure can also be applied when segmenting into two or four or more classes.

[0053] Furthermore, while the above embodiments segment the lungs, liver, and other regions included in the target image G0, the invention is not limited to this. For example, the technology of this disclosure can also be applied when segmenting hemorrhagic regions, infarcted regions, and normal regions in the brain. The technology of this disclosure can also be applied when segmenting the lungs into regions of ground-glass opacities, honeycomb lungs, and normal lungs, etc.

[0054] Furthermore, in the above embodiment, the hardware structure of the Processing Unit that executes various processes, such as the image acquisition unit 21, the first conversion unit 22, the first segmentation unit 23, the second conversion unit 24, the second segmentation unit 25, and the display control unit 26, can be the following types of processors. As mentioned above, these types of processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to execute a particular process, such as an ASIC (Application Specific Integrated Circuit).

[0055] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0056] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, as exemplified by client and server computers, and this processor functions as multiple processing units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0057] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits (Circuitry) that combine circuit elements such as semiconductor elements. [Explanation of symbols]

[0058] 1 Computer 2. Imaging device 3. Image storage server 4 Network 11 CPU 12 Image Processing Programs 13 Storage 14 displays 15 Input Devices 16 memory 17 Network Interface 18 bus 20 Image Processing Devices 21 Image acquisition unit 22 First Conversion Unit 23. First Segmentation Department 23A Segmentation Model 24 Second Conversion Unit 25. Second Segmentation Department 26 Display Control Unit 30 Convolutional Neural Networks 31 encoders 32 Decoders 33 Output Layer 34 Processing Layers 40 display screen 41, 42 masks 51 Maximum value buffer 52 Segmentation Results C0~C2 Class Images CG0~CG2 Conversion Class Image R0, R1 segmentation results

Claims

1. Equipped with at least one processor, The aforementioned processor, By converting the size of the target image, a resized image is derived. By using a segmentation model constructed by machine learning a neural network, the resized image is segmented into regions of at least one class, thereby deriving multiple class images in which the pixel value of each pixel represents the classness of that at least one class. The at least one class image is converted to the size of the target image by an interpolation operation other than the nearest neighbor method, and at least one converted class image is sequentially derived for each class. An image processing apparatus that sequentially segments the target image according to the class based on the pixel value of each pixel in the at least one transformation class image.

2. The image processing apparatus according to claim 1, wherein the size conversion is scaling, reduction, or normalization in at least one direction in which pixels are arranged in the target image.

3. The image processing apparatus according to claim 1 or 2, wherein the pixel values ​​of the class image are scores derived by the neural network that represent the probability of being at least one of the classes.

4. The image processing apparatus according to any one of claims 1 to 3, wherein the processor segments the target image by deriving the argmax of the pixel value of the corresponding pixel in the at least one transformed class image.

5. By converting the size of the target image, a resized image is derived. A segmentation model built by machine learning a neural network By using the `L` method to segment the resized image into regions of at least one class, a plurality of class images are derived in which the pixel value of each pixel represents the classness of the at least one class. The at least one class image is converted to the size of the target image by an interpolation operation other than the nearest neighbor method, and at least one converted class image is sequentially derived for each class. An image processing method that sequentially segments the target image into classes based on the pixel values ​​of each pixel in the at least one transformation class image.

6. The procedure for deriving a resized image by converting the size of the target image, A procedure for deriving multiple class images in which the pixel value of each pixel represents the classness of the at least one class by segmenting the resized image into regions of at least one class using a segmentation model constructed by machine learning a neural network, A procedure for sequentially deriving at least one transformed class image for each class by converting the at least one class image to the size of the target image using an interpolation operation other than the nearest neighbor method, An image processing program that causes a computer to perform a procedure of sequentially segmenting the target image according to the class based on the pixel value of each pixel of the at least one transformation class image.

Images