Image processing method, image processing apparatus, and program

Abstract

To obtain an image segmentation model with adjustable boundary smoothness, and to improve the accuracy and efficiency of segmentation inference while improving boundary smoothness. [Solution] The image processing method according to the embodiment is an image processing method that performs segmentation inference on a specific region included in a medical image using an image segmentation model, and includes the steps of: receiving a labeled first medical image and a first smoothness constraint parameter for restricting the smoothness of the boundary of the segmentation inference result as input, training an image segmentation model to obtain an image segmentation model in which the first smoothness constraint parameter can be adjusted; and performing segmentation inference on a second medical image using the trained image segmentation model, the input second smoothness constraint parameter and the second medical image, and outputting the segmentation inference result.

Description

[Technical Field]

[0001] Embodiments disclosed herein and in the drawings relate to image processing methods, image processing devices, and programs. [Background technology]

[0002] Medical image segmentation inference is an image processing process that segments a medical image into several regions based on similar or different features between those regions. Currently, medical image segmentation inference typically involves processing images of cells, tissues, or organs to extract specific information. Conventional medical image segmentation inference methods include image processing methods that perform segmentation inference based on spatial local features such as the uniformity of grayscale, texture, and other statistical properties of pixels in the medical image, and image processing methods that determine the boundaries of the segmentation target using gradient information.

[0003] For example, when performing segmentation inference on medical images containing liver segments, lung segments, or brain regions, segmentation inference may be possible for only some regions by referring to the physiological or physical boundaries of the liver, lungs, or brain (e.g., anatomical structures such as blood vessels, sulci, and gyri), while segmentation inference for other regions may only be possible by extending existing boundaries. However, when segmentation inference can only be performed by extending existing boundaries, the boundaries of the extended portions may not be smooth.

[0004] Specifically, we will explain using the example of performing segmentation inference processing on liver segments. For liver segments, segmentation inference processing is usually performed using the Couinaud segmentation rule. The Couinaud segmentation rule mainly refers to the hepatic veins and portal vein trunk of the liver to segment (segment) the liver. However, when segmenting liver segments using the Couinaud segmentation rule, a problem arises in that fully automated end-to-end liver segmentation cannot refer to blood vessels, resulting in segmentation inference results that are not smooth. Furthermore, this problem of segmentation inference results being not smooth is more likely to occur at the distal end of the liver.

[0005] However, liver segmentation plays a crucial role in disease diagnosis, specifically in the localization (identification) of the liver segment of the lesion. In disease diagnosis, non-smoothing issues in segmentation inference results can lead to errors in lesion localization and further affect the outcome of disease diagnosis and surgical planning.

[0006] One conventional technique addresses the problem of non-smoothness in segmentation inference results by employing a method that increases the smoothness constraint loss to derive a model that yields smoother segmentation inference results. However, while using a large smoothness constraint parameter improves the smoothness of the segmentation inference results, it also has the problem of decreasing the accuracy of the segmentation inference. Furthermore, adjusting the smoothness constraint parameter requires retraining multiple models, leading to increased training costs and decreased efficiency.

[0007] Another conventional technique employs image post-processing methods to mitigate the non-smoothness problem in segmentation inference results by utilizing surrounding information of the non-smooth regions. However, image post-processing methods cannot resolve the phenomenon of large non-smooth regions, and therefore cannot completely eliminate the non-smoothness problem in segmentation inference results. Furthermore, there is the problem that it takes time to execute the image post-processing method after segmentation inference is complete. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Special Publication No. 2020-510463 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] One of the problems that the embodiments disclosed herein and in the drawings aim to solve is to obtain an image segmentation model with adjustable boundary smoothness, and to improve the accuracy and efficiency of segmentation inference while improving boundary smoothness. However, the problems that the embodiments disclosed herein and in the drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0010] The image processing method according to the embodiment is an image processing method that performs segmentation inference on a specific region included in a medical image using an image segmentation model based on a deep neural network, and includes a model training step and a segmentation inference step. In the model training step, a labeled first medical image and a first smoothness constraint parameter for constraining the smoothness of the boundary of the segmentation inference result are input, and an image segmentation model is trained to obtain an image segmentation model in which the first smoothness constraint parameter can be adjusted. In the segmentation inference step, segmentation inference is performed on a second medical image using the trained image segmentation model, the input second smoothness constraint parameter and the second medical image, and the segmentation inference result is output. [Brief explanation of the drawing]

[0011] [Figure 1]Figure 1 is a block diagram showing an example configuration of an image processing apparatus according to the first embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the configuration of the model learning unit in the first embodiment. [Figure 3] Figure 3 is a schematic diagram showing an image segmentation model based on smoothness constraint parameters. [Figure 4] Figure 4 is a block diagram showing an example of the configuration of the segmentation inference unit in the first embodiment. [Figure 5] Figure 5 is a flowchart showing the flow of the image processing method according to the first embodiment. [Figure 6] Figure 6 is a block diagram showing an example configuration of an image processing apparatus according to the second embodiment. [Figure 7] Figure 7 is a block diagram showing an example configuration of the segmentation inference unit in the second embodiment. [Figure 8] Figure 8 is a flowchart showing a part of the flow of the image processing method according to the second embodiment. [Figure 9] Figure 9 is a schematic diagram showing a specific example of the second embodiment. [Modes for carrying out the invention]

[0012] The image processing method, image processing apparatus, and program according to the embodiments described below can obtain an image segmentation model with adjustable boundary smoothness, and can improve the accuracy and efficiency of segmentation inference while improving boundary smoothness.

[0013] In addition, the image processing method, image processing apparatus, and program according to the embodiments described below do not require re-learning a plurality of models to adjust the smoothness constraint parameter as compared with the conventional method of increasing the smoothness constraint loss. Therefore, the learning cost of the image segmentation model can be reduced, and the efficiency can be improved. Further, since the segmentation inference is performed using an appropriate smoothness constraint parameter with respect to the difference in the physical reference boundary in the medical image for which the segmentation inference should be performed, the accuracy of the segmentation inference can be improved.

[0014] In addition, the image processing method, image processing apparatus, and program according to the embodiments described below can eliminate large non-smooth regions by increasing the smoothness constraint parameter as compared with the conventional method using image post-processing. Further, since there is no need to execute the remaining processing after the segmentation inference is completed, the efficiency of the segmentation inference can be improved.

[0015] The image processing method according to the embodiment is an image processing method for performing segmentation inference on a specific region included in a medical image using an image segmentation model based on a deep neural network. A labeled first medical image and a first smoothness constraint parameter for constraining the smoothness of the boundary of the segmentation inference result are input, and the image segmentation model is learned to obtain the image segmentation model capable of adjusting the first smoothness constraint parameter. A model learning step, and a segmentation inference step of performing segmentation inference on the second medical image using the learned (trained) image segmentation model, the input second smoothness constraint parameter, and the second medical image, and outputting a segmentation inference result.

[0016] Thereby, in the model learning process of the image segmentation model, the image segmentation model is learned based on the first smoothness constraint parameter. Therefore, the learned image segmentation model can perform segmentation inference based on the input second smoothness constraint parameter. In the segmentation inference process of the image segmentation model, based on the input second smoothness constraint parameter, segmentation inference is performed on the second medical image using the learned image segmentation model. Therefore, for the differences in the physical reference boundaries in the medical images for which segmentation inference should be performed, segmentation inference can be performed using appropriate smoothness constraint parameters, improving the smoothness of the boundaries and improving the accuracy and efficiency of segmentation inference. Also, an image segmentation model with adjustable boundary smoothness can be obtained.

[0017] In the image processing method according to the embodiment, the model learning step includes: a first input step in which the labeled first medical image and the first smoothness constraint parameter that is a random number within the parameter range are input; a combining step of combining the first smoothness constraint parameter with the labeled first medical image; and a model parameter optimization step of learning the image segmentation model using the combined first smoothness constraint parameter, calculating an evaluation value Loss based on the learning result, and optimizing the parameters of the image segmentation model.

[0018] This allows multiple first smoothness constraint parameters to be combined with a labeled first medical image during the model training process of the image segmentation model, thereby training the image segmentation model. Therefore, the trained image segmentation model can perform segmentation inference based on multiple smoothness constraint parameters. Furthermore, appropriate smoothness constraint parameters can be used for differences in the physical reference boundary in the medical image to be segmented, improving boundary smoothness while enhancing the accuracy and efficiency of segmentation inference. Additionally, an image segmentation model with adjustable boundary smoothness can be obtained.

[0019] In the image processing method according to the embodiment, the segmentation inference step includes a second input step into which the second medical image and the second smoothness constraint parameter manually entered by an operator are input; an inference step into which segmentation inference is performed using the image segmentation model, the second smoothness constraint parameter, and the second medical image; and an output step into which the segmentation inference result of a specific region included in the second medical image is output.

[0020] This allows the image segmentation model to perform segmentation inference on a second medical image using the trained image segmentation model based on a second smoothness constraint parameter input by the user. Therefore, segmentation inference can be performed using the smoothness constraint parameter input by the user, improving the accuracy and efficiency of segmentation inference while improving boundary smoothness. Furthermore, it becomes possible to obtain an image segmentation model with adjustable boundary smoothness.

[0021] In the image processing method according to the embodiment, the segmentation inference step includes: a second input step in which the medical image and an initial value of the smoothness constraint parameter are input; an inference step in which segmentation inference is performed using the image segmentation model, the smoothness constraint parameter, and the medical image; a smoothness determination step in which it is determined whether the smoothness index of the segmentation inference result of a specific region included in the medical image satisfies a preset smoothness condition, and if it is determined that the threshold is not met, the smoothness constraint parameter is increased and the inference step is re-executed; and an output step in which, if it is determined in the smoothness determination step that the smoothness condition is met, the segmentation inference result is output.

[0022] This allows the image segmentation model to gradually increase the smoothness constraint parameter from its initial value λ during the segmentation inference process until the smoothness index of the segmentation inference result satisfies the pre-set smoothness conditions. Therefore, medical images can be segmented based on more appropriate smoothness constraint parameters, improving boundary smoothness and increasing the accuracy of segmentation inference. Furthermore, by automatically selecting appropriate smoothness constraint parameters and obtaining an image segmentation model with automatically adjustable boundary smoothness, the efficiency of inference can be improved.

[0023] In the image processing method according to this embodiment, the smoothness index is the number of holes in the segmentation inference result.

[0024] This makes it possible to more appropriately determine whether the smoothness index of the segmentation inference result satisfies the pre-set smoothness conditions during the segmentation inference process of the image segmentation model. Therefore, medical images can be segmented based on more appropriate smoothness constraint parameters, improving the accuracy of segmentation inference while improving boundary smoothness.

[0025] The image processing apparatus according to this embodiment is an image processing apparatus that performs segmentation inference on a specific region included in a medical image using an image segmentation model based on a deep neural network, and comprises: a model learning unit that receives a labeled first medical image and a first smoothness constraint parameter for restricting the smoothness of the boundary of the segmentation inference result as input, and trains the image segmentation model to obtain an image segmentation model in which the first smoothness constraint parameter can be adjusted; and a segmentation inference unit that performs segmentation inference on the second medical image using the trained image segmentation model, the input second smoothness constraint parameter and the second medical image, and outputs the segmentation inference result.

[0026] The program according to the embodiment is a program that causes a computer to perform segmentation inference on a specific region included in a medical image using an image segmentation model based on a deep neural network, and includes a model learning process in which a labeled first medical image and a first smoothness constraint parameter for restricting the smoothness of the boundary of the segmentation inference result are input, and the image segmentation model is trained to obtain an image segmentation model in which the first smoothness constraint parameter can be adjusted; and a segmentation inference process in which segmentation inference is performed on the second medical image using the trained image segmentation model, the input second smoothness constraint parameter and the second medical image, and the segmentation inference result is output.

[0027] The image processing method, image processing apparatus, and program according to the embodiment will be described below with reference to the drawings. In the following description, elements identical to those previously mentioned will be denoted by the same reference numerals, and detailed explanations will be omitted as appropriate.

[0028] In describing the embodiments, we will use as an example the performance of segmentation inference for liver regions in medical images using an image segmentation model based on a deep neural network. That is, we will use as an example the performance of segmentation inference for liver segments in medical images using an image segmentation model. In the following description, "specific region" refers to, for example, a liver segment. Of course, "specific region" in medical images may refer to regions of other parts, tissues, or structures, such as lung segments or brain regions. The image processing method, image processing apparatus, and program according to the embodiments can be applied to segmentation inference of liver segments and can also be applied to segmentation inference of other regions such as lung segments and brain regions.

[0029] (First Embodiment) Hereinafter, an example of the configuration of the image processing apparatus 100 according to the first embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing an example of the configuration of the image processing apparatus 100 according to the first embodiment.

[0030] The image processing device 100 comprises a processing circuit 1, a memory 2, and a communication interface 3. Each component is connected to the others via a bus, which is a common signal transmission path, enabling communication between them.

[0031] Memory 2 is a device for storing various data and information. Memory 2 may be a storage medium readable by the processor (e.g., a magnetic storage medium, an electromagnetic storage medium, an optical storage medium, or a semiconductor memory), or it may be a drive device that reads and writes data and information to and from the storage medium. Memory 2 stores, for example, the computer programs that enable each functional unit to be implemented in the processing circuit 1.

[0032] Communication IF3 is an interface for communicating various data and information between the image processing device 100 and external devices. For example, communication IF3 communicates various data and information with an operating terminal or display device (not shown).

[0033] Processing circuit 1 is a circuit that controls the overall operation of the image processing device 100. Processing circuit 1 includes at least one processor. A processor refers to a circuit such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), Application Specific Integrated Circuit (ASIC), or Programmable Logic Device (e.g., Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), Field Programmable Gate Array (FPGA)). If the processor is a CPU, the CPU realizes each function by reading and executing each program stored in memory 2. If the processor is an ASIC, each function is directly incorporated as a logic circuit within the ASIC's circuitry. The processor may be configured as a single circuit or as a combination of multiple independent circuits. Processing circuit 1 realizes the functions of each functional unit described later.

[0034] In this embodiment, the processing circuit 1 comprises a model learning unit 11 and a segmentation inference unit 12.

[0035] The following describes an example of the configuration of the model learning unit 11 in the first embodiment with reference to Figure 2. Figure 2 is a block diagram showing an example of the configuration of the model learning unit 11 in the first embodiment.

[0036] The model learning unit 11 is a functional unit for training an image segmentation model M based on a deep neural network. As shown in Figure 2, the model learning unit 11 also includes a first input unit 111, a coupling unit 112, and a parameter optimization unit 113.

[0037] During the model training process of the image segmentation model M, the first input unit 111 receives labeled image data and a smoothness constraint parameter λ for constraining the smoothness of the boundaries of the segmentation inference results. The smoothness constraint parameter λ is, for example, a random number (or decimal) within the parameter range. For example, the first input unit 111 may be implemented by an input interface such as a mouse or keyboard and a processor such as a CPU (Central Processing Unit).

[0038] Specifically, a set of input data consists of training medical images, ground truth (GT), and a smoothness constraint parameter λ. During the model training process for an image segmentation model M of one batch size, the smoothness constraint parameter λ in one set of input data is set to an arbitrary value, for example, a random number within the range of [0-60].

[0039] Here, the medical images used as training data are, for example, labeled image data in which data labels have been assigned to medical images through a labeling process. The smoothness constraint parameter λ is a parameter used to constrain the smoothness of the boundaries of the segmentation inference results of the image segmentation model M. As mentioned above, the smoothness constraint parameter λ is, for example, a random number within the parameter range. In the following explanation, the "parameter range" of the smoothness constraint parameter λ is, for example, the range [0-60].

[0040] The image segmentation model M based on the smoothness constraint parameter λ will be explained below with reference to Figure 3. Figure 3 is a schematic diagram showing the image segmentation model M based on the smoothness constraint parameter λ.

[0041] In the following explanation, we will use the nnUnet hyperparameter model with Laplacian regularization constraints as the image segmentation model M based on a deep neural network as an example. However, this embodiment is not limited to this, and other deep learning models such as Vnet may be used.

[0042] In Figure 3, the solid lines represent multiple cascade layers of the image segmentation model M, each containing a convolutional layer, an input layer, and an activation function (3*3*3Conv-IN-ReLU). The dashed lines represent skip connections between each cascade layer, the arrows represent convolutional transposes between each cascade layer, and the dashed boxes represent multiple smoothness constraint parameters λ.

[0043] The coupling section 112 combines the smoothness constraint parameter λ with the labeled image data.

[0044] Specifically, the coupling unit 112 sets n smoothness constraint parameters λ1~λ within a predetermined parameter range for n cascade layers in the image segmentation model M. n The following are selected. Furthermore, the coupling section 112 has a fully connectable layer that allows for learning of n smoothness constraint parameters λ1~λ n n tensors t1~t (number of input channels) n This is converted to the following. Here, the number of input channels n in the image segmentation model M is the same as the number of cascade layers n. Furthermore, the coupling section 112 has n smoothness constraint parameters λ1~λ n n tensors t1~t transformed from n The smoothness constraint parameter λ is combined with the labeled image data by multiplying each input channel by the parameter. For example, the merging unit 112 may be implemented by a processor such as a CPU.

[0045] The parameter optimization unit 113 trains the image segmentation model M using the combined smoothness constraint parameter λ, calculates an evaluation value Loss based on the training results, and optimizes the parameters of the image segmentation model M. For example, the parameter optimization unit 113 may be implemented by a processor such as a CPU.

[0046] Specifically, the parameter optimization unit 113 calculates the evaluation value Loss using the following equation (1).

[0047]

number

[0048] In this embodiment, the case in which the normalization region is limited to the region inside the liver using a background probability threshold will be explained as an example. However, this embodiment is not limited to this, and the normalization region may be limited to other normalization regions, or the normalization region may be limited by other methods.

[0049] Furthermore, the parameter optimization unit 113 trains the image segmentation model M using the smoothness constraint parameter λ coupled as described above, and obtains the training result. Then, the parameter optimization unit 113 calculates an evaluation value Loss based on the training result of the image segmentation model M, and optimizes the parameters of the image segmentation model M using the calculated evaluation value Loss. For example, the parameter optimization unit 113 can optimize the parameters of the image segmentation model M so that the evaluation value Loss becomes smaller.

[0050] As a result, during the model learning process of the image segmentation model M, the model learning unit 11 optimizes the parameters of the image segmentation model M, which is obtained. Here, during the model learning process of the image segmentation model M, multiple smoothness constraint parameters λ are combined by the coupling unit 112 of the model learning unit 11, so the learned image segmentation model M can perform segmentation inference adapted to each smoothness constraint parameter λ, that is, an image segmentation model M with adjustable boundary smoothness is obtained. The learned image segmentation model M is a model obtained by optimizing the parameters, and is therefore a trained model.

[0051] The following describes an example of the configuration of the segmentation inference unit 12 in the first embodiment with reference to Figure 4. Figure 4 is a block diagram showing an example of the configuration of the segmentation inference unit 12 in the first embodiment.

[0052] The segmentation inference unit 12 is a functional unit that performs segmentation inference using medical images and outputs the segmentation inference results. As shown in Figure 4, the segmentation inference unit 12 includes a second input unit 121, an inference unit 122, and an output unit 123.

[0053] In the segmentation inference step based on the trained image segmentation model M, the segmentation inference unit 12 performs segmentation inference on the medical image using the trained image segmentation model M, the input smoothness constraint parameter λ, and the medical image, and outputs the segmentation inference result.

[0054] In the first embodiment, the segmentation inference unit 12 performs segmentation inference on a medical image using a learned image segmentation model M based on smoothness constraint parameters λ manually input by an operator (e.g., a doctor or technician).

[0055] Specifically, the second input unit 121 receives a medical image and a smoothness constraint parameter λ manually entered by the operator. Here, the input medical image is the medical image for which segmentation inference should be performed using the image segmentation model M. Also, as shown in Figure 4, the operator selects, for example, a smoothness constraint parameter λ=30 from a list containing multiple smoothness constraint parameters λ. Here, the list containing multiple smoothness constraint parameters λ coincides with the parameter range of the smoothness constraint parameter λ used in the model learning process of the image segmentation model M.

[0056] In this embodiment, the operator may appropriately select or input other smoothness constraint parameters λ depending on the prominence of the physical reference boundary in the medical image on which segmentation inference is to be performed, provided that the selected or input smoothness constraint parameters λ are within the pre-set parameter range.

[0057] For example, when performing segmentation inference on medical images with inconspicuous physical reference boundaries, a large smoothness constraint parameter λ can be manually input, while when performing segmentation inference on medical images with prominent physical reference boundaries, a small smoothness constraint parameter λ can be manually input. For example, the second input unit 121 may be implemented by an input interface such as a mouse or keyboard and a processor such as a CPU.

[0058] Furthermore, the inference unit 122 performs segmentation inference using the learned image segmentation model M, the smoothness constraint parameter λ, and the medical image. For example, the inference unit 122 may be implemented by a processor such as a CPU.

[0059] Specifically, the inference unit 122 performs segmentation inference on a medical image using the image segmentation model M learned by the model learning unit 11, based on the smoothness constraint parameter λ manually entered by the operator, and obtains a specific region as a result of the segmentation inference.

[0060] Furthermore, the output unit 123 outputs the segmentation inference result for a specific region included in the medical image. For example, the output unit 123 may be implemented by a processor such as a CPU and a display. When the output unit 123 is implemented by a processor and a display, the processor causes the segmentation inference result to be displayed on the display.

[0061] As a result, according to the image processing apparatus 100 of the first embodiment, an image segmentation model M with adjustable boundary smoothness can be obtained, and the accuracy and efficiency of segmentation inference can be improved.

[0062] This is because, during the model learning process of the image segmentation model M, the model learning unit 11 combines multiple smoothness constraint parameters λ with labeled image data to train the image segmentation model M. As a result, the trained image segmentation model M can perform segmentation inference based on the multiple smoothness constraint parameters λ. During the segmentation inference process of the image segmentation model M, the segmentation inference unit 12 performs segmentation inference of medical images using the image segmentation model M trained based on the smoothness constraint parameters λ input by the user. This allows for segmentation inference to be performed using appropriate smoothness constraint parameters λ in response to differences in the physical reference boundary in the medical image to be segmented, thereby improving the smoothness of the boundary and enhancing the accuracy and efficiency of segmentation inference.

[0063] (Image processing method of the first embodiment) The flow of the image processing method according to the first embodiment will be described below with reference to Figure 5. Figure 5 is a flowchart showing the flow of the image processing method according to the first embodiment.

[0064] In step S101, the first input unit 111 receives labeled image data and a smoothness constraint parameter λ for constraining the smoothness of the boundary of the segmentation inference result. Specifically, for example, medical images, ground truth GT (Ground Truth), and the smoothness constraint parameter λ are input as input data to the first input unit 111. Here, the first input unit 111 may further receive a preset parameter range of the smoothness constraint parameter λ. Next, it proceeds to step S102.

[0065] In step S102, the combining unit 112 combines the smoothness constraint parameter λ input via the first input unit 111 with the labeled image data. Specifically, as described above, for the n cascade layers in the image segmentation model M, the combining unit 112 selects n smoothness constraint parameters λ1~λ n within a preset parameter range, and uses a learnable fully connected layer to convert the n smoothness constraint parameters λ1~λ n into n tensors t1~t n of the number of input channels, and multiplies the n tensors t1~t n converted from the n smoothness constraint parameters λ1~λ n by the input channels respectively, thereby combining the smoothness constraint parameter λ with the labeled image data. Next, it proceeds to step S103.

[0066] In step S103, the parameter optimization unit 113 uses the smoothness constraint parameter λ combined by the combining unit 112 to train the image segmentation model M. Next, it proceeds to step S104.

[0067] In step S104, the parameter optimization unit 113 calculates an evaluation value Loss using the above formula (1) based on the training result in step S103. Next, it proceeds to step S105.

[0068] In step S105, the parameter optimization unit 113 determines whether the calculated evaluation value Loss satisfies a preset loss condition. Here, the preset loss condition for the evaluation value Loss is, for example, the condition that the evaluation value Loss is below the evaluation value threshold. Note that the preset loss condition for the evaluation value Loss may be any other condition, as long as it results in the best learning result of the image segmentation model M.

[0069] In step S105, if the parameter optimization unit 113 determines that the calculated evaluation value Loss does not satisfy the preset loss condition (NO in step S105), it proceeds to step S106.

[0070] In step S106, the parameter optimization unit 113 modifies the parameters of the image segmentation model M. For example, the parameter optimization unit 113 modifies the parameters of the image segmentation model M so that the evaluation value Loss of the learning result becomes smaller. Next, the process proceeds to step S103, where the parameter optimization unit 113 trains the image segmentation model M using the modified parameters and obtains the learning result. In other words, the parameter optimization unit 113 repeatedly executes steps S103 to S106 until the parameters of the image segmentation model M are optimized.

[0071] In step S105, if the parameter optimization unit 113 determines that the calculated evaluation value Loss satisfies the preset loss conditions (YES in step S105), it proceeds to step S107.

[0072] In step S107, the second input unit 121 of the segmentation inference unit 12 receives the medical image and the smoothness constraint parameter λ manually entered by the operator. Here, the operator selects, for example, the smoothness constraint parameter λ=30 from a list containing multiple smoothness constraint parameters λ. Next, the process proceeds to step S108.

[0073] In step S108, the inference unit 122 performs segmentation inference using the image segmentation model M learned by the model learning unit 11, the smoothness constraint parameter λ input by the second input unit 121, and the medical image. Specifically, the inference unit 122 performs segmentation inference on the medical image input via the second input unit 121, using the image segmentation model M learned by the model learning unit 11, based on the smoothness constraint parameter λ manually input by the operator via the second input unit 121, and obtains a specific region as the segmentation inference result. Next, the process proceeds to step S109.

[0074] In step S109, the output unit 123 outputs the segmentation inference result for a specific region included in the medical image that has been segmented inferred by the inference unit 122. Then, the flow is terminated.

[0075] As a result, according to the image processing method of this embodiment, an image segmentation model M with adjustable boundary smoothness can be obtained, and the accuracy and efficiency of segmentation inference can be improved while improving boundary smoothness.

[0076] (Second embodiment) The image processing apparatus 200 according to the second embodiment will be described below with reference to Figures 6 and 7. Figure 6 is a block diagram showing an example configuration of the image processing apparatus 200 according to the second embodiment. Figure 7 is a block diagram showing an example configuration of the segmentation inference unit 22 in the second embodiment. In describing the second embodiment, only the differences from the first embodiment will be described, and the same content as in the first embodiment will be omitted.

[0077] In the second embodiment, the image processing apparatus 200 differs from the image processing apparatus 100 according to the first embodiment in that it includes a processing circuit 1A shown in Figure 6 instead of processing circuit 1. Furthermore, the processing circuit 1A differs from the processing circuit 1 according to the first embodiment in that it includes a segmentation inference unit 22 instead of segmentation inference unit 12. The processing circuit 1A is implemented, for example, by a processor. In the image processing apparatus 200 of the second embodiment, the model learning unit 11 included in the processing circuit 1A and the model learning process of the image segmentation model M by the model learning unit 11 are the same as in the first embodiment, so a description is omitted.

[0078] As shown in Figure 7, the segmentation inference unit 22 includes a second input unit 221, an inference unit 222, a smoothness determination unit 223, and an output unit 224.

[0079] The second input unit 221 receives the medical image and the initial value λ0 of the smoothness constraint parameter.

[0080] Specifically, the second input unit 221 receives a medical image to which segmentation inference should be performed using the image segmentation model M. In the following explanation, the initial value of the smoothness constraint parameter λ0 is assumed to be 1. The initial value of the smoothness constraint parameter λ0 may be any other value within a pre-set parameter range. For example, the second input unit 221 may be implemented by an input interface such as a mouse or keyboard and a processor such as a CPU.

[0081] The inference unit 222 performs segmentation inference using the trained image segmentation model M, the initial value λ0 of the smoothness constraint parameter, and the medical image.

[0082] Specifically, the inference unit 222 performs segmentation inference on a medical image using the image segmentation model M learned by the model learning unit 11, based on the initial value λ0 of the smoothness constraint parameter, and obtains a specific region as the segmentation inference result. For example, the inference unit 222 may be implemented by a processor such as a CPU.

[0083] The smoothness determination unit 223 determines whether the smoothness index of the segmentation inference result of a specific region included in a medical image based on the image segmentation model M satisfies the pre-set smoothness conditions. For example, the smoothness determination unit 223 may be implemented by a processor such as a CPU.

[0084] Here, the "smoothness index" can be an index such as the curvature or the number of holes in the segmentation inference result. Note that "holes" refer to a case where, for example, multiple liver segments are obtained by segmentation inference by the inference unit 222. In such a case, one liver segment obtained by segmentation inference may contain another liver segment. Specifically, the area of ​​the other liver segment may be enclosed within the area of ​​the first liver segment, thus being included within the first liver segment. In such a case, the area of ​​the other liver segment within the first liver segment is counted as a "hole". In this embodiment, the "smoothness index" is described as the number of holes in the segmentation inference result. The pre-set "condition" is that the number of holes in the segmentation inference result is less than a threshold. In the following explanation, the threshold is set to 4. The pre-set smoothness condition is that the number of holes in the segmentation inference result is #holes<4. When using the curvature of the segmentation inference results or other metrics as a "smoothing index," the conditions corresponding to that smoothing index should be set appropriately.

[0085] Specifically, the smoothness determination unit 223 calculates the smoothness index Metric #holes using the following formula (2).

[0086] Metric #holes= Σx,y,z hole ···(2) Here, in equation (2), Σx,y,z hole represents the number of holes in the segmentation inference result of a specific region (a three-dimensional region of the x, y, and z axes) contained in the medical image.

[0087] Next, the smoothness determination unit 223 determines whether the calculated smoothness index Metric #holes satisfies the pre-set smoothness conditions.

[0088] If the smoothness determination unit 223 determines that the calculated smoothness index Metric #holes does not satisfy the pre-set smoothness conditions, it increases the smoothness constraint parameter λ. For example, if the inference unit 222 performs segmentation inference on a medical image using the image segmentation model M based on the initial value λ0 of the smoothness constraint parameter and obtains a segmentation inference result, the smoothness determination unit 223 increases the smoothness constraint parameter λ from the initial value λ0.

[0089] At this point, the inference unit 222, based on the increased smoothness constraint parameter λ, uses the image segmentation model M learned by the model learning unit 11 to perform segmentation inference on the medical image and obtains a specific region as a result of the segmentation inference.

[0090] Furthermore, the output unit 224 outputs the segmentation inference result when the smoothness determination unit 223 determines that the calculated smoothness index Metric #holes satisfies the pre-set smoothness conditions. For example, the output unit 224 may be implemented by a processor such as a CPU and a display. When the output unit 224 is implemented by a processor and a display, the processor displays the segmentation inference result on the display.

[0091] In other words, the smoothness determination unit 223 repeatedly increases the smoothness constraint parameter λ until the smoothness index Metric #holes of the segmentation inference result based on the smoothness constraint parameter λ satisfies the pre-set smoothness conditions.

[0092] As a result, according to the image processing apparatus 200 of the second embodiment, an image segmentation model M with adjustable boundary smoothness can be obtained, and the accuracy and efficiency of segmentation inference can be further improved.

[0093] This is because, in the segmentation inference process of the image segmentation model M, the segmentation inference unit 22 gradually increases the smoothness constraint parameter λ from its initial value λ0 until it determines that the smoothness index Metric #holes of the segmentation inference result satisfies the preset smoothness conditions. This allows for segmentation inference of medical images based on a more appropriate smoothness constraint parameter λ, improving boundary smoothness and increasing the accuracy of segmentation inference. Furthermore, by automatically selecting an appropriate smoothness constraint parameter λ, an image segmentation model M with automatically adjustable boundary smoothness is obtained, improving the efficiency of inference.

[0094] (Image processing method of the second embodiment) The following describes a portion of the flow of the image processing method according to the second embodiment, with reference to Figure 8. Figure 8 is a flowchart showing a portion of the flow of the image processing method according to the second embodiment. In describing the image processing method of the second embodiment, only the procedures that differ from the first embodiment will be described, and the procedures that are the same as those of the first embodiment will be omitted. In the image processing method of the second embodiment, the model learning process of the image segmentation model M (steps S101 to S106) is the same as in the first embodiment, so the description will be omitted. In addition, the image processing method of the second embodiment differs from the image processing method of the first embodiment in that the processes in steps S207 to S213 are executed instead of steps S107 to S109 in the image processing method of the first embodiment.

[0095] In step S207, the second input unit 221 of the segmentation inference unit 22 receives the medical image and the initial value λ0 of the smoothness constraint parameter. Here, for example, the initial value λ0 of the smoothness constraint parameter is set to 1. Next, the process proceeds to step S208.

[0096] In step S208, the inference unit 222 performs segmentation inference using the image segmentation model M trained by the model learning unit 11, the initial value λ0 of the smoothness constraint parameter input by the second input unit 221, and the medical image. Specifically, the inference unit 122 performs segmentation inference on the medical image input via the second input unit 221 using the image segmentation model M trained by the model learning unit 11, based on the initial value λ0 of the smoothness constraint parameter input via the second input unit 221, and obtains a specific region as the segmentation inference result. Next, the process proceeds to step S209.

[0097] In step S209, the smoothness determination unit 223 calculates the smoothness index Metric #holes of the segmentation inference result performed by the inference unit 222 using the formula (2) described above. Next, the process proceeds to step S210.

[0098] In step S210, the smoothness determination unit 223 determines whether the calculated smoothness index Metric #holes satisfies a preset smoothness condition (for example, #holes < 4). If the smoothness determination unit 223 determines that the calculated smoothness index Metric #holes does not satisfy the preset smoothness condition (Metric #holes ≥ 4) (NO in step S210), the process proceeds to step S211. If the smoothness determination unit 223 determines that the calculated smoothness index Metric #holes satisfies the preset smoothness condition (Metric #holes < 4) (YES in step S210), the process proceeds to step S213.

[0099] In step S211, the smoothness determination unit 223 increases the smoothness constraint parameter λ. For example, during the first execution of step S211, the smoothness determination unit 223 increases the smoothness constraint parameter λ from its initial value λ0, and in subsequent executions of step S211, the smoothness determination unit 223 increases the smoothness constraint parameter λ during the execution process. Next, the process proceeds to step S212. For example, in step S211, the smoothness determination unit 223 increases the smoothness constraint parameter λ by a predetermined value.

[0100] In step S212, the inference unit 222 performs segmentation inference on the medical image using the learned image segmentation model M based on the increased smoothness constraint parameter λ and obtains the segmentation inference result. Next, proceeding to step S209, the smoothness determination unit 223 calculates the smoothness index Metric #holes of the segmentation inference result performed by the inference unit 222 using the above-described equation (2), and repeats the processing from step S210 onwards.

[0101] In step S213, the output unit 224 outputs the segmentation inference result determined by the smoothness determination unit 223, which determines that the calculated smoothness index Metric #holes satisfies the pre-set smoothness conditions.

[0102] In other words, the step of increasing the smoothness constraint parameter λ is repeatedly performed until the smoothness index Metric #holes of the segmentation inference result based on the smoothness constraint parameter λ satisfies the predetermined smoothness condition.

[0103] As a result, according to the image processing method of the second embodiment, it is possible to obtain an image segmentation model M in which the smoothness of the boundaries can be adjusted, and the accuracy and efficiency of segmentation inference can be further improved.

[0104] (Specific example) The following describes a specific example of the second embodiment with reference to Figure 9. Figure 9 is a schematic diagram showing a specific example of the second embodiment. In describing the specific example, it is assumed that the pre-set smoothness condition is #holes<4 and the initial value of the smoothness constraint parameter λ0=1. The first column of Figure 9 (the leftmost column in Figure 9) shows the medical images (Case 1, Case 2) to be segmented using the image segmentation model M, and the second column of Figure 9 (the middle column in Figure 9) shows the segmentation inference results obtained by segmenting the medical images using the image segmentation model M based on the initial value of the smoothness constraint parameter λ0. The third column of Figure 9 (the rightmost column in Figure 9) shows the segmentation inference results obtained by segmenting the medical images using the image segmentation model M based on an increased smoothness constraint parameter λ (for example, smoothness constraint parameter λ=30).

[0105] As shown in Figure 9, the smoothness of a specific region in the medical image of Case 1 is good, while the smoothness of a specific region in the medical image of Case 2 is poor.

[0106] When segmentation inference is performed on the medical image of Case 1 using the image processing device 200 according to the second embodiment, the smoothness index Metric #holes of the segmentation inference result based on the initial value λ0 of the smoothness constraint parameter satisfies the preset smoothness condition (Metric #holes < 4). At this time, the segmentation inference result based on the initial value λ0 of the smoothness constraint parameter is output by the output unit 224.

[0107] When segmentation inference is performed on the medical image of Case 2 using the image processing apparatus 200 according to the second embodiment, the segmentation inference result based on the initial value λ0 of the smoothness constraint parameter shows that the smoothness of the region indicated by the ellipse in Figure 9 is poor, and the smoothness index Metric #holes does not satisfy the preset smoothness condition (Metric #holes≧4). At this time, the smoothness determination unit 223 increases the smoothness constraint parameter λ. When the smoothness constraint parameter λ is increased to 30, the segmentation inference result based on this smoothness constraint parameter λ shows that the smoothness of the region indicated by the ellipse in Figure 9 is improved, and the smoothness index Metric #holes satisfies the preset smoothness condition (Metric #holes<4). At this time, the segmentation inference result based on the initial value λ=30 of the smoothness constraint parameter is output by the output unit 224.

[0108] As a result, during the segmentation inference process of the image segmentation model M, the segmentation inference unit 22 gradually increases the smoothness constraint parameter λ from its initial value λ0 until it is determined that the smoothness index Metric #holes of the segmentation inference result satisfies the preset smoothness conditions. This allows for segmentation inference of medical images based on a more appropriate smoothness constraint parameter λ, improving boundary smoothness and increasing the accuracy of segmentation inference. Furthermore, by automatically selecting an appropriate smoothness constraint parameter λ, an image segmentation model with automatically adjustable boundary smoothness is obtained, improving the efficiency of inference.

[0109] In the above explanation, the example described is that in the segmentation inference step of the image segmentation model M, the smoothness constraint parameter λ is selected by the operator from a list containing multiple smoothness constraint parameters λ and input to the second input unit 121. However, the embodiments are not limited to this. For example, the operator may input the smoothness constraint parameter λ within a preset parameter range.

[0110] Furthermore, in the embodiments described above, the processing circuit is not limited to being implemented by a single processor, but may be composed of a combination of multiple independent processors, with each processor executing a program to realize each processing function. Also, each processing function of the processing circuit may be implemented by appropriately distributing or integrating it across one or more processing circuits. Moreover, each processing function of the processing circuit may be implemented by a mixture of hardware such as circuits and software. In addition, although an example in which the programs corresponding to each processing function are stored in a single memory circuit has been described here, the embodiments are not limited to this. For example, the programs corresponding to each processing function may be distributed and stored in multiple memory circuits, and the processing circuit may read and execute each program from each memory circuit. Furthermore, while the embodiments described above illustrate examples where each functional unit in this specification is implemented by a processing circuit, the embodiments are not limited to these. For example, each functional unit in this specification may be implemented not only by the processing circuits described in the embodiments, but also by hardware alone, software alone, or a combination of hardware and software.

[0111] Furthermore, the term "processor" used in the above-described embodiments refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). Here, instead of storing the computer program in a memory circuit, the computer program may be directly incorporated into the processor's circuit. In this case, the processor realizes its function by reading and executing the program incorporated into the circuit. Moreover, each processor in this embodiment is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor and realize its function.

[0112] Furthermore, the "computer program product" used in the description of the embodiments includes a computer program executed by a processor, and this computer program is provided pre-installed in ROM (Read Only Memory) or memory circuits. This computer program may also be provided as a file in an installable or executable format on a computer-readable non-transient storage medium such as a CD (Compact Disk)-ROM, FD (Flexible Disk), CD-R (Recordable), or DVD (Digital Versatile Disk). Alternatively, this computer program may be stored on a computer connected to a network such as the Internet and provided or distributed by downloading it via the network. For example, this computer program consists of modules containing the processing functions described above. In actual hardware, the CPU reads the computer program from a storage medium such as ROM and executes it, loading each module onto the main memory and generating it in the main memory.

[0113] Furthermore, in the embodiments described above, each component of each illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific form of distribution or integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads and usage conditions. Moreover, each processing function performed by each device can be implemented, in whole or in any part, by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0114] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0115] According to at least one embodiment or modification described above, an image segmentation model with adjustable boundary smoothness can be obtained, and the accuracy and efficiency of segmentation inference can be improved while improving boundary smoothness.

[0116] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0117] 1.1A Processing Circuit 11 Model Learning Section 12,22 Segmentation Inference Unit

Claims

1. An image processing method that performs segmentation inference on a specific region contained in a medical image using an image segmentation model based on a deep neural network, A model learning step is performed in which a first labeled medical image and a first smoothness constraint parameter for restricting the smoothness of the boundary of the segmentation inference result are input, and the image segmentation model is trained to obtain the image segmentation model in which the first smoothness constraint parameter can be adjusted. A segmentation inference step which involves performing segmentation inference on the second medical image using the learned image segmentation model, the input second smoothness constraint parameters, and the second medical image, and outputting the segmentation inference result, Image processing methods, including those mentioned above.

2. The aforementioned model training step is, An input step in which the labeled first medical image and the first smoothness constraint parameter, which is a random number within the parameter range, are input, A coupling step of coupling the first smoothness constraint parameter to the labeled first medical image, A model parameter optimization step involves training the image segmentation model using the combined first smoothness constraint parameters, calculating an evaluation value Loss based on the training results, and optimizing the parameters of the image segmentation model. The image processing method according to claim 1, including the method described in claim 1.

3. The segmentation inference step is as follows: An input step in which the second medical image and the second smoothness constraint parameter manually entered by the operator are input, An inference step in which segmentation inference is performed using the image segmentation model, the second smoothness constraint parameter, and the second medical image, An output step that outputs the segmentation inference result for a specific region included in the second medical image, The image processing method according to claim 1, including the method described in claim 1.

4. The segmentation inference step is as follows: An input step in which the second medical image and the initial value of the second smoothness constraint parameter are input, An inference step in which segmentation inference is performed using the image segmentation model, the second smoothness constraint parameter, and the second medical image, A smoothness determination step which determines whether the smoothness index of the segmentation inference result of a specific region included in the second medical image satisfies a predetermined smoothness condition, and if it is determined that the smoothness condition is not met, increases the second smoothness constraint parameter and re-executes the inference step, If it is determined in the smoothness determination step that the smoothness condition is met, an output step is performed to output the segmentation inference result, The image processing method according to claim 1, including the method described in claim 1.

5. The smoothness index is the number of holes in the segmentation inference result. The image processing method according to claim 4.

6. An image processing device that performs segmentation inference on a specific region contained in a medical image using an image segmentation model based on a deep neural network, A model learning unit receives a labeled first medical image and a first smoothness constraint parameter for restricting the smoothness of the boundary of the segmentation inference result as input, trains the image segmentation model, and obtains an image segmentation model in which the first smoothness constraint parameter can be adjusted. A segmentation inference unit that performs segmentation inference on the second medical image using the learned image segmentation model, the input second smoothness constraint parameters, and the second medical image, and outputs the segmentation inference result, An image processing device equipped with the following features.

7. A program that causes a computer to perform segmentation inference on specific regions contained in medical images using an image segmentation model based on a deep neural network, A model learning process is performed in which a labeled first medical image and a first smoothness constraint parameter for restricting the smoothness of the boundary of the segmentation inference result are input, and the image segmentation model is trained to obtain the image segmentation model in which the first smoothness constraint parameter can be adjusted. A segmentation inference process that performs segmentation inference on the second medical image using the learned image segmentation model, the input second smoothness constraint parameters, and the second medical image, and outputs the segmentation inference result, A program that causes a computer to perform a process that includes [a specific action].

Images