Image processing apparatus and image processing method

Abstract

According to the image processing apparatus and the image processing method of the present invention, weighted values can be appropriately given to boundaries, and the accuracy of boundary segmentation can be improved. The image processing apparatus includes: a receiving unit that receives training data including an image and a plurality of pieces of tag information associated with pixels in the image, and divides the plurality of pieces of tag information into a plurality of tag channels according to tag categories of the tag information; the weight map generation unit is used for generating a weight map according to the label information, and the weight map comprises weight information of pixels in the image; the loss calculation unit is used for calculating loss according to weight information in the weight map; a model training unit; and the image segmentation unit is used for training the prediction model based on the calculated loss in the training process of the prediction model, and extracting the boundary of the target area in the image by the image segmentation unit by using the trained prediction model in the prediction process of the prediction model.

Description

Technical Field

[0001] This invention relates to an image processing apparatus and an image processing method. Background Technology

[0002] In medical image segmentation tasks, high accuracy of boundaries is typically required. For example, in clinical examinations, it is necessary to accurately generate the boundaries of the target region. In radiotherapy planning, it is necessary to accurately plan the target region to maximize tumor eradication while protecting surrounding tissues. Furthermore, to distinguish between benign and malignant tumors, clear surface morphology is required.

[0003] Convolutional neural networks based on semantic segmentation suffer from low performance when processing the boundaries of target regions. In medical images, accurate tracking of disease biomarkers requires high precision in understanding the surface and volume of biological tissues; for example, the morphological and shape features of biological tissues are extremely important.

[0004] Furthermore, by setting weights through a loss function, the accuracy of boundary segmentation in medical images can be improved. In supervised learning of machine learning, a series of training samples are typically provided in advance, and the mapping relationships within these training samples are learned. Given an element that does not belong to the training samples, a predicted value is output, making the predicted value as close as possible to the true value. The loss function can be used to estimate the degree of inconsistency between the predicted value and the true value; the smaller the loss function, the better the robustness of the model.

[0005] In a prior art, there is a known image processing method that sets a loss function based on a distance map. This method generates a distance map from real data and creates a penalty term for the loss function based on this generated distance map, thereby directing the neural network's attention to segmentation-difficult boundaries in the target region.

[0006] However, this existing technique calculates the Euclidean distance between each point and the boundary, assigning greater weight to points that are closer, thus performing weighted processing. Therefore, the weights based on the distance map only reflect the distance to the boundary, assigning the same weight to all boundaries while ignoring the impact of boundary complexity on segmentation difficulty, resulting in inaccurate segmentation of complex boundaries. Furthermore, this technique requires a huge computational workload; when facing 3D image segmentation tasks, it is necessary to first calculate the distance map for all views, increasing computational costs and making it unsuitable for 3D image segmentation tasks.

[0007] In another existing technology, the target region is classified, and different weights are assigned to different categories of target regions. Typically, smaller target regions are assigned larger weights.

[0008] However, this existing technology can only improve the detection rate for smaller target regions. It is prone to oversegmentation when dealing with small boundaries, failing to properly assign weights to the boundaries and improve the accuracy of boundary segmentation. Summary of the Invention

[0009] The present invention was made in view of the above problems, and its object is to provide an image processing apparatus and an image processing method that can appropriately assign weight values ​​to boundaries and improve the accuracy of boundary segmentation.

[0010] According to the image processing apparatus of the present invention, a target region is segmented from an image using a prediction model, characterized by comprising: a receiving unit for receiving training data, the training data including an image and multiple label information corresponding to pixels in the image; a weight map generation unit for generating a weight map based on the label information, the weight map including weight information of the pixels in the image; a loss calculation unit for calculating a loss based on the weight information in the weight map; a model training unit; and an image segmentation unit, wherein during the training of the prediction model, the model training unit trains the prediction model based on the calculated loss, and during the prediction process of the prediction model, the image segmentation unit uses the trained model to extract the boundary of the target region in the image.

[0011] Therefore, a weight map is generated based on the label information of each pixel in the image, and the loss is calculated based on the weight information in the weight map. During the training process of the prediction model, the prediction model is trained based on the calculated loss. During the prediction process of the prediction model, the trained prediction model is used to extract the boundary of the target region in the image.

[0012] This allows for the appropriate assignment of weights to boundaries and improves the accuracy of boundary segmentation. This is because the weight map generated from the label information of each pixel includes more boundary information. Weighting the boundary of the target region based on the weight information in the weight map enables the prediction model to accurately focus its attention on the weighted boundary of the target region.

[0013] In addition, the weight information in the weight map generated based on the label information of each pixel reflects the complexity of the boundary of the target region. More weights are assigned to the more complex boundaries, so that the attention of the prediction model is accurately focused on the more complex boundaries. Therefore, it is possible to further assign appropriate weight values ​​to the boundaries and improve the accuracy of boundary segmentation.

[0014] In addition, the weight information in the weight map generated based on the label information of each pixel reflects the size of the boundary of the target region. Appropriate weights can be assigned to both large and small boundaries, so that the prediction model can appropriately allocate attention to boundaries of different sizes. Therefore, it is possible to further assign appropriate weight values ​​to the boundaries and improve the accuracy of boundary segmentation.

[0015] Furthermore, during the training process of the prediction model, the prediction model is trained based on the calculated loss. Therefore, the image processing apparatus of the present invention can be applied to existing neural networks without modifying the existing neural network training process or network structure.

[0016] According to the image processing apparatus of the present invention, the weight map generation unit divides a plurality of label information into a plurality of label channels according to the label category of the label information, wherein the label channel is a probability map including probability values ​​representing the relationship between the label information of the pixel and the label category, calculates the neighborhood weight of the label information based on any one of the plurality of label information and the label information of the neighboring label information, calculates the global weight of the label information based on the neighborhood weight, and generates the weight map based on the neighborhood weight and the global weight.

[0017] Therefore, the neighborhood weight of the label information is calculated based on the label information and the label information of the neighboring neighborhood of the label information, and the global weight is calculated based on the neighborhood weight. A weight graph is generated based on the neighborhood weight and the global weight, the loss is calculated based on the weight information in the weight graph, and the prediction model is trained based on the calculated loss.

[0018] This allows for the appropriate assignment of weights to boundaries, improving the accuracy of boundary segmentation. This is because, at the boundaries of target regions in an image, the label information of adjacent pixels typically differs. Furthermore, when a pixel is located on a highly complex boundary, the label information of its neighborhood changes significantly, while when a pixel is located on a less complex boundary, the change is smaller. Therefore, by generating a weight map based on neighborhood weights and global weights and calculating the loss, the weight information and loss in the weight map reflect the label information in a pixel's neighborhood as well as the global label information. The weight values ​​and loss include more boundary information, allowing the prediction model to more accurately focus its attention on the weighted boundaries of the target region.

[0019] According to the image processing apparatus of the present invention, the weight generation unit calculates the average of the probability values ​​of the plurality of pixels in the neighborhood of any one of the plurality of pixels in each of the label channels as the first-order neighborhood weight of that pixel.

[0020] According to the image processing apparatus of the present invention, the weight generation unit performs k downsampling operations on the probability map of the label channel in each label channel, calculates the average of the probability values ​​of the plurality of pixels in the neighborhood of the pixel as the k+1 order sampling weight of the pixel for any one of the plurality of pixels after downsampling, generates a k+1 order sampling weight map representing the k+1 order sampling weight of the pixel, performs k upsampling operations on the k+1 order sampling weight map, and generates the k+1 order neighborhood weight of the pixel, wherein k is an integer greater than 0 and less than n, and n is the maximum order of the prediction model.

[0021] According to the image processing apparatus of the present invention, the weight generation unit calculates the average of the first-order to n-order neighborhood weights of any one of the plurality of pixels in each of the label channels, and uses this average as the neighborhood weight of the pixel.

[0022] Therefore, in each label channel, for any pixel, the first-order to n-order neighborhood weights of the pixel are calculated based on the probability values ​​of multiple pixels in the pixel's neighborhood, and the average of the first-order to n-order neighborhood weights of the pixel is calculated as the neighborhood weight of the pixel.

[0023] This allows for the appropriate assignment of weights to boundaries and improves the accuracy of boundary segmentation. This is because the neighborhood weights of a pixel, calculated based on the probability values ​​of multiple pixels in its neighborhood, reflect the label information of that pixel's neighborhood. Consequently, the weight information in the weight map reflects more boundary information, enabling the appropriate assignment of weights to boundaries of varying complexity or size. This allows the prediction model to more accurately focus its attention on the weighted boundary of the target region.

[0024] According to the image processing apparatus of the present invention, the weight generation unit extracts the neighborhood weights of pixels whose label categories are consistent with the label categories of the label channel in each label channel, integrates the extracted neighborhood weights into a full label map, and calculates the global weight of any pixel in the full label map based on the proportion of the label category of the pixel in the full label map.

[0025] Therefore, the neighborhood weights of pixels with the same label category in each label channel are integrated into a full label map, and the global weight of the pixel is calculated based on the proportion of the label category of any pixel in the full label map.

[0026] This allows for the appropriate assignment of weights to boundaries and improves the accuracy of boundary segmentation. This is because global weights reflect the label information across the entire system; therefore, the weight information in the weight graph reflects more boundary information, enabling the global differentiation of boundaries of varying complexity or size and the appropriate assignment of weights. This allows the prediction model to focus its attention more accurately on the weighted boundaries of the target region.

[0027] According to the image processing apparatus of the present invention, the loss calculation unit calculates the loss based on the neighborhood weights and the global weights.

[0028] This allows for the appropriate assignment of weights to boundaries, improving the accuracy of boundary segmentation. This is because neighborhood weights and global weights reflect the label information of a pixel's neighborhood and the global label information, incorporating more boundary information into the calculated loss. Consequently, weights can be appropriately assigned to boundaries of varying complexity or size, enabling the prediction model to focus its attention more accurately on the weighted boundaries of the target region.

[0029] According to the image processing apparatus of the present invention, the loss is at least one of Dice loss, cross-entropy loss, and Focal loss.

[0030] Therefore, different losses can be used to train the prediction model according to the needs of the segmentation task or the neural network. That is, the image processing apparatus of the present invention can be applied to existing neural networks without modifying the training process or network structure of existing neural networks. Furthermore, it can be easily integrated with existing neural networks.

[0031] Furthermore, when faced with new segmentation tasks, it can adaptively train on new training data, making it easily applicable to any segmentation task.

[0032] According to the image processing apparatus of the present invention, the average weight of the weighted graph is 1.

[0033] Therefore, the image processing apparatus of this invention can be applied to existing neural networks without changing the overall training step size of the neural network or modifying the existing neural network training process or network structure. Furthermore, it can be easily integrated with existing neural networks.

[0034] According to the image processing apparatus of the present invention, the maximum order n is set according to the receptive field of the neural network of the prediction model.

[0035] Therefore, the maximum order of the weight map can be set according to the characteristics of the neural network, and the image processing device of the present invention can be applied to existing neural networks without modifying the training process or network structure of the existing neural network.

[0036] According to the image processing apparatus of the present invention, a fill pixel is provided outside the edge of the probability map of the label channel.

[0037] Therefore, neighborhood weights can be appropriately calculated even at the edges of the probability graph.

[0038] According to the image processing apparatus of the present invention, in each of the label channels, if the label information of any pixel belongs to the label category corresponding to the label channel, the probability value of the pixel is represented as 0, and if the label information of any pixel does not belong to the label category corresponding to the label channel, the probability value of the pixel is represented as 1.

[0039] Therefore, in the probability map of the label channel, the probability value of pixels whose label information belongs to the label category corresponding to the label channel is represented as 0, and the probability value of pixels whose label information does not belong to the label category corresponding to the label channel is represented as 1. The neighborhood weight and global weight are calculated based on the probability map set in this way.

[0040] This allows for the appropriate assignment of weights to the boundaries and improves the accuracy of boundary segmentation. This is because the probability map, set in this way, more accurately reflects boundary information, enabling the prediction model to focus its attention more precisely on the weighted boundaries of the target region.

[0041] According to the image processing apparatus of the present invention, the prediction model segments the target region from a three-dimensional image, the loss calculation unit further calculates a cross-entropy loss, and adds the calculated loss to the cross-entropy loss as a second loss, and trains the prediction model based on the second loss.

[0042] According to the image processing apparatus of the present invention, it is characterized by further comprising a display unit, wherein during the prediction process of the prediction model, the weight generation unit generates a prediction weight map based on a probability map, and the display unit outputs and displays the prediction weight map.

[0043] According to the image processing apparatus of the present invention, during the prediction process of the prediction model, the display unit displays the prediction weight map overlapping the segmentation mask.

[0044] This allows for the segmentation of target regions from 3D images. Furthermore, the predicted weight map can be displayed visually along with the segmentation mask.

[0045] This enables the visualization of highly complex, ambiguous, or multi-class boundary regions, and allows attention to be more accurately focused on the indivisible boundaries within the target region.

[0046] According to the image processing method of the present invention, a target region is segmented from an image using a prediction model, characterized by comprising the following steps: receiving training data, the training data including an image and multiple label information corresponding to pixels in the image; generating a weight map based on the label information, the weight map including weight information of the pixels in the image; calculating a loss based on the weight information in the weight map; training the prediction model; and segmenting the image, wherein during the training of the prediction model, the prediction model is trained based on the calculated loss, and during the prediction process of the prediction model, the trained prediction model is used to extract the boundary of the target region in the image.

[0047] Invention Effects

[0048] The image processing apparatus and image processing method according to the present invention can appropriately assign weight values ​​to boundaries and improve the accuracy of boundary segmentation.

[0049] Furthermore, according to the image processing apparatus and image processing method of the present invention, no modification is required to the training process or network structure of the existing neural network, and the image processing apparatus and image processing method of the present invention can be applied to the existing neural network.

[0050] Furthermore, the image processing apparatus and image processing method according to the present invention can adaptively train on new training data when faced with a new segmentation task, and are easily applicable to any segmentation task.

[0051] Furthermore, the image processing apparatus and image processing method according to the present invention can be easily integrated with existing neural networks.

[0052] Furthermore, the image processing apparatus and image processing method according to the present invention can visualize complex, blurry, or multi-class boundary regions, and can further accurately focus attention on the difficult-to-divide boundaries in the target region. Attached Figure Description

[0053] Figure 1 This is a block diagram illustrating the configuration of the image processing apparatus according to the first embodiment.

[0054] Figure 2 This is a schematic diagram representing the label information of an image.

[0055] Figure 3This is a schematic diagram representing the label channel.

[0056] Figure 4 This is a diagram representing the filled pixels.

[0057] Figure 5 This is a schematic diagram illustrating the calculation process of the first-order neighborhood weight.

[0058] Figure 6 This is a schematic diagram illustrating the calculation process of the second-order neighborhood weights.

[0059] Figure 7 This is a schematic diagram illustrating the calculation process of the k+1 order neighborhood weights.

[0060] Figure 8 This is a schematic diagram illustrating the calculation process of neighborhood weights.

[0061] Figure 9 This is a schematic diagram illustrating the process of extracting neighborhood weights in the second label channel.

[0062] Figure 10 This is a schematic diagram representing the full label image.

[0063] Figure 11 This is a schematic diagram illustrating the calculation process of global weights.

[0064] Figure 12 This is a flowchart of the image processing apparatus according to the first embodiment.

[0065] Figure 13 (a) is a schematic diagram representing the true value of the segmentation task.

[0066] Figure 13 (b) is a schematic diagram representing the weight values ​​of the present invention.

[0067] Figure 13 (c) is a diagram showing the weight values ​​of the comparison examples.

[0068] Figure 14 This is a block diagram illustrating the configuration of the image processing apparatus according to the second embodiment.

[0069] Figure 15 (a) is a schematic diagram representing the prediction weight map.

[0070] Figure 15 (b) is a schematic diagram showing the prediction weight map and the segmentation mask displayed overlapping.

[0071] Figure 15 (c) is a schematic diagram representing a probability graph. Detailed Implementation

[0072] Hereinafter, preferred embodiments of the image processing apparatus and image processing method of the present invention will be described with reference to the accompanying drawings. In each embodiment, the same reference numerals are used for the same parts as in the prior drawings, and detailed descriptions of those parts are omitted; descriptions will focus primarily on the different parts.

[0073] The image processing apparatus of this invention comprises multiple functional modules. It can be installed as software in a standalone computer or other device with a CPU (central processing unit) and memory, or it can be distributed across multiple devices, with a processor executing the various functional modules of the image processing apparatus stored in memory. Alternatively, it can be implemented in hardware as circuitry capable of executing the various functions of the image processing apparatus. The circuitry implementing the image processing apparatus can transmit and receive data or acquire data via a network such as the Internet. Furthermore, the image processing apparatus of this invention can also be directly installed as part of an image acquisition device.

[0074] Furthermore, in the following description, the size, value, included information, order, and shape of each image, label channel, probability map, full label map, weight map, etc., are shown schematically for ease of explanation and are not intended to limit the scope of the present invention. The values ​​of each label information, probability value, neighborhood weight, and global weight, etc., are shown schematically for ease of explanation and are not intended to limit the scope of the present invention.

[0075] (First Implementation)

[0076] (Image processing device 100)

[0077] use Figure 1 The configuration of the image processing apparatus 100 according to the first embodiment will be described. Figure 1 This is a block diagram showing the configuration of the image processing apparatus 100 according to the first embodiment.

[0078] like Figure 1 As shown, the image processing apparatus 100 includes a receiving unit 1, a weight map generation unit 2, a loss calculation unit 3, a model training unit 4, an image segmentation unit 5, and a memory 6. The image processing apparatus 100 segments the target region T from the prediction image using a prediction model M, described later. The memory 6 stores the prediction model M.

[0079] (Receiver Unit 1)

[0080] The receiving unit 1 receives training data, which includes images and multiple label information corresponding to the pixels in the images.

[0081] use Figure 2 Provide explanations for the image's label information. Figure 2 This is a schematic diagram representing the label information of an image.

[0082] like Figure 2 As shown, the training data received by receiving unit 2 includes an image, which is used to segment target regions from the image using the prediction model M described later. The image comprises multiple pixels; here, an image with 12 × 12 = 144 pixels is used as an example for illustration. Furthermore, the training data received by receiving unit 2 also includes label information corresponding to each pixel in the image. Here, based on the different label categories, the label information is represented as first label information (“1” in the figure), second label information (“2” in the figure), and third label information (“3” in the figure). “1” in the first label information indicates that the label information belongs to the first label category, “2” in the second label information indicates that the label information belongs to the second label category, and “3” in the third label information indicates that the label information belongs to the third label category.

[0083] In this invention, the image is, for example, a medical image captured by a medical image acquisition device, such as a two-dimensional abdominal ultrasound image captured by a medical image acquisition device during pregnancy. However, the image processed by the image processing device 100 is not limited to this; it may also be a medical image other than a two-dimensional abdominal ultrasound image, an image captured by other medical acquisition devices, or a three-dimensional image, or an image other than a medical image.

[0084] Furthermore, in this invention, an image comprising 12 × 12 = 144 pixels is used as an example for explanation. However, the number of pixels in an image is not limited to this and can be other numbers.

[0085] Furthermore, in this invention, the label information is represented as first label information (“1” in the figure), second label information (“2” in the figure), and third label information (“3” in the figure) as an example for explanation. However, the number of label information, the way it is represented, etc., are not limited to this, and may include fewer than 3 or more label categories, or may use other methods to represent labels.

[0086] (Weight Graph Generation Unit 2)

[0087] use Figures 3 to 11 The weighted graph generation unit 2 will be explained.

[0088] The label channel, as described later, is a probability graph that includes probability values ​​P representing the relationships between the first label information, the second label information, the third label information, and the first label category, the second label category, and the third label category. The label channel includes the first label channel, the second label channel, and the third label channel.

[0089] In addition, in each label channel, if the label information of any pixel belongs to the label category corresponding to that label channel, the probability value P of that pixel is 0, and if the label information of any pixel does not belong to the label category corresponding to that label channel, the probability value P of that pixel is 1.

[0090] use Figure 3 Explanation of the label channel. Figure 3 This is a schematic diagram representing the label channel.

[0091] Specifically, such as Figure 3 As shown, the weighted graph generation unit 2 generates weighted graphs based on the label categories to which the first label information, second label information, and third label information belong. Figure 2 The label information shown is one-hot encoded to generate... Figure 3 The top shows the unique heatmaps of the first label channel, the second label channel, and the third label channel.

[0092] In the unique heatmap of the first label channel, the value of a pixel with first label information belonging to the first label category is represented as 1, and the value of a pixel with second label information or third label information not belonging to the first label category is represented as 0.

[0093] In the unique heatmap of the second label channel, the value of a pixel with second label information belonging to the second label category is represented as 1, and the value of a pixel with first label information or third label information not belonging to the second label category is represented as 0.

[0094] In the unique heatmap of the third label channel, the value of a pixel with third label information belonging to the third label category is represented as 1, and the value of a pixel with first label information or second label information that does not belong to the third label category is represented as 0.

[0095] The weighted map generation unit 3 reverses the pixel values ​​in the unique heatmaps of the first label channel, the second label channel, and the third label channel, that is, it reverses the value of a pixel with a value of 1 to 0 and the value of a pixel with a value of 0 to 1, thereby generating (reversed) probability maps of the first label channel, the second label channel, and the third label channel.

[0096] In the probability map of the first label channel, the probability value P of a pixel having first label information belonging to the first label category is represented as 0, and the probability value P of a pixel having second label information or third label information that does not belong to the first label category is represented as 1.

[0097] In the probability map of the second label channel, the probability value P of a pixel having second label information belonging to the second label category is 0, and the probability value P of a pixel having first label information or third label information that does not belong to the second label category is 1.

[0098] In the probability map of the third label channel, the probability value P of a pixel having third label information belonging to the third label category is 0, and the probability value P of a pixel having first label information or second label information that does not belong to the third label category is 1.

[0099] Then, the weight graph generation unit 2 calculates the neighborhood weight W of the label information based on any label information in the label information and the label information of the neighboring neighborhood of the label information, and calculates the global weight C of the label information based on the neighborhood weight W, and generates a weight graph ф based on the neighborhood weight W and the global weight C.

[0100] use Figures 4 to 8 The calculation of the neighborhood weight W is explained. Figure 4 This is a diagram representing the filled pixels. Figure 5 This is a schematic diagram illustrating the calculation process of the first-order neighborhood weight W1. Figure 6 This is a schematic diagram illustrating the calculation process of the second-order neighborhood weight W2. Figure 7 It represents the weight W of the (k+1)th order neighborhood. k+1 A schematic diagram of the calculation process. Figure 8 This is a schematic diagram illustrating the calculation process of the neighborhood weight W.

[0101] Weight generation unit 2 sets fill pixels outside the edges of the probability maps of the first label channel, second label channel, and third label channel, and sets the probability value P of the fill pixels to 0. Thus, the probability map including the fill pixels comprises 14 × 14 = 196 pixels. In the accompanying figures, the borders of the fill pixels are represented by dashed lines.

[0102] Setting padding pixels outside the edges of the probability map is to facilitate the calculation of the neighborhood weights W, which will be described later. In this invention, setting padding pixels outside the edges of the probability map is used as an example, but it is not limited to this, and the calculation of the neighborhood weights W can also be implemented in other ways.

[0103] Next, in the first label channel, the second label channel, and the third label channel, the weight generation unit 2 calculates the average of the probability values ​​P of multiple pixels in the neighborhood of any pixel as the first-order neighborhood weight W1 of that pixel.

[0104] by Figure 4 , 5 The calculation of the first-order neighborhood weight W1 is explained using the probability map of the second label channel as an example.

[0105] In the probability map including the second label channel of the fill pixel, examples of the pixel at coordinates (2, 2) and the pixel at coordinates (3, 7) are illustrated.

[0106] The probability value P of the pixel with coordinates (2,2) is 1. The probability value P of 3 of the 8 other pixels in the neighborhood of pixel (2,2) is 1, and the probability value P of the remaining 5 pixels is 0.

[0107] For pixel (2,2), the mean of the probability values ​​P of the 8 pixels in the 3×3 neighborhood of pixel (2,2) is calculated as the first-order neighborhood weight W1 of pixel (2,2). That is, the first-order neighborhood weight W1 of pixel (2,2) is 0.375.

[0108] The probability value P of the pixel with coordinates (3, 7) is 1, and the probability value P of the 8 pixels in the neighborhood of pixel (3, 7) is also 1.

[0109] For pixel (3, 7), the mean of the probability values ​​P of the 8 pixels in the 3×3 neighborhood of pixel (3, 7) is calculated as the first-order neighborhood weight W1 of pixel (3, 7). That is, the first-order neighborhood weight W1 of pixel (3, 7) is 1.

[0110] Therefore, based on the above-described method for calculating the first-order neighborhood weight W1, weight generation unit 2 calculates the first-order neighborhood weight W1 of all pixels in the probability map of the second label channel. Similarly, weight generation unit 2 calculates the first-order neighborhood weight W1 of all pixels in the probability map of the first label channel, and calculates the first-order neighborhood weight W1 of all pixels in the probability map of the third label channel.

[0111] Next, weight generation unit 2 performs k downsampling operations on the probability maps of the first, second, and third label channels. For any pixel among the downsampled pixels, the mean of the probability values ​​P of the pixels in its neighborhood is calculated as the (k+1)th order sampling weight of that pixel, generating a (k+1)th order sampling weight map representing the (k+1)th order sampling weight of the pixel. The (k+1)th order sampling weight map is then upsampled k times to generate the (k+1)th order neighborhood weight W of the pixel. k+1 , where k is an integer greater than 0 and less than n, and n is the maximum order of the prediction model M.

[0112] First of all, with Figure 6 The calculation of the second-order neighborhood weight W2 is explained using the probability map of the second label channel as an example.

[0113] In the calculation of the second-order neighborhood weight W2, the weight generation unit 2 downsamples the probability map of the second label channel once. In the probability map of the second label channel after downsampling, including the filling pixels, the second-order sampling weights of all pixels are calculated in a similar manner to the calculation of the first-order neighborhood weight W1.

[0114] For example, the probability value P of the downsampled pixel at coordinates (2, 2) is 1, the probability value P of 3 of the 8 other pixels in the neighborhood of pixel (2, 2) is 1, and the probability value P of the remaining 5 pixels is 0.

[0115] For pixel (2, 2), the mean of the probability values ​​P of the 8 pixels in its 3×3 neighborhood is used as the second-order sampling weight for pixel (2, 2). That is, the second-order sampling weight for pixel (2, 2) is 0.375.

[0116] The downsampled probability value P of the pixel at coordinate (5, 5) is 1. The probability values ​​P of 3 out of the 8 other pixels in the neighborhood of pixel (5, 5) are 1, and the probability values ​​P of the remaining 5 pixels are 0.

[0117] For pixel (5, 5), the mean of the probability values ​​P of the 8 pixels in its 3×3 neighborhood is used as the second-order sampling weight for pixel (5, 5). That is, the second-order sampling weight for pixel (5, 5) is 0.375.

[0118] Therefore, based on the above-described method for calculating second-order sampling weights, weight generation unit 2 calculates the second-order sampling weights for all pixels in the probability map of the second label channel. Furthermore, as... Figure 6 As shown, weight generation unit 2 generates a second-order sampled weight map representing the second-order sampled weights of pixels.

[0119] Next, the weight generation unit 2 performs one upsampling on the second-order sampled weight map. The probability value P of each pixel in the upsampled probability map represents the second-order neighborhood weight W2 of that pixel.

[0120] Here, the second-order neighborhood weight W2 of pixel (2,2) is 0.375, and the second-order neighborhood weight W2 of pixel (5,5) is 0.375.

[0121] Therefore, weight generation unit 2 calculates the second-order neighborhood weight W2 of all pixels in the probability map of the second label channel. Similarly, weight generation unit 2 calculates the second-order neighborhood weight W2 of all pixels in the probability map of the first label channel and in the probability map of the third label channel.

[0122] Based on the above method, weight generation unit 2 generates the k+1 order neighborhood weight W for each pixel in the first label channel, the second label channel, and the third label channel. k+1 .

[0123] by Figure 7 Taking the probability map of the second label channel in the image as an example, consider the k+1 order neighborhood weight W. k+1 The calculations will be explained. Figure 7The probability map of the second label channel after k downsampling, the k-th order sampling weight map, and the k+1 order neighborhood weight W are shown in the figure. k+1 This is just an illustration; the values ​​have been omitted for ease of explanation (represented by "..."). The size of this probability graph may not necessarily match the actual situation.

[0124] Similarly, in the k+1 order neighborhood weight W k+1 In the calculation, the weight generation unit 2 downsamples the probability map of the second label channel k times. In the probability map of the second label channel after k downsampling including the filling pixels, the k+1 order sampling weights of all pixels are calculated in a similar manner to the calculation of the first order neighborhood weight W1. A k+1 order sampling weight map representing the k+1 order sampling weights of the pixels is generated.

[0125] Next, weight generation unit 2 performs k upsampling on the k+1 order sampled weight map. The probability value P of each pixel in the probability map after k upsampling represents the k+1 order neighborhood weight W of that pixel. k+1 .

[0126] Therefore, in the probability map of the second label channel, weight generation unit 2 calculates the k+1 order neighborhood weight W of all pixels. k+1 Similarly, in the probability map of the first label channel, weight generation unit 2 calculates the k+1 order neighborhood weights W for all pixels. k+1 In the probability map of the third label channel, calculate the k+1 order neighborhood weight W for all pixels. k+1 .

[0127] Here, k is an integer greater than 0 and less than n, and n is the maximum order of the prediction model M.

[0128] Therefore, weight generation unit 2 calculates the second-order neighborhood weight W2 to the nth-order neighborhood weight W for all pixels in the first label channel, the second label channel, and the third label channel. n Therefore, weight generation unit 2 calculates the first-order neighborhood weights W1 to the nth-order neighborhood weights Wn of all pixels. n .

[0129] Alternatively, the maximum order n can be set based on the receptive field of the neural network of the prediction model M.

[0130] Therefore, the maximum order of the weight map can be set according to the characteristics of the neural network, and the image processing apparatus 100 of the present invention can be applied to existing neural networks without modifying the training process or network structure of the existing neural network. Furthermore, it can be easily integrated with existing neural networks.

[0131] Next, in the first label channel, the second label channel, and the third label channel, the weight generation unit 2 calculates the first-order neighborhood weight W1 to the nth-order neighborhood weight W for any pixel. n The average value is used as the neighborhood weight W of that pixel.

[0132] by Figure 8 The calculation of the neighborhood weight W is explained using the neighborhood weight W of the second label channel as an example.

[0133] like Figure 8 As shown, in the second label channel, the weight generation unit 2 calculates the first-order neighborhood weight W1 to the nth-order neighborhood weight Wn for any pixel. n The average value is used as the neighborhood weight W of the pixel. Similarly, in the first label channel and the third label channel, the weight generation unit 2 calculates the first-order neighborhood weight W1 to the nth-order neighborhood weight W1 of any pixel. n The average value is used as the neighborhood weight W of the pixel.

[0134] Furthermore, in this invention, the neighborhood weight W is not calculated for the filled pixels.

[0135] Furthermore, in the above explanation, the neighborhood of any pixel is defined as a range of 3×3 pixels adjacent to it. However, the range of a pixel's neighborhood is not limited to 3×3 pixels; it can also be a range larger than 3×3 but smaller than the maximum range of the probability map. Additionally, when changing the neighborhood range, the number of fill pixels can be appropriately changed. Furthermore, the neighborhood can also be a three-dimensional range, as described later.

[0136] Therefore, the neighborhood weight W can be appropriately calculated at the edge of the probability graph.

[0137] use Figures 9 to 11 The calculation of global weights is explained. Figure 9 This is a schematic diagram illustrating the process of extracting the neighborhood weight W in the second label channel. Figure 10 This is a schematic diagram representing the full label graph N. Figure 11 This is a schematic diagram illustrating the calculation process of the global weight C.

[0138] The weight generation unit 2 extracts the neighborhood weights W of pixels whose label categories are consistent with the label categories of the first label channel, the second label channel, and the third label channel. The extracted neighborhood weights W are integrated into a full label map N. For any pixel in the full label map N, the global weight C of the pixel is calculated according to the proportion of the label category of the pixel in the full label map N.

[0139] by Figure 9 The neighborhood weight W of the second label channel in the diagram will be used as an example for illustration. Figure 9 In the label information, the label information of the four columns of pixels (the fifth to the eighth columns) represented by solid lines belongs to the second label category, while the label information of the remaining pixels does not belong to the second label category.

[0140] Therefore, in the second label channel, weight generation unit 2 extracts the neighborhood weights W of the four columns of pixels whose label category matches the second label category. Similarly, in the first label channel, weight generation unit 2 extracts the neighborhood weights W of pixels (columns 1 to 4) whose label category matches the first label category, and in the third label channel, it extracts the neighborhood weights W of pixels (columns 9 to 12) whose label category matches the third label category.

[0141] Next, the weight generation unit 2 integrates the neighborhood weights W extracted from the first label channel, the second label channel, and the third label channel into a full label graph N.

[0142] like Figure 10 As shown, the neighborhood weights W extracted from the first label channel, the neighborhood weights W extracted from the second label channel, and the neighborhood weights W extracted from the third label channel are integrated into a full-label graph N.

[0143] For ease of explanation, Figure 9 , 10 The process of extracting the neighborhood weights W in the first and third label channels is omitted, and the values ​​of the extracted neighborhood weights W in the first and third label channels are also omitted (represented as "...").

[0144] Next, for any pixel in the full label map N, the weight generation unit 2 calculates the global weight C of the pixel based on the proportion of the label category of the pixel in the full label map N.

[0145] by Figure 11 The calculation of the global weight C is explained using the fully labeled graph N as an example. Figure 11 In the fully labeled graph N, the neighborhood weights W in columns 1 to 4 come from the first label channel, the neighborhood weights W in columns 5 to 8 come from the second label channel, and the neighborhood weights W in columns 9 to 12 come from the third label channel. That is, in Figure 11 In the full-label image N shown, 1 / 3 of the pixels are labeled with the first label category, 1 / 3 of the pixels are labeled with the second label category, and 1 / 3 of the pixels are labeled with the third label category.

[0146] Therefore, for example, for a pixel from the second label channel in the full label image N, the weight generation unit 2 obtains the proportion of the second label category of that pixel in the full label image N, which is 1 / 3. Furthermore, the weight generation unit 2 calculates the global weight of that pixel C = 1 - (the proportion of the label category of that pixel in the full label image N). That is, for a pixel from the second label channel in the full label image N, the global weight of that pixel is C = 1 - 1 / 3 = 0.667.

[0147] Similarly, for example, for a pixel from the first label channel in the full label map N, weight generation unit 2 calculates the global weight C of that pixel based on the proportion of the first label category of that pixel in the full label map N. The global weight C of the pixel from the first label channel is 1 - 1 / 3 = 0.667.

[0148] For example, for a pixel from the third label channel in the full label image N, weight generation unit 2 calculates the global weight C of the pixel based on the proportion of the third label category of the pixel in the full label image N. The global weight C of the pixel from the third label channel is 1 - 1 / 3 = 0.667.

[0149] Therefore, weight generation unit 2 calculates the global weight C for all pixels.

[0150] Furthermore, in this invention, the label information and label category of each pixel in the training data are set for ease of explanation and are not limited thereto. That is, the number of label categories is not necessarily 3, and the proportion of a certain label category in the full label image N is not necessarily 1 / 3.

[0151] Next, weight generation unit 2 generates a weight graph ф based on the full label graph N and the global weight C. Specifically, weight generation unit 2 generates the weight graph ф using Equation 1 below.

[0152] Formula 1

[0153] ф=(N / mean(N)+C / mean(C)) / 2

[0154] Where N is the value in the full-label map N, and C is the global weight C of the pixel. The weight map ф includes the weight information of the pixels in the received training data.

[0155] Therefore, for the images and label information in the received training data, the weight generation unit 2 generates a weight map ф including the weight information of each pixel. The weight map ф is used to train the prediction model M by the method described later.

[0156] In addition, weight generation unit 2 makes the average weight of weight graph ф 1 (not shown).

[0157] Therefore, the image processing apparatus 100 of the present invention can be applied to existing neural networks without changing the overall training step size of the neural network or modifying the training process or network structure of the existing neural network. Furthermore, it can be easily integrated with existing neural networks.

[0158] (Loss Calculation Unit 3)

[0159] Loss calculation unit 3 calculates the loss based on the weight information in the weight graph ф. Here, the weight information in the weight graph ф reflects the calculated neighborhood weights W and the calculated global weights C.

[0160] Specifically, loss calculation unit 3 uses Equation 2 below to calculate the loss.

[0161] Formula 2

[0162]

[0163] Where AW Dice(p,y) is the calculated loss, ф is the weight information in the weight map ф, y is the label information in the form of a one-heatmap, and p is the probability map output by the prediction model M.

[0164] Furthermore, in this invention, the loss calculated by loss calculation unit 3 is used as an example for explanation. However, the loss calculated by loss calculation unit 3 can also be a variant of Dice loss (IOU loss, Tversky loss), cross-entropy loss, Focal loss, or at least one of Dice loss and its variants, cross-entropy loss, and Focal loss. For example, it can also be a combination of Dice loss and cross-entropy loss. The type of loss is not limited, as long as it can be used to train or update the prediction model M using the methods described later.

[0165] (Model Training Unit 4)

[0166] Model training unit 4 uses the loss AW Dice(p,y) calculated by loss calculation unit 3 to train the prediction model M in a way that minimizes the loss AW Dice(p,y) between the predicted value and the true value of the prediction model M.

[0167] During the training of the prediction model M, the model training unit 4 trains the prediction model M based on the calculated loss AW Dice(p,y).

[0168] (Image segmentation unit 5)

[0169] During the prediction process of prediction model M, image segmentation unit 5 uses the trained prediction model M to extract the boundary of the target region in the image.

[0170] (Flowchart of image processing device 100)

[0171] use Figure 12 The processing flow of the image processing apparatus 100 according to the first embodiment will be described. Figure 12 This is a flowchart of the processing of the image processing apparatus 100 according to the first embodiment.

[0172] In step S101, the weight generation unit 2 receives training data, which includes an image and multiple label information corresponding to the pixels in the image. Then, the process proceeds to step S102.

[0173] In step S102, the weight generation unit 2 generates probability maps for the first label channel, the second label channel, and the third label channel.

[0174] Specifically, firstly, the weight generation unit 2 performs one-hot encoding on the tag information according to the tag category to which the first tag information, second tag information, and third tag information belong, and generates one-hot maps of the first tag channel, second tag channel, and third tag channel.

[0175] Next, weight generation unit 2 inverts the pixel values ​​in the unique heatmaps of the first label channel, the second label channel, and the third label channel to generate (inverted) probability maps of the first label channel, the second label channel, and the third label channel. Then, proceed to step S103.

[0176] In step S103, the weight generation unit 2 calculates the average of the probability values ​​P of multiple pixels in the neighborhood of any pixel in the first label channel, the second label channel, and the third label channel as the first-order neighborhood weight W1 of that pixel. Then, the process proceeds to step S104.

[0177] In step S104, the weight generation unit 2 performs k downsampling operations on the probability maps of the first, second, and third label channels. For any pixel among the downsampled pixels, the mean of the probability values ​​P of the pixels in the neighboring area is calculated as the k+1 order sampling weight of that pixel, generating a k+1 order sampling weight map representing the k+1 order sampling weight of the pixel. The k+1 order sampling weight map is then upsampled k times to generate the k+1 order neighborhood weight W of the pixel. k+1 Here, k is an integer greater than 0 and less than n. Therefore, in the first label channel, the second label channel, and the third label channel, the second-order neighborhood weight W2 to the nth-order neighborhood weight W are calculated for all pixels. n Next, proceed to step S105.

[0178] In step S105, the weight generation unit 2 calculates the first-order neighborhood weight W1 to the nth-order neighborhood weight W for any pixel in the first label channel, the second label channel, and the third label channel. n The average value is used as the neighborhood weight W for that pixel. Then, proceed to step S106.

[0179] In step S106, the weight generation unit 2 extracts the neighborhood weights W of pixels whose label categories match those of the first, second, and third label channels, and integrates the extracted neighborhood weights W into a full label map N. Then, the process proceeds to step S107.

[0180] In step S107, weight generation unit 2 calculates the global weight C of any pixel in the full label map N based on the proportion of the pixel's label category in the full label map N. Then, the process proceeds to step S108.

[0181] In step S108, weight generation unit 2 generates a weight map ф based on the full label map N and the global weight C. Then, proceed to step S109.

[0182] In step S109, the loss calculation unit 3 calculates the loss AW Dice(p,y) based on the weight information in the weight map ф. Then, proceed to step S110.

[0183] In step S110, model training unit 4 trains the prediction model M using the calculated loss AW Dice(p,y). Here, step S110 is a step in the training process of the prediction model M. Next, proceed to step S111.

[0184] In step S111, image segmentation unit 5 uses the trained prediction model M to extract the boundary of the target region in the image. Here, step S111 is the prediction process of prediction model M. Then, the process ends.

[0185] (Effect)

[0186] The image processing apparatus 100 according to the present invention can appropriately assign weight values ​​to boundaries and improve the accuracy of boundary segmentation.

[0187] This is because, at the boundaries of target regions in an image, adjacent pixels typically have significantly different label information. By generating a weight map ф based on neighborhood weights W and global weights C, and calculating the loss, the weight map ф and the loss can reflect the label information in the neighborhood of a pixel as well as the label information in the global scope.

[0188] Therefore, the weight values ​​include more boundary information, allowing the prediction model M to accurately focus its attention on the boundary of the weighted target region.

[0189] In addition, the weight information in the weight map ф generated based on the neighborhood weight W of each pixel and the global weight C reflects the complexity of the boundary of the target region, and greater weights are assigned to boundaries with higher complexity.

[0190] Therefore, the attention of the prediction model M can be more accurately focused on the highly complex boundaries, and the boundaries can be appropriately weighted and the accuracy of boundary segmentation can be improved.

[0191] Furthermore, the weight information in the weight map ф generated based on the neighborhood weights W and global weights C of each pixel reflects the size of the target region's boundary. Appropriate weights can be assigned to both large and small boundaries, allowing the prediction model M to appropriately allocate attention to boundaries of different sizes. Therefore, it is possible to further assign appropriate weight values ​​to the boundaries and improve the accuracy of boundary segmentation.

[0192] Furthermore, the image processing apparatus 100 of the present invention can be applied to existing neural networks without modifying their training processes or network structures. Moreover, it can adaptively train on new training data when faced with new segmentation tasks, making it easily applicable to any segmentation task. Furthermore, the prediction model M can be easily integrated with existing neural networks.

[0193] (Comparative Example)

[0194] use Figure 13 The differences between the weight values ​​of the present invention and the weight values ​​of the comparative examples will be explained. Figure 13 This is a schematic diagram illustrating the difference between the weight values ​​of the present invention and the weight values ​​of the comparative examples. Figure 13 This refers to a two-dimensional abdominal ultrasound image captured by a medical image acquisition device during pregnancy. The prediction model M of this invention and the comparative examples are used to segment the pubic symphysis and the fetal head included in the image, respectively.

[0195] Figure 13 (a) is a schematic diagram representing the true value of the segmentation task. Figure 13 (b) is a schematic diagram representing the weight values ​​of the present invention. Figure 13 (c) is a diagram showing the weight values ​​of the comparison examples. Figure 13 (b) and Figure 13 In (c), the grayscale value represents the magnitude of the weight value.

[0196] like Figure 13 As shown in (a), the ground truth values ​​of the target region of the image have boundaries with different levels of complexity and size.

[0197] like Figure 13 As shown in (c), in the comparative example, all boundaries are assigned the same weight value. Therefore, the difficulty of segmentation varies for boundaries of different complexities, making it impossible to accurately segment highly complex boundaries.

[0198] Here, a comparative example is a method for calculating loss based on a distance map.

[0199] like Figure 13 As shown in (b), the image processing apparatus 100 of the present invention can appropriately assign different weight values ​​to boundaries of different complexities or sizes. For example, Figure 13 (b) The boundary of the upper target region has large and complex changes, so it is given a higher weight value. Figure 13 (b) The lower part of the target region has a gentle boundary and low complexity, so it is given a lower weight value.

[0200] Therefore, it can be seen that through the image processing device 100 of the present invention, the weight map ф and the loss can reflect the label information in the neighborhood of the pixel and the label information in the global range. The weight value reflects the complexity and size of the boundary and includes more boundary information, so that the attention of the prediction model M is accurately focused on the boundary of the weighted target region.

[0201] (Second Implementation)

[0202] use Figure 14 The configuration of the image processing apparatus 200 according to the second embodiment will be described. Figure 14 This is a block diagram illustrating the configuration of the image processing apparatus 200 according to the second embodiment.

[0203] like Figure 14 As shown, the image processing apparatus 200 includes a receiving unit 1', a weight map generation unit 2', a loss calculation unit 3', a model training unit 4, an image segmentation unit 5, and a memory 6. Additionally, the image processing apparatus 200 also includes a display unit 7.

[0204] The receiving unit 1', weight map generation unit 2', loss calculation unit 3', model training unit 4, image segmentation unit 5, and memory 6 of the image processing apparatus 200 of the second embodiment are substantially the same as those of the image processing apparatus 100 of the first embodiment. In the following description, only the differences between the image processing apparatus 200 of the second embodiment and the image processing apparatus 100 of the first embodiment will be described, and the description of the similarities will be omitted.

[0205] The second embodiment differs from the first embodiment in that the image processing apparatus 200 uses a prediction model M to segment the target region from the three-dimensional image. Therefore, the image, label information, probability maps of the first label channel, the second label channel, and the third label channel, the full label map N, the weight map ф, etc., of the training data all become three-dimensional, and the neighborhood of each pixel becomes an n×n×n three-dimensional space.

[0206] Furthermore, the receiving unit 1', weight generation unit 2', and loss calculation unit 3' of the image processing apparatus 200 are substantially the same as those of the image processing apparatus 100.

[0207] The difference lies in that receiving unit 1' receives the probability map generated by the prediction model M. Weight generation unit 2' performs an argmax function on the received probability map to determine the label category of each pixel in the probability map. Then, similarly to weight generation unit 2 of image processing device 100, it calculates the neighborhood weights W and global weights C of all pixels and generates a prediction weight map. Loss unit 3' calculates the cross-entropy loss using Equation 3 below.

[0208] Formula 3

[0209] AW CE(p,y)=φ⊙CE(p,y)

[0210] Next, loss unit 3' adds the calculated loss AW Dice(p,y) to the cross-entropy loss AW CE(p,y) to obtain the second loss = AW Dice(p,y) + AW CE(p,y).

[0211] Where p is the probability map output by the prediction model M, and y is the label information in the form of a one-heatmap.

[0212] Furthermore, the prediction model M is trained based on the second loss.

[0213] Here, the method for generating the predicted weight map is largely the same as the method for generating the weight map ф in the first embodiment, so the description is omitted.

[0214] (Display Unit 7)

[0215] use Figure 15 The display content of the display unit 7 of the image processing apparatus 200 according to the second embodiment will be described. Figure 15 This is a schematic diagram showing the display content of the display unit 7 of the image processing apparatus 200 according to the second embodiment.

[0216] During the prediction process of prediction model M, display unit 7 outputs and displays the prediction weight map.

[0217] Specifically, display unit 7 can display the prediction weight map separately, or it can display the prediction weight map overlapping the segmentation mask.

[0218] Figure 15 (a) is a schematic diagram representing the prediction weight map. Figure 15 (b) is a schematic diagram showing the prediction weight map and the segmentation mask displayed overlapping. Figure 15 (c) is a schematic diagram representing a probability graph.

[0219] (Effect)

[0220] Therefore, as Figure 15 As shown, in the prediction weight map, higher weight values ​​can be appropriately assigned to boundaries that are difficult to segment, such as those with high complexity, fuzziness, or multiple types of boundary regions. The prediction weight map can accurately reflect boundaries with different segmentation difficulties.

[0221] In addition, compared to simply displaying the probability map, displaying the prediction weight map, the segmentation mask, or displaying the prediction weight map and the segmentation mask together can visualize complex, ambiguous, or multi-class boundary regions, and can further focus attention more accurately on the difficult-to-segment boundaries in the target region.

[0222] The constituent elements and units of the devices described in the above embodiments are functional concepts and do not necessarily need to be physically configured as shown in the illustrations. That is, the specific forms of the distributed and integrated devices and units are not limited to those shown in the illustrations, and they can be configured in any unit, functionally or physically, in a distributed and integrated manner, to correspond to various loads and usage conditions. Furthermore, all or any part of the processing functions performed in each device can be implemented by a CPU and the program parsed and executed by the CPU, or they can be implemented as hardware based on wiring logic.

[0223] Furthermore, the image processing apparatus and image processing method described in the above embodiments can be implemented by executing a pre-prepared program by a computer such as a personal computer or workstation. This program can be distributed via a network such as the Internet. Additionally, the program can be recorded on a computer-readable non-transitory recording medium such as a hard disk, floppy disk (FD), CD-ROM, MO, or DVD, and then read from the recording medium by a computer for execution.

[0224] Several embodiments of the present invention have been described, but these embodiments are merely illustrative and are not intended to limit the scope of the invention. These embodiments can be implemented in a wide variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention, as well as within the scope of the claims and their equivalents.

Claims

1. An image processing apparatus that segments a target region from an image by a prediction model, characterized by comprising: Possessing: a receiving unit, receiving training data, the training data including images and a plurality of label information respectively corresponding to pixels in the images; a weight map generating unit, generating a weight map according to the label information, the weight map including weight information of the pixels in the images; a loss calculating unit, calculating a loss according to the weight information in the weight map; a model training unit; and an image segmentation unit, in the training process of the prediction model, the model training unit trains the prediction model based on the calculated loss, in the prediction process of the prediction model, the image segmentation unit extracts the boundary of the target region in the image using the trained prediction model.

2. The image processing apparatus according to claim 1, wherein the weight map generating unit, divides a plurality of the label information into a plurality of label channels according to label categories of the label information, the label channel being a probability map including probability values representing relationships between the label information of the pixels and the label categories, calculates a neighborhood weight of any one of a plurality of the label information and label information of a neighborhood of the label information according to the neighborhood weight and a global weight of the label information according to the neighborhood weight, generates the weight map based on the neighborhood weight and the global weight.

3. The image processing apparatus according to claim 2, wherein the weight generating unit, in each of the label channels, for any one of a plurality of the pixels, calculates a mean value of the probability values of a plurality of the pixels of a neighborhood of the pixel as a first-order neighborhood weight of the pixel.

4. The image processing apparatus according to claim 3, wherein the weight generating unit, in each of the label channels, k-times down-samples the probability map of the label channel, for any one of a plurality of the pixels after down-sampling, calculates a mean value of the probability values of a plurality of the pixels of a neighborhood of the pixel as a k+1-order sampling weight of the pixel, generates a k+1-order sampling weight map representing the k+1-order sampling weight of the pixel, k-times up-samples the k+1-order sampling weight map, and generates a k+1-order neighborhood weight of the pixel, wherein k is an integer greater than 0 and less than n, and n is a maximum order number of the prediction model.

5. The image processing apparatus according to claim 4, wherein the weight generating unit, in each of the label channels, for any one of a plurality of the pixels, calculates an average value of the first-order neighborhood weight to the n-order neighborhood weight of the pixel as the neighborhood weight of the pixel.

6. The image processing apparatus according to claim 5, wherein the weight generating unit, in each of the label channels, extracts the neighborhood weight of the pixel whose label category is consistent with the label category of the label channel, integrates the extracted neighborhood weight into a full label map, ​ For each pixel in the full label map, a proportion of the label class of the pixel in the full label map is calculated as the global weight of the pixel.

7. The image processing apparatus according to claim 2, wherein the loss calculation unit calculates the loss based on the neighborhood weight and the global weight.

8. The image processing apparatus according to claim 1, wherein the loss is at least one of a Dice loss, a cross-entropy loss, and a Focal loss.

9. The image processing apparatus according to claim 2, wherein an average weight of the weight map is 1.

10. The image processing apparatus according to claim 4, wherein the maximum order n is set based on a receptive field of a neural network of the prediction model.

11. The image processing apparatus according to claim 3, wherein a padding pixel is set outside an edge of the probability map of the label channel.

12. The image processing apparatus according to claim 2, wherein in each of the label channels, in a case where the label information of any pixel belongs to the label class corresponding to the label channel, the probability value of the pixel is represented as 0, in a case where the label information of any pixel does not belong to the label class corresponding to the label channel, the probability value of the pixel is represented as 1.

13. The image processing apparatus according to claim 1, wherein the prediction model segments the target region from a three-dimensional image, the loss calculation unit further calculates a cross-entropy loss, and adds the calculated loss and the cross-entropy loss as a second loss, the prediction model is trained based on the second loss.

14. The image processing apparatus according to claim 13, further comprising a display unit, in a prediction process of the prediction model, the weight generation unit generates a prediction weight map based on a probability map, the display unit outputs and displays the prediction weight map.

15. The image processing apparatus according to claim 14, wherein in a prediction process of the prediction model, the display unit displays the prediction weight map overlaid with a segmentation mask. comprising:

16. An image processing method of segmenting a target region from an image by a prediction model, characterized by, receiving training data including an image and a plurality of label information respectively corresponding to pixels in the image, generating a weight map including weight information of the pixels in the image based on the label information, calculating a loss based on the weight information in the weight map, training the prediction model, and segmenting the target region in the image, in a training process of the prediction model, training the prediction model based on the calculated loss, in a prediction process of the prediction model, extracting a boundary of the target region in the image using the trained prediction model. ​

Images