Image processing device, image processing method, and image processing program

Abstract

To provide an image processing device with improved learning accuracy by utilizing, in addition to local inference that performs semi-supervised learning using small images partly extracted from an image, global inference that extracts and performs inference on the overall image features.SOLUTION: An image processing device 100 of the present invention uses a neural network to perform region segmentation of a processing object image. The image processing device 100 includes: a global inference unit 130 configured to extract global features of an image and perform inference on the extracted features; a local inference unit 140 configured to extract local features of the image and perform inference on the extracted features; and a learning unit 120 configured to perform semi-supervised learning for each of the global inference unit 130 and the local inference unit 140, based on inference results of both the global inference unit 130 and the local inference unit 140.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program that perform image region segmentation using a neural network. [Background technology]

[0002] In the visual inspection of products in factories and in medical diagnoses using image data in clinical settings, it is important to accurately identify the shape and size of regions containing some kind of abnormality. Image processing techniques that achieve accurate pixel-level region segmentation have long been researched in the field of computer vision. In particular, in recent years, region segmentation techniques using deep neural networks (DNNs) have begun to achieve practical performance for a variety of problems. However, supervised learning-based methods require a large number of region mask images for training DNNs. Creating region mask images for training requires manually labeling every pixel in the image, which is a costly process. Furthermore, creating accurate region mask images requires a high level of expertise in the images to be segmented. Thus, creating training data for DNN-based image segmentation techniques requires significant effort and cost, and technological innovations that can simplify this process are needed.

[0003] Patent Document 1, by one of the inventors of the present applicant, discloses an image processing and analysis device that uses a neural network to segment an image to be processed, and is equipped with a classifier that uses a trained model that has undergone semi-supervised learning so as to input the image to be processed and output a segmented image, and the dataset used for semi-supervised learning includes a plurality of foreground patch images in which parts of the image have been cropped and assigned positive labels, a plurality of background patch images in which parts of the image have been cropped and assigned negative labels, and a plurality of unlabeled patch images in which parts of the image have been cropped. This image processing and analysis device reduces the burden of image annotation work required for semi-supervised learning and makes it easier to create training data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2023 / 189662 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention aims to provide an image processing device that improves learning accuracy by utilizing global inference that extracts features of the entire image and performs inference, in addition to local inference that performs semi-supervised learning using multiple small images (patch images) in which parts of the image have been cut out and labeled, and multiple small images in which parts of the image have been cut out and unlabeled. [Means for solving the problem]

[0006] The image processing device of the present invention is an image processing device that uses a neural network to perform region division of an image to be processed, and is equipped with a global inference unit that extracts global features of the image and performs inference, a local inference unit that extracts local features of the image and performs inference, and a learning unit that performs semi-supervised learning in each of the global inference unit and the local inference unit using the inference results of both the global inference unit and the local inference unit.

[0007] The image processing method of the present invention is an image processing method that uses a neural network to perform region segmentation of an image to be processed, and includes a global inference step that extracts global features of the image and performs inference, a local inference step that extracts local features of the image and performs inference, and a learning step that performs semi-supervised learning in each of the global inference step and the local inference step using the inference results of both the global inference step and the local inference step. [Effects of the Invention]

[0008] According to the present invention, in addition to local inference, which cuts out small images from an image and performs inference on each small image, global inference, which extracts features of the entire image and performs inference, can be used to improve learning accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a configuration of an image processing device for a training phase according to an embodiment. [Figure 2] 1 is a flowchart showing the operation of an image processing device in a training phase according to an embodiment. [Figure 3] FIG. 10 is a diagram showing a display example of learning data before annotation work in an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of annotation work according to an embodiment. [Figure 5] FIG. 1 shows labeled points generated by annotation in an embodiment. [Figure 6] FIG. 1 shows unlabeled points generated automatically in an embodiment. [Figure 7] FIG. 10 shows a small image generated by cutting out a part of an image according to an embodiment. [Figure 8] 1 is a block diagram showing the configuration of a global inference unit according to an embodiment of the present invention; [Figure 9] 1 is a block diagram showing a configuration of a local inference unit according to an embodiment of the present invention; [Figure 10] A block diagram showing the configuration of an image processing device in the inference phase according to an embodiment. [Figure 11] 1 is a flowchart showing the operation of an image processing device in an inference phase according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described, but the present invention is not limited to the following embodiments.

[0011] (Embodiment) The image processing device according to this embodiment will be described by dividing it into a training phase and an inference phase.

[0012] [1. Training Phase]

[0013] [1-1.Configuration] 1 is a block diagram showing the configuration of an image processing device 100 in the training phase of this embodiment. The image processing device 100 includes an image output unit 102 that outputs an image to a display device 101, an operation input unit 104 that accepts an input operation by a user using an input device 103, a training data generation unit 105 that generates training data (training data) based on the input operation by the user, a training data storage unit 110 that stores data used for training, such as image data for training, coordinate information related to specific points on an image, and class levels, and a learning unit 120 that performs training using the training data stored in the training data storage unit 110. The learning unit 120 includes a global inference unit 130 and a local inference unit 140.

[0014] The display device 101 is a device that displays images, such as a display or tablet. The input device 103 is an input device such as a touch panel that inputs a position on an image displayed on the display device 101 by touching a specific location on the image, a mouse, or a keyboard.

[0015] The learning data holding unit 110 is a storage medium that holds data, such as an SSD, a hard disk drive, a memory, etc. The learning data holding unit 110 stores a large amount of image data used for learning.

[0016] The image processing device 100 is composed of a memory and a processor, and each block shown inside the image processing device 100 in Fig. 1 is realized by software that operates in cooperation with the memory and processor. It may also be realized as a program that runs on a computer. Note that it is not necessary to realize all of the blocks by software, and some may be realized by hardware.

[0017] [1-2. Operation] The operation of the image processing device 100 will be described below. The image processing device 100 performs supervised learning or semi-supervised learning using point annotation data generated by a user specifying points on an image displayed on the display device 101. Figure 2 is a flowchart showing the operation of the image processing device 100.

[0018] The user uses the input device 103 to specify image data stored in the learning data storage unit 110, and displays the image to be annotated on the display device 101 (step S201). FIG. 3 is a diagram showing an example of the learning data before annotation displayed on the display device 101. In FIG. 3, the image to be annotated is an image of a part. The user performs region segmentation to identify abnormalities such as defects on the part. Here, defects refer to cosmetic defects such as scratches, dents, hair or other attachments, distortions, etc. Region segmentation classifies each pixel in the image into a specific class.

[0019] The user performs point annotation on the image displayed on the display device 101 (step S202). Point annotation involves identifying and labeling a limited number of pixels on the image. As shown in FIG. 4, the user touches a specific point on the image on the display device 101 using a touch panel, which is the input device 103. When a specific point on the image is touched on the display device 101, a mark is displayed at the touched point. The user then specifies a class label for the point using the input device 103, such as a mouse. Class labels include "Class 1," which indicates a normal area on a component that is free of defects, "Class 2," which indicates an abnormal area on a component that is free of defects, and "Background," which indicates a background area that is not a component. The image processing device 100 changes the display format of the mark for the point touched by the user for each class. Note that, although three classes are used in this embodiment, two classes, four or more classes may be used depending on the type of image to be segmented. Alternatively, the user may first specify a class label and then touch a point on the image. In this case, when touched, a mark is displayed on the display device 101 in a display format corresponding to the class label.

[0020] The operation input unit 104 reads the coordinate information and class label of the specified point and outputs it to the learning data generation unit 105. The learning data generation unit 105 generates labeled points consisting of the coordinate information and class label of the input point (step S203). FIG. 5 is a diagram showing the generated labeled points. The learning data generation unit 105 stores the coordinate information and class label of the labeled points generated by the point annotation work in the learning data storage unit 110 as training data.

[0021] The learning data generation unit 105 automatically generates points randomly on the image and acquires coordinate information of the points (step S204). The coordinate information automatically generated by the learning data generation unit 105 is unlabeled points. FIG. 6 is a diagram showing the generated unlabeled points. The number of unlabeled points is greater than the number of labeled points. This is because unlabeled points are automatically generated and can be increased more easily than labeled points. The learning data generation unit 105 stores the coordinate information of the automatically generated unlabeled points in the learning data holding unit 110 as learning data.

[0022] The images, coordinate information of labeled points, class labels, and coordinate information of unlabeled points stored in the learning data holding unit 110 are used as learning data for the global inference unit 130 in the learning unit 120.

[0023] The training data generation unit 105 generates small images by cutting out a portion of an image as training data for the local inference unit 140 in the training unit 120 (step S205). The training data generation unit 105 generates small images by cutting out a portion of the image stored in the training data storage unit 110, with a predetermined size centered on a labeled point or an unlabeled point. FIG. 7 shows the generated small images. The generated small images have the same shape and size. Note that the cut-out small images may overlap each other. The image size when cut out differs for each data set, and is determined empirically or based on evaluation values ​​in verification data. Note that, in this embodiment, the shape of the cut-out small images is rectangular, but is not limited to a rectangular shape and may be circular or another shape. The training data generation unit 105 stores the generated small images in the training data storage unit 110 as training data for the local inference unit 140.

[0024] Through the above operations, the learning data generation unit 105 completes the generation of learning data for one image. The user performs similar processing on other image data stored in the learning data storage unit 110 to generate learning data (No in step S206).

[0025] When the generation of the learning data is completed (Yes in step S206), the learning unit 120 performs learning using the learning data stored in the learning data holding unit 110 (step S207). The main components of the learning unit 120 are a global inference unit 130, a local inference unit 140, and four cost terms: L g , L l , R l2 , R con Hereinafter, each of the main components of the learning unit 120 will be described in detail.

[0026] The global inference unit 130 processes global features of images used as training data. The global inference unit 130 uses the images and coordinate information of labeled points initially stored in the training data storage unit 110 as input data for training.

[0027] 8 is a block diagram showing the configuration of the global inference unit 130. The global inference unit 130 includes a feature extraction unit 131 configured using a trained convolutional neural network (CNN), an extended pooling unit 132, a point feature extraction unit 133, and a point probability inference unit 134.

[0028] The feature extraction unit 131 extracts a feature map containing various hierarchical representations from the input image. ResNet-50 is used as an example of the feature extraction unit 131. ResNet has a structure called skip connection, which can alleviate the gradient vanishing problem. As an example of the extracted hierarchical feature map, a high-level feature map C5 and multiple low-level feature maps C2, C3, and C4 are used. Here, in the feature map Cr, r-1 is the number of downsamplings. When the height of the input image is H and the width is W, the size of Cr is H / 2. r-1 ×W / 2 r-1C5 is a feature map output from a layer close to the final layer in a trained CNN. C5 has high context but low resolution. C2, C3, and C4 are feature maps output from multiple early layers in a trained CNN concatenated in the channel direction. C2, C3, and C4 have low context but high resolution. Each feature map from C2 to C5 is selected from the layers that make up ResNet-50, from which feature maps that satisfy the above conditions are extracted.

[0029] In this embodiment, we will explain a form in which feature maps C2 to C5 are used, with downsampling times of 1 to 4 and image sizes of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the height and width of the input image, respectively. However, the number of downsamplings, image sizes, and number of feature maps are not limited to these numbers and may be other numbers.

[0030] The atrous spatial pyramid pooling (ASPP) is an atrous spatial pyramid pooling (ASPP) algorithm. The atrous pooling 132 combines a convolutional layer, a pooling layer, and a convolutional layer with different dilation rates. The atrous pooling 132 uses C5 as input and extracts high-level features at multiple scales.

[0031] The point feature extraction unit 133 upsamples the low-level feature maps C2, C3, and C4 to generate an image resized to the same size as the input image. Bilinear interpolation is used as an example of upsampling. Point features are features sampled in the channel direction based on arbitrary coordinate information from a high-resolution feature map in which low-context information is encoded. The point feature extraction unit 133 samples features from the resized feature map in the channel direction according to the input coordinates. The point feature extraction unit 133 outputs the sampled features as point features.

[0032] The point probability inference unit 134 uses parameters generated from the high-level features at multiple scales output by the extended pooling unit 132 to convert the point features output by the point feature extraction unit 133 into point probabilities, which are the predicted probabilities of those points. Point probabilities are the class probabilities to which the input coordinate information belongs. The point probability inference unit 134 uses a technology called Implicit PointRend. Implicit PointRend consists of two mechanisms: a parameter head and a point head. The parameter head generates weight and bias parameters for the point head from high-level features at multiple scales. However, in conventional Implicit PointRend, the parameter head surrounds objects in an image with a frame called a bounding box, treats objects within the bounding box as the foreground, and assigns foreground and background class labels to only a portion of the pixels. Then, it generates parameters for the point head from the features within the bounding box of the object detected from each input image. Meanwhile, the parameter head in the point probability inference unit 134 generates parameters for the point head from each input image. The Point head is structurally similar to a Multilayer Perceptron (MLP). However, unlike a normal MLP, it does not have learnable parameters; instead, its parameters are received from the Parameter head. Therefore, the Parameter head learns how to generate appropriate parameters for transforming point features. The Point head consists of three layers, with the last layer using a softmax function and the remaining layers using the ReLU function.

[0033] In this way, the global inference unit 130 infers point probabilities, which are class probabilities to which coordinates belong, from the input image and its coordinate information. The learning unit 120 inputs information on each coordinate that constitutes the image to the global inference unit 130 and outputs point probabilities for those coordinates.

[0034] Next, the local inference unit 140 will be described in detail. The local inference unit 140 processes the local features of the image used as training data. The local inference unit 140 learns only with small images cut out at an arbitrary size from the center of the coordinates in the image. By using the classification of the small image as an inference of the class at the center of the small image, the local inference unit 140 can treat the class probability for the small image as the point probability for the input coordinates.

[0035] The configuration of the local inference unit 140 is shown in Figure 9. The local inference unit 140 is composed of a feature extraction unit 141 and a classification unit 142. The feature extraction unit 141 of the local inference unit 140 uses a smaller model than the convolutional CNN that is the feature extraction unit 131 of the global inference unit 130. This is because when a small image is input, a large model results in redundant performance. ResNet-18 is used as an example of the feature extraction unit 141. The classification unit 142 is a normal MLP that uses a softmax function that outputs class probabilities as an activation function. Because a small image is information that has been artificially localized from the entire image, the local inference unit 140 is a model that is more specialized for local information processing than the global inference unit 130.

[0036] The learning unit 120 calculates a loss value using a cost function with four cost terms based on the point probabilities inferred by each of the global inference unit 130 and the local inference unit 140. The learning unit 120 simultaneously trains the inference models of the global inference unit 130 and the local inference unit 140 using backpropagation.

[0037] Furthermore, the training unit 120 utilizes the coordinate information of unlabeled points for each epoch in training. An epoch is a unit of training in which training is completed using a set of training data. The training unit 120 uses the coordinate information of these unlabeled points for entropy minimization and consistency regularization.

[0038] The cost function will be explained in detail. The cost function of the learning unit 120 is L, which is the loss function of the extended focal loss for the global inference unit 130. g, L, which is the loss function of the extended focal loss for the local inference unit 140 l , R for L2 regularization l2 , Global-Local Consistency Regularization (GLCR) in R con Combining these four terms, the cost function is

[0039]

number

[0040] The details of these four terms are as follows. First, we consider the loss function L for the global inference unit 130 and the local inference unit 140. g and L l is defined as follows:

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045]

number

[0046]

number

[0047] L2 regularization R l2 teeth,

[0048]

number

[0049] Global and local consistency regularization R con is defined as follows:

[0050]

number

[0051]

number

[0052] The learning unit 120 calculates these four terms: g ,L l ,R l2 ,R con The global inference unit 130 and the local inference unit 140 are trained simultaneously so as to minimize a cost function L consisting of: When the training is completed, the inference models of the global inference unit 130 and the local inference unit 140 are completed (step S208), and the training phase ends.

[0053] [2. Inference Phase]

[0054] [2-1.Configuration] 10 is a block diagram showing the configuration of an image processing device 200 in the inference phase of this embodiment. The image processing device 200 includes an image input unit 210 that inputs a processing target image to be segmented into regions, an area division unit 220 that performs area division on the input image, and an image output unit 230 that outputs the image segmented by the area division unit 220.

[0055] The image input unit 210 reads an image to be processed stored in an SSD, hard disk drive, memory, or the like in response to a user instruction, and outputs the image to the region segmentation unit 220. The region segmentation unit 220 includes a global inference unit 130 that is configured using a trained inference model learned in a training phase, and performs region segmentation of the image using the global inference unit 130. The image output unit 230 outputs the region-segmented image to a display or the like. The image output unit 230 may be configured to store the region-segmented image in an SSD, hard disk drive, memory, or the like.

[0056] The image processing device 200 is composed of a memory and a processor, and each block shown inside the image processing device 200 in Fig. 10 is realized by software that operates in cooperation with the memory and processor. It may also be realized as a program that runs on a computer. Note that it is not necessary to realize all of the blocks by software, and some may be realized by hardware.

[0057] In this embodiment, the configurations of the image processing device 100 and the image processing device 200 have been described separately for the training phase and the inference phase. The image processing device 100 and the image processing device 200 do not need to be configured as separate computers, and the blocks that make up each may be operated on a device such as a single computer.

[0058] [2-2. Operation] The following describes the operation of the image processing device 200. FIG.

[0059] The image processing device 200 receives input of a processing target image to be segmented via the image input unit 210 (step S1101). The region segmentation unit 220 inputs the processing target image and coordinate information to the trained global inference unit 130, and outputs the point probability at those coordinates. The region segmentation unit 220 sequentially uses the coordinate information of all pixels constituting the processing target image while scanning all pixels as the coordinate information to be input to the global inference unit 130. The region segmentation unit 220 classifies each pixel in the processing target image into a specific class based on the point probabilities of all pixels output by the global inference unit 130, and outputs an image segmented into regions on a pixel-by-pixel basis. The image output unit 230 outputs the image segmented by the region segmentation unit 220 to a display or the like (step S1103).

[0060] [3. Effects, etc.] In this embodiment, the image processing device 100 performs region segmentation of a processing target image using a neural network, and includes a global inference unit 130 that extracts global features of the image and performs inference, a local inference unit 140 that extracts local features of the image and performs inference, and a learning unit 120 that performs semi-supervised learning in each of the global inference unit 130 and the local inference unit 140 using inference results from both the global inference unit 130 and the local inference unit 140. The image processing device 100 can improve learning accuracy by using two different inference methods, namely, inference performed by extracting global features of the image and inference performed by extracting local features of the image, and performing semi-supervised learning of each of the two different inference methods based on the inference results from both methods.

[0061] In this embodiment, the image processing device 100 is an image processing device that uses a neural network to perform region segmentation of a processing target image, and includes a global inference unit 130 that extracts global features of the image and infers a predicted probability of each pixel that constitutes the image, a local inference unit 140 that extracts local features of the image and infers a predicted probability of each pixel that constitutes the image, and a learning unit 120 that performs semi-supervised learning in each of the global inference unit 130 and the local inference unit 140 using a cost function that includes a difference between the predicted probabilities inferred for each pixel by the global inference unit 130 and the local inference unit 140. The image processing device 100 can improve learning accuracy by using two different inference methods, namely, inference performed by extracting global features of the image and inference performed by extracting local features of the image, and performing semi-supervised learning on each of the two different inference methods so as to minimize a cost function that includes a difference between the predicted probabilities that are the inference results of each inference.

[0062] The system further includes an image output unit 102 that outputs an image to a display device 101 and an operation input unit 104 that inputs any point selected on the image and its label information. The learning unit 120 may use the image, coordinate information on the selected point on the image, and label information as training data for the global inference unit 130, and a small image clipped from the image containing the selected point and its label information as training data for the local inference unit 140. A user can select points on the image and perform point annotation to label the selected points, thereby constructing training data and reducing the work cost of creating training data. Furthermore, by performing semi-supervised learning based on the inference results of two different methods, global inference and local inference, the system can simultaneously improve learning accuracy, improve region segmentation performance, and simplify annotation work.

[0063] The learning unit 120 may automatically generate coordinate information for any point on an image and use the automatically generated coordinate information as training data for the global inference unit 130 and the local inference unit 140. The user can construct training data by automatically generating points without having to perform point annotation work of selecting points on the screen, thereby reducing the work cost of creating training data. This makes it possible to achieve both improved region segmentation performance and simplification of annotation work.

[0064] The cost function may include a loss function of the global inference unit 130, a loss function of the local inference unit 140, and an L2 regularization term. The image processing device 100 uses two different inference methods, namely, inference performed by extracting global features of an image and inference performed by extracting local features of an image, and performs semi-supervised learning of each of the inference methods so that the loss function based on each inference result and the cost function including the L2 regularization term are minimized, thereby improving the learning accuracy.

[0065] In this embodiment, the image processing device 200 is an image processing device 200 that performs region segmentation of a processing target image using a neural network, and includes an image input unit 210 that inputs the processing target image, a global inference unit 130 that extracts global features of the image and performs inference, a local inference unit 140 that extracts local features of the image and performs inference, a region segmentation unit 220 that performs semi-supervised learning in each of the global inference unit 130 and the local inference unit 140 using the inference results of both the global inference unit 130 and the local inference unit 140, and performs region segmentation of the processing target image using the trained inference model of the global inference unit 130, and an image output unit 230 that outputs the region-segmented image. The image processing device 200 uses two different inference methods, one that extracts global features of an image and one that extracts local features of an image, and performs semi-supervised learning of each of the two different inference methods based on the results of both inference methods, thereby improving the performance of region segmentation by using a global inference unit 130 with improved learning accuracy.

[0066] In this embodiment, an image processing method for performing region segmentation of a processing target image using a neural network includes a global inference step for extracting global features of the image and performing inference, a local inference step for extracting local features of the image and performing inference, and a learning step for performing semi-supervised learning in each of the global inference step and the local inference step using inference results from both the global inference step and the local inference step.The image processing method in this embodiment can improve learning accuracy by using inference methods of two different techniques, inference performed by extracting global features of the image and inference performed by extracting local features of the image, and performing semi-supervised learning of each of the two inference methods based on the inference results from both.

[0067] (Other embodiments) As described above, the embodiment has been described as an example of the present invention. However, the present invention is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Therefore, other embodiments will be exemplified below.

[0068] In the embodiment, the global inference unit has been described as having a configuration including a feature extraction unit, extended pooling, a point feature extraction unit, and a point probability inference unit, but the global inference unit is not limited to this configuration. The global inference unit may have any configuration as long as it can receive training images and coordinate information as input, extract global features of the images, perform inference, and output point probabilities for the coordinate information.

[0069] In the embodiment, a configuration has been described in which the region segmentation unit of the image processing device in the inference phase uses an inference model of a global inference unit that has been trained in the training phase. Instead of the global inference unit, the region segmentation unit may use an inference model of a local inference unit that has been trained in the training phase to perform region segmentation using the local inference unit. In this case, the region segmentation unit generates small images centered on all coordinates within the image to be processed, and generates a region-segmented image from the results of classifying all of these small images. [Explanation of symbols]

[0070] 100 Image processing device 101 Display Devices 102 Image output unit 103 Input Devices 104 Operation input section 105 Learning data generation unit 110 Learning data storage unit 120 Learning Department 130 Global Inference 131 Feature Extraction Unit 132 Enhanced Pooling 133-point feature extraction unit 134-point probability inference section 140 Local Inference 141 Feature Extraction Unit 142 Classification Department 200 Image processing device 210 Image input unit 220 Area division part 230 Image output unit

Claims

1. An image processing device that performs area division of a processing target image using a neural network, a global inference unit that extracts global features of an image and performs inference; a local inference unit that extracts local features of the image and performs inference; a learning unit that performs semi-supervised learning in each of the global inference unit and the local inference unit using inference results from both the global inference unit and the local inference unit; An image processing device comprising:

2. An image processing device that performs area division of a processing target image using a neural network, a global inference unit that extracts global features of an image and infers a predicted probability of each pixel that constitutes the image; a local inference unit that extracts local features of the image and infers a predicted probability of each pixel that constitutes the image; a learning unit that performs semi-supervised learning in each of the global inference unit and the local inference unit using a cost function that includes a difference between the predicted probabilities inferred by each of the global inference unit and the local inference unit for each pixel; An image processing device comprising:

3. an image output unit that outputs an image to a display device; an operation input unit for inputting an arbitrary point selected on the image and label information of the point; the learning unit uses the image, coordinate information on the image of the selected point, and the label information as training data for the global inference unit, and uses a small image obtained by cutting out a part of the image including the selected point and the label information as training data for the local inference unit.

3. The image processing device according to claim 1.

4. the learning unit automatically generates coordinate information of an arbitrary point on the image, and uses the automatically generated coordinate information as training data for the global inference unit and the local inference unit. The image processing device according to claim 3 .

5. The cost function includes a loss function of the global inference unit, a loss function of the local inference unit, and an L2 regularization term. The image processing device according to claim 2 .

6. An image processing device that performs area division of a processing target image using a neural network, an image input unit for inputting the processing target image; a global inference unit that extracts global features of an image and performs inference; a local inference unit that extracts local features of the image and performs inference; and a region segmentation unit that performs semi-supervised learning in each of the global inference unit and the local inference unit using inference results from both the global inference unit and the local inference unit, and performs region segmentation of the image to be processed using an inference model of the trained global inference unit. an image output unit that outputs the image that has been subjected to the region division; An image processing device comprising:

7. An image processing method for performing region segmentation of a processing target image using a neural network, comprising: a global inference step of extracting global features of an image and performing inference; a local inference step of extracting local features of the image and performing inference; a learning step of performing semi-supervised learning in each of the global inference step and the local inference step using inference results from both the global inference step and the local inference step; An image processing method comprising:

8. An image processing program that causes a computer to execute an image processing method for performing region segmentation of a processing target image using a neural network, a global inference step of extracting global features of an image and performing inference; a local inference step of extracting local features of the image and performing inference; a learning step of performing semi-supervised learning in each of the global inference step and the local inference step using inference results from both the global inference step and the local inference step; An image processing program that executes the following.

Images