Image processing apparatus, image processing method, and image processing program

Abstract

Provided is an image processing apparatus in which learning accuracy is enhanced by utilizing global inference for performing inference by extracting features of an entire image, in addition to local inference for performing semi-supervised learning by using a small image obtained by cutting out a part of an image. An image processing apparatus 100, which uses a neural network to perform region segmentation on an image to be processed, comprises: a global inference unit 130 that performs inference by extracting a global feature of an image; a local inference unit 140 that performs inference by extracting a local feature of the image; and a learning unit 120 that performs semi-supervised learning in each of the global inference unit 130 and the local inference unit 140, using inference results from both the global inference unit 130 and the local inference unit 140.

Description

Image processing device, image processing method, and image processing program

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program that perform image region segmentation using a neural network.

[0002] In visual inspections of products in factories and medical diagnoses using image data in clinical settings, it is important to accurately identify the shape and size of regions containing some kind of abnormality. Image processing techniques that achieve accurate pixel-level region segmentation have long been researched in the field of computer vision. In particular, in recent years, region segmentation techniques using deep neural networks (DNNs) have begun to achieve practical performance for a variety of problems. However, supervised learning-based methods require a large number of region mask images for training the DNN. Creating region mask images for training requires manually labeling every pixel in the image, which is a costly process. In addition, creating accurate region mask images requires a high level of expertise in the images to be segmented. As such, image region segmentation techniques using DNNs require significant effort and cost to create training data, and technological innovations that can simplify this process are needed.

[0003] Patent Literature 1, by one of the inventors of the present applicant, discloses an image processing and analysis device that uses a neural network to segment an image to be processed, and is equipped with a classifier that uses a trained model that has undergone semi-supervised learning so as to input the image to be processed and output a segmented image, and the dataset used for the semi-supervised learning includes multiple foreground patch images in which portions of the image have been cropped and assigned positive labels, multiple background patch images in which portions of the image have been cropped and assigned negative labels, and multiple unlabeled patch images in which portions of the image have been cropped. This image processing and analysis device reduces the burden of image annotation work required for semi-supervised learning and makes it easier to create training data.

[0004] International Publication No. 2023 / 189662

[0005] The present invention aims to provide an image processing device that improves learning accuracy by utilizing global inference that extracts features of the entire image and performs inference, in addition to local inference that performs semi-supervised learning using multiple small images (patch images) in which parts of the image have been cut out and labeled, and multiple small images in which parts of the image have been cut out and unlabeled.

[0006] The image processing device of the present invention is an image processing device that uses a neural network to perform region division of an image to be processed, and is equipped with a global inference unit that extracts global features of the image and performs inference, a local inference unit that extracts local features of the image and performs inference, and a learning unit that performs semi-supervised learning in each of the global inference unit and the local inference unit using the inference results of both the global inference unit and the local inference unit.

[0007] The image processing method of the present invention is an image processing method that uses a neural network to perform region segmentation of an image to be processed, and includes a global inference step that extracts global features of the image and performs inference, a local inference step that extracts local features of the image and performs inference, and a learning step that performs semi-supervised learning in each of the global inference step and the local inference step using the inference results of both the global inference step and the local inference step.

[0008] According to the present invention, in addition to local inference, which cuts out small images from an image and performs inference on each small image, global inference, which extracts features of the entire image and performs inference, can be used to improve learning accuracy.

[0009] FIG. 1 is a block diagram showing the configuration of an image processing device in the training phase in the first embodiment; FIG. 2 is a flowchart showing the operation of the image processing device in the training phase in the first embodiment; FIG. 3 is a diagram showing an example of a display in which learning data before annotation work in the first embodiment is displayed; FIG. 4 is a diagram showing an example of annotation work in the first embodiment; FIG. 5 is a diagram showing labeled points generated by annotation in the first embodiment; FIG. 6 is a diagram showing unlabeled points generated by automatic generation in the first embodiment; FIG. 7 is a diagram showing a small image generated by cutting out a part of an image in the first embodiment; A flowchart showing the operation of an image processing device in a phase. A block diagram showing the overall configuration of an image processing system in a second embodiment. A diagram showing the configuration of a head-mounted display in a second embodiment. A flowchart showing the processing flow in an inspection mode of an image processing device in a second embodiment. A diagram showing an example display of a head-mounted display in a second embodiment. A flowchart showing the processing flow in a re-learning mode of an image processing device in a second embodiment. A diagram showing an example display of a re-learning mode of a head-mounted display in a second embodiment. A diagram showing a schematic diagram of learning data in a second embodiment. A flowchart showing the processing flow in a learning mode of an image processing device in a second embodiment. A diagram showing an example display of a learning mode of a head-mounted display in a second embodiment.

[0010] Hereinafter, embodiments of the present invention will be described, but the present invention is not limited to the following embodiments.

[0011] [First Embodiment] The image processing device according to the first embodiment will be described in terms of a training phase and an inference phase. <1. Training Phase> <1-1. Configuration> FIG. 1 is a block diagram showing the configuration of an image processing device 100 in the training phase of the first embodiment. The image processing device 100 includes an image output unit 102 that outputs images to a display device 101, an operation input unit 104 that accepts input operations by a user (also referred to as an operator) using an input device 103, a training data generation unit 105 that generates training data (training data) based on the user's input operations, a training data storage unit 110 that stores data used for training, such as image data for training, coordinate information related to specific points on an image, and class labels, and a learning unit 120 that performs training using the training data stored in the training data storage unit 110. The learning unit 120 includes a global inference unit 130 and a local inference unit 140.

[0012] The display device 101 is a device that displays images, such as a display or tablet. The input device 103 is an input device such as a touch panel that inputs a position on an image displayed on the display device 101 by touching a specific location on the image, a mouse, or a keyboard.

[0013] The learning data holding unit 110 is a storage medium that holds data, such as an SSD, a hard disk drive, a memory, etc. The learning data holding unit 110 stores a large amount of image data used for learning.

[0014] The image processing device 100 is composed of a memory and a processor, and each block shown inside the image processing device 100 in Fig. 1 is realized by software that operates in cooperation with the memory and processor. It may also be realized as a program that runs on a computer. Note that it is not necessary to realize all of the blocks by software, and some may be realized by hardware.

[0015] <1-2. Operation> The operation of the image processing device 100 will be described below. The image processing device 100 performs supervised learning or semi-supervised learning using point annotation data generated by a user specifying points on an image displayed on the display device 101. Figure 2 is a flowchart showing the operation of the image processing device 100.

[0016] The user uses the input device 103 to specify image data stored in the learning data storage unit 110, and the image to be annotated is displayed on the display device 101 (step S201). FIG. 3 is a diagram showing an example of the learning data before annotation displayed on the display device 101. In FIG. 3, the image to be annotated is an image of a component. The user performs region segmentation to identify abnormalities such as defects on the component. Here, defects refer to cosmetic defects such as scratches, dents, hair or other attachments, distortions, etc. Region segmentation classifies each pixel in the image into a specific class.

[0017] The user performs point annotation on the image displayed on the display device 101 (step S202). Point annotation involves identifying and labeling a limited number of pixels on the image. As shown in FIG. 4, the user touches a specific point on the image on the display device 101 using a touch panel, which is the input device 103. When a specific point on the image is touched on the display device 101, a mark is displayed at the touched point. The user then specifies a class label for the point using the input device 103, such as a mouse. The user specifies one of the following class labels: "Class 1," which indicates a normal area on a component that is free of defects; "Class 2," which indicates an abnormal area on a component that is free of defects; or "Background," which indicates a background area that is not a component. The image processing device 100 displays the mark for the point touched by the user in a different display format for each class. Note that, although three classes are used in this embodiment, two classes, four or more classes may be used depending on the type of image to be segmented. Alternatively, the user may first specify a class label and then touch the point on the image. In this case, when touched, a mark is displayed on the display device 101 in a display format corresponding to the class label.

[0018] The operation input unit 104 reads the coordinate information and class label of the specified point and outputs them to the training data generation unit 105. The training data generation unit 105 generates labeled points consisting of the input coordinate information and class label of the point (step S203). Figure 5 is a diagram showing the generated labeled points. The training data generation unit 105 stores the coordinate information and class label of the labeled points generated by the point annotation work in the training data storage unit 110 as training data.

[0019] The learning data generation unit 105 automatically generates points randomly on the image and acquires coordinate information of the points (step S204). The coordinate information automatically generated by the learning data generation unit 105 is unlabeled points. FIG. 6 is a diagram showing the generated unlabeled points. The number of unlabeled points is greater than the number of labeled points. This is because unlabeled points are automatically generated and can be increased more easily than labeled points. The learning data generation unit 105 stores the coordinate information of the automatically generated unlabeled points in the learning data storage unit 110 as learning data.

[0020] The images, coordinate information of labeled points, class labels, and coordinate information of unlabeled points stored in the learning data holding unit 110 are used as learning data for the global inference unit 130 in the learning unit 120 .

[0021] The training data generation unit 105 generates small images by cutting out portions of an image as training data for the local inference unit 140 in the training unit 120 (step S205). The training data generation unit 105 generates small images by cutting out portions of the image stored in the training data storage unit 110, centered on labeled or unlabeled points, to a predetermined size. FIG. 7 shows the generated small images. The generated small images are of the same shape and size. Note that the cut-out small images may overlap each other. The image size used for cutting out differs for each dataset, and is determined empirically or based on evaluation values ​​in validation data. Note that, in this embodiment, the shape of the cut-out small images is rectangular, but this is not limited to a rectangular shape and may be circular or another shape. The training data generation unit 105 stores the generated small images in the training data storage unit 110 as training data for the local inference unit 140.

[0022] Although the display device 101 has been described using devices such as a display and a tablet as examples, the present invention is not limited thereto. For example, a head-mounted display (HMD) or the like may be used as the display device 101. Furthermore, although a touch panel, a mouse, and a keyboard have been described as examples of the input device 103, the present invention is not limited thereto. For example, a gesture device that detects the movement of a user's hand and inputs instructions through hand gestures may be used as the input device 103. By using a combination of a head-mounted display and a gesture device, it is expected that the efficiency of point annotation work, for example, can be improved. Specifically, while viewing an image displayed on the head-mounted display, a user can specify a class label by pointing at any point with their hand or finger. Since point annotation can be performed with such intuitive actions, it is expected that the work efficiency can be improved. Configuration examples of a head-mounted display and a gesture device will be described in detail in the second embodiment.

[0023] Through the above operations, the training data generation unit 105 completes the generation of training data for one image. The user performs similar processing on other image data stored in the training data storage unit 110 to generate training data (No in step S206).

[0024] When the generation of the learning data is completed (Yes in step S206), the learning unit 120 performs learning using the learning data stored in the learning data holding unit 110 (step S207). The main components of the learning unit 120 are a global inference unit 130, a local inference unit 140, and four cost terms: L ｇ , L ｌ , R l２ , R ｃｏｎ Hereinafter, each of the main components of the learning unit 120 will be described in detail.

[0025] The global inference unit 130 processes global features of images used as training data. The global inference unit 130 uses the images and coordinate information of labeled points initially stored in the learning data storage unit 110 as input data for training.

[0026] 8 is a block diagram showing the configuration of the global inference unit 130. The global inference unit 130 includes a feature extraction unit 131 configured by a trained convolutional neural network (CNN), an extended pooling unit 132, a point feature extraction unit 133, and a point probability inference unit 134.

[0027] The feature extraction unit 131 extracts a feature map containing various hierarchical expressions from the input image. The global inference unit 130 uses ResNet-50 as an example of the feature extraction unit 131. ResNet has a structure called skip connection, which can alleviate the gradient vanishing problem. An example of the extracted hierarchical feature map is a high-level feature map C5 and multiple low-level feature maps C2, C3, and C4. Here, in the feature map Cr, r-1 is the number of downsamplings. When the height of the input image is H and the width is W, the size of Cr is H / 2. r-1 ×W / 2 r-1 C5 is a feature map output from a layer close to the final layer in the trained CNN. C5 has high context but low resolution. C2, C3, and C4 are feature maps output from multiple initial layers in the trained CNN concatenated in the channel direction. C2, C3, and C4 have low context but high resolution. Each feature map from C2 to C5 is selected from the layers constituting ResNet-50 from which a feature map satisfying the above conditions is extracted.

[0028] In this embodiment, a form is described in which feature maps C2 to C5 are used, with downsampling times of 1 to 4 and image sizes of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the height and width of the input image, respectively. However, the number of downsamplings, image sizes, and number of feature maps are not limited to these numbers and may be other numbers.

[0029] The dilated pooling 132 is an atrous spatial pyramid pooling (ASPP) algorithm. The dilated pooling 132 combines a convolution layer, a pooling layer, and a convolution layer with different dilation rates. The dilated pooling 132 uses C5 as input and extracts high-level features at multiple scales.

[0030] The point feature extraction unit 133 upsamples the low-level feature maps C2, C3, and C4 to generate an image resized to the same size as the input image. The point feature extraction unit 133 uses bilinear interpolation as an example of upsampling. A point feature is a feature sampled in the channel direction based on arbitrary coordinate information from a high-resolution feature map in which low-context information is encoded. The point feature extraction unit 133 samples a feature from the resized feature map in the channel direction according to the input coordinates. The point feature extraction unit 133 outputs the sampled feature as a point feature.

[0031] The point probability inference unit 134 converts the point feature output by the point feature extraction unit 133 into a point probability, which is the predicted probability of that point, using parameters generated from the high-level feature at multiple scales output by the extended pooling 132. The point probability is the probability of the class to which the input coordinate information belongs. The point probability inference unit 134 uses a technique called Implicit PointRend. Implicit PointRend is composed of two mechanisms: a Parameter head and a Point head. The Parameter head generates weight and bias parameters for the Point head from the high-level feature at multiple scales. However, in conventional Implicit PointRend, the parameter head surrounds the object in the image with a frame called a bounding box, and then assigns foreground and background class labels to only a portion of the pixels, with the object in the bounding box being the foreground. Then, parameters for the point head are generated from the feature values ​​within the bounding box of the object detected from each input image. Meanwhile, the parameter head in the point probability inference unit 134 generates parameters for the point head from each input image. The point head is structurally similar to a multilayer perceptron (MLP). However, unlike a normal MLP, it does not have learnable parameters; instead, the parameters are received from the parameter head. Therefore, the parameter head learns how to generate appropriate parameters for converting point features. The Point head is composed of three layers, and the last layer uses a softmax function, while the other layers use a ReLU function.

[0032] In this way, the global inference unit 130 infers point probabilities, which are class probabilities to which coordinates belong, from the input image and its coordinate information. The learning unit 120 inputs information on each coordinate that constitutes the image to the global inference unit 130 and outputs point probabilities for those coordinates.

[0033] Next, the local inference unit 140 will be described in detail. The local inference unit 140 processes local features of images used as training data. The local inference unit 140 learns only with small images cut out at an arbitrary size from the center of the coordinates in the image. By using the classification of the small image as an inference of the class at the center of the small image, the local inference unit 140 can treat the class probability for the small image as a point probability for the input coordinates.

[0034] The configuration of the local inference unit 140 is shown in Figure 9. The local inference unit 140 is composed of a feature extraction unit 141 and a classification unit 142. The feature extraction unit 141 of the local inference unit 140 uses a smaller model than the convolutional CNN that is the feature extraction unit 131 of the global inference unit 130. This is because when a small image is input, a large model results in redundant performance. The local inference unit 140 uses ResNet-18 as an example of the feature extraction unit 141. The classification unit 142 is a normal MLP that uses a softmax function that outputs class probabilities as an activation function. Because a small image is information that is artificially localized from the entire image, the local inference unit 140 is a model that is more specialized for local information processing than the global inference unit 130.

[0035] The learning unit 120 calculates a loss value using a cost function with four cost terms based on the point probabilities inferred by each of the global inference unit 130 and the local inference unit 140. The learning unit 120 simultaneously trains the inference models of the global inference unit 130 and the local inference unit 140 using backpropagation.

[0036] Furthermore, the training unit 120 utilizes the coordinate information of the unlabeled points for each epoch in training. An epoch is a unit of training in which training is completed using a set of training data. The training unit 120 uses the coordinate information of the unlabeled points for entropy minimization and consistency regularization.

[0037] The cost function will be described in detail. The cost function of the learning unit 120 is the loss function of the extended focal loss for the global inference unit 130, L ｇ , L, which is the loss function of the extended focal loss for the local inference unit 140ｌ , L ２ Regularization R l２ ,Global-Local Consistency Regularization (GLCR) R ｃｏｎ Combining these four terms, the cost function is

[0038] where λ １ is R ｌ２ The magnitude of the value of λ ２ is R ｃｏｎ is a hyperparameter that adjusts the magnitude of the value of

[0039] The details of these four terms are as follows: First, we define the loss function L for the global inference unit 130 and the local inference unit 140. ｇ and L ｌ is defined as follows:

[0040] Here, t is a symbol that represents g if it is the loss function of the global inference unit 130, and l if it is the loss function of the local inference unit 140. ｔ Ｕ is a hyperparameter that adjusts the magnitude of the loss function for normal supervised data. ｔ Ｌ , the loss function for special unlabeled data is f ｔ Ｕ The usual focal loss has an adaptive term that underestimates the loss value for correctly classified samples and overestimates the loss value for misclassified samples, and is a loss function that is robust to class imbalance. The loss functions for these two supervised / unsupervised data are defined as

[0041]

[0042] Here, T is a symbol that represents L if it is a loss function for labeled data, and U if it is a loss function for unlabeled data. Also, N is the number of input images, and M Ｔrepresents the number of labeled or unlabeled points in the image. The loss function has two types of hyperparameters, α and γ. The number of unlabeled points and the number of labeled points are M Ｕ >M Ｌ This is because unlabeled points can be generated automatically and can be increased more easily than labeled points. ｔ ｉｊ ∈R Ｋ represents the predicted class probability from the global inference unit 130 or the local inference unit 140 for the j-th coordinate information in the i-th image. K represents the number of classes. q, which corresponds to the objective variable of the loss function, ｔ ｉｊ ∈R Ｋ The value of changes depending on the type of data. In the case of a loss function for labeled data, the objective variable q ｔ ｉｊ is the one-hot encoded class label y ｉｊ ∈R Ｋ In the case of a loss function for unlabeled data, the objective variable q ｔ ｉｊ represents the pseudo label. ｔ teeth,

[0043] where σ() is the softmax function, and u ｔ ｉｊ ∈R Ｋ represents the logit of the global inference unit 130 or the local inference unit 140 before applying σ(). Logit is the output of the model with a range of [-∞, ∞]. Also, τ represents the temperature parameter. The temperature parameter is a hyperparameter that controls the sharpness of the class probability distribution in the output of the softmax function. The lower the temperature parameter τ, the sharper the distribution becomes, so the entropy decreases, and the higher the temperature parameter τ, the smoother the distribution becomes, so the entropy increases. In the loss function for unlabeled data, the point probability p ｔ Since the objective variable is x, the learning is self-referenced to increase the point probability. This is a technique based on entropy minimization and pseudo-labeling. ｉThe j-th coordinate information for z ｉｊ Then, the set of labeled points including class labels and coordinate information generated by point annotation for the i-th image is

[0044] The set of unlabeled points is expressed as

[0045] It is expressed as:

[0046] L ２ Regularization R ｌ２ teeth,

[0047] Here, W and B represent the weight and bias parameters for the point head generated from the parameter head in the global inference unit 130. These parameters are not directly learned by the backpropagation method.

[0048] Global and local consistency regularization R ｃｏｎ is defined as follows:

[0049]

[0050] Here, K is the number of classes, and d( , ) is Kullback-Leibler Divergence (KLD). KLD is a function used in knowledge distillation and mutual learning. By maintaining consistency between the inferences for the same point between the global inference unit 130 and the local inference unit 140, the two inference units can acquire a hierarchical representation of the feature space. In addition, since this concept of consistency can be applied to any coordinate in the image, it can also be used for unlabeled points. ｃｏｎcan be calculated. Therefore, even with only point annotation work, which is sparse and prone to bias in image space, practical region segmentation can be achieved by utilizing a large number of unlabeled points. While many conventional consistency regularizations based on semi-supervised learning use perturbations, which are artificial noise, the learning unit 120 achieves consistency regularization based on semi-supervised learning that does not require perturbations. Therefore, it is not necessary to consider the impact of perturbations on the feature distribution of the data, and degradation of generalization performance due to perturbations can be suppressed.

[0051] The learning unit 120 calculates these four terms: L ｇ , L ｌ , R ｌ２ , R ｃｏｎ The global inference unit 130 and the local inference unit 140 are trained simultaneously so as to minimize a cost function L consisting of: When the training is completed, the inference models of the global inference unit 130 and the local inference unit 140 are completed (step S208), and the training phase ends.

[0052] 10 is a block diagram showing the configuration of the image processing device 200 in the inference phase of the first embodiment. The image processing device 200 includes an image input unit 210 that inputs a processing target image to be segmented into regions, an area division unit 220 that performs area division on the input image, and an image output unit 230 that outputs the image segmented by the area division unit 220.

[0053] The image input unit 210 reads an image to be processed stored in an SSD, hard disk drive, memory, or the like in response to a user instruction, and outputs the image to the region segmentation unit 220. The region segmentation unit 220 includes a global inference unit 130 formed from a learned inference model learned in the training phase, and performs region segmentation of the image using the global inference unit 130. The image output unit 230 outputs the region-segmented image to a display or the like. The image output unit 230 may be configured to store the region-segmented image in an SSD, hard disk drive, memory, or the like.

[0054] The image processing device 200 is composed of a memory and a processor, and each block shown inside the image processing device 200 in Fig. 10 is realized by software that operates in cooperation with the memory and processor. It may also be realized as a program that runs on a computer. Note that it is not necessary to realize all of the blocks by software, and some may be realized by hardware.

[0055] In this embodiment, the configurations of the image processing device 100 and the image processing device 200 have been described separately for the training phase and the inference phase. The image processing device 100 and the image processing device 200 do not need to be configured as separate computers, and the blocks that make up each may be operated on a device such as a single computer.

[0056] 2-2. Operation The following describes the operation of the image processing device 200. FIG.

[0057] The image processing device 200 receives a target image to be segmented via the image input unit 210 (step S1101). The region segmentation unit 220 inputs the target image and coordinate information to the trained global inference unit 130 and outputs point probabilities at those coordinates. The region segmentation unit 220 sequentially uses the coordinate information of all pixels constituting the target image while scanning all pixels as the coordinate information to be input to the global inference unit 130. The region segmentation unit 220 classifies each pixel in the target image into a specific class based on the point probabilities of all pixels output by the global inference unit 130, and outputs an image segmented into pixel units. The image output unit 230 outputs the segmented image output by the region segmentation unit 220 to a display or the like (step S1103).

[0058] 3. Effects, etc. In this embodiment, the image processing device 100 performs region segmentation of a processing target image using a neural network, and includes a global inference unit 130 that extracts global features of the image and performs inference, a local inference unit 140 that extracts local features of the image and performs inference, and a learning unit 120 that performs semi-supervised learning in each of the global inference unit 130 and the local inference unit 140 using inference results from both the global inference unit 130 and the local inference unit 140. The image processing device 100 can improve learning accuracy by using two different inference methods, namely, inference performed by extracting global features of the image and inference performed by extracting local features of the image, and performing semi-supervised learning of each of the two different inference methods based on the inference results from both methods.

[0059] In this embodiment, the image processing device 100 is an image processing device 100 that performs region segmentation of a processing target image using a neural network, and includes a global inference unit 130 that extracts global features of the image and infers a predicted probability of each pixel that constitutes the image, a local inference unit 140 that extracts local features of the image and infers a predicted probability of each pixel that constitutes the image, and a learning unit 120 that performs semi-supervised learning in each of the global inference unit 130 and the local inference unit 140 using a cost function that includes a difference between the predicted probabilities inferred for each pixel by the global inference unit 130 and the local inference unit 140. The image processing device 100 can improve learning accuracy by using two different inference methods, namely inference performed by extracting global features of the image and inference performed by extracting local features of the image, and performing semi-supervised learning on each of the two different inference methods so as to minimize a cost function that includes a difference between the predicted probabilities that are the inference results of each inference.

[0060] The system further includes an image output unit 102 that outputs an image to a display device 101, and an operation input unit 104 that inputs any point selected on the image and its label information. The learning unit 120 may use the image, coordinate information on the selected point on the image, and label information as training data for the global inference unit 130, and a small image clipped from a portion of the image containing the selected point and its label information as training data for the local inference unit 140. A user can select points on the image and perform point annotation to label the selected points, thereby constructing training data and reducing the work cost of creating training data. Furthermore, by performing semi-supervised learning based on the inference results of two different methods, global inference and local inference, the learning accuracy can be improved, thereby achieving both improved region segmentation performance and simplified annotation work.

[0061] The display device 101 may be a head-mounted display, and the operation input unit 104 may input points selected by hand gestures and label information for those points. With this configuration, a user can specify a class label by pointing at any point with their hand or finger while looking at the image displayed on the head-mounted display. This allows point annotation to be performed with intuitive actions, which is expected to improve work efficiency.

[0062] The learning unit 120 may automatically generate coordinate information for any point on the image and use the automatically generated coordinate information as training data for the global inference unit 130 and the local inference unit 140. The user can construct training data by automatically generating points without having to perform point annotation work of selecting points on the screen, thereby reducing the work cost of creating training data. This makes it possible to achieve both improved region segmentation performance and simplified annotation work.

[0063] The cost function may include a loss function of the global inference unit 130, a loss function of the local inference unit 140, and an L2 regularization term. The image processing device 100 uses two different inference methods, namely, inference performed by extracting global features of an image and inference performed by extracting local features of an image, and performs semi-supervised learning on each of the inference methods so that the loss function based on each inference result and the cost function including the L2 regularization term are minimized, thereby improving the learning accuracy.

[0064] In this embodiment, the image processing device 200 is an image processing device 200 that performs region segmentation of a processing target image using a neural network, and includes an image input unit 210 that inputs the processing target image, a global inference unit 130 that extracts global features of the image and performs inference, a local inference unit 140 that extracts local features of the image and performs inference, a region segmentation unit 220 that performs semi-supervised learning in each of the global inference unit 130 and the local inference unit 140 using the inference results of both the global inference unit 130 and the local inference unit 140, and performs region segmentation of the processing target image using the inference model of the trained global inference unit 130, and an image output unit 230 that outputs the region-segmented image. The image processing device 200 uses two different inference methods, one that extracts global features of an image and one that extracts local features of an image, and performs semi-supervised learning of each of the two different inference methods based on the results of both inference methods, thereby improving the performance of region segmentation by using a global inference unit 130 with improved learning accuracy.

[0065] In this embodiment, an image processing method for performing region segmentation of a processing target image using a neural network includes a global inference step for extracting global features of the image and performing inference, a local inference step for extracting local features of the image and performing inference, and a learning step for performing semi-supervised learning in each of the global inference step and the local inference step using inference results from both the global inference step and the local inference step.The image processing method in this embodiment can improve learning accuracy by using two different inference methods, inference performed by extracting global features of the image and inference performed by extracting local features of the image, and performing semi-supervised learning of each of the two different inference methods based on the inference results from both methods.

[0066] 4. Other Embodiments As described above, the first embodiment has been described as an example of the present invention. However, the present invention is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Therefore, other embodiments will be described below as examples.

[0067] In the first embodiment, the global inference unit is described as having a configuration including a feature extraction unit, an extended pooling unit, a point feature extraction unit, and a point probability inference unit, but the global inference unit is not limited to this configuration. The global inference unit may have any configuration as long as it can receive training images and coordinate information as input, extract global features of the images, perform inference, and output point probabilities of the coordinate information.

[0068] In the first embodiment, the region segmentation unit of the image processing device in the inference phase uses the inference model of the global inference unit that has been trained in the training phase. Instead of the global inference unit, the region segmentation unit may use the inference model of the local inference unit that has been trained in the training phase to perform region segmentation using the local inference unit. In this case, the region segmentation unit generates small images centered on all coordinates within the image to be processed, and generates a region-segmented image from the results of classifying all of these small images.

[0069] In the first embodiment, a model based on a convolutional neural network is used for the global inference unit and the local inference unit, but this is not limiting. For example, a model based on the Vision Transformer (ViT) may be used. When a model based on ViT is used, it becomes possible to process global features more effectively than a convolutional neural network. For example, by using ViT for the global inference unit and a convolutional neural network for the local inference unit, it is expected that global features and local features can be more effectively utilized in inference.

[0070] Second Embodiment In the second embodiment, an image processing device that performs visual inspection of components and the like using a head-mounted display will be described. When visual inspection is performed using a trained inference model, a non-defective part may be erroneously determined to be defective, or a defective part may be erroneously determined to be non-defective. Because there is a limit to the number and types of learning data that can be trained in advance, for example, erroneous determinations are likely to occur for unknown defects. Furthermore, a factor in erroneous determinations may also be that the environment in which the visual inspection is performed is different from the environment during learning.

[0071] To reduce such misidentifications, it is desirable to retrain an existing trained model to adapt it to the new inspection environment. For example, by retraining the model by adding new training data to the misidentified objects, it is expected that the detection accuracy will improve.

[0072] However, re-learning requires the complex work of taking a new image of the object to be learned using a camera or other device, importing the image data into a terminal such as a computer, identifying the area to be learned within the imported image, creating learning data, and then performing the learning process again, which increases the workload on-site.

[0073] Therefore, in the second embodiment, an image processing device will be described that allows the user to visually check the test results while wearing a head-mounted display, and also allows the user to create training data and perform learning and relearning processes.

[0074] 12 shows the overall configuration of an image processing system according to the second embodiment. A head-mounted display 301, which is a display device, a camera 302, and an input device 303 are connected to an image processing device 300. A user, who is an inspector, wears the head-mounted display 301 to perform an inspection. The head-mounted display 301 may be either a transmissive or non-transmissive type.

[0075] The image processing device 300 includes an information receiving unit 304 that receives position information and rotation information from the head-mounted display 301, an image receiving unit 305 that receives image data from the camera 302, an operation control unit 306 that outputs a control signal for controlling the operation of the image processing device 300 based on an input from the input device 303, a region estimation unit 307 that estimates a predetermined feature region from the image data using a learned inference model 314, and an image control unit 308 that displays the feature region estimated by the region estimation unit 307 on the head-mounted display 301 via an image output unit 309. The operation control unit 306 also functions as an operation input unit that receives information from the input device 303 (e.g., user operation information).

[0076] The image processing device 300 also includes an area selection unit 310 that selects an arbitrary area of ​​image data, a learning data generation unit 311 that generates learning data, a learning data storage unit 312 that stores the learning data, and a learning unit 313 that performs learning processing using a neural network. The learning data (training data) is data used for learning the neural network, and in the case of supervised learning, for example, it is so-called teacher data.

[0077] The image processing device 300 is composed of a memory and a processor, and each block shown inside the image processing device 300 in Fig. 12 is realized by software that operates in cooperation with the memory and processor. It may also be realized as a program that runs on a computer. Note that it is not necessary to realize all of the blocks by software, and some may be realized by hardware.

[0078] The head-mounted display 301 is connected to the image processing device 300 via a wired or wireless connection. The camera 302 is configured integrally with the head-mounted display 301 and captures a scene (foreground) in real space that the user visually recognizes through the head-mounted display 301. Note that the camera 302 only needs to be able to capture a scene that the user can visually recognize, and may be configured separately from the head-mounted display 301, using, for example, a wide-angle camera or an omnidirectional camera.

[0079] The input device 303 is an input device such as a keyboard or a mouse. Of course, other input devices may be used as long as they can be used in combination with the head-mounted display 301. For example, by using a gesture input device (also referred to as a gesture device) that can detect the movements of the user's hands and fingers, input by hand gestures becomes possible. The gesture input device may be, for example, a device in which a sensor is attached to the user's hand or finger, or a device in which the user's hand or finger is photographed with a camera. In the case of a sensor type, the movements of the user's hand or finger are detected by a sensor attached to the user's hand. In the case of a camera type, the movements of the user's hand or finger are detected by analyzing an image taken by a camera photographing the user's hand.

[0080] When capturing images of the movements of the user's hands and fingers, the camera 302 may be used for capturing the images. In this case, the image data captured by the camera 302 is received by the image receiving unit 305, and the operation control unit 306 analyzes the hand movements and generates a control signal, thereby substituting for the operation of the input device 303. Furthermore, the operation of the image processing device 300 can be controlled in combination with the input device 303.

[0081] Furthermore, if the head-mounted display 301 is equipped with a gaze detection function, gaze-based input may also be used. Gaze information detected by the head-mounted display 301 is sent to the operation control unit 306 via the information receiving unit 304. The operation control unit 306 then analyzes the movement of the user's gaze and converts it into a control signal for controlling the operation of the image processing device 300. This makes it possible to control the operation of the image processing device 300 using the gaze movement. Furthermore, the operation of the image processing device 300 can also be controlled in combination with the input device 303, hand gestures, etc.

[0082] 13 shows an example of the configuration of the head-mounted display 301. Here, a see-through head-mounted display will be described as an example. The head-mounted display 301 includes a half mirror 410, a liquid crystal display unit 411, a gyro sensor 412 that detects the position and rotation of the head-mounted display 301, and an eye-gaze sensor 413 that detects the movement of the user's eyes 401.

[0083] The user can view the real space through the half mirror 410. Specifically, the user views a foreground 400, which is the scenery in front of the user's eyes. The foreground 400 is also photographed by the camera 302 and input to the image processing device 300. The image processing device 300 acquires position information and rotation information of the head-mounted display 301 from the gyro sensor 412. The image processing device 300 then corrects the display position and display angle of the processed image based on this position information and rotation information, and displays it on the liquid crystal display unit 411. This correction is performed so that the output image of the image processing device 300 is superimposed on the foreground 400. This allows the user to view the image output from the image processing device 300 superimposed on the foreground 400.

[0084] Although the configuration of the head-mounted display 301 has been described using an example, the configuration is not limited to this. Any other configuration may be used as long as it has a function of superimposing an image output from the image processing device 300 on the real-world scenery visually recognized by the user.

[0085] <2. Operation> The operation of the image processing device 300 will be described with reference to the drawings. In this embodiment, image processing when performing a visual inspection of parts used in a factory or the like will be described as an example. The operation of the image processing device 300 can be broadly divided into three modes: inspection mode, re-learning mode, and learning mode. The operation of each mode will be described below. <2-1. Inspection Mode> Inspection mode is a mode in which a visual inspection of a part is performed. A region containing a predetermined feature (feature region) is identified using a pre-trained inference model. In this embodiment, a region containing a defect in the appearance of the part (hereinafter referred to as an abnormal region) is treated as a feature region. Here, a defect refers to a defect in appearance, such as a scratch, a dent, a hair or other attachment, or a distortion.

[0086] 14 is a flowchart showing the processing in the inspection mode of the image processing device 300. First, when a user wears the head-mounted display 301, the foreground 400 comes into the user's field of vision. Then, the camera 302 attached to the head-mounted display 301 captures an image of the foreground 400, and image data of the captured foreground 400 is input to the image receiving unit 305 (step S501).

[0087] Next, the region estimation unit 307 estimates a feature region using the trained inference model 314 of the neural network (step S502).

[0088] Generally, when a feature region is estimated using a neural network, an inference model is used that has been trained so that the output value of the neural network is high when an image having a predetermined feature is input. By using this inference model, a relatively high numerical value is output when an image having the predetermined feature is input, and a relatively low numerical value is output otherwise. Based on this output value, it is estimated whether the input image contains the predetermined feature.

[0089] In this embodiment, an inference model that has been trained to have a high output value when an image containing a defect is input and a low output value when an image containing no defect is input is used as the trained inference model 314. The output value of the trained inference model 314 is used as an evaluation value, and if the evaluation value exceeds a predetermined value, it is estimated to be an abnormal region containing a defect. The evaluation value is, for example, a value between 0 and 1, and in this case, the predetermined value is, for example, 0.5. The evaluation value can also be expressed as a percentage, and in this case, the predetermined value is, for example, 50%.

[0090] Learning can also be performed for each pixel. For example, if a neural network is configured to input an image of 512 x 512 pixels and output 512 x 512 pixels, learning is performed so that when an image having a predetermined feature is input, the output value of the part corresponding to the pixel in the feature region becomes high. Using an inference model trained in this way, an evaluation value can be calculated for each pixel of the input image. Then, a group of pixels whose evaluation value exceeds a predetermined value is estimated to be an abnormal region. This makes it possible to estimate the size and shape of the abnormal region.

[0091] The image control unit 308 displays this estimated abnormal region on the head-mounted display 301 (step S503). In Fig. 14, HMD (Head Mounted Display) refers to the head-mounted display 301. Here, the image control unit 308 displays the estimated abnormal region on the liquid crystal display unit 411 so as to be superimposed on the foreground 400 on the head-mounted display 301. Specifically, the superimposed display is performed by correcting the display position, display angle, etc. of the image indicating the abnormal region based on position information and rotation information of the head-mounted display 301. An example display on the head-mounted display 301 is shown in Fig. 15.

[0092] 15 shows examples of displays on the head-mounted display 301. In Figures 15(a) to 15(d), reference numeral 600 schematically shows an image presented to the user via the half mirror 410 of the head-mounted display 301. That is, it shows a state in which the image transmitted through the half mirror 410 and the image displayed on the liquid crystal display unit 411 are combined.

[0093] 15A is a display example when there is no defect in the part 602. Because the part 602 does not contain any defects, there is no output from the image processing device 300, and the user can view the foreground 400 including the platform 601 and the part 602 through the half mirror.

[0094] 15(b) is a display example in which a part 602 contains a defect. Marks 611 and 612 indicate estimated abnormal areas. In this way, the abnormal area is superimposed on the part in real space and can be visually recognized, so the user can clearly identify the location of the abnormal area in the part 602. The marks 611 and 612 are displayed on the liquid crystal display unit 411 of the head-mounted display 301 by the image processing device 300. These marks are superimposed on the foreground 400 by the half mirror 410 and are visually recognized by the user.

[0095] Mark 613 is a mark indicating that an abnormal area exists on the back side of the component. An abnormal area on the back side of component 602 cannot be detected from an image captured from only one direction, but by rotating the base 601 while wearing the head-mounted display 301 or by moving around the periphery of the base 601, the user can capture images of component 602 from various angles. This makes it possible to estimate the abnormal area on the back side of component 602. Mark 613 is displayed in a different manner from marks 611 and 612 to indicate that an abnormal area exists on the back side of the component.

[0096] Next, the user visually checks whether or not there is a defect in the part 602 using the marks 611 to 613 displayed on the head-mounted display 301 as clues. Marks 611 to 613 are displayed to indicate areas that may contain defects, so the user can focus on checking the areas indicated by marks 611 to 613. This allows for more efficient inspection than if all areas were checked visually. If a defect is confirmed visually, the part is determined to be defective and the inspection is terminated.

[0097] If the abnormal region is estimated correctly, the user can make a final judgment simply by visually inspecting it. However, depending on the inspection environment, good estimation results may not be obtained. For example, this may occur when inspecting a large number of unlearned parts or when the work environment has changed. In such cases, the reliability of the evaluation value indicating the abnormal region may decrease, and an abnormal region may be estimated even though it does not contain any defects. In such a situation, the user must carefully visually inspect the region indicated by the marks 611 to 613 to determine whether it is a true abnormal region, which reduces the efficiency of the inspection work.

[0098] Therefore, in such a case, the user can display auxiliary information on the head-mounted display 301. The auxiliary information is information that serves as a reference for the user when performing a visual inspection.

[0099] If auxiliary information is necessary, the user instructs the image processing device 300 to display the auxiliary information (step S504). The image control unit 308 then outputs the auxiliary information together (step S505). This is shown in FIG. 15C.

[0100] Figure 15(c) is an example display showing auxiliary information about the part. 621 to 623 are evaluation values ​​obtained by converting the output values ​​of the trained inference model 314 into percentages. As mentioned above, the higher the evaluation value, the more likely the part is to contain a defect. In this example, the evaluation values ​​for the areas marked 611 and 613 are high at "90%" and "80%, respectively, so these areas are likely to contain defects. Therefore, the user can make a final determination that a defect is present simply by casually checking visually.

[0101] On the other hand, if the evaluation value is low, there is a possibility that the estimation of the abnormal region is incorrect. In this case, the evaluation value of mark 612 is low at "51%, so there is a possibility that the estimation of this region is incorrect. Therefore, the user carefully checks the region of mark 612 visually. If a defect is confirmed visually, the estimation result is maintained. On the other hand, if a defect is not confirmed, the estimation result is corrected. In this way, by referring to the evaluation value, it is possible to prioritize and focus on checking regions that are likely to have been misjudged, allowing the user to perform inspections efficiently.

[0102] Another example of auxiliary information related to a part is shown in 624. Here, the part model number and the number of abnormal areas estimated by the area estimation unit 307 are displayed. By displaying the model number, it is possible to prevent misidentification of the part. Furthermore, if there are many abnormal areas, it is possible to prevent oversight of checking by displaying the total number of abnormal areas. By performing an inspection with reference to such auxiliary information, it is possible to improve work efficiency. When the inspection is completed, the process ends (step S506).

[0103] FIG. 15(d) shows an example of an enlarged display of a component. If an enlarged display is required, the user instructs the image processing device 300 to perform an enlarged display (step S504). The image control unit 308 then outputs the enlarged image (step S505). The enlarged image 630 enlarged by the image processing device 300 is displayed on the LCD display unit 411 and superimposed on the half mirror 410, bringing it into the user's field of view. The magnification ratio of the enlarged image 430 is controlled by the image control unit 308, allowing it to be enlarged and displayed at a desired magnification. For example, as shown in FIG. 15(d), the entire component can be enlarged, or by specifying only the area of ​​the mark 612, only the area of ​​the mark 612 can be enlarged and displayed. This is an effective function when more careful inspection is required, such as when the defective part is small. Of course, a reduced display is also possible. This is effective when a bird's-eye view of the entire component is desired. The user visually checks the enlarged or reduced image, and ends the process when all inspections are complete (step S506).

[0104] The auxiliary information described in FIG. 15( c) is not limited to this, and other information such as information necessary for inspection can also be displayed. Although not shown here, the aggregated results according to the level of the evaluation value may be displayed. For example, evaluation value ranks may be defined, and evaluation values ​​over 90% may be classified as rank A, evaluation values ​​over 70 to 90% as rank B, and evaluation values ​​over 50 to 70% as rank C, and the number of abnormal areas for each rank may be displayed. For example, areas with rank A are highly likely to contain defects, so parts with a large number of rank A areas are presumed to have a large number of defects. By displaying the number of areas for each rank, the condition of the parts can be checked at a glance.

[0105] Although the evaluation value numbers themselves are presented to the user as an example of auxiliary information, this is not limiting. It is sufficient to present the user with an approximate magnitude of the evaluation value, and for example, instead of displaying numbers, the marks 611 to 613 may be displayed in different colors or shapes for each rank.

[0106] Note that the instruction for auxiliary information can be given by any means, such as the input device 303, hand gestures, or gaze input. Although not shown here, when displaying auxiliary information such as that shown in FIG. 15( c), for example, a button labeled "Display auxiliary information" may be displayed on the screen, and the button may be operated by an input device such as a mouse, hand gestures, gaze input, etc. When wearing the head-mounted display 301, it may be difficult to use input devices such as a mouse or keyboard, so it is desirable to be able to give instructions by hand gestures, gaze input, etc.

[0107] Similarly, enlargement of the display can be performed by any means, such as the input device 303, hand gestures, or eye gaze input. When using hand gestures, enlargement or reduction can be achieved by pinching in or out using two fingers, for example. This allows for intuitive and easy-to-understand operations.

[0108] <2-2. Re-learning Mode> Next, the re-learning mode will be described. If the area estimation unit 307 can stably and satisfactorily estimate the abnormal area, there is no need to re-learn the inference model. However, an inference model trained in advance under specified conditions is not omnipotent, and as mentioned above, there are cases where the abnormal area is not estimated correctly when the inspection environment changes, for example. In this case, it is desirable to re-learn the trained inference model. This is because by customizing the trained inference model trained under specified conditions based on the actual environment, it is possible to construct an inference model that is more suited to the actual environment.

[0109] 16 is a flowchart showing the processing of the image processing device 300 in the re-learning mode. The processing in the re-learning mode is executed following the processing in the inspection mode. This will be described with reference to FIG. 17 as well.

[0110] 17 shows examples of displays on the head-mounted display 301. In Figures 17(a) to 17(d), reference numeral 800 schematically shows an image presented to the user via the half mirror 410 of the head-mounted display 301. That is, it shows a state in which the image transmitted through the half mirror 410 and the image displayed on the liquid crystal display unit 411 are combined.

[0111] FIG. 17( a) shows the inspection mode, with abnormal regions indicated by marks 801 and 802. Mark 802 has a relatively low evaluation value of "60%," so the value is underlined. If the evaluation value is lower than a predetermined value (e.g., 70%), the user is alerted to this and prompted to visually check or re-learn. FIG. 17( b) shows an enlarged view of part 602. If necessary, the user can display an enlarged image of the part and visually check the area indicated by mark 802 to determine whether or not a defect is present.

[0112] If it is determined that the area of ​​the mark 802 does not contain any defects, i.e., if the estimation by the area estimation unit 307 is incorrect, the user can switch to re-learning mode and re-learn the trained inference model 314. The switch to re-learning mode is performed based on an instruction from the user. For example, the switch to re-learning mode is performed based on an instruction from the input device 303 or a hand gesture.

[0113] When the system enters the relearning mode, the process shown in FIG. 16 begins. First, the user specifies a target area for relearning using the area selection unit 310 (step S701). The area marked with mark 802 is determined to be an abnormal area, even though it is not actually a defective area. Therefore, as shown in FIG. 17(c), the area marked with mark 802 is specified as a target area for relearning, i.e., a learning area. Then, as shown in FIG. 17(d), the specified area is displayed by mark 805.

[0114] The user can specify an area by directly pointing at a part in real space. As described above, the movement of the user's finger 804 is captured by the camera 302 and analyzed by the operation control unit 306 to detect the location where the user is pointing. Since an area can be specified by directly pointing at a part in real space, an intuitive and easy-to-use operation can be realized. The area can also be selected using the gaze sensor 413. For example, the area the user is gazing at may be detected and that area may be selected.

[0115] Once the region is selected, learning data for re-learning is generated (step S702). FIG. 18 is a diagram schematically illustrating the learning data. Reference numeral 900 denotes learning data when the entire component is the learning target. Region 901 is the region selected by region selection unit 310. Region 901 may be designated by the region indicated by mark 802 in inspection mode, or the user may designate the region by drawing an arbitrary rectangle. Alternatively, the user may designate the region by designating only one point and creating a rectangle of a predetermined width and height centered on that point. Learning data 900 is then generated by associating region 901 with information indicating that it is a "normal region."

[0116] The training data 900 includes a region 902 that is estimated to be an abnormal region. This abnormal region has been estimated well by the existing trained inference model 314, so basically, there is no need to re-learn it. However, if re-learning is performed with a focus on only the region 901, the accuracy of the region estimation that was previously performed well may decrease. In such a case, the region that has been estimated well may also be included in the targets of re-learning. This is expected to suppress the decrease in accuracy. In this case, the training data 900 may be generated by associating the region 901 with information indicating that it is a "normal region" and the region 902 with information indicating that it is an "abnormal region."

[0117] Although the regions 901 and 902 have been described as rectangular regions, the target regions may be specified in more detail, and the defective regions themselves may be specified. For example, the regions 903 and 904 in FIG. 18B may be specified as regions to be re-learned.

[0118] While the present embodiment has been described with reference to a case where the entire component is the learning target, it is also possible to use only the defective portion as the learning target. In this case, learning data can be generated by using regions 901 and 902 in Fig. 18(b) as learning data, associating region 903 with information indicating that it is a "normal region," and associating region 904 with information indicating that it is an "abnormal region."

[0119] When the generation of the learning data is completed, the learning unit 313 performs a re-learning process (step S703). The re-learning is started by pressing a button 806 on the screen. The button 806 may be pressed using the input device 303, or may be operated by hand gestures, eye gaze input, or the like.

[0120] The re-learning process is performed using the new learning data generated in step S702, with the trained inference model 314 as the initial value. The trained inference model 314 is then updated with the re-learned inference model (step S704). When all re-learning is complete, the re-learning mode process ends (step S705). By using the updated trained inference model 314 in the inspection mode, it is expected that the area indicated by the mark 802 will be estimated as a normal area.

[0121] In this embodiment, it has been described that re-learning is performed using the new training data generated in step S702, but re-learning may also be performed using training data that has already been trained. This is effective when it is not desired to make major changes to the existing inference model. In this case, this can be achieved by reading out the existing training data from the training data storage unit 312 and performing re-learning together with the new training data.

[0122] <2-3. Learning Mode> Next, the learning mode will be explained. Up until now, the explanation has been given on the assumption that a trained inference model 314 exists. The learning mode is a mode in which new training data is created and an inference model is generated based on this new training data.

[0123] 19 is a flowchart showing the processing in the learning mode of the image processing device 300. The processing in the learning mode is performed while the user is wearing the head-mounted display 301. When the processing in the learning mode starts, first, the area selection unit 310 specifies an area to which a learning tag is to be attached (step S1001). Next, the learning data generation unit 311 associates tag information (step S1002). Then, learning data is generated (step S1003). This process will be described using FIG. 20.

[0124] Fig. 20 shows an example of a display on the head-mounted display 301. Reference numeral 1200 schematically shows an image presented to the user via the half mirror 410 of the head-mounted display 301. Fig. 20(a) shows a state in which a part 1201 in real space is visible. Here, an example will be described in which learning data is generated by associating tag information indicating "failure" with a defect 1202, which is one of the feature regions of the image.

[0125] First, the area designation (step S1001) will be described. FIG. 20B shows how an area is designated by a hand gesture. Finger 1204 is the user's finger in real space, and shows how finger 1204 is pointing near defect 1202. When the operation control unit 306 analyzes the operation of the user's finger, area selection unit 310 designates an area within a predetermined range from the tip of finger 1204 as a nearby area. Then, image control unit 308 displays an image showing this nearby area 1203.

[0126] Next, a defective area 1205 is identified from this nearby area 1203, as shown in FIG. 20( c). The defective area 1205 can be identified by applying a technique such as area division to the nearby area 1203. This allows the user to automatically perform the process up to identifying the defective area 1205 simply by pointing to the area near the defect 1202. Note that although the nearby area 1203 has been described as the target of area division here, the entire screen may be the target of area division without providing a nearby area. Alternatively, the defective area 1205 may be manually identified by finger movement, the input device 303, or the like.

[0127] Once the defective region 1205 is identified, it is tagged for learning (step S1002). In this embodiment, tag information indicating "defect" is associated with the defective region 1205. Specifically, a "class label" that classifies the type of defect is associated with the defective region. The "class label" is expressed as a numerical value according to the type of defect, such as "0" for normal, "1" for a scratch, or "2" for a dent. More specifically, the "class label," which is label information, is associated with the coordinates of each point included in the defective region 1205.

[0128] Once tagging is complete, learning data is generated (step S1003). Here, the explanation has been given based on an image displayed on the head-mounted display 301, but in reality, the identification of the defective area 1205 and the association of tag information are performed on image data captured by the camera 302. The tag information is input, for example, from the input device 303. The image data associated with the tag information is then converted to a predetermined size and generated as learning data. Here, an image corresponding to 1200 in FIG. 20(c) is generated as learning data.

[0129] Although the entire component is the learning target here, if only the defective region is the learning target, for example, only the neighboring region 1203 or the defective region 1205 may be output as learning data. The generated learning data is held in the learning data holding unit 312.

[0130] When the creation of the training data is completed (step S1004), the training unit 313 performs a training process for the neural network using the training data stored in the training data storage unit 312 (step S1005). When the training is completed, a trained inference model 314 is created (step S1006), and the training mode ends.

[0131] 20(a) to 20(c), the vicinity region 1203 and the defective region 1205 are superimposed images output from the image processing device 300, and the rest are explained as images in real space displayed transparently. However, because the images in real space are captured by the camera 302, it is also possible to output part or all of an object in real space from the image processing device 300.

[0132] Furthermore, although the learning mode has been described as a mode in which learning data is created and new learning is performed, it can also be applied, for example, to re-learning. In the above-described re-learning mode, re-learning is performed on the area identified in the inspection mode. However, for example, if a defect is found in an area where no abnormal area has been estimated in the state shown in FIG. 15B, learning data can be generated using steps S1001 to S1003 of this mode. This makes it possible to re-learn not only for areas where a location that is not actually defective is estimated to be defective, but also for areas that are actually defective but are estimated to be non-defective.

[0133] As described above, the second embodiment has been described as an example of the present invention. However, the present invention is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. For example, the image processing device of the present invention can achieve similar operation even if the head-mounted display 301 in FIG. 12 is replaced with a non-transparent head-mounted display. In the previous embodiments, information regarding the characteristic region and the foreground 400 are combined using the half mirror 410. In contrast, when a non-transparent head-mounted display is used, the image control unit 308 combines the information regarding the characteristic region onto the image of the foreground 400 so as to superimpose it. This operation will be described with reference to FIG. 12.

[0134] The image receiving unit 305 receives an image of the foreground 400 from the camera 302 mounted on the head-mounted display 301. The image control unit 308 combines information about the characteristic region estimated by the region estimation unit 307 with the image of the foreground 400, and outputs the combined image to the head-mounted display 301 via the image output unit 309. This allows the user to visually recognize the image in which the information about the characteristic region is superimposed on the foreground 400. Other operations are the same as those in the embodiments described above, and can be similarly implemented for any of the operations in the inspection mode, re-learning mode, and learning mode described with reference to FIGS. 14 to 20. With this configuration, the user can perform inspection and learning processes while wearing the head-mounted display.

[0135] Furthermore, the learning unit 313 and the trained inference model 314 described in the second embodiment may be configured to include the global inference unit 130 and the local inference unit 140 in the first embodiment. The inference and learning operations in this case have been described in detail in the first embodiment, so a description thereof will be omitted here.

[0136] 3. Effects, etc. In this embodiment, the image processing device 300 is an image processing device that performs region segmentation of a processing target image using a neural network, and includes an image output unit 309 that outputs an image to a display device (head-mounted display 301), an operation input unit (operation control unit 306) to which an arbitrary point selected on the image by a hand gesture and label information of the point are input, and a learning unit 313 that trains an inference model that extracts features of the image, and the learning unit 313 uses the image and the coordinate information and label information of the selected point on the image as training data for the inference model 314. A user can specify a class label by pointing at an arbitrary point with their hand or finger while viewing the image displayed on the display device, which is expected to improve work efficiency.

[0137] The display device may be a head-mounted display, and the operation input unit (operation control unit 306) may receive input operations from a gesture device that detects the operator's hand movements as hand gestures. By using gesture input with hands or fingers, an arbitrary point on an image can be selected while wearing the head-mounted display 301.

[0138] The system may further include a learning data generation unit 311 that associates label information with points on an image specified by hand gestures and generates learning data based on the image, the coordinate information on the image of the selected point, and the label information, and the learning unit 313 may use the learning data generated by the learning data generation unit 311 to train the inference model 314.

[0139] The system may further include a region estimation unit 307 that estimates a feature region of the image using the inference model 314, and an image control unit 308 that superimposes auxiliary information related to the feature region of the image estimated by the region estimation unit 307 on the image and outputs the information to a display device (head-mounted display 301). Because the auxiliary information related to the feature region is superimposed, various pieces of information can be visually recognized simply by looking at the screen displayed on the display device.

[0140] The auxiliary information may include information (such as the evaluation value 621) indicating the inference result by the inference model 314. Since it is possible to confirm whether the feature region has been appropriately extracted by the inference model 314, the auxiliary information can be used as reference information for determining whether re-learning is necessary.

[0141] The image control unit 308 may enlarge or reduce a portion of the superimposed image and output it to the display device (head-mounted display 301). By enlarging a portion of the image, the state of the image can be confirmed in more detail. Furthermore, by displaying a reduced image, the entire image can be viewed from above.

[0142] The system further includes a learning data holding unit 312 that holds learned learning data, and the inference model 314 is a trained inference model 314 trained using the learning data held in the learning data holding unit 312. The learning unit 313 may perform training on the trained inference model 314 using both the learning data held in the learning data holding unit 312 and the learning data generated by the learning data generation unit 311. The trained inference model can be updated by adding new learning data according to the actual usage environment and re-training, for example, and therefore the accuracy of inference can be improved even in a real environment.

[0143] In this embodiment, the image processing device 300 is an image processing device that displays an image on a head-mounted display 301 and includes: an image receiving unit 305 that receives image data captured of a scene in front of the head-mounted display 301; a region estimation unit 307 that estimates a predetermined feature region from the image data using a neural network inference model 314; an image control unit 308 that outputs information regarding the feature region; a region selection unit 310 that selects an arbitrary region in the image data; a training data generation unit 311 that generates an image included in the region selected by the region selection unit 310 as training data; and a learning unit 313 that performs training using at least the training data generated by the training data generation unit 311 to generate an inference model. Since inspection and training can be performed while referring to the image displayed on the head-mounted display 301, work efficiency can be improved. Furthermore, by adding objects in an actual work site to the training data, an inference model adapted to the real environment can be generated.

[0144] The image control unit 308 may output display data in which information about the characteristic region is superimposed on the image data received by the image receiving unit 305. This causes information about the estimated characteristic region to be superimposed on the foreground, allowing tasks such as inspection and learning to be performed while wearing the head-mounted display.

[0145] The head-mounted display 301 may also be a see-through head-mounted display. In this case, the image control unit 308 outputs display data for superimposing information about the characteristic region on a foreground that is transparently displayed on the see-through head-mounted display. Since the user can directly view the foreground, the user can perform work in a state closer to reality.

[0146] The system may further include a learning data holding unit 312 that holds learned learning data, and the learning unit 313 may perform learning on a trained inference model 314 using the learning data held in the learning data holding unit 312 and the learning data generated by the learning data generation unit 311. By performing learning based on an existing inference model, it is possible to perform existing region estimation well while preventing erroneous determination in specific situations.

[0147] The image control unit 308 may output the display data including the area selected by the area selection unit 310. The selected area is displayed as a mark 805 in Fig. 17(d), so that the selected area can be confirmed.

[0148] The image control unit 308 may enlarge or reduce the display data before outputting it. If the defect is small or if you want to check the condition of the part more carefully, you can enlarge the image and display it. Also, by reducing the image and displaying it, you can get an overview of the entire part.

[0149] The image control unit 308 may use the output of the inference model 314 as an evaluation value and output this evaluation value as part of the display data. By referring to the evaluation value of the inference model, it is possible to confirm the accuracy of the estimation in each estimated area. Furthermore, by referring to this evaluation value, it is possible to determine whether re-learning is necessary.

[0150] The image receiving unit 305 may further receive image data of the user's actions, and may further include an operation control unit 306 that analyzes the image data of the user's actions and outputs control information, and the area selecting unit 310 may select an area based on the control information. For example, since the user can operate the image processing device 300 by hand gestures, a series of operations can be performed while wearing the head-mounted display 301.

[0151] The information receiving unit 304 may further receive user's gaze information from the head-mounted display 301, and the area selecting unit 310 may select an area based on the gaze information. Since gaze input can be used as part of the interface, area selection can be easily performed while wearing the head-mounted display 301.

[0152] In this embodiment, the image processing method is an image processing method for displaying an image on a head-mounted display, and includes the steps of receiving image data captured of a scene in front of the head-mounted display, estimating a predetermined feature region from the image data using a neural network inference model, outputting information about the feature region, selecting an arbitrary region in the image data, generating an image included in the selected region as training data, and performing training using at least the generated training data to generate an inference model. Since inspection and training can be performed while referring to the image displayed on the head-mounted display 301, work efficiency can be improved. Furthermore, by adding objects in an actual work site to the training data, an inference model adapted to the real environment can be generated.

[0153] In this embodiment, the image processing program causes a computer to execute an image processing method for displaying an image on a head-mounted display, and the image processing program executes the following steps: receiving image data of a scene in front of the head-mounted display; estimating a predetermined feature region from the image data using a neural network inference model; outputting information about the feature region; selecting an arbitrary region from the image data; generating an image included in the selected region as training data; and performing training using at least the generated training data to generate an inference model. Since inspection and training can be performed while referring to the image displayed on the head-mounted display 301, work efficiency can be improved. Furthermore, by adding objects in an actual work site to the training data, an inference model adapted to the real environment can be generated.

[0154] 100 Image processing device 101 Display device 102 Image output unit 103 Input device 104 Operation input unit 105 Learning data generation unit 110 Learning data storage unit 120 Learning unit 130 Global inference unit 131 Feature extraction unit 132 Extended pooling 133 Point feature extraction unit 134 Point probability inference unit 140 Local inference unit 141 Feature extraction unit 142 Classification unit 200 Image processing device 210 Image input unit 220 Area division unit 230 Image output unit 300 Image processing device 301 Head-mounted display 302 Camera 303 Input device 304 Information receiving unit 305 Image receiving unit 306 Operation control unit 307 Area estimation unit 308 Image control unit 309 Image output unit 310 Area selection unit 311 Learning data generation unit 312 Learning data storage unit 313 Learning unit 314 Trained inference model 400 Foreground 401 Eye 410 Half mirror 411 Liquid crystal display unit 412 Gyro sensor 413 Eye gaze sensor 400, 600, 700, 1200 Image 601 Base 602, 1201 Part 611 to 613, 801, 802 Mark 621 to 623 Evaluation value 624 Auxiliary information 630, 803 Enlarged image 804, 1204 Finger 805 Nearby area 806 Button 901 to 904 Area 1202 Defect 1203 Nearby area 1205 Defect area

Claims

1. An image processing apparatus that performs region division of a target image using a neural network, the apparatus comprising: a global inference unit that extracts global features of the image and performs inference; a local inference unit that extracts local features of the image and performs inference; and a learning unit that performs semi-supervised learning in each of the global inference unit and the local inference unit using the inference results of both the global inference unit and the local inference unit.

2. An image processing apparatus that performs region division of a target image using a neural network, the apparatus comprising: a global inference unit that extracts global features of the image and infers the prediction probability of each pixel constituting the image; a local inference unit that extracts local features of the image and infers the prediction probability of each pixel constituting the image; and a learning unit that performs semi-supervised learning in each of the global inference unit and the local inference unit using a cost function including the difference between the prediction probabilities inferred by each of the global inference unit and the local inference unit for each pixel.

3. The image processing apparatus according to claim 1 or 2, further comprising: an image output unit that outputs an image to a display device; and an operation input unit to which an arbitrary point selected on the image and label information of the point are input, wherein the learning unit uses, as training data for the global inference unit, the image, the coordinate information of the selected point on the image, and the label information, and uses, as training data for the local inference unit, a small image obtained by cutting out a part of the image including the selected point and the label information.

4. The image processing apparatus according to claim 3, wherein the learning unit automatically generates coordinate information of an arbitrary point on the image and uses the automatically generated coordinate information as training data for the global inference unit and the local inference unit.

5. The image processing apparatus according to claim 2, wherein the cost function includes a loss function of the global inference unit, a loss function of the local inference unit, and an L2 regularization term.

6. An image processing apparatus that performs region division of a processing target image using a neural network, comprising: an image input unit that inputs the processing target image; a global inference unit that extracts global features of the image and performs inference; a local inference unit that extracts local features of the image and performs inference; a region division unit that performs semi-supervised learning in both the global inference unit and the local inference unit using the inference results of both the global inference unit and the local inference unit, and performs region division of the processing target image using the learned inference model of the global inference unit; and an image output unit that outputs the image on which the region division has been performed.

7. The image processing apparatus according to claim 3, wherein the display device is a head-mounted display, and the operation input unit inputs a point selected by a hand gesture and label information of the point.

8. An image processing apparatus that performs region division of a processing target image using a neural network, comprising: an image output unit that outputs an image to a display device; an operation input unit that inputs an arbitrary point selected by a hand gesture on the image and label information of the point; and a learning unit that performs learning of an inference model that extracts features of the image, wherein the learning unit uses the image, coordinate information of the selected point on the image, and the label information as learning data for the inference model.

9. The image processing apparatus according to claim 8, wherein the display device is a head-mounted display, and the operation input unit receives an input operation from a gesture device that detects a movement of an operator's hand as a hand gesture.

10. The image processing apparatus according to claim 8 or 9, further comprising a learning data generation unit that associates the label information with a point on the image specified by a hand gesture, and generates learning data based on the image, coordinate information of the selected point on the image, and the label information, wherein the learning unit performs learning of the inference model using the learning data generated by the learning data generation unit.

11. The image processing apparatus according to claim 10, further comprising: a region estimation unit that estimates a feature region of an image using the inference model; and an image control unit that superimposes auxiliary information regarding the feature region of the image estimated by the region estimation unit on the image and outputs the image to the display device.

12. The image processing apparatus according to claim 11, wherein the auxiliary information includes information indicating an inference result by the inference model.

13. The image processing apparatus according to claim 11, wherein the image control unit enlarges or reduces a part of the superimposed image and outputs the result to the display device.

14. The image processing apparatus according to claim 12, further comprising a learning data holding unit that holds learned learning data, wherein the inference model is a learned inference model learned using the learning data held by the learning data holding unit, and the learning unit performs learning on the learned inference model using both the learning data held by the learning data holding unit and the learning data generated by the learning data generating unit.

15. An image processing method for performing region division of a target image using a neural network, the method comprising: a global inference step of extracting global features of the image and performing inference; a local inference step of extracting local features of the image and performing inference; and a learning step of performing semi-supervised learning in each of the global inference step and the local inference step using the inference results of both the global inference step and the local inference step.

16. An image processing program for causing a computer to execute an image processing method for performing region division of a target image using a neural network, the method comprising: a global inference step of extracting global features of the image and performing inference; a local inference step of extracting local features of the image and performing inference; and a learning step of performing semi-supervised learning in each of the global inference step and the local inference step using the inference results of both the global inference step and the local inference step.

Images