Image processing device and image processing method

Abstract

To provide an image processing device that performs appropriate class classification and label application.SOLUTION: An image processing device includes: a feature extraction unit that has a first neural network that extracts a feature of a source image and a first target image; a class classification unit that has a second neural network that classifies, on the basis of the extracted feature, the source image and the first target image; a domain classification unit that has a third neural network that performs domain classification; a processing unit 8 that assigns a pseudo label to the first target image using the class classification unit having the second neural network that is in the middle of learning; a self-learning unit 9 that performs, on the basis of a feature obtained by inputting the first target image to which the pseudo label is applied to the feature extraction unit, self-learning of the first to third neural networks; and a learning unit that learns, on the basis of the class classification and a domain classification result and a self-learning result, the first to third neural networks through backpropagation processing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Image classification techniques using machine learning have been attracting attention. For example, a method has been proposed in which images are classified into several classes and each image is assigned a label corresponding to the class. This method uses a trainable neural network to perform class classification and label assignment.

[0003] However, to perform proper classification and labeling, neural networks need to be trained thoroughly, which requires a large amount of training data, and therefore requires a large cost (time, data, etc.) for training.

[0004] Furthermore, depending on the image, even if learning is performed, it may not be possible to properly classify and label the image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent Publication US2020 / 0410388 Summary of the Invention [Problem to be solved by the invention]

[0006] Therefore, one embodiment of the present invention provides an image processing device and an image processing method that are capable of performing appropriate class classification and label assignment. [Means for solving the problem]

[0007] In order to solve the above problem, according to one embodiment of the present invention, there is provided a method for generating a labeled source image using a first image acquisition unit; a second image acquisition unit that acquires a first target image that is not labeled; a label acquisition unit that acquires a label; a feature extraction unit having a first neural network for extracting features of the source image and features of the first target image; a classification unit having a second neural network that performs classification of the source image and the first target image based on the plurality of features extracted by the feature extraction unit; a domain classification unit having a third neural network that performs domain classification of the source image and the first target image based on the features extracted by the feature extraction unit; a processing unit that assigns a pseudo label to the first target image using the classification unit having the second neural network in the middle of training; a self-learning unit that performs self-learning of the first neural network, the second neural network, and the third neural network based on features obtained by inputting the first target image to which the pseudo label has been assigned to the feature extraction unit; a learning unit that learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the classification result of the class classification unit, the classification result of the domain classification unit, and the self-learning result of the self-learning unit; a first loss calculation unit that calculates a first loss that indicates reliability of classification of the source image by the classification unit; a second loss calculation unit that calculates a second loss that indicates reliability of domain classification of the source image and the first target image by the domain classification unit; a third loss calculation unit that calculates a third loss that indicates reliability of class classification of the first target image by the class classification unit, the self-learning unit inputs the first target image to which the pseudo label has been assigned to the feature extraction unit, and causes the third loss calculation unit to calculate the third loss; the learning unit comprises: a first learning unit that performs backpropagation processing based on the first loss and the second loss to learn the first neural network, the second neural network, and the third neural network; and a second learning unit that performs backpropagation processing based on the third loss to learn the first neural network and the second neural network. An image processing device is provided. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic configuration of an image processing apparatus 1 according to a first embodiment. [Figure 2] FIG. 2 is a block diagram of the main parts of the image processing device 1 of FIG. [Figure 3] 4 is a flowchart showing the processing operation of the image processing device 1 according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating the layer configuration of a first neural network of a feature extraction unit, a second neural network of a class classification unit, and a third neural network of a domain classification unit. [Figure 5] 10 is a flowchart showing the processing operation of the image processing device according to a first modified example of the first embodiment. [Figure 6] 10 is a flowchart showing the processing operation of the image processing device according to a second modified example of the first embodiment. [Figure 7A] FIG. 2 is a diagram for explaining the basic operation of the image processing device according to the first embodiment. [Figure 7B] FIG. 1 is a diagram illustrating the operation of maximum logit masking. [Figure 7C] FIG. 1 is a diagram illustrating the operation of full logit masking. [Figure 8] 7C is a flowchart showing the processing operation of the maximum logit masking of FIG. 7B. [Figure 9] 7D is a flowchart showing the processing operation of the full logit masking of FIG. 7C. [Figure 10] FIG. 10 is a diagram illustrating the operation of the fourth modified example of the first embodiment. [Figure 11] 10 is a flowchart showing the processing operation of the image processing device according to a fourth modified example of the first embodiment. [Figure 12A] FIG. 11 is a diagram illustrating the operation of the fifth modified example of the first embodiment. [Figure 12B] FIG. 11 is a diagram illustrating the operation of the fifth modified example of the first embodiment. [Figure 13] 10 is a flowchart showing the processing operation of an image processing device according to a fifth modified example of the first embodiment. [Figure 14A] A diagram showing scores for multiple first target images in a mini-batch. [Figure 14B] FIG. 10 is a diagram showing an example in which the number of first target images to be extracted when the score is equal to or greater than a predetermined threshold is limited. [Figure 15]10 is a flowchart showing the processing operation of an image processing device according to a sixth modified example of the first embodiment. [Figure 16] FIG. 13 is a diagram for explaining a seventh modified example of the first embodiment. [Figure 17] 13 is a flowchart showing the processing operation of the image processing device according to the seventh modified example of the first embodiment. [Figure 18] FIG. 10 is a block diagram of the main parts of an image processing apparatus according to a second embodiment. [Figure 19] 10 is a flowchart showing the processing operation of an image processing device according to a second embodiment. [Figure 20] FIG. 10 is a block diagram showing a schematic configuration of an image processing apparatus according to a third embodiment. [Figure 21] 10 is a flowchart showing the processing operation of an image processing device according to a third embodiment. [Figure 22] FIG. 13 is a block diagram showing a schematic configuration of an image processing apparatus according to a first modified example of the third embodiment. [Figure 23] 13 is a flowchart showing the processing operation of the image processing device according to a first modified example of the third embodiment. [Figure 24] FIG. 13 is a block diagram showing a schematic configuration of an image processing device according to a second modified example of the third embodiment. [Figure 25A] A diagram showing the distribution of the number of source images per class in a mini-batch. [Figure 25B] FIG. 4 is a diagram for explaining processing of the DOA unit. [Figure 26A] 13 is a flowchart showing the processing operation of the image processing device according to a second modified example of the third embodiment. [Figure 26B] Flowchart following Figure 26A. [Figure 27] FIG. 13 is a block diagram showing a schematic configuration of an image processing apparatus according to a third modified example of the third embodiment. [Figure 28A] FIG. 10 is a diagram showing the distribution of the number of source images or first target images for each class. [Figure 28B] FIG. 4 is a diagram showing the processing operations of the first DOA unit and the second DOA unit. [Figure 29A] FIG. 13 is a block diagram showing a schematic configuration of an image processing apparatus according to a third modified example of the third embodiment. [Figure 29B] Flowchart following Figure 29A. [Figure 30] 10 is a flowchart showing the processing operation of an inference process. [Figure 31] FIG. 2 is a diagram showing the layer configuration of a first neural network and the layer configuration of a second neural network. [Figure 32] FIG. 1 is a block diagram showing a hardware configuration for performing processing operations of image processing devices according to first to third embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an image processing device and an image processing method will be described with reference to the drawings. The following description will focus on the main components of the image processing device, but the image processing device may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0010] (First embodiment) Fig. 1 is a block diagram showing a schematic configuration of an image processing device 1 according to a first embodiment, and Fig. 2 is a block diagram showing a main part of the image processing device 1 shown in Fig. 1. The processing operations of the image processing device 1 shown in Fig. 1 can be realized, for example, by a computer executing a program stored in a recording medium. Alternatively, a hardware device that executes the processing operations of the image processing device 1 shown in Fig. 1 may be provided.

[0011] 1 and 2, an image processing apparatus 1 according to the first embodiment includes a first image acquisition unit 2, a second image acquisition unit 3, a label acquisition unit 4, a feature extraction unit 5, a class classification unit 6, a domain classification unit 7, a self-learning unit 9 including a processing unit 8, a learning unit 10, a first data combination unit 11, a second data combination unit 12, a first judgment unit 13, a second judgment unit 14, a first cross-entropy unit 15, a second cross-entropy unit 16, a third cross-entropy unit 17, and a weight loss generation unit 18. Of the components shown in FIGS. 1 and 2, the first data combination unit 11, the second data combination unit 12, the first judgment unit 13, the second judgment unit 14, the first cross-entropy unit 15, the second cross-entropy unit 16, the third cross-entropy unit 17, and the weight loss generation unit 18 may be omitted as appropriate.

[0012] TIFF0007778607000001.tif20170

[0013] A label is identification information for each class when classifying a source image. A class is information that indicates the type of image. For example, if there are multiple types of images, such as numbers, animals, or plants, each image is classified into one of the classes: numbers, animals, or plants. The source images are assigned valid labels and are used as training data.

[0014] TIFF0007778607000002.tif25170

[0015] As will be described later, a pseudo label is assigned to the first target image by the inference process of the image processing device 1 in Fig. 1. A pseudo label is a label that is assigned in a pseudo manner by the inference process based on extracted features of the first target image.

[0016] TIFF0007778607000003.tif24170

[0017] TIFF0007778607000004.tif19170

[0018] The first neural network 5n has multiple layers, and can extract features from each of the source image and the first target image by updating parameters such as the weights of the nodes in each layer through a learning process. The source image and the first target image are input to the feature extraction unit 5. The feature extraction unit 5 also receives as input the first target image to which a pseudo-label has been assigned, predicted by the classification unit 6 (described later).

[0019] The classification unit 6 has a second neural network 6n that performs classification of the source image and the first target image based on the features extracted by the feature extraction unit 5.

[0020] The domain classification unit 7 includes a third neural network 7n that performs domain classification of the source image and the first target image based on the features extracted by the feature extraction unit 5. A domain refers to the environment in which an image is used. For example, the domain classification unit 7 classifies an image into different domains depending on whether the background of the image is colored or monochrome.

[0021] The processing unit 8 assigns a pseudo label to the first target image using the classification unit 6 having the second neural network 6n undergoing training and the second judgment unit 14. The pseudo label is associated with a class. Once the classification unit 6 infers the class of the first target image, the processing unit 8 can assign a pseudo label to the first target image based on the inferred class.

[0022] The self-learning unit 9 performs self-learning using the feature unit 5, the processing unit 8, and the third cross-entropy unit 17. The self-learning unit 9 inputs the first target image, to which a pseudo-label has been assigned by the processing unit 8, to the feature extraction unit 5. Based on the features obtained thereby, the self-learning unit 9 performs self-learning of the first neural network 5n, the second neural network 6n, and the third neural network 7n. The self-learning unit 9 also outputs the self-learning loss from the third cross-entropy unit 17.

[0023] TIFF0007778607000005.tif19170

[0024] The classification unit 6 in the self-learning unit 9 performs classification using the same second neural network 6n as the classification unit 6 connected to the first data association unit 11 and the first cross-entropy unit 15. Therefore, for convenience, two classification units 6 are shown in FIG. 1 and in FIGS. 2, 18, 22, 24, and 27 described below in order to clearly show the processing operations of the self-learning unit 9. However, the input and output data of the second neural network 6n of the classification unit 6 connected between the first data association unit 11 and the first cross-entropy unit 15 and the classification unit 6 in the self-learning unit 9 are different.

[0025] The learning unit 10 performs backpropagation processing based on the classification results of the class classification unit 6 and the domain classification unit 7 and the self-learning results of the self-learning unit 9, thereby learning the first neural network 5n, the second neural network 6n, and the third neural network 7n.

[0026] The first data combination unit 11 inputs data obtained by combining feature data of the source image and feature data of the first target image to which a pseudo label has been assigned to the class classification unit 6. The second data combination unit 12 inputs data obtained by combining feature data of the first target image without a label and feature data of the first target image to which a pseudo label has been assigned to the domain classification unit 7.

[0027] The first determination unit 13 determines whether or not to perform self-learning by the self-learning unit 9. If self-learning is not performed, learning is performed by the learning unit 10 using the results of the class classification by the classifying unit 6 and the results of the domain classification by the domain classifying unit 7.

[0028] TIFF0007778607000006.tif24170

[0029] The second determination unit 14 determines whether or not the first target image to which the pseudo label has been assigned by the classification unit 6 is to be used as training data.

[0030] TIFF0007778607000007.tif15170

[0031] The first cross entropy unit 15 calculates a source loss (first loss) that represents the reliability of class classification of the source image by the class classification unit 6. The second cross entropy unit 16 calculates a domain loss (second loss) that represents the reliability of domain classification of the source image and the first target image by the domain classification unit 7. The third cross entropy unit 17 calculates a self-learning loss (third loss) that represents the reliability of class classification of the first target image to which a pseudo label has been assigned.

[0032] The self-learning unit 9 assigns pseudo labels to the unlabeled first target image and calculates the self-learning loss. The self-learning unit 9 also inputs the pseudo-labeled first target image to the feature extraction unit 5, and performs training on the first to third neural networks 5n, 6n, and 7n.

[0033] The weight loss generator 18 generates weight losses by adjusting the weights of the source loss, domain loss, and self-learning loss when performing backpropagation processing. The learning unit 10 performs backpropagation processing based on the weight losses to learn the first to third neural networks 5n, 6n, and 7n.

[0034] 3 is a flowchart showing the processing operation of the image processing device 1 according to the first embodiment. The processing operation of the image processing device 1 according to the first embodiment will be described below with reference to the flowchart in FIG.

[0035] The process of the flowchart in Fig. 3 is performed for each mini-batch including a plurality of source images and a plurality of first target images. In the flowchart in Fig. 3, learning is performed for the first to third neural networks 5n, 6n, and 7n in units of mini-batches.

[0036] TIFF0007778607000008.tif14170

[0037] TIFF0007778607000009.tif14170

[0038] Furthermore, the classification unit 6 calculates a logit value for each class using the features of the source image Xs extracted by the feature extraction unit 5, and outputs a Softmax value from the logit value. The Softmax value is a value that indicates the confidence level for each class, and is set for each class. The confidence level for each class means the likelihood that the image belongs to the class. The first cross-entropy unit 15 calculates the cross-entropy error of the Softmax value for each class, and outputs a source loss (S2). The source loss is a value that quantifies the reliability of the classification by the classification unit 6. The smaller the source loss, the higher the reliability of the classification.

[0039] TIFF0007778607000010.tif29170

[0040] The domain loss is a numerical value that represents the reliability of the domain classification by the domain classification unit 7. The smaller the domain loss, the higher the reliability of the domain classification.

[0041] TIFF0007778607000011.tif14170

[0042] The labels assigned by the domain classification unit 7 are called domain labels, and are different from the labels corresponding to the classes classified by the class classification unit 6.

[0043] Next, it is determined whether the source loss is on a decreasing trend or whether the number of processed epochs is less than a predetermined ratio of the total number of epochs (S4). Here, one epoch refers to multiple mini-batches.

[0044] If S4 is YES, i.e., if the source loss is decreasing or the number of processed epochs is less than a predetermined percentage of the total number of epochs, it is determined that it is too early to perform self-learning. The learning unit 10 performs backpropagation processing using the weight loss generated by the weight loss generation unit 18 (S5). This generates gradients for each parameter of the first to third neural networks 5n, 6n, and 7n that each of the feature extraction unit 5, the class classification unit 6, and the domain classification unit 7 has. The weight loss generated by the weight loss generation unit 18 includes source loss and domain loss, and in some cases, also includes self-learning loss.

[0045] Next, based on the generated gradient, parameters such as the weights of each layer of the first to third neural networks 5n, 6n, and 7n are updated (S6), thereby training the first to third neural networks 5n, 6n, and 7n.

[0046] The learning of the above-mentioned first to third neural networks 5n, 6n, and 7n is performed for each source image included in the mini-batch.

[0047] If the result of S4 is NO while the first to third neural networks 5n, 6n, and 7n are being learned, i.e., if the source loss is not decreasing, or if the number of processed epochs reaches a predetermined percentage of the total number of epochs, self-learning by the self-learning unit 9 is started.

[0048] TIFF0007778607000012.tif24170

[0049] TIFF0007778607000013.tif26170

[0050] Next, the processing unit 8 calculates the cross entropy error of the pseudo-labeled first target image in the third cross entropy unit 17 and outputs the self-learning loss (S9). After the process of S9 is completed, the processes from S5 onwards are performed. The processes of S7 to S9 are performed for each first target image included in the mini-batch.

[0051] When the processes S1 to S9 are completed for all source images and the first target image included in the mini-batch to be processed, one iteration is considered to be completed. When one iteration is completed, processing of the next mini-batch to be processed is performed. The image processing device 1 according to this embodiment performs multiple iterations for multiple mini-batches based on the flowchart of FIG.

[0052] In this way, each time class classification and domain classification are performed for each source image and first target image included in a mini-batch, the first to third neural networks 5n, 6n, and 7n are trained by the training unit 10. Similarly, self-training by the self-training unit 9 is also performed each time a pseudo label is assigned to each first target image included in a mini-batch.

[0053] FIG. 4 is a diagram illustrating the layer configuration of the first neural network 5n of the feature extraction unit 5, the second neural network 6n of the classification unit 6, and the third neural network 7n of the domain classification unit 7. As illustrated, the first to third neural networks 5n, 6n, and 7n each include multiple layers. More specifically, the first neural network 5n has multiple layers 5L, such as convolutional layers, BN (Batch Normalization), and ReLU layers, with the number of channels increased for each layer. Because an image only has three channels (RGB), the number of channels is increased to extract features beyond these three channels. The second neural network 6n and the third neural network 7n each have multiple fully connected layers 6L and 7L, with the number of channels reduced for each layer. The final fully connected layers 6L and 7L obtain logit values ​​equal to the number of classes. The layers 6S and 7S connected to the second and third neural networks 6n and 7n, respectively, calculate Softmax values ​​based on the logit values. The second and third neural networks 6n and 7n output as many Softmax values ​​as the number of classes. The Softmax values ​​are values ​​that quantify the confidence levels corresponding to the corresponding classes.

[0054] As described above, in the first embodiment, a self-learning unit 9 is provided, and the first target image to which a pseudo-label has been assigned by the classification unit 6 is input to the feature extraction unit 5 to train the first to third neural networks 5n, 6n, and 7n, and the self-learning loss is calculated. Furthermore, a learning unit 10 is provided, and the first to third neural networks 5n, 6n, and 7n are trained by performing backpropagation processing based on the source loss output from the first cross-entropy unit 15, the domain loss output from the second cross-entropy unit 16, and the self-learning loss output from the third cross-entropy unit 17. This allows an appropriate label to be assigned to an unlabeled first target image. Furthermore, since the self-learning unit 9 performs self-learning only when it is expected that a valid label can be assigned, processing efficiency can be improved.

[0055] (First Modification of the First Embodiment) In the first modification of the first embodiment, the pseudo label of the first target image assigned during the processing of one mini-batch is not used in other mini-batches.

[0056] The block configuration of the image processing device 1 according to the first modified example is the same as that of Figures 1 and 2. Figure 5 is a flowchart showing the processing operation of the image processing device 1 according to the first modified example of the first embodiment. The flowchart of Figure 5 is the flowchart of Figure 3 with S10 added.

[0057] When steps S1 to S9 are completed for all source images and the first target image included in a mini-batch, one iteration is completed, but the pseudo-label assigned to the first target image during that iteration is not used in the next iteration (S10). After that, the process returns to S1 and starts the iterative process for the next mini-batch.

[0058] The pseudo label assigned to the first target image in the iterative processing of a certain mini-batch may not necessarily be appropriate. If the first to third neural networks 5n, 6n, and 7n are trained using a first target image to which an inappropriate pseudo label has been assigned, this may adversely affect the iterative processing of other mini-batches. Therefore, in the first modification, the pseudo label assigned to the first target image in the iterative processing of a certain mini-batch is not used in the iterative processing of other mini-batches. This prevents the adverse effect of an inappropriate pseudo label from extending to the iterative processing of all other mini-batches, and allows the first target image to be assigned an appropriate pseudo label.

[0059] (Second Modification of the First Embodiment) In the second modification of the first embodiment, a condition for stopping self-learning after it has started is defined.

[0060] The block configuration of the image processing device 1 according to the second modified example is the same as that shown in FIGS. 1 and 2. The image processing device 1 according to the second modified example has one additional condition for performing self-learning. More specifically, in the second modified example, even if self-learning by the self-learning unit 9 has started, a condition is added that the self-learning is stopped if the source loss subsequently becomes equal to or greater than a threshold. If the source loss is equal to or greater than the threshold, there is a risk that an appropriate label may not be assigned, so the self-learning is stopped and learning is continued using labeled source images.

[0061] FIG. 6 is a flowchart showing the processing operation of the image processing device 1 according to the second modified example of the first embodiment. The flowchart in FIG. 6 is the flowchart in FIG. 3 with S11 added. S11 is executed after a YES determination is made in S4. In S11, it is determined whether the source loss output from the second cross entropy unit 16 is smaller than a threshold. If the source loss is smaller than the threshold, the self-learning process of steps S7 to S9 continues, but if the source loss is equal to or greater than the threshold, the self-learning is stopped and the process of S5 is performed. In this case, the processes of S1 to S6 are performed without further self-learning being performed.

[0062] In this way, in the second variant of the first embodiment, if the self-learning effect is not sufficient, the self-learning is stopped, thereby eliminating the risk of assigning an inappropriate pseudo label to the first target image by performing unreliable self-learning.

[0063] (Third Modification of the First Embodiment) In the third modification of the first embodiment, the calculation result of the classification unit 6 for the first target image is masked as necessary. Here, masking refers to forcibly setting the logit value, which is the calculation result of the classification unit 6, to zero.

[0064] An image processing device 1 according to the third modified example of the first embodiment has the same block configuration as that shown in Figures 1 and 2. Figures 7A, 7B, and 7C are diagrams illustrating the operation of the third modified example of the first embodiment.

[0065] Fig. 7A is a diagram illustrating the basic operation of the image processing device 1 according to the first embodiment, Fig. 7B is a diagram illustrating the operation of maximum logit masking, and Fig. 7C is a diagram illustrating the operation of total logit masking.

[0066] 7A, 7B, and 7C are bar graphs showing the Softmax values ​​for each class of four first target images IM1 to IM4 in a mini-batch. Each of the first target images IM1 to IM4 shows an example in which it has Softmax values ​​for three classes. In FIGS. 7A, 7B, and 7C, the threshold for the Softmax value shown on the vertical axis is indicated by a dotted line. In the examples of FIGS. 7A, 7B, and 7C, the Softmax value of class 3 for first target image IM1 exceeds the threshold. The Softmax value of class 2 for first target image IM2 exceeds the threshold. The Softmax values ​​of all classes for first target images IM3 and IM4 are below the threshold.

[0067] In the case of FIG. 7A, the first target images IM1 and IM2 have Softmax values ​​that exceed the threshold, and therefore pseudo labels are assigned to them. Specifically, a pseudo label corresponding to class 3 is assigned to the first target image IM1, and a pseudo label corresponding to class 2 is assigned to the first target image IM2. The first target images IM3 and IM4 do not have Softmax values ​​that exceed the threshold, and therefore no pseudo labels are assigned to them. However, the third cross-entropy unit 17 calculates the self-learning loss using the Softmax values ​​of all classes in each of the first target images IM1 to IM4.

[0068] In the maximum logit masking of Figure 7B, the first target images IM1 and IM2 are treated in the same way as in Figure 7A. For the first target images IM3 and IM4, the Softmax values ​​of the three classes are all below the threshold, so pseudo labels are assigned, but the maximum logit value is masked. The third cross-entropy unit 17 masks the maximum value of the logit values ​​of the three classes of the first target images IM3 and IM4, and calculates the self-learning loss using a Softmax value calculated again using the logit values ​​of the other classes. The maximum logit value of the Softmax value below the threshold is masked in order to increase the cross-entropy error of the corresponding first target image and provide feedback that the classification result was incorrect.

[0069] In the full logit masking of Figure 7C, the first target images IM1 and IM2 are treated in the same way as in Figure 7A. For the first target images IM3 and IM4, the Softmax values ​​of the three classes are all below the threshold, so pseudo labels are assigned, but all logit values ​​are masked. The third cross-entropy unit 17 masks all Softmax values ​​of the three classes of the first target images IM3 and IM4 and calculates the self-learning loss. Masking all logit values ​​with Softmax values ​​below the threshold provides feedback that the classification results were incorrect due to the influence of the first target image with a small Softmax value.

[0070] 8 and 9 are flowcharts showing the processing operations of the image processing device 1 according to the third modified example of the first embodiment. Fig. 8 shows the processing operations of maximum logit masking shown in Fig. 7B, and Fig. 9 shows the processing operations of total logit masking shown in Fig. 7C.

[0071] The flowchart in Fig. 8 adds S12 and S16 instead of S8 in the flowchart in Fig. 3. S12 is executed after the maximum Softmax value of the first target image is calculated as a score in S7 during self-learning.

[0072] In S12, if the score calculated in S7 is smaller than a threshold, only the logit value of the Softmax value class of that score is set to zero. In this specification, setting the logit value to zero is sometimes referred to as "masking to zero." When masking a logit value, the logit value may be set to any value other than zero. After masking the logit value, the classification unit 6 calculates the Softmax value again. This results in different Softmax values ​​before and after masking the logit value. By performing logit masking, the cross-entropy error in the third cross-entropy unit 17 is increased, providing negative feedback. In contrast, if logit masking is not performed, the cross-entropy error is reduced, providing positive feedback.

[0073] After the process of S12 is completed, the process of S16 is performed. In S16, the class corresponding to the score is set as a pseudo label for all the first target images.

[0074] The flowchart in Fig. 9 adds S13 and S16 instead of S8 in the flowchart in Fig. 3. S13 is executed after the maximum Softmax value of the first target image is calculated as a score in S7 during self-learning.

[0075] In S13, if the score calculated in S7 is smaller than the threshold, the logit values ​​of all classes in the first target image are set to 0. In the next step S16, the classes corresponding to the scores are set as pseudo labels for all first target images.

[0076] As described above, in the third modified example of the first embodiment, maximum logit masking or full logit masking is performed. In maximum logit masking, if the score, which is the maximum Softmax value output from the classification unit 6, is smaller than the threshold, the logit value of the class of that score is set to zero, resulting in a large cross-entropy error. For this reason, negative feedback is performed for some first target images. In addition, in full logit masking, if the score, which is the maximum Softmax value output from the classification unit 6, is smaller than the threshold, all logit values ​​are set to zero. For this reason, neutral feedback is performed for first target images whose Softmax value is equal to or smaller than the threshold.

[0077] (Fourth Modification of the First Embodiment) The fourth modification of the first embodiment performs masking differently from that of the third modification.

[0078] An image processing device 1 according to the fourth modified example of the first embodiment has the same block configuration as that shown in Figures 1 and 2. Figure 10 is a diagram illustrating the operation of the fourth modified example of the first embodiment.

[0079] FIG. 10 is a bar graph showing the Softmax values ​​for each class of six first target images IM1 to IM6 in a mini-batch. The first target images IM1 to IM6 each have Softmax values ​​for three classes. In FIG. 10, two thresholds t1 and t2 for the Softmax values ​​shown on the vertical axis are indicated by dotted lines. In the example of FIG. 10, the Softmax value for class 3 of the first target image IM1 is equal to or greater than the first threshold t1, and the Softmax values ​​for classes 1 and 2 are less than the second threshold t2. The Softmax value for class 2 of the first target image IM2 is equal to or greater than the first threshold t1, and the Softmax values ​​for classes 1 and 3 are less than the second threshold t2. A pseudo label corresponding to class 3 is assigned to the first target image IM1. A pseudo label corresponding to class 2 is assigned to the first target image IM2.

[0080] For the first target image IM3, the Softmax value of class 3 is less than the first threshold t1 and greater than or equal to the second threshold t2, and the Softmax values ​​of classes 1 and 2 are less than the second threshold t2. For the first target image IM4, the Softmax value of class 2 is less than the first threshold t1 and greater than or equal to the second threshold t2, and the Softmax values ​​of classes 1 and 3 are less than the second threshold t2. For the first target image IM3 and the first target image IM4, the maximum Softmax values ​​are less than the first threshold t1 and greater than or equal to the second threshold t2, so the logit values ​​of all classes are masked to zero.

[0081] For the first target image IM5 and the first target image IM6, the Softmax values ​​of all classes are less than the second threshold t2. In this case, only the class with the maximum logit value is masked to zero. That is, for the first target image IM5, the logit value of class 3 is masked to zero, and for the first target image IM6, the logit value of class 2 is masked to zero.

[0082] As shown in FIG. 10, in the fourth modification of the embodiment, when the Softmax value for each class of each first target image is within a predetermined range, the logit values ​​of all classes are masked to zero.

[0083] FIG. 11 is a flowchart showing the processing operation of the image processing device 1 according to the fourth modified example of the first embodiment. The flowchart in FIG. 11 adds S14 to S16 instead of S8 in the flowchart in FIG. 3. S14 is executed after the maximum Softmax value of the first target image is calculated as the score in S7. In S14, if the score is less than the second threshold t2 (in the case of first target images IM5 and IM6 in FIG. 10), only the logit value of the maximum value is masked to zero. In S15, if the score is less than the first threshold t1 but equal to or greater than the second threshold t2 (in the case of first target images IM3 and IM4 in FIG. 10), the logit values ​​of all classes of the first target image are masked to zero. In S16, for all first target images, the class labels of the scores are set as pseudo labels of the first target images.

[0084] After the process of S16, the third cross entropy unit 17 performs a process to calculate the self-learning loss (S9).

[0085] As described above, in the fourth modification of the first embodiment, the Softmax value for each class output from the classification unit 6 for the first target image is compared with the first threshold t1 and the second threshold t2. If the maximum Softmax value is less than the first threshold t1 and greater than or equal to the second threshold t2, the logit values ​​of all classes are masked to zero, and if the maximum Softmax value is less than the second threshold t2, only the maximum logit value is masked to zero. This makes it possible to avoid the risk of inappropriate learning being performed using a first target image whose Softmax value is within a predetermined range (less than the first threshold t1 and greater than or equal to the second threshold t2).

[0086] (Fifth Modification of the First Embodiment) The fifth modified example of the first embodiment is intended to eliminate the imbalance in the number of sheets per class.

[0087] An image processing device 1 according to a fifth modified example of the first embodiment has the same block configuration as in FIGS. 1 and 2. FIGS. 12A and 12B are diagrams explaining the operation of the fifth modified example of the first embodiment. FIGS. 12A and 12B are diagrams showing the classification results of the class classification unit 6 for multiple first target images in a certain mini-batch. The horizontal axis of FIGS. 12A and 12B represents the type of class, and the vertical axis represents the number of images. The class number distribution cn1 on the left side of FIG. 12A shows the classification results of the class classification unit 6 for all first target images in a certain mini-batch, and the class number distribution cn2 on the right side of FIG. 12A shows the classification results of the class classification unit 6 for first target images whose Softmax values ​​are equal to or greater than a threshold (for example, 0.8 or greater) among the multiple first target images in the mini-batch. The class number distribution cn2 on the right side of FIG. 12A shows an example in which the number of images in class 2 is significantly higher.

[0088] If there is a large imbalance in the number of images per class, there is a risk that the first to third neural networks 5n, 6n, and 7n may not be able to learn properly. Therefore, in the fifth modified example, the number of images per class is adjusted.

[0089] The class number distribution cn3 on the left side of FIG. 12B is the same as the class number distribution cn1 in FIG. 12A. The class number distribution cn4 in the center of FIG. 12B shows the classification results of the class classification unit 6 for first target images with Softmax values ​​equal to or greater than a threshold (for example, 0.8 or greater) among the multiple first target images in the mini-batch. The class number distribution cn5 on the right side of FIG. 12B shows the same number of first target images in each class with a Softmax value equal to or greater than a threshold. This allows the same number of pseudo labels to be assigned to the multiple first target images for each class. In the class number distributions cn3 and cn4, the threshold for the number of images shown on the vertical axis is indicated by a dotted line.

[0090] Fig. 13 is a flowchart showing the processing operation of the image processing device 1 according to the fifth modified example of the first embodiment. The flowchart in Fig. 13 is obtained by adding S17 to the flowchart in Fig. 3. S17 is executed after the process of assigning a pseudo label to the first target image having a Softmax value (score) equal to or greater than the threshold value in S8.

[0091] In S17, the number of pseudo labels for each class assigned to multiple first target images in a mini-batch is set to not exceed a threshold. After the processing of S17 is completed, the third cross-entropy unit 17 performs processing to calculate the self-learning loss (S9). Here, the thresholds in S8 and S17 are different values ​​because the targets are different.

[0092] In this way, in the fifth variant of the first embodiment, when pseudo labels are assigned to multiple first target images, the number of pseudo labels for each class is made uniform, thereby avoiding the risk of learning being strongly influenced by a particular class.

[0093] (Sixth Modification of the First Embodiment) The sixth modification of the first embodiment adjusts the total number of pseudo labels regardless of the class.

[0094] An image processing device 1 according to the sixth modified example of the first embodiment has the same block configuration as in FIGS. 1 and 2. FIGS. 14A and 14B are diagrams illustrating the operation of the fifth modified example of the first embodiment. FIGS. 14A and 14B are diagrams showing the scores of multiple first target images in a mini-batch, with the horizontal axis representing the type of first target image and the vertical axis representing the score. The score is the maximum Softmax value output from the classification unit 6 for the first target image.

[0095] Fig. 14A shows an example in which all first target images sc2 having scores equal to or greater than a threshold are extracted from the first target image group sc1. In the case of Fig. 14A, pseudo labels are assigned to all first target images sc2 having scores equal to or greater than the threshold. Fig. 14B shows an example in which the number of first target images sc3 having scores equal to or greater than the threshold and to be extracted is limited.

[0096] Pseudo labels are assigned to the first target images of the number finally extracted based on Fig. 14B, which makes it possible to limit the number of pseudo labels assigned.

[0097] Fig. 15 is a flowchart showing the processing operation of the image processing device 1 according to the sixth modified example of the first embodiment. The flowchart in Fig. 15 is obtained by adding S18 to the flowchart in Fig. 3. S18 is executed after identifying the first target image having a Softmax value (score) equal to or greater than the threshold value in S8.

[0098] In S18, if the number of first target images to be assigned pseudo labels is less than a predetermined ratio of the total number of first target images included in the mini-batch (i.e., less than the predetermined ratio), pseudo labels are assigned to the corresponding first target images. After the processing of S18 is completed, processing of the third cross-entropy unit 17 is performed, and the self-learning loss is calculated (step S9).

[0099] In this way, in the sixth modification of the first embodiment, the total number of first target images to which pseudo labels are assigned is limited, and therefore the processing load on the image processing device 1 can be reduced.

[0100] (Seventh Modification of the First Embodiment) In the seventh modification of the first embodiment, the weights of the source loss and the domain loss are adjusted according to the progress of self-learning.

[0101] An image processing device 1 according to a seventh modification of the first embodiment has the same block configuration as those shown in FIGS. 1 and 2. FIG. 16 is a diagram illustrating the seventh modification of the first embodiment. The horizontal axis of FIG. 6 represents the period (learning period) since the start of self-learning, and the vertical axis represents the weight values ​​of the source loss and the domain loss. The weight loss generator 18 shown in FIG. 1 adjusts the weights of the source loss output from the first cross-entropy unit 15, the domain loss output from the second cross-entropy unit 16, and the self-learning loss output from the third cross-entropy unit 17 to obtain a final weight loss. The weight loss generator 18 may vary the weights of the source loss and the domain loss depending on the progress of self-learning by the self-learning unit 9.

[0102] 16, the weights of the source loss and the domain loss are decreased as the self-learning progresses. That is, as the self-learning progresses, the self-learning unit 9 performs self-learning by placing more importance on the self-learning loss than on the source loss and the domain loss.

[0103] The weight loss generator 18 calculates the final weight loss (loss) based on, for example, the following equation (1). loss=theta d ×domain_loss+theta s ×source_loss+theta p × selftrain_loss …(1)

[0104] The first term on the right side of equation (1) is the weighting coefficient theta d The second term on the right side is the source loss (source_loss) multiplied by the weighting coefficient theta s The third term on the right side is the self-training loss (selftrain_loss) multiplied by the weighting coefficient theta p These weighting coefficients theta d , theta s , theta p can be changed according to the progress of self-study.

[0105] Weighting coefficient thetad , theta s , theta p may be a value obtained by multiplying a function f(λ) shown in the following equation (2) by an arbitrary fixed value.

number

[0106] The formula for calculating the weight loss is not limited to the above formulas (1) and (2).

[0107] FIG. 17 is a flowchart showing the processing operation of the image processing device 1 according to the seventh modified example of the first embodiment. The flowchart in FIG. 17 is the flowchart in FIG. 3 with S19 added. S19 calculates the weight loss used when performing the backpropagation process in S5. The weight loss is calculated using, for example, the above-mentioned formulas (1) and (2). As learning progresses, the weight loss is calculated so that the weights of the source loss and domain loss become smaller. After the process in S19 is completed, the processes from S6 onwards are performed. Here, the order of the processes in S5 and S19 may be reversed.

[0108] In this way, in the seventh modification of the first embodiment, the weights of the source loss and the domain loss are changed according to the progress of self-learning, so that learning can be performed efficiently.

[0109] (Second embodiment) The learning unit 10 in the first embodiment learns the first to third neural networks 5n, 6n, and 7n by performing backpropagation processing based on the source loss, domain loss, and self-learning loss, but the processing operation of the learning unit 10 in the second embodiment differs from that in the first embodiment.

[0110] 18 is a block diagram of the main parts of an image processing device 1a according to the second embodiment. In the image processing device 1a according to the second embodiment, a first learning unit 10a and a second learning unit 10b each perform backpropagation processing independently to train the first to third neural networks 5n, 6n, and 7n.

[0111] The first learning unit 10a learns the first to third neural networks 5n, 6n, and 7n by performing backpropagation processing based on the source loss output from the first cross entropy unit 15 and the domain loss output from the second cross entropy unit 16. The weight loss generation unit 18 generates a weight loss based on the source loss and the domain loss. The first learning unit 10a performs backpropagation processing based on the weight loss generated based on the source loss and the domain loss.

[0112] On the other hand, the second learning unit 10b performs backpropagation processing based on the self-learning loss output from the third cross entropy unit 17, thereby learning the first and second neural networks 5n and 6n.

[0113] As described above, the second embodiment differs from the first embodiment in that learning is performed in two stages.

[0114] Fig. 19 is a flowchart showing the processing operation of the image processing device 1a according to the second embodiment. The flowchart in Fig. 19 is obtained by adding S20 and S21 to the flowchart in Fig. 3 and by partially changing the order in which the processes in S1 to S9 are executed.

[0115] 19, steps S1, S2, S3, S5, S6, and S4 are executed in this order. Learning is performed by the first learning unit 10a through these processes.

[0116] When self-learning is started according to the determination result of S4, the processes of S7 to S9 are executed in order. Thereafter, learning is performed by the second learning unit 10b in S20 and S21. In S20, gradients of each parameter of the first and second neural networks 5n and 6n are generated by performing backpropagation processing based on the self-learning loss. In S21, each parameter is updated based on the parameter gradients. When the process of S21 is completed, the processes from S1 onwards are repeated.

[0117] In this way, in the second embodiment, learning of the first to third neural networks 5n, 6n, 7n by backpropagation processing based on source loss and domain loss, and learning of the first and second neural networks 5n, 6n by backpropagation processing based on self-learning loss can be performed in two stages.

[0118] (Third embodiment) The first and second embodiments are based on the premise that a first target image without a label is acquired, whereas the third embodiment is based on the premise that a second target image with a label is acquired in addition to the first target image without a label.

[0119] Fig. 20 is a block diagram showing a schematic configuration of an image processing device 1b according to the third embodiment. The image processing device 1b in Fig. 20 includes a first image acquisition unit 2, a second image acquisition unit 3, a third image acquisition unit 20, a label acquisition unit 4, a feature extraction unit 5, a class classification unit 6, a domain classification unit 7, a first data combination unit 11, a second data combination unit 12, a first cross-entropy unit 15 (first loss calculation unit), a second cross-entropy unit (second loss calculation unit) 16, and a weight loss generation unit 18. Of these, the first data combination unit 11, the second data combination unit 12, the first cross-entropy unit 15 (first loss calculation unit), the second cross-entropy unit (second loss calculation unit) 16, and the weight loss generation unit 18 can be omitted as appropriate.

[0120] As described above, in the image processing device 1b of FIG. 20, compared to the configuration of FIG. 1, the third image acquisition unit 20 is newly added, but the self-learning unit 9 is omitted.

[0121] TIFF0007778607000015.tif30170

[0122] TIFF0007778607000016.tif22170

[0123] The classification unit 6 has a second neural network 6n that classifies the source image and the second target image based on the features of the source image and the second target image.

[0124] The domain classification unit 7 includes a third neural network 7n that performs domain classification of the source image, the first target image, and the second target image based on the features of the source image, the first target image, and the second target image.

[0125] The first cross entropy unit 15 calculates a source loss and a few-shot loss that represent the reliability of class classification of the source image, the first target image, and the second target image by the classifier 6. The second cross entropy unit 16 calculates a domain loss that represents the reliability of domain classification of the source image, the first target image, and the second target image by the domain classifier 7. The weighted loss generator 18 generates a weighted loss by weighting the source loss, the few-shot loss, and the domain loss.

[0126] The learning unit 10 performs backpropagation processing based on the weight loss, thereby learning the first to third neural networks 5n, 6n, and 7n.

[0127] 20 may include a first data combination unit 11 and a second data combination unit 12. The first data combination unit 11 inputs data obtained by combining feature data of a source image and feature data of a second target image to the class classification unit 6. The second data combination unit 12 inputs data obtained by combining feature data of the first target image and feature data of the second target image to the domain classification unit 7.

[0128] Fig. 21 is a flowchart showing the processing operation of the image processing device 1b according to the third embodiment. The processing of the flowchart in Fig. 21 is performed for each mini-batch including a plurality of source images, a plurality of first target images, and a small number of second target images. In the flowchart in Fig. 21, the first to third neural networks 5n, 6n, and 7n are trained in units of mini-batches.

[0129] TIFF0007778607000017.tif14170

[0130] TIFF0007778607000018.tif51170

[0131] TIFF0007778607000019.tif36170

[0132] Next, backpropagation processing is performed using a weighted loss obtained by weighting the source loss, domain loss, and few-shot loss to generate gradients of each parameter in the feature extraction unit 5, class classification unit 6, and domain classification unit 7 (S34). Next, based on the generated gradients, parameters such as the weights of each layer of the first to third neural networks 5n, 6n, and 7n are updated (step S35).

[0133] As described above, in the third embodiment, a small number of labeled second target images are input to the feature extraction unit 5, and a forward propagation process is performed to calculate the small-shot loss. The first to third neural networks 5n, 6n, and 7n are then trained by a backpropagation process based on the source loss, domain loss, and small-shot loss. Using a small number of labeled second target images can improve the learning effect.

[0134] (First modified example of the third embodiment) In the first modified example of the third embodiment, an image processing device 1b according to the third embodiment is provided with a self-learning function similar to that of the image processing devices 1 and 1a according to the first and second embodiments.

[0135] Fig. 22 is a block diagram showing a schematic configuration of an image processing device 1c according to a first modified example of the third embodiment. The image processing device 1c in Fig. 22 is configured by adding a self-learning unit 9 including a processing unit 8 to the configuration in Fig. 20.

[0136] The processing unit 8 assigns a pseudo label to the first target image using the classification unit 6 having the second neural network 6n that is currently being trained.

[0137] The self-learning unit 9 inputs the first target image with the pseudo-label assigned to the feature extraction unit 5 and performs self-learning of the first to third neural networks 5n, 6n, 7n. The self-learning unit 9 also extracts features of the unlabeled first target image using the feature extraction unit 5, and calculates the self-learning loss using the third cross-entropy unit 17 based on the Softmax value for each class output from the classification unit 6 in the processing unit 8 based on the extracted features.

[0138] The weight loss generator 18 generates a weight loss by weighting the source loss and few-shot loss output from the first cross entropy unit 15, the domain loss output from the second cross entropy unit 16, and the self-learning loss output from the third cross entropy unit 17.

[0139] The learning unit 10 performs backpropagation processing based on the weight loss, thereby learning the first to third neural networks 5n, 6n, and 7n.

[0140] 23 is a flowchart showing the processing operation of an image processing device 1c according to a first modified example of the third embodiment. The flowchart in Fig. 23 is obtained by adding S36 to S39 to the flowchart in Fig. 21.

[0141] In S31 to S33 of FIG. 23, similarly to S31 to S33 of FIG. 21, forward propagation processing is performed to calculate the source loss, domain loss, and minority shot loss.

[0142] After performing the processes of S31 to S33 for all source images, first target images, and second target images in the mini-batch, it is determined whether the source loss is on a decreasing trend or whether the number of processed epochs is less than a predetermined percentage of the total number of epochs (S36).

[0143] TIFF0007778607000020.tif37170

[0144] TIFF0007778607000021.tif20170

[0145] TIFF0007778607000022.tif20170

[0146] Next, the learning unit 10 performs backpropagation processing based on the source loss, domain loss, few-shot loss, and, in some cases, self-learning loss, thereby generating gradients of the parameters of the first to third neural networks 5n, 6n, and 7n (S34a).

[0147] Next, the learning unit 10 updates the parameters of the first to third neural networks 5n, 6n, and 7n based on the gradients of the generated parameters (S35). When the process of S35 ends, the processes from S31 onwards are repeated.

[0148] In this way, in the first variant of the third embodiment, not only are a small number of labeled second target images obtained, but a self-learning unit 9 is also provided, so that appropriate labels can be assigned to unlabeled first target images.

[0149] (Second modified example of the third embodiment) The second modification of the third embodiment is intended to eliminate variations in the number of source images for each class.

[0150] Fig. 24 is a block diagram showing a schematic configuration of an image processing device 1d according to a second modified example of the third embodiment. In addition to the configuration of Fig. 22, the image processing device 1d of Fig. 24 includes a DOA (Data Oversampling Augmentation) unit 21 and a preprocessing unit 22. The DOA unit 21 increases the number of source images for classes with a small number of images in order to eliminate variations in the number of source images for each class. When increasing the number of source images, new source images are generated, for example, by rotating existing source images by a predetermined angle.

[0151] By providing the DOA unit 21, source images with no variation in the number of images per class are input to the feature extraction unit 5. Therefore, when the classification unit 6 classifies the source images, there is no variation in the number of source images between classes, and the learning effect for each class can be made uniform.

[0152] The preprocessing unit 22 performs preprocessing on the source images and the unlabeled first target images added by the DOA unit 21. The preprocessing includes filtering to remove unnecessary images such as noisy images. Each image preprocessed by the preprocessing unit 22 is input to the feature extraction unit 5.

[0153] 25A is a diagram showing the distribution nd1 of the number of source images per class in a mini-batch input to the image processing device 1d. As shown in Fig. 25A, the source images are classified into multiple classes, and the number of source images in each class is not necessarily the same, but varies depending on the class.

[0154] 25B is a diagram illustrating the processing of the DOA unit 21. The DOA unit 21 increases the number of source images in a class with a small number of source images so that the number of source images belonging to each class is the same (number distribution nd2). As a result, the source images output from the DOA unit 21 do not vary in number between classes.

[0155] Fig. 26 is a flowchart showing the processing operation of the image processing device 1d according to the second modified example of the third embodiment. The flowchart in Fig. 26 is obtained by adding S40 to the flowchart in Fig. 23. In S40, before starting the processing of S31, the DOA unit 21 performs processing (augmentation processing) to increase the number of source images so that the number of source images for each class is equal. Thereafter, similar to Fig. 23, the processing from S31 onwards is started.

[0156] In this way, in the second variant of the third embodiment, the number of source images input to the feature extraction unit 5 is increased so that the number of images of each class is equal, thereby eliminating variation in the accuracy of pseudo-label assignment between classes.

[0157] (Third Modification of the Third Embodiment) In the third modified example of the third embodiment, not only the number of source images is increased, but also the number of labeled second target images is increased.

[0158] Fig. 27 is a block diagram showing a schematic configuration of an image processing device 1e according to a third modified example of the third embodiment. The image processing device 1e in Fig. 27 includes a first DOA unit 21a, a second DOA unit 21b, and a pre-processing unit 22 in addition to the configuration in Fig. 22.

[0159] The first DOA unit 21a is the same as the DOA unit 21 in Fig. 24, and increases the number of source images so that the number of source images for each class is equal. The second DOA unit 21b increases the number of second target images so that the number of labeled second target images for each class is equal.

[0160] The preprocessing unit 22 performs preprocessing on the source image added by the first DOA unit 21 a, the labeled second target image added by the second DOA unit 21 b, and the unlabeled first target image. Each image preprocessed by the preprocessing unit 22 is input to the feature extraction unit 5.

[0161] Fig. 28A is a diagram showing a number distribution nd3 for each class of source images before being input to the first DOA unit 21a, or first target images before being input to the second DOA unit 21b. Fig. 28B is a diagram showing the processing operations of the first DOA unit 21a and the second DOA unit 21b. As shown in Fig. 28B, both the first DOA unit 21a and the second DOA unit 21b increase the number of images so that the number of images for each class is equal (number distribution nd4).

[0162] Fig. 29 is a block diagram showing a schematic configuration of an image processing device 1f according to a third modified example of the third embodiment. The flowchart in Fig. 29 is obtained by adding S41 to the flowchart in Fig. 26. S40 in Fig. 29 is executed by the first DOA unit 21a instead of the DOA unit 21 in Fig. 24.

[0163] In S41, before starting the process of S31, the second DOA unit 21b performs a process (augmentation process) to increase the number of second target images so that the number of labeled second target images for each class is equal. The total number of second target images is made equal to the total number of source images. Then, the process from S31 onwards is started.

[0164] In this way, in the third modification of the third embodiment, not only the source images but also the labeled second target images are increased, so that the accuracy variation of the pseudo labels for each class can be further suppressed.

[0165] (Fourth embodiment) The fourth embodiment describes the inference processing performed after the learning of the first to third neural networks 5n, 6n, and 7n is completed in the image processing devices 1, 1a, 1b, 1c, 1d, 1e, and 1f according to the first to third embodiments described above.

[0166] Fig. 30 is a flowchart showing the processing operations of the inference process. Fig. 30 shows the processing operations of the classifying unit 6 when a first target image is input. First, at least one image of the inference target captured by an imaging device is acquired (S51). The image acquired in S51 is the first target image. Next, the acquired first target image is input to the feature extracting unit 5, and forward propagation processing is performed to extract features (S52).

[0167] Next, the extracted features are input to the classification unit 6, and a forward propagation process is performed to calculate a logit value for each class of each first target image (S53). Next, the classification unit 6 calculates a Softmax value from the logit value, obtains the class with the largest Softmax value, and calculates the Softmax value of the first target image (S54). The class with the largest Softmax value is assigned as the pseudo label.

[0168] Figure 31 is a diagram that schematically shows the layer structure of the first neural network 5n used when the feature extraction unit 5 extracts the features of the first target image, and the layer structure of the second neural network 6n used when the class classification unit 6 classifies the first target image, and is the same as that shown in Figure 4.

[0169] In this way, the first to third neural networks 5n, 6n, 7n trained by the image processing devices 1, 1a, 1b, 1c, 1d, 1e, 1f according to the first to third embodiments can be used to classify the first target image, thereby assigning an appropriate pseudo label to the first target image.

[0170] At least a part of the image processing devices 1, 1a, 1b, 1c, 1d, 1e, and 1f described in the first to third embodiments may be configured with hardware or software. If configured with software, a program that realizes at least a part of the functions of the image processing devices 1, 1a, 1b, 1c, 1d, 1e, and 1f may be stored on a recording medium such as a flexible disk or a CD-ROM and read and executed by a computer. The recording medium is not limited to removable media such as magnetic disks and optical disks, but may also be fixed recording media such as hard disk drives and memories.

[0171] In addition, a program that realizes at least some of the functions of the image processing devices 1, 1a, 1b, 1c, 1d, 1e, and 1f may be distributed via a communication line (including wireless communication) such as the Internet. Furthermore, the program may be encrypted, modulated, or compressed and distributed via a wired or wireless line such as the Internet, or stored on a recording medium.

[0172] 32 is a block diagram showing a hardware configuration for performing processing operations of the image processing devices 1, 1a, 1b, 1c, 1d, 1e, and 1f according to the first to third embodiments. The image processing devices 1, 1a, 1b, 1c, 1d, 1e, and 1f in FIG. 32 are realized by an arithmetic circuit 100. The arithmetic circuit 100 performs processing operations of the first image acquisition unit 2, the second image acquisition unit 3, the label acquisition unit 4, the feature extraction unit 5, the class classification unit 6, the domain classification unit 7, the self-learning unit 9 including the processing unit 8, the learning unit 10, the first data combination unit 11, the second data combination unit 12, the first judgment unit 13, the second judgment unit 14, the first cross-entropy unit 15, the second cross-entropy unit 16, the third cross-entropy unit 17, and the weight loss generation unit 18 shown in FIG. The arithmetic circuit 100 may be, for example, a computer device such as a general-purpose PC (Personal Computer) or a server, or may be a semiconductor chip such as a CPU (Central Processing Unit), a signal processor, or a GPU (Graphic Processing Unit).

[0173] The memory 200 stores programs executed by the arithmetic circuit 100, data used by the arithmetic circuit 100 for arithmetic processing, and the like. Specifically, the layer configurations, weighting coefficients, input data, output data, and the like of the first to third neural networks 5n, 6n, and 7n are stored in the memory 200. The various programs and data stored in the memory 200 are read out by the arithmetic circuit 100 as needed, and the results of calculations by the arithmetic circuit 100 are stored in the memory 200 as needed. The memory 200 may be a stationary storage device such as a hard disk device or an optical disk device, or may be a semiconductor storage device such as a volatile memory or a non-volatile memory.

[0174] This embodiment can be summarized as follows. [Appendix 1] a first image acquisition unit that acquires a source image to which a label has been assigned; a second image acquisition unit that acquires a first target image that is not labeled; a label acquisition unit that acquires a label; a feature extraction unit having a first neural network for extracting features of the source image and features of the first target image; a classification unit having a second neural network that performs classification of the source image and the first target image based on the plurality of features extracted by the feature extraction unit; a domain classification unit having a third neural network that performs domain classification of the source image and the first target image based on the features extracted by the feature extraction unit; a processing unit that assigns a pseudo label to the first target image using the classification unit having the second neural network in the middle of training; a self-learning unit that performs self-learning of the first neural network, the second neural network, and the third neural network based on features obtained by inputting the first target image to which the pseudo label has been assigned to the feature extraction unit; and a learning unit that learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the classification result of the class classification unit, the classification result of the domain classification unit, and the self-learning result of the self-learning unit. [Appendix 2] a first loss calculation unit that calculates a first loss that indicates reliability of classification of the source image by the classification unit; a second loss calculation unit that calculates a second loss that indicates reliability of domain classification of the source image and the first target image by the domain classification unit; a third loss calculation unit that calculates a third loss that indicates reliability of the class classification of the first target image to which the pseudo label is assigned, the self-learning unit inputs the first target image to which the pseudo label has been assigned to the feature extraction unit, and causes the third loss calculation unit to calculate the third loss; 2. The image processing device according to claim 1, wherein the learning unit learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the first loss, the second loss, and the third loss. [Appendix 3] a first loss calculation unit that calculates a first loss that indicates reliability of classification of the source image by the classification unit; a second loss calculation unit that calculates a second loss that indicates reliability of domain classification of the source image and the first target image by the domain classification unit; a third loss calculation unit that calculates a third loss that indicates reliability of class classification of the first target image by the class classification unit, the self-learning unit inputs the first target image to which the pseudo label has been assigned to the feature extraction unit, and causes the third loss calculation unit to calculate the third loss; The image processing device described in Appendix 1, wherein the learning unit comprises: a first learning unit that performs backpropagation processing based on the first loss and the second loss to learn the first neural network, the second neural network, and the third neural network; and a second learning unit that performs backpropagation processing based on the third loss to learn the first neural network and the second neural network. [Appendix 4] 4. The image processing device according to claim 2, wherein the self-learning unit stops the self-learning when the second loss becomes equal to or greater than a threshold value after starting the self-learning. [Appendix 5] a weight loss generation unit that generates a weight loss by adjusting weights of the first loss, the second loss, and the third loss when performing the backpropagation process; 5. The image processing device according to claim 2, wherein the learning unit performs the backpropagation process based on the weight loss to learn the first neural network, the second neural network, and the third neural network. [Appendix 6] The image processing device according to claim 5, wherein the weight loss generation unit generates the weight loss by lowering weights of the first loss and the second loss and increasing a weight of the third loss as the self-learning by the self-learning unit progresses. [Appendix 7] the class classification unit and the domain classification unit perform class classification and domain classification for a plurality of epochs, with an image group including the plurality of source images and the first target image as one epoch; The image processing device according to any one of appendix 2 to 6, wherein the self-learning unit starts labeling the first target image and the self-learning when the first loss is equal to or less than a threshold value or when the number of processed epochs exceeds a predetermined ratio of the total number of epochs. [Appendix 8] a third image acquisition unit that acquires second target images having a number smaller than that of the first target images and having labels attached thereto; the feature extraction unit extracts features of the second target image; the class classification unit classifies the second target image based on the features of the second target image extracted by the feature extraction unit, the domain classification unit performs domain classification of the second target image based on the features of the second target image extracted by the feature extraction unit; The image processing device according to any one of appendixes 1 to 7, wherein the learning unit learns the first neural network, the second neural network, and the third neural network based on class and domain classification results of the source image, the first target image, and the second target image. [Appendix 9] The image processing device according to any one of appendices 1 to 8, wherein the self-learning unit performs the self-learning by inputting, into the feature extraction unit as training data, first target images to which pseudo labels have been assigned by the class classification unit, the first target images having a certainty of the pseudo label equal to or greater than a threshold. [Appendix 10] the class classification unit and the domain classification unit perform class classification and domain classification for each image group including the plurality of source images and the first target image, The image processing device according to any one of appendices 1 to 9, wherein the self-learning unit determines whether or not to assign the pseudo label to the first target image each time one of the first target images in the image group is input to the second image acquisition unit, and performs the self-learning based on the first target image to which the pseudo label has been assigned. [Appendix 11] the class classification unit and the domain classification unit perform class classification and domain classification for each image group including the plurality of source images and the first target image, The image processing device according to any one of claims 1 to 10, wherein the self-learning unit assigns a pseudo label to the first target image in one of the image groups without referring to the pseudo label assigned to the first target image in another of the image groups. [Appendix 12] the class classification unit outputs a Softmax value that quantifies a degree of certainty that the first target image is classified into each of a plurality of classes; 12. The image processing device according to claim 1, wherein, when a maximum value of the Softmax values ​​exceeds a threshold, the self-learning unit assigns the pseudo label corresponding to a class of the maximum value. [Appendix 13] the class classification unit outputs a Softmax value that quantifies a degree of certainty that the first target image is classified into each of a plurality of classes; 12. The image processing device according to claim 1, wherein, when the maximum value of the Softmax values ​​is equal to or less than a threshold, the self-learning unit performs the self-learning by masking a value related to the maximum value of the Softmax values. [Appendix 14] the class classification unit outputs a Softmax value that quantifies a degree of certainty that the first target image is classified into each of a plurality of classes; 12. The image processing device according to claim 1, wherein the self-learning unit performs the self-learning by masking values ​​related to all of the Softmax values ​​corresponding to the plurality of classes that constitute the Softmax values ​​when the maximum value of the Softmax values ​​is equal to or less than a threshold. [Appendix 15] the class classification unit and the domain classification unit perform class classification and domain classification for each image group including the plurality of source images and the first target image, The image processing device according to any one of claims 1 to 14, wherein the self-learning unit adjusts the number of images for each class to which the pseudo label is assigned among the plurality of first target images included in the image group. [Appendix 16] a first image acquisition unit that acquires a source image to which a label has been assigned; a second image acquisition unit that acquires a first target image that is not labeled; a third image acquisition unit that acquires second target images, the number of which is smaller than that of the first target images, and to which labels are assigned; a label acquisition unit that acquires a label; a feature extraction unit having a first neural network that extracts features of the source image, features of the first target image, and features of the second target image; a classification unit having a second neural network that performs classification of the source image and the second target image based on the plurality of features extracted by the feature extraction unit; a domain classification unit having a third neural network that performs domain classification of the source image, the first target image, and the second target image; a learning unit that learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the classification results of the class classification unit and the classification results of the domain classification unit. [Appendix 17] a first loss calculation unit that calculates a first loss that indicates reliability of class classification of the source image and the second target image by the class classification unit; a second loss calculation unit that calculates a second loss that indicates reliability of domain classification of the source image, the first target image, and the second target image by the domain classification unit, 17. The image processing device of claim 16, wherein the learning unit learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the first loss and the second loss. [Appendix 18] a processing unit that assigns a pseudo label to the first target image using the classification unit having the second neural network in the middle of training; 18. The image processing device of claim 17, further comprising a self-learning unit that performs self-learning of the first neural network, the second neural network, and the third neural network by inputting the first target image to which the pseudo-label has been assigned to the feature extraction unit. [Appendix 19] a third loss calculation unit that calculates a third loss that represents reliability of class classification of the first target image by the class classification unit; the self-learning unit inputs the first target image to which the pseudo label has been assigned to the feature extraction unit, and causes the third loss calculation unit to calculate the third loss; 19. The image processing device of claim 18, wherein the learning unit learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the first loss, the second loss, and the third loss. [Appendix 20] An image processing device as described in Appendix 8 or 16, comprising an augmentation unit that increases the number of second target images acquired by the third image acquisition unit so that the number of second target images for each class is equal.

[0175] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents. [Explanation of symbols]

[0176] 1, 1a, 1b, 1c, 1d, 1e, 1f image processing device, 2 first image acquisition unit, 3 second image acquisition unit, 4 label acquisition unit, 5 feature extraction unit, 5L layer, 5n first neural network, 6 class classification unit, 6L fully connected layer, 6n second neural network, 6S layer, 7 domain classification unit, 7L fully connected layer, 7n third neural network, 7S layer, 8 processing unit, 9 self-learning unit, 10 learning unit, 10a first learning unit, 10b second learning unit, 11 first data combination unit, 12 second data combination unit, 13 first judgment unit, 14 second judgment unit, 15 first cross-entropy unit, 16 second cross-entropy unit, 17 third cross-entropy unit, 18 loss generation unit, 20 third image acquisition unit, 21 DOA unit, 21a first DOA unit, 21b second DOA unit, 22 Preprocessing section

Claims

1. a first image acquisition unit that acquires a source image to which a label has been assigned; a second image acquisition unit that acquires a first target image that is not labeled; a label acquisition unit that acquires a label; a feature extractor having a first neural network for extracting features of the source image and features of the first target image; a classification unit having a second neural network that performs classification of the source image and the first target image based on the plurality of features extracted by the feature extraction unit; a domain classification unit having a third neural network that performs domain classification of the source image and the first target image based on the features extracted by the feature extraction unit; a processing unit that assigns a pseudo label to the first target image using the classification unit having the second neural network in the middle of training; a self-learning unit that performs self-learning of the first neural network, the second neural network, and the third neural network based on features obtained by inputting the first target image to which the pseudo label has been assigned to the feature extraction unit; a learning unit that learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the classification result of the class classification unit, the classification result of the domain classification unit, and the self-learning result of the self-learning unit; a first loss calculation unit that calculates a first loss that indicates reliability of classification of the source image by the classification unit; a second loss calculation unit that calculates a second loss representing reliability of domain classification of the source image and the first target image by the domain classification unit; a third loss calculation unit that calculates a third loss that indicates reliability of class classification of the first target image by the class classification unit, the self-learning unit inputs the first target image to which the pseudo label has been assigned to the feature extraction unit, and causes the third loss calculation unit to calculate the third loss; The image processing device includes: a first learning unit that performs backpropagation processing based on the first loss and the second loss to learn the first neural network, the second neural network, and the third neural network; and a second learning unit that performs backpropagation processing based on the third loss to learn the first neural network and the second neural network.

2. a weight loss generator that generates a weight loss by adjusting weights of the first loss, the second loss, and the third loss when performing the backpropagation process; The image processing device according to claim 1 , wherein the weighted loss generator generates the weighted loss by decreasing weights of the first loss and the second loss and increasing a weight of the third loss as the self-learning by the self-learning unit progresses.

3. the class classification unit outputs a Softmax value that quantifies a degree of certainty that the first target image is classified into each of a plurality of classes; The image processing device according to claim 1 , wherein the self-learning unit assigns the pseudo label corresponding to a class of the maximum Softmax value when the maximum Softmax value exceeds a threshold.

4. The class classification unit and the domain classification unit perform class classification and domain classification for each image group including a plurality of the source images and the first target image, 4. The image processing device according to claim 1, wherein the self-learning unit determines whether or not to assign the pseudo label to the first target image each time one of the first target images in the image group is input to the second image acquisition unit, and performs the self-learning based on the first target image to which the pseudo label has been assigned.

5. The class classification unit and the domain classification unit classify the plurality of source images and the first target image. For each image group including the target image, class classification and domain classification are performed. The self-learning unit is configured to: assigning a pseudo-label to the first target image in the set of other images without reference to the label. The image processing device according to claim 1 .

6. a first image acquisition unit that acquires a source image to which a label has been assigned; a second image acquisition unit that acquires a first target image that is not labeled; a third image acquisition unit configured to acquire second target images, the number of which is smaller than that of the first target images, and to which labels are assigned; a label acquisition unit that acquires a label; a feature extractor having a first neural network that extracts features of the source image, features of the first target image, and features of the second target image; a classification unit having a second neural network that performs classification of the source image and the second target image based on the plurality of features extracted by the feature extraction unit; a domain classification unit having a third neural network that performs domain classification of the source image, the first target image, and the second target image; a learning unit that learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the classification results of the class classification unit and the classification results of the domain classification unit.

7. a first loss calculation unit that calculates a first loss that indicates reliability of class classification of the source image and the second target image by the class classification unit; a second loss calculation unit that calculates a second loss that indicates reliability of domain classification of the source image, the first target image, and the second target image by the domain classification unit, The image processing device according to claim 6 , wherein the learning unit learns the first neural network, the second neural network, and the third neural network by performing backpropagation processing based on the first loss and the second loss.

Images