Learning device, learning method, learning program, object detection device, object detection method, object detection program, learning support system, learning support method, and learning support program

Abstract

To improve the performance of object detection by a detection network obtained by domain adaptation.SOLUTION: A learning apparatus includes a learning section that generates a detection network for detecting an object from an image belonging to a target domain. The learning section computes detection task loss that is loss relating to a detection position and class identification, by supervised learning of the detection network on the basis of a target domain image. The learning section computes domain identification loss that is loss relating to identification of the target domain and a source domain, by adversarial learning of the detection network on the basis of the target domain image and a source domain image. The learning section computes self-learning loss that is loss relating to class identification by self-learning, by self-learning of the detection network on the basis of the target domain image. The learning section computes the total loss on the basis of weighting addition of the domain identification loss, the detection task loss and / or the self-learning loss. The learning section updates learning parameters of the detection network so as to minimize the total loss.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a learning device, a learning method, a learning program, an object detection device, an object detection method, an object detection program, a learning support system, a learning support method, and a learning support program. [Background technology]

[0002] In recent years, many object detection methods using convolutional neural networks (CNNs) have been proposed. Among these, domain adaptation has been proposed, which efficiently trains a detection network that has already been trained on data belonging to an existing domain (source domain) using data belonging to a new domain (target domain). In particular, in domain adaptation using weakly supervised learning, a detection network is trained on data belonging to a new domain, to which no teaching labels (annotations) such as object detection positions are given. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-200685 [Non-patent literature]

[0004] [Non-Patent Document 1] Kuniaki Saito, et al. “Strong-Weak Distribution Alignment for Adaptive Object Detection”, arXiv:1812.04798v3 [cs.CV] 5 Apr 2019 [Non-patent document 2] Seunghyeon Kim, et al. “Self-Training and Adversarial Background Regularization for Unsupervised Domain Adaptive One-Stage Object Detection”, arXiv:1909.00597v1 [cs.CV] 2 Sep 2019 [Non-patent document 3] Vikas Verma, et al. “Manifold Mixup: Better Representations by Interpolating Hidden States” , arXiv:1806.05236v7 [stat.ML] 11 May 2019 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 trains an object detector using only image-level category labels without location information. It does not address domain adaptation when the class is unknown or when the domain gap is large. Non-Patent Document 1 proposes a function that uses adversarial learning to align feature representations of the source and target domains. Non-Patent Document 2 introduces weak self-supervised learning and adversarial background score regularization to perform training using target domain data. Non-Patent Document 3, regarding data augmentation aimed at increasing training data, proposes a method for effectively learning boundary surfaces using Manifold Mixup, which uses a linear combination of the outputs of the hidden layers of a network, in MixUp, which increases the number of image data by linearly combining images. Both of the functions in Non-Patent Documents 1 and 3 have been implemented in two-stage object detection and other tasks. However, there have been no examples of their implementation in a one-stage object detector and their combined verification. Furthermore, Non-Patent Document 2 incorporates self-supervised learning to mitigate the adverse effects of inaccurate pseudo-labels, but there is still room for improvement in the pseudo-label generation method.

[0006] The problem to be solved by the present invention is to provide a learning device, a learning method, a learning program, an object detection device, an object detection method, an object detection program, a learning support device, a learning support method, and a learning support program that are capable of improving the performance of object detection using a detection network obtained by domain adaptation. [Means for solving the problem]

[0007] A learning device according to an embodiment includes a learning unit that generates a detection network for detecting objects from images belonging to a target domain. The learning unit includes a detection task loss calculation unit that calculates a detection task loss, which is a loss related to detection position and class identification, through supervised learning of the detection network based on images belonging to the target domain; a domain identification loss calculation unit that calculates a domain identification loss, which is a loss related to identification between the target domain and the source domain, through adversarial learning of the detection network based on images belonging to the target domain and images belonging to a source domain; a self-learning loss calculation unit that calculates a self-learning loss, which is a loss related to class identification through self-learning, through self-learning of the detection network based on images belonging to the target domain; an overall loss calculation unit that calculates an overall loss based on a weighted sum of the domain identification loss, the detection task loss, and / or the self-learning loss; and an update unit that updates learning parameters of the detection network so as to minimize the overall loss. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a learning device according to a first embodiment; [Figure 2] An example of the input and output of a detection network [Figure 3] Diagram showing the process of training a detection network using domain adaptation [Figure 4] FIG. 1 is a flowchart showing domain adaptive learning by a learning device. [Figure 5]A diagram showing an example of the network configuration of a detection network used in domain adaptation learning. [Figure 6] A diagram showing an example of a network configuration for a detection network for domain identification loss. [Figure 7] Diagram showing the process of generating integrated pseudo labels [Figure 8] FIG. 10 is a pseudocode of the filtering algorithm in step S304. [Figure 9] Figure showing the predicted detection position for one image [Figure 10] A diagram showing the overlay of prediction results for multiple detection positions for multiple images. [Figure 11] Diagram showing integrated pseudo-labels [Figure 12] A diagram showing the calculation process of the manifold synthesis self-learning loss. [Figure 13] FIG. 10 is a diagram showing an example of the configuration of an object detection device according to a second embodiment; [Figure 14] FIG. 1 is a flowchart showing object detection by an object detection device. [Figure 15] FIG. 10 is a diagram showing an example of the configuration of a learning support system according to a third embodiment. [Figure 16] A flowchart showing learning support by the learning support system. [Figure 17] FIG. 10 is a diagram showing an example of a display screen for evaluation results and learning status. [Figure 18] An example of a display screen for nine evaluation results. [Figure 19] A diagram showing the aggregated results of the nine evaluation results shown in Figure 18. [Figure 20] Venn diagram showing the aggregation results in Figure 19 [Figure 21] Figure showing the performance comparison results of domain adaptive learning DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a learning device, a learning method, a learning program, an object detection device, an object detection method, an object detection program, a learning support device, a learning support method, and a learning support program according to the present embodiment will be described with reference to the drawings.

[0010] (First embodiment) The first embodiment relates to a learning device, a learning method, and a learning program.

[0011] FIG. 1 is a diagram showing an example of the configuration of a learning device 1 according to the first embodiment. The learning device 1 according to the first embodiment is a computer that trains a deep learning network for object detection. The deep learning network for object detection will be called a detection network.

[0012] FIG. 2 is a diagram showing an example of input and output of the detection network N20. The detection network N20 is a deep learning network in which learning parameters are trained so that an image I1 is input and a class name I11 and a detection position I12 of a specific object contained in the image I1 are output. The learning parameters refer to parameters assigned to the detection network N20 that are optimized during the training process. Examples of learning parameters include weight parameters and biases. FIG. 2 shows an example in which a bicycle is set as the specific object. In this case, text information "bike," which means a bicycle, is output as the class name I11, and a rectangle (bounding box) surrounding the bicycle is output as the detection position I12. Note that the specific object is not limited to one type, and multiple types may be set.

[0013] The detection network is used to detect specific objects in images captured by any imaging device. The imaging device and specific object are not particularly limited. For example, the detection network is expected to be used for detecting people in images captured by security cameras or detecting vehicles in footage captured by in-vehicle cameras.

[0014] 1, the learning device 1 is a computer having a processing circuit 101, a storage device 102, an input device 103, a communication device 104, and a display device 105. Data communication between the processing circuit 101, the storage device 102, the input device 103, the communication device 104, and the display device 105 is performed via a bus.

[0015] The processing circuit 101 includes a processor such as a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory). The processing circuit 101 includes a learning unit 110, an acquisition unit 120, and an output unit 130. The processing circuit 101 executes a learning program related to machine learning according to this embodiment to realize the functions of the above-described units 110 to 130. The learning program is stored in a non-transitory computer-readable recording medium such as a storage device 102. The learning program may be implemented as a single program that describes all the functions of the above-described units 110 to 130, or may be implemented as multiple modules divided into several functional units. Furthermore, the above-described units 110 to 130 may be implemented by integrated circuits such as application-specific integrated circuits (ASICs). In this case, the units may be implemented on a single integrated circuit or individually on multiple integrated circuits.

[0016] The learning unit 110 generates a detection network for detecting objects from images belonging to a target domain. Specifically, the learning unit 110 generates a detection network adapted to the target domain from a detection network trained in the source domain by domain adaptation based on images belonging to the target domain and images belonging to the source domain. The target domain refers to a domain in which no teaching labels exist in any of the images belonging to the target domain, or the number of teaching labels is small even if they exist. Teaching labels refer to labels used as teaching data in supervised learning, etc., and in the detection task of this embodiment, they refer to class names and detection positions. The source domain refers to a domain in which teaching labels exist in almost all images belonging to the domain. Images belonging to the target domain are referred to as target domain images, and images belonging to the source domain are referred to as source domain images.

[0017] FIG. 3 illustrates the process of training a detection network using domain adaptation. As shown in FIG. 3, first, the learning unit 110 generates a detection network N31 for the source domain through supervised learning based on a source domain image with a training label (S101). The detection network N31 is a detection network optimized for detecting a specific object from the source domain image. Next, the learning unit 110 generates a detection network N32 for the target domain through domain adaptation learning based on a target domain image without a training label and a source domain image with a training label (S102). The detection network N32 for the target domain is a detection network optimized for detecting a specific object from the target domain image. The first embodiment primarily aims to generate the detection network N32 for the target domain. Unless otherwise specified, the term "detection network" refers to the detection network for the target domain. If the target domain image does not have a training label, a pseudo-label may be attached to the target domain image.

[0018] As shown in FIG. 1, the learning unit 110 includes a detection task loss calculation unit 111, a domain discrimination loss calculation unit 112, a self-learning loss calculation unit 113, an overall loss calculation unit 114, an update unit 115, a pseudo label generation unit 116, and a learning control unit 117.

[0019] The detection task loss calculation unit 111 calculates the detection task loss by supervised learning of the detection network based on the target domain image. The detection task loss is a loss related to the detection position and class identification. If a training label exists in the target domain image, the training label is used as training data. That is, supervised learning is performed based on the training label and the target domain image. If no training label exists in the target domain image, a pseudo label is used as training data. That is, supervised learning is performed based on the pseudo label and the target domain image. Note that a pseudo label is different from a training label whose reliability is guaranteed by being assigned manually, and refers to a label whose reliability is not guaranteed because it is obtained by calculation by the pseudo label generation unit 116.

[0020] The domain discrimination loss calculation unit 112 calculates a domain discrimination loss by adversarial learning of a detection network based on the target domain image and the source domain image. The domain discrimination loss is a loss related to discrimination between the target domain and the source domain.

[0021] The self-learning loss calculation unit 113 calculates a self-learning loss by self-learning the detection network based on the target domain image. The self-learning loss is a loss related to class classification by self-learning.

[0022] As shown in FIG. 1 , the self-learning loss calculation unit 113 includes a first self-learning loss calculation unit 118 and a second self-learning loss calculation unit 119. The first self-learning loss calculation unit 118 calculates an integrated pseudo-label self-learning loss as the self-learning loss. The integrated pseudo-label self-learning loss is a loss related to class identification by self-learning using the integrated pseudo label. The integrated pseudo label is a type of pseudo label generated by the pseudo label generation unit 116. The second self-learning loss calculation unit 119 calculates a manifold synthesis self-learning loss as the self-learning loss. The manifold synthesis self-learning loss is a loss related to class identification by self-learning using the manifold synthesis pseudo label. The manifold synthesis pseudo label is a type of pseudo label generated by the pseudo label generation unit 116.

[0023] The overall loss calculation unit 114 calculates the overall loss based on a weighted sum of the domain identification loss and the detection task loss and / or the self-learning loss. Specifically, the overall loss may be calculated based on the domain identification loss and the detection task loss, the domain identification loss and the integrated pseudo-label self-learning loss, the domain identification loss and the manifold synthesis self-learning loss, the domain identification loss, the detection task loss and the integrated pseudo-label self-learning loss, the domain identification loss, the detection task loss and the manifold synthesis self-learning loss, the domain identification loss, the integrated pseudo-label self-learning loss and the manifold synthesis self-learning loss, or the domain identification loss, the detection task loss, the integrated pseudo-label self-learning loss and the manifold synthesis self-learning loss.

[0024] The update unit 115 updates the learning parameters of the detection network so as to minimize the overall loss. The learning parameters are updated according to a predetermined optimization method.

[0025] The pseudo label generation unit 116 generates a pseudo label for the target domain image based on the target domain image. As an example, the pseudo label generation unit 116 outputs a class name and / or a detection position as a pseudo label by forward propagating the target domain image through the detection network in the current iteration unit. The pseudo label generation unit 116 also generates an integrated pseudo label and a manifold synthesis pseudo label. The integrated pseudo label is a pseudo label obtained by integrating multiple pseudo labels based on multiple transformed images generated by applying multiple types of image transformation to the target domain image. The manifold synthesis pseudo label is a pseudo label based on a linear combination of intermediate features output from the feature extraction layer of the detection network.

[0026] The learning control unit 117 comprehensively controls domain adaptive learning according to the first embodiment. In domain adaptive learning, learning parameters are iteratively updated based on the overall loss. For each predetermined iteration unit (hereinafter referred to as a stage), the learning control unit 117 determines whether the current stage is a stage for performing self-learning. If it is determined that the current stage is a stage for performing self-learning, the learning control unit 117 instructs the self-learning loss calculation unit 113 to calculate the self-learning loss. If it is determined that the current iteration unit is not a stage for performing self-learning, the learning control unit 117 instructs the self-learning loss calculation unit 113 not to calculate the self-learning loss. At each stage, the learning control unit 117 determines whether a learning stop condition is met, and repeatedly updates the learning parameters using the update unit 115 until the learning stop condition is met.

[0027] The acquiring unit 120 acquires various data. For example, the acquiring unit 120 acquires training data for the detection network, i.e., a dataset of target domain images and a dataset of source domain images. The acquiring unit 120 may acquire the various data from an external device via the communication device 104 or from the storage device 102.

[0028] The output unit 130 outputs various data. For example, the output unit 130 outputs the detection network trained by the training unit 110 to the storage device 102, or to an external device via the communication device 104. The output unit 130 also displays the target domain image, class name, detection position, etc. on the display device 105.

[0029] The storage device 102 is configured by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), an integrated circuit storage device, etc. The storage device 102 stores various data such as learning data and learning programs.

[0030] The input device 103 inputs various commands from a user. The input device 103 may be a keyboard, a mouse, various switches, a touchpad, a touch panel display, or the like. An output signal from the input device 103 is supplied to the processing circuit 101. Note that the input device 103 may also be an input device of a computer connected to the processing circuit 101 via a wired or wireless connection.

[0031] Communication device 104 is an interface for performing data communication between learning device 1 and external devices connected to the learning device 1 via a network.

[0032] The display device 105 displays various data. As the display device 105, a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display known in the art can be appropriately used. The display device 105 may also be a projector.

[0033] The details of domain adaptation learning by the learning device 1 are described below. In this embodiment, data already used for an existing domain is available, and the objective is to improve object detection performance for data in a new domain. However, it is necessary to reduce the annotation cost for data in the new domain. Therefore, the problem of unsupervised domain adaptation (UDA) for cross-domain object detection is addressed. In the unsupervised domain adaptation scenario, complete annotations for the source domain dataset are available. In the unsupervised domain adaptation scenario, a detection network is trained with complete annotations for the source domain dataset and no annotations for the target domain dataset. In addition, a method for assigning reliable pseudo-labels to the target data is introduced to facilitate self-learning. While conventional UDA methods have achieved significant performance improvements for target datasets, the UDA method with self-learning aims to further reduce the domain gap and thereby improve the accuracy of the detection network.

[0034] Fig. 4 is a flowchart of domain adaptive learning by the learning device 1. The processing circuit 101 starts the domain adaptive learning shown in Fig. 4 by reading and executing a learning program from the storage device 102. Note that before the start of step S201, it is assumed that a data set of a target domain image and a data set of a source domain image, which are learning data, are prepared in the storage device 102 or an external database or the like.

[0035] As shown in Fig. 4, the learning control unit 117 inputs training data corresponding to one mini-batch into the detection network (S201). One mini-batch is composed of a predetermined number of target domain images arbitrarily selected from a dataset of target domain images prepared as training data. Note that although some of the target domain images prepared as training data may be assigned training labels, in this embodiment, it is assumed that no training labels are assigned to the target domain images.

[0036] The detection network 10 according to the first embodiment includes VGG (Reference 1) as a base network layer (backbone). <Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014)> ) and ResNet (Reference 2 <He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.> ) and a single-stage object detector, the Single Shot Multibox Detector (SSD) (Reference 3), is used to estimate the object detection position and class name. <Liu Wei, et al. "SSD: Single shot multibox detector." European conference on computer vision. Springer, Cham, 2016.> ) and CenterNet (Reference 4<Xingyi Zhou, Dequan Wang, Philipp Krahenbuhl. “Objects as Points.”, arXiv 1904.07850 (2019)> ) directly performs object classification and regression of the detected position for each pixel of the feature map. <Ren, Shaoqing, et al. "Faster r-cnn: Towards real-time object detection with region proposal networks." Advances in neural information processing systems. 2015.> ) may be used, which extracts object candidate regions, and then classifies the target object and regresses the detected position.

[0037] In the following, as an example, it is assumed that the detection network according to the first embodiment is based on SSD, which is a one-stage object detector designed to detect specific objects in real time.

[0038] Fig. 5 is a diagram showing an example of the network configuration of an SSD-type detection network N50 used in the domain adaptation learning shown in Fig. 4. The detection network N50 has a network configuration for inputting a target domain image and outputting the class name and detected position of a specific object from the target domain image.

[0039] As shown in FIG. 5, the detection network N50 includes a base network layer N51, an auxiliary network layer N52, an NMS module N53, a class identification module N54, and a detection position regression module N55. The base network layer N51 includes a feature extraction layer (convolutional layer) that generates a feature map representing the image features of an input image. As an example, a network layer based on VGG16 or the like is used as the base network layer N51. In the base network layer N51, the fully connected layer of VGG16 is replaced with multiple convolutional layers. The auxiliary network layer N52 has a chain structure of multiple feature extraction layers (convolutional layers) that extract feature maps at multiple scales. The auxiliary network layer N52 is capable of detecting objects of various sizes.

[0040] Each feature map has target points called anchors set at equal intervals at multiple scales. For each anchor, multiple reference rectangles (default boxes) with different sizes and aspect ratios are set for each class. In SSD, 8732 reference rectangles are set for each class. Class features and position features are extracted for each reference rectangle for each feature map.

[0041] The NMS module N53 performs non-maximum suppression (NMS) on the feature maps output from the base network layer N51 and the auxiliary network layer N52, suppressing rectangles with low confidence to prevent multiple rectangles from being detected for one anchor. The NMS module N53 outputs class features and position features for each rectangle. In this embodiment, the NMS module N53 is a network module included in the detection network N50, as shown in FIG. 5.

[0042] The class identification module N54 outputs a classification probability value for each class based on the output from the NMS module N53, and outputs a class name corresponding to a classification probability value higher than a threshold. In this embodiment, the class identification module N54 is a network module included in the detection network N50, as shown in FIG.

[0043] The detected position regression module N55 outputs the detected position for each class based on the output from the detection network N50. The detected position is defined by parameters such as the coordinates of the reference point of the rectangle, its vertical width, and its horizontal width. In this embodiment, the detected position regression module N55 is a network module included in the detection network N50, as shown in FIG.

[0044] As described above, the detection network N50 includes the base network layer N51, the auxiliary network layer N52, the NMS module N53, the class identification module N54, and the detection location regression module N55. However, this embodiment is not limited to this. However, the NMS module N53, the class identification module N54, and the detection position regression module N55 may be program modules independent of the detection network N50. That is, the minimum configuration of the detection network N50 is the base network layer N51 and the auxiliary network layer N52.

[0045] When step S201 is performed, the detection task loss calculation unit 111 calculates the detection task loss, and the domain discrimination loss calculation unit 112 calculates the domain discrimination loss (S202).

[0046] First, the calculation of the detection task loss by the detection task loss calculation unit 111 will be described. The detection task loss calculation unit 111 calculates the detection task loss for the detection position and class classification based on the target domain image with pseudo labels. The detection task loss L SSD is expressed by the following equation (1): A positive example means an image area in which the object to be detected is depicted, and a negative example means an image area in which the object to be detected is not depicted.

[0047]

number

[0048] The above (1) is defined using the probability value P(·) for each class and background for the set of positive examples Pos and the set of negative examples Neg. loss represents the loss due to localization error.

[0049] Next, we will explain the calculation of the domain discrimination loss by the domain discrimination loss calculation unit 112. The domain discrimination loss calculation unit 112 calculates the domain discrimination loss related to discrimination between the target domain and the source domain by adversarial learning of a detection network based on the target domain image and the source domain image.

[0050] Fig. 6 is a diagram showing an example of the network configuration of a detection network for domain discrimination loss. As shown in Fig. 6, for adversarial learning, a base network layer N51 of the detection network is connected to a discrimination network N61 via a gradient reversal layer (GRL).

[0051] As described above, the base network layer N51 is a network layer that extracts image features contained in the input image. The base network layer N51 generally includes a first local feature extraction layer N511, a second local feature extraction layer N512, and a global feature extraction layer N513. The first local feature extraction layer N511 corresponds, for example, to conv3_2 in the base network layer N51. The first local feature extraction layer N511 extracts a local feature F1, which is a local feature of the input image. The second local feature extraction layer N512 corresponds, for example, to conv3_3 in the base network layer N51. The second local feature extraction layer N512 extracts a local feature F2, which is a local feature of the input image. Although the local feature F2 is local, it represents a more global feature than the local feature F1. The global feature extraction layer N513 corresponds to conv4_3 in the base network layer N51, i.e., the first detection head. The global feature extraction layer N513 extracts a global feature F3, which is a global feature of the input image.

[0052] As shown in Figure 6, the classification network N61 has a convolutional layer N611, a convolutional layer N612, and a domain classification layer N613. The first local feature extraction layer N511 is connected to the convolutional layer N611 via a gradient inversion layer N621. The local feature F1 is supplied to the convolutional layer N611 via the gradient inversion layer N621. The convolutional layer N611 performs a convolution operation on the local feature F1 to generate a domain prediction map. The second local feature extraction layer N512 is connected to the convolutional layer N612 via a gradient inversion layer N622. The local feature F2 is supplied to the convolutional layer N612 via the gradient inversion layer N622. The convolutional layer N612 performs a convolution operation on the local feature F2 to generate a domain prediction map. The global feature extraction layer N513 is connected to the domain classification layer N613 via a gradient inversion layer N623. The global feature F3 is supplied to the domain identification layer N613 via the gradient inversion layer N623. The domain identification layer N613 calculates a domain identification value based on the global feature F3.

[0053] The domain identification loss calculation unit 112 inputs a target domain image or a source domain image to the detection network N51, outputs domain prediction maps from each of the convolution layers N611 and N612, outputs a domain identification value from the domain identification layer N613, and calculates a domain identification loss based on a first domain identification loss related to the domain prediction maps output from the convolution layers N611 and N612 and a second domain identification loss related to the domain identification value output from the domain identification layer N613.

[0054] Below, we explain the domain discrimination loss in more detail. In this model, for global-level feature alignment, we use a discriminative network N61 to align the image features of the target image to the source domain image. Target images that are easy to discriminate are far from the source domain image in feature space, while target images that are difficult to discriminate are close to the source domain image. Forcing a hard alignment on global image features may degrade the model's performance, as there may be larger shifts that affect the scene layout (e.g., the number of objects and their co-occurrence). Therefore, we apply a weak alignment to global features. To do this, the discriminative network N61 needs to focus less on easy-to-discriminate images and more on difficult-to-discriminate images. This is the approach described in Reference 6. <T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll´ar. Focal loss for dense object detection. In ICCV, 2017.> This is achieved by using the FL (Focal Loss) function shown in the following equation (2). Here, γ controls the weighting for images that are difficult to distinguish, and f(p t ) is the modulation factor of the cross-entropy loss.

[0055]

number

[0056] To align the domain distributions, the model must minimize the domain discrimination loss while optimizing the parameters of the base network layer N51 to maximize this loss. This can be achieved by using gradient inversion layers N621, N622, and N623, while normal gradient descent is applied to train the discrimination network N61. When passing through the gradient inversion layers N621, N622, and N623, the sign of the gradient is reversed to optimize the base network layer N51.

[0057] Weak alignment to global features is suitable for large domain shifts, while strong alignment to local features matches the texture and color of the domain and should improve performance. Dg is calculated using the loss function shown in the following equation (3). Dg is the output D of the domain identification layer N613 g is calculated for each set of source domain images S and target domain images T.

[0058]

number

[0059] As shown in Figure 6, the network configuration of the convolutional layers N611 and N612 is designed with a focus on local image features. The convolutional layers N611 and N612 are fully convolutional networks with a kernel size of 1. The convolutional layers N611 and N612 output a domain prediction map with the same width and height as the input feature. The loss L related to local domain discrimination is Dl The loss function is defined as the following equation (4). In equation (4), the loss L Dl outputs features with width W and height H. Convolutional layers N611 and N612 are trained based on the least squares error loss.

[0060]

number

[0061] Loss L for domain adaptation loss adv The loss function of (F, D) is the feature F1 and the domain prediction map D based on it, as shown in equation (4). l and loss L based on Dl (F1,D l ) and feature F2 and its based domain prediction map D l and loss L based on Dl (F2,D l ) and the feature F3 and the domain discrimination value D based on it, as shown in equation (3). g and loss L based on Dg (F3,D g ) is defined as in the following equation (5): adv (F,D) is the loss L Dl (F1,D l ), loss L Dl (F2,D l ) and loss L Dg (F3,D g ) may be defined as a weighted sum of

[0062]

number

[0063] To reduce the domain gap, image feature alignment is necessary not only at the image level (e.g., image scale, image style, lighting) but also at the local scale (e.g., object texture, shape, and color features). However, as described in Non-Patent Document 1, forcing invariance to global image features degrades performance. As described above, this embodiment applies weak global feature alignment to accommodate image-level domain shifts. Conversely, strong alignment of local features can help the model achieve better invariance to such features, potentially reducing the domain gap. This alignment of strong and weak features is achieved by employing discriminative networks at various stages of the detection network and training them adversarially.

[0064] After step S202, the learning control unit 117 determines whether the current stage is a stage in which self-learning is performed (S203). As an example, it is assumed that whether or not self-learning is performed is associated with each stage using a look-up table (LUT) or the like. Stages in which self-learning is performed and stages in which self-learning is not performed may be determined in advance experimentally or empirically. In this case, the learning control unit 117 may check the current stage against the LUT to determine whether or not the current stage is a stage in which self-learning is performed.

[0065] If it is determined that the current stage is a stage for self-learning (S203: YES), the self-learning loss calculation unit 113 calculates the self-learning loss (S204). As described above, the self-learning loss includes the integrated pseudo-label self-learning loss calculated by the first self-learning loss calculation unit 118 and the manifold synthesis self-learning loss calculated by the second self-learning loss calculation unit 119. In this embodiment, both the integrated pseudo-label self-learning loss and the manifold synthesis self-learning loss are calculated. Pseudo labels are used in self-learning. If the quality of the pseudo labels is high, a better detection network can be generated. Therefore, the pseudo label generation unit 116 generates integrated pseudo labels for calculating the integrated pseudo-label self-learning loss and generates manifold synthesis pseudo labels for calculating the manifold synthesis self-learning loss.

[0066] The following describes the calculation of the integrated pseudo label self-learning loss by the first self-learning loss calculation unit 118. To calculate the integrated pseudo label self-learning loss, first, an integrated pseudo label is generated by the pseudo label generation unit 116. The pseudo label generation unit 116 applies multiple types of image transformation to the target domain image to generate multiple transformed images, applies the multiple transformed images to a detection network to calculate multiple predicted labels, and generates an integrated pseudo label based on an ensemble of the multiple predicted labels.

[0067] FIG. 7 is a diagram showing the process of generating an integrated pseudo label. As shown in FIG. 7, the pseudo label generation unit 116 first performs multiple types of image transformation on the target domain image to generate multiple transformed images (S301). The image transformation is similar to that used in data augmentation. That is, the image transformation may be simple transformation such as sliding the image, inverting color values, enlarging or reducing, etc. In FIG. 7, it is assumed that N types of image transformation are performed. As a result, N transformed images TIn (n is the index of the transformed image, 2≦n≦N) are generated.

[0068] After step S301, the pseudo label generation unit 116 applies the same position detector N70 to each of the N transformed images TIn to generate a predicted label Boxn (2≦n≦N) (S302). The predicted label Boxn corresponds to the detected position output by the detection network, in other words, a bounding box. The position detector N70 may be the detection network in the current stage.

[0069] After step S302, the pseudo label generation unit 116 ensembles the N predicted labels Boxn to generate a single detected label (hereinafter referred to as an integrated detected label) (S303). The ensemble corresponds to calculating the average value of multiple detected labels obtained at the same position.

[0070] After step S303, the pseudo label generation unit 116 filters the integrated detection labels (S304). By filtering, unreliable integrated detection labels are removed, and highly reliable integrated detection labels are extracted. The extracted integrated detection labels are used as integrated pseudo labels.

[0071] 8 is a diagram showing a pseudo code of the filtering algorithm in step S304. As shown in FIG. * , ε, and δ are input. As shown in line 1, the set of rectangles O * From any region (rectangle) r l* As shown in line 2, the set O of all prediction results output from Boxn is read out and an arbitrary region (rectangle) r i As shown in line 3, the region r l * and region r i The IoU value with l * ,r i ) is calculated, and IoU(r l * ,r i ) is compared against a threshold δ. As shown in line 4, IoU(r l * ,r i ) is greater than or equal to the threshold δ, then O to r i is collected. i The processes in lines 2 to 6 are repeated while changing the i When the processes in lines 2 to 6 are repeated for , the SRRS is calculated according to the following equation (6), as shown in line 7. As shown in line 8, the SRRS is compared with a threshold ε. If the SRRS is equal to or greater than the threshold ε, the region r is added to the set Y of pseudo labels. l * If the SRRS is less than the threshold ε, the region r l * As shown in line 11, the processing in lines 1 to 11 is performed on set O * All regions r belonging to l * is performed for all regions r l * When executed for, set Y is output. Region r belonging to set Y l * means the integrated pseudo label.

[0072]

number

[0073] Figure 9 shows the predicted detection position for one image. Figure 10 shows the overlay of predicted detection positions for multiple images. Figure 11 shows the integrated pseudo label. As shown in Figure 9, all predicted labels are accumulated at the scale of the original image. As shown in Figure 10, if an object is detected in many transformed images, there should be multiple bounding boxes around the object. This increases the reliability of the detection. Therefore, the bounding boxes around the object are averaged to obtain a bounding box (integrated detection label). This process corresponds to ensembling. Once all bounding boxes are detected, the algorithm shown in Figure 8 is applied to reject less reliable bounding boxes, and the final bounding box (integrated pseudo label) is obtained. Figure 11 shows the final bounding box (integrated pseudo label). Applying the algorithm shown in Figure 8 ensures that the integrated pseudo label is reliable. The integrated pseudo label is useful as a reliable pseudo label in weak self-learning. Weak self-learning helps to learn categorical information of detected objects in order to learn better domain-invariant models in a semi-supervised learning environment.

[0074] Once the integrated pseudo labels are generated, the first self-learning loss calculation unit 118 calculates an integrated pseudo label self-learning loss based on the target domain images with the integrated pseudo labels.

[0075] Models trained on large-scale annotated datasets achieve the best results. On the other hand, in domain adaptation scenarios, the target dataset is unannotated. Annotations for the target dataset would have helped the model learn the underlying data distribution. Therefore, generating correct labels is essential to improving model performance. Labels generated for target data from a model trained only on source data often result in incorrect outputs even with high confidence scores due to large domain shifts. Once pseudo labels are generated, they are treated as positive examples during training. Negative examples (Neg) for training are generally obtained by hard negative mining. However, if false negatives are selected by hard negative mining, this can hinder training. In this embodiment, to ignore negative examples that may be foreground, we use ||Neg, which has the smallest confidence loss among negative examples. ~ Select || / 3 samples. This process is called weak negative mining, and Neg ~ We can obtain the integrated pseudo-label self-learning loss L wst The loss function L is wst In this paper, the localization loss is ignored compared to the loss function of the detection task loss (Eq. 1).

[0076]

number

[0077] After unsupervised domain adaptation learning stabilizes, weak self-learning is applied to generate pseudo-labels. Weak self-learning is applied only for a few stages because self-learning may have a negative impact on the model parameters if the pseudo-labels are incorrect.

[0078] Next, we will explain the calculation of the manifold synthesis self-learning loss by the second self-learning loss calculation unit 119. The manifold synthesis self-learning loss is used to improve the accuracy of self-learning using pseudo labels.

[0079] Reference 7 <H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond Empirical Risk Minimization," in arXiv:1710.09412, 2017> discloses a data augmentation method called mixup, which mixes two pairs of training samples to create new training samples. In contrast, Non-Patent Document 3 proposes a manifold mixup method that uses linear interpolation of features in the hidden layer and linear combination of corresponding labels for training. This method functions as a regularization method, providing robustness against adversarial samples and corrupted labels and improving the generalization ability of deep neural networks. Providing additional input signals through linear interpolation in the hidden layer and flattening class representations leads to the generation of smoother decision boundaries.

[0080] Figure 12 shows the manifold synthesis self-learning loss L mixup 12 is a diagram illustrating the calculation process of the pseudo label generation unit 116. j and the kth target domain image I k and get the target domain image I j and target domain image I k The pseudo label generator 116 generates the pseudo label for the target domain image I j Feature extraction process F θ Then, the feature F θ (I j ) and extract the target domain image I k Feature extraction process F θ Then, the feature F θ (I k ) is extracted (S401). θ is the target domain image I in the feature extraction layer included in the base network layer N51 or the auxiliary network layer N52 included in the detection network at the current stage. jand target domain image I k This may be performed by applying

[0081] When step S401 is performed, the pseudo label generation unit 116 generates the feature F θ (I j ) and feature F θ (I k ) to calculate a composite feature (S402). The composite feature is calculated according to the following formula (8). Formula (8) expresses that two vectors, a and b, are linearly combined using λ sampled from a beta distribution. In step S402, for example, F is applied to vector a. θ (I j ) and assign F to vector b θ (I k ) to obtain the composite feature MIX(F θ (I j ),F θ (I k )) is obtained.

[0082]

number

[0083] When step S402 is performed, the pseudo label generation unit 116 generates the combined feature MIX(F θ (I j ),F θ (I k )) and perform position detection processing to obtain the manifold synthesis pseudo label y m t The position detection process generates a composite feature MIX(F θ (I j ),F θ (I k )) can be performed by forward propagating

[0084] On the other hand, the pseudo label generation unit 116 generates a pseudo label y for the j-th image generated from the detection network in the previous stage. θ jand the pseudo-label y for the kth image θ k and the manifold synthetic pseudo-label y m t-1 (S404) The manifold synthesis pseudo label y m t-1 can be calculated according to the above formula (8). For example, if vector a is θ j Substitute y into vector b θ k Substituting MIX(y θ j ,y θ k ) to obtain the manifold synthesis pseudo-label y m t-1 is obtained.

[0085] After steps S403 and S404 are performed, the second self-learning loss calculation unit 119 calculates the manifold synthesis pseudo label y m t and the manifold composition pseudo-label y m t-1 and based on the manifold synthesis self-learning loss L mixup (S405) The manifold synthesis self-learning loss L mixup is calculated according to the following formula (9): x m is the mixed feature, y m represents the corresponding mixed class label. For the model parameters, we use intermediate features F1, F2, and F3. L mixup uses the sum of binary cross-entropy loss calculated for each set of positive and negative examples.

[0086]

number

[0087] When step S204 is performed or when it is determined in step S203 that the current stage is not a stage for performing self-learning (S203: NO), the total loss calculation unit 114 calculates the detection task loss L SSD and the domain discrimination loss Ladv and the self-learning loss are weighted and added together to calculate the total loss (S205). Specifically, the total loss calculation unit 114 first calculates the loss L det Calculate the loss L det is the detection task loss L when the current stage is a self-learning stage. SSD and the integrated pseudo-label self-learning loss L wst and the manifold synthesis self-learning loss L mixup If the current stage is not a self-learning stage, the loss L det is the detection task loss L SSD Matches.

[0088]

number

[0089] loss L det The total loss calculation unit 114 calculates the loss L according to the following equation (11): det and the domain discrimination loss L adv Based on the weighted sum of TOTAL The weighting coefficient λ can be set to any value.

[0090]

number

[0091] When step S205 is performed, the update unit 115 updates the total loss L TOTAL (S206). Specifically, the update unit 115 updates the learning parameters so as to optimize an objective function based on the overall loss, as defined in the following equation (12). More specifically, the loss L det While minimizing the domain discrimination loss L adv The learning parameters are updated to maximize the overall loss. This minimizes the overall loss. Any optimization method, such as stochastic gradient descent (SGD) or ADAM, can be used.

[0092]

number

[0093] After step S206 is performed, the learning control unit 117 determines whether a learning termination condition is met (S207). The learning termination condition is set, for example, to reach a predetermined total number of iterations. If it is determined that the learning termination condition is not met (S207: NO), the learning control unit 117 inputs a new mini-batch of training data into the detection network. Then, steps S202 to S207 are executed for the new mini-batch. In this way, steps S201 to S207 are repeated while changing the mini-batch until it is determined that the learning termination condition is met.

[0094] If it is determined that the learning termination condition is met (S207: YES), the learning control unit 117 terminates the domain adaptive learning.

[0095] 4 is an example of the flow of domain adaptive learning, and the present embodiment is not limited to this. Various modifications and application examples will be described below.

[0096] As described above, the detection task loss calculation unit 111 may calculate the detection task loss based on the target domain image with the teaching labels. In this case, the detection task loss can be calculated by using the teaching labels assigned to the target domain image instead of the pseudo labels.

[0097] The loss L in the self-learning stage according to the above embodiment det is the detection task loss L SSD and the integrated pseudo-label self-learning loss L wst and the manifold synthesis self-learning loss L mixup However, in domain adaptation that does not use source domain images, the loss L in the self-learning stage is calculated as shown in the following equation (13). det is the integrated pseudo-label self-learning loss Lwst Without using the detection task loss L SSD and the manifold synthesis self-learning loss L mixup It is preferable to calculate it by weighted addition of

[0098]

number

[0099] In addition, depending on whether or not source domain images are used and the network configuration of the detection network, the loss L in the self-learning stage can be adjusted. det can be changed in various ways. For example, the loss L det is the integrated pseudo-label self-learning loss L wst The detection task loss L may be calculated based only on SSD and the integrated pseudo-label self-learning loss L wst It may be a weighted addition of

[0100] In the above embodiment, the detection task loss calculation unit 111 calculates the detection task loss based only on the target domain image, but the detection task loss may also be calculated using not only the target domain image but also the source domain image with teaching labels. This makes it possible to effectively utilize the source domain image and improve the performance of the detection network.

[0101] As described above, the learning device 1 according to the first embodiment includes a learning unit 110 that generates a detection network for detecting objects from images belonging to a target domain. The learning unit 110 includes a detection task loss calculation unit 111, a domain classification loss calculation unit 112, a self-learning loss calculation unit 113, an overall loss calculation unit 114, and an update unit 115. The detection task loss calculation unit 111 calculates a detection task loss, which is a loss related to detection position and class classification, through supervised learning of the detection network based on the target domain image. The domain classification loss calculation unit 112 calculates a domain classification loss, which is a loss related to classification between the target domain and the source domain, through adversarial learning of the detection network based on the target domain image and the source domain image. The self-learning loss calculation unit 113 calculates a self-learning loss, which is a loss related to class classification through self-learning, through self-learning of the detection network based on the target domain image. The overall loss calculation unit 114 calculates an overall loss based on a weighted sum of the domain classification loss, the detection task loss, and / or the self-learning loss. The update unit 115 updates the learning parameters of the detection network so as to minimize the overall loss.

[0102] According to the configuration of the first embodiment, in domain adaptation training, the learning parameters of the detection network are updated to minimize the overall loss, which is based on a weighted sum of the domain discrimination loss, the detection task loss, and / or the self-training loss. By considering the domain discrimination loss, it is possible to bring the feature representations of the source domain and the target domain closer together through adversarial training. Furthermore, it is possible to consider the self-training loss in the self-training stage and consider the detection task loss without considering the self-training loss in the non-self-training stage. This makes it possible to improve the learning performance of domain adaptation training for a target domain with few or no training labels. Furthermore, by using the trained detection network generated by the domain adaptation training, it is possible to improve the performance of object detection. By considering the integrated pseudo-label self-training loss as the self-training loss, self-training is performed using reliable pseudo labels (integrated pseudo labels) generated by the ensemble method, thereby improving the accuracy of self-training. By considering the manifold synthesis self-training loss as the self-training loss, self-training is performed using reliable pseudo labels (manifold synthesis pseudo labels) generated by the manifold mixup method, thereby improving the accuracy of self-training.

[0103] (Second embodiment) The second embodiment relates to an object detection device, an object detection method, and an object detection program.

[0104] 13 is a diagram showing an example of the configuration of an object detection device 2 according to the second embodiment. The object detection device 2 is a computer that detects a specific object using a trained detection network generated by the learning device 1 according to the first embodiment. The object detection device 2 is used, for example, to detect people in images captured by security cameras or vehicles in images captured by in-vehicle cameras.

[0105] 13, the object detection device 2 is a computer having a processing circuit 201, a storage device 202, an input device 203, a communication device 204, and a display device 205. Data communication between the processing circuit 201, the storage device 202, the input device 203, the communication device 204, and the display device 205 is performed via a bus.

[0106] The processing circuit 201 includes a processor such as a CPU and a memory such as a RAM. The processing circuit 201 includes an acquisition unit 210, a processing unit 220, and an output unit 230. The processing circuit 201 executes an object detection program related to object detection according to this embodiment to realize the functions of the above-mentioned units 210 to 230. The object detection program is stored in a non-transitory computer-readable recording medium such as the storage device 202. The object detection program may be implemented as a single program that describes all the functions of the above-mentioned units 210 to 230, or may be implemented as multiple modules divided into several functional units. Furthermore, the above-mentioned units 210 to 230 may be implemented using integrated circuits such as ASICs. In this case, they may be implemented on a single integrated circuit or individually on multiple integrated circuits.

[0107] The acquisition unit 210 acquires various data. For example, the acquisition unit 210 acquires an image to be processed for object detection. The acquisition unit 210 may acquire the various data from an external device via the communication device 204 or from the storage device 202.

[0108] The processing unit 220 outputs the detected position and class name of an object included in the processing target image based on the processing target image and the trained detection network generated by the learning device 1 according to the first embodiment.

[0109] The output unit 230 outputs various data. For example, the output unit 230 displays on the display device 205 the detected position and class name of the object included in the processing target image output by the processing unit 220.

[0110] The storage device 202 is configured by a ROM, HDD, SSD, integrated circuit storage device, etc. The storage device 202 stores various data such as the detection network, the image to be processed, and the object detection program.

[0111] The input device 203 inputs various commands from the user. The input device 203 may be a keyboard, a mouse, various switches, a touchpad, a touch panel display, or the like. An output signal from the input device 203 is supplied to the processing circuit 201. Note that the input device 203 may also be an input device of a computer connected to the processing circuit 201 via a wired or wireless connection.

[0112] The communication device 204 is an interface for performing data communication with an external device connected to the object detection device 2 via a network.

[0113] The display device 205 displays various data. As the display device 205, a CRT display, a liquid crystal display, an organic EL display, an LED display, a plasma display, or any other display known in the art can be appropriately used. The display device 205 may also be a projector.

[0114] The object detection by the object detection device 2 will be described in detail below.

[0115] Fig. 14 is a diagram showing a flowchart of object detection by the object detection device 2. The processing circuit 201 starts the object detection shown in Fig. 14 by reading and executing an object detection program from the storage device 202. It is assumed that the detection network is stored in advance in the storage device 202 or the like. The detection network may be any object detector that is domain-adapted to the target domain, but is assumed to be a one-stage SSD as in the first embodiment.

[0116] 14, first, the acquisition unit 210 acquires a processing target image (S201). The processing target image is an image that belongs to a domain that has become a new adaptation destination through domain adaptation, that is, a target domain.

[0117] After step S201 is performed, the processing unit 220 shapes the image to be processed to the input size of the detection network (S202).

[0118] After step S202, the processing unit 220 applies the shaped processing target image to the detection network to predict class names and detection positions (S203). Specifically, the processing unit 220 first forward propagates the shaped processing target image through the base network layer N51 and auxiliary network layer N52 shown in FIG. 5 to generate a feature map for each class. Next, the processing unit 220 executes the NMS module N53 to perform non-maximum suppression on the feature map for each class, remove overlapping rectangles, and output class features and position features for each class for each rectangle. The processing unit 220 then executes the class identification module N54 to output a classification probability value for each class from the class features and output the class name of a class having a classification probability value exceeding a set threshold. The processing unit 220 also executes the detection position regression module N55 to output a detection position for each class from the position feature value and output the detection position of a class having a classification probability value exceeding a set threshold.

[0119] After step S203 is performed, the output unit 230 outputs the class name and the detection position (S204). As an example, the display device 205 displays the processing target image in which text representing the class name and a rectangle representing the detection position are superimposed.

[0120] According to the second embodiment, it is possible to perform object detection using the trained detection network generated by the first embodiment, thereby enabling high-performance object detection.

[0121] (Third embodiment) The third embodiment is a learning support system, a learning support method, and a learning support program. FIG. 15 is a diagram showing an example configuration of a learning support system 3 according to the third embodiment. The learning support system 3 is a computer network system that supports learning of a trained deep learning network. The deep learning network according to the third embodiment is not limited to the detection network according to the first and second embodiments, and may be a network that executes any task. However, for the sake of specificity in the following description, the deep learning network according to the third embodiment is assumed to be the detection network according to the first and second embodiments.

[0122] The learning support system 3 domain-adapts learning data using data taken at the camera installation location, for example, for person detection in images taken by a security camera or vehicle detection in images taken by an in-vehicle camera, and provides the user with information that allows them to visually see the correspondence between learning data and recognition performance when aiming to improve recognition performance using learning data taken at various locations.

[0123] 15, the learning support system 3 includes a learning device 1, an object detection device 2, an existing domain data storage device 4, a new domain data storage device 5, a history storage device 6, and an evaluation device 7. The learning device 1, the object detection device 2, the existing domain data storage device 4, the new domain data storage device 5, the history storage device 6, and the evaluation device 7 are connected to each other via a communication network such as a LAN so as to be able to communicate data with each other.

[0124] The existing domain data storage device 4 is a storage device that stores a data set of existing domain data with instruction labels. The existing domain data is image data that has instruction labels such as the positions and class names of objects that can be detected.

[0125] The new domain data storage device 5 is a storage device that stores a dataset of new domain data without teaching labels. The new domain data is image data that is newly collected in a different situation or place and does not have teaching labels such as object positions and class names.

[0126] The learning device 1 has the same functions as the learning device 1 according to the first embodiment. The learning device 1 trains multiple detection networks based on multiple learning situations having various learning data and learning condition parameters. More specifically, for existing domain data, the learning device 1 trains a detection network for the existing domain through supervised learning based on the existing domain data. For new domain data, the learning device 1 trains a detection network for the new domain through domain adaptive learning based on the new domain data and the existing domain data.

[0127] The object detection device 2 is equipped with multiple detection networks trained by the learning device 1. In response to a command from the evaluation device 7, the object detection device 2 applies evaluation data to each of the multiple detection networks to predict multiple class names and detection positions. The class names and detection positions predicted based on the evaluation data are called evaluation results. The evaluation data may be existing domain data, new domain data, or other image data.

[0128] The history storage device 6 is a storage device that stores history information. As the history information, the history storage device 6 stores a plurality of networks that execute a predetermined task and that have been trained in a plurality of learning situations in which learning data and / or learning parameters differ from one another, in association with the plurality of learning situations. Specifically, the history storage device 6 stores a plurality of detection networks trained by the learning device 1 and a plurality of pieces of learning situation information corresponding to the plurality of detection networks, in association with one another. Furthermore, the history storage device 6 may store a plurality of detection networks and a plurality of evaluation results corresponding to the plurality of detection networks, in association with one another, as history information.

[0129] The evaluation device 7 is a computer that uses the object detection device 2 to apply evaluation data to multiple detection networks to generate multiple evaluation results, and displays the multiple evaluation results and multiple learning situations side by side on a display device.

[0130] 15, the evaluation device 7 is a computer having a processing circuit 701, a storage device 702, an input device 703, a communication device 704, and a display device 705. Data communication between the processing circuit 701, the storage device 702, the input device 703, the communication device 704, and the display device 705 is performed via a bus.

[0131] The processing circuit 701 includes a processor such as a CPU and a memory such as a RAM. The processing circuit 701 includes an acquisition unit 710, an evaluation unit 720, a selection unit 730, and an output unit 740. The processing circuit 701 executes a learning assistance program for learning assistance according to this embodiment to realize the functions of the above-mentioned units 710 to 740. The learning assistance program is stored in a non-transitory computer-readable recording medium such as a storage device 702. The learning assistance program may be implemented as a single program that describes all the functions of the above-mentioned units 710 to 740, or may be implemented as multiple modules divided into several functional units. Furthermore, the above-mentioned units 710 to 740 may be implemented using integrated circuits such as ASICs. In this case, they may be implemented on a single integrated circuit or individually on multiple integrated circuits.

[0132] The acquiring unit 710 acquires various data. For example, the acquiring unit 710 acquires evaluation data and evaluation results from the object detection device 2.

[0133] The evaluation unit 720 uses the object detection device 2 to apply the evaluation data to a plurality of detection networks to generate a plurality of evaluation results.

[0134] The selector 730 selects a learning context useful for training the detection network from among the plurality of learning contexts based on the plurality of evaluation results.

[0135] The output unit 740 outputs various data. For example, the output unit 740 displays the multiple evaluation results output by the evaluation unit 720 on the display device 705. The output unit 740 also displays the learning status selected by the selection unit 730 on the display device 705.

[0136] The storage device 702 is configured by a ROM, HDD, SSD, integrated circuit storage device, etc. The storage device 702 stores various data such as the detection network and the learning support program.

[0137] The input device 703 inputs various commands from the user. The input device 703 may be a keyboard, a mouse, various switches, a touchpad, a touch panel display, or the like. An output signal from the input device 703 is supplied to the processing circuit 701. Note that the input device 703 may also be an input device of a computer connected to the processing circuit 701 via a wired or wireless connection.

[0138] The communication device 704 is an interface for communicating data with external devices such as the learning device 1, the object detection device 2, the existing domain data storage device 4, the new domain data storage device 5, and the history storage device 6, which are connected to the evaluation device 7 via a network.

[0139] The display device 705 displays various data. A CRT display, a liquid crystal display, an organic EL display, an LED display, a plasma display, or any other display known in the art can be used as the display device 705. The display device 705 may also be a projector.

[0140] The learning support provided by the learning support system 3 will be described in detail below.

[0141] Fig. 16 is a diagram showing a flowchart of learning support by the learning support system 3. As shown in Fig. 16, the learning device 1 trains a detection network based on existing domain data and learning conditions (S601).

[0142] After step S601, the learning device 1 trains a detection network based on new domain data (S602). In S602, the learning device 1 trains a detection network that has already been trained for an existing domain by domain adaptive learning, as in the first embodiment.

[0143] When step S602 is performed, the history storage device 6 stores the learning situation (existing domain data and learning conditions) used in step S601 and the detection network trained according to the learning situation, and stores the learning situation used in step S602 and the detection network trained according to the learning situation (new domain data and learning conditions) (S603).

[0144] After step S603 is performed, the evaluation unit 720 of the evaluation device 7 applies the evaluation data to the detection network to generate an evaluation result (S604). An image (hereinafter, test image) arbitrarily selected from new domain data or existing domain data is used as the evaluation data. Step S605 will be described in detail. First, the evaluation device 7 transmits a test image to the object detection device 2 along with a command to generate an evaluation result. The object detection device 2 applies the test image to each of the multiple trained detection networks and outputs the class name and detection position of a specific object contained in the test image as the evaluation result. The object detection device 2 transmits the test image to the evaluation device 7 and the history storage device 6. The history storage device 6 stores the evaluation result in association with each of the multiple detection networks.

[0145] When step S604 is performed, the output unit 740 of the evaluation device 7 displays the evaluation result and the learning situation on the display device 705 (S605).

[0146] FIG. 17 is a diagram showing an example of a display screen I13 for displaying evaluation results and learning status. As shown in FIG. 17, three test images I131, I132, and I133 are displayed on the display screen I13. Each test image I131, I132, and I133 displays a class name such as "A," "B," or "C" as an evaluation result, and a detection position represented by a detection rectangle, superimposed on the class name and detection position. The class name and detection position correspond to the evaluation result. Each evaluation result displays identification information for learning data such as "D1," "D2," or "D3," and learning condition parameters such as "P1," "P2," or "P3," side by side. The learning data and learning condition parameters correspond to learning conditions.

[0147] As shown in FIG. 17, the performance of a detection network can be evaluated by visually correlating the evaluation results with the learning status. For example, in FIG. 17, class "B" is an overdetection that does not actually exist in the test image. In this case, the performance of the detection network used for test image I131, which overdetects an object of class "B," is poor, while the performance of the detection networks used for test images I132 and I133, which do not overdetect, is good. Therefore, it can be seen that the training data D2 and training condition parameters P2 used to train the detection network applied to test image I132, or the training data D3 and training condition parameters P3 used to train the detection network applied to test image I133, are useful for training a detection network for the domain to which the test image belongs.

[0148] Note that the display example shown in FIG. 17 is an example and is not limited to this. Other display examples will be described below. Generally, there are multiple types of learning condition parameters. Therefore, it is preferable to display only learning condition parameters whose values ​​are different from each other. This makes it possible to easily know the learning condition parameters that contribute to differences in evaluation results.

[0149] After step S604, the evaluation device 7 determines whether or not to perform learning using other new domain data (S605). For example, it may determine whether or not to perform further learning in accordance with an instruction from the user via the input device 703. Alternatively, it may determine whether or not to perform further learning depending on whether or not learning has been completed for all of the new domain data to be learned that have been set in advance. If it is determined that learning using other new domain data is to be performed (S605: YES), steps S602 to S605 are repeated.

[0150] If it is determined that learning using other new domain data is not to be performed (S605: NO), the learning assistance provided by the learning assistance system 3 ends.

[0151] Next, a description will be given of the process of selecting a learning situation by the selection unit 730. The selection unit 730 selects a learning situation that is useful for training the detection network from among a plurality of learning situations based on a plurality of evaluation results. The process by the selection unit 730 will be described in detail below.

[0152] FIG. 18 is a diagram showing an example of a display screen I18 of nine evaluation results. FIG. 19 is a diagram showing an aggregated result 19 of the nine evaluation results shown in FIG. 18. FIG. 20 is a Venn diagram 20 showing the aggregated result of FIG. 19. The display screen I18 is displayed, for example, in S605 of FIG. 16. As in FIG. 17, the class name and detection position are superimposed on each test image as the evaluation result. The nine evaluation results are obtained by applying nine detection networks to the same test image.

[0153] As shown in Figures 18 and 19, class "A" was detected by seven detection networks, class "B" was detected by two detection networks, and class "C" was detected by nine detection networks. Objects of classes "A" and "C" were present in the test image, but objects of class "B" were not present in the test image, resulting in an overdetection. The oval for each class in Venn diagram 20 represents the set of learning situations used to train the detection network that detected that class. For example, the oval for class "A" represents the set of learning situations used to train the detection network that correctly detected class "A."

[0154] The selection unit 730 calculates the difference set between the intersection of the set of learning situations used to train the detection network that correctly detected the specific object and the set of learning situations used to train the detection network that overdetected the specific object. In the example of FIG. 20 , the selection unit 730 calculates the intersection of the set of learning situations used to train the detection network that correctly detected class "A" and the set of learning situations used to train the detection network that correctly detected class "C," and then calculates the difference set 21 by subtracting from the intersection the set of learning situations used to train the detection network that correctly detected class "C." The selection unit 730 then selects the learning situations belonging to the difference set 21 as learning situations useful for training the detection network. By training the detection network using the training data and learning condition parameters included in the selected learning situations, it is expected that a detection network with fewer overdetections will be generated.

[0155] At this time, information is required for each detected rectangle to indicate whether the object has been correctly detected. This can be determined if existing teaching data exists, but if no teaching data exists, manual teaching is required. For example, an input interface may be added to the display device 705, and teaching information may be provided by interactively clicking on the rectangle to determine which detected rectangle is the correct object.

[0156] Figure 21 shows the comparison results of the performance of domain adaptive learning. The PASCAL VOC dataset was used as the source domain, and the Clipart dataset was used as the target domain. mAP (mean average precision) was used as the performance index. mAP is the mean average precision (AP) value, which represents the area under the precision-recall curve for each class, averaged over all classes. mAP represents the performance of class identification for target domain images.

[0157] Figure 21 (A) shows the results of SSD trained only on the source domain. (B) shows the results of SSD trained only on the target domain with teaching labels. (C1), (C2), and (C3) are the results of the conventional methods described in Non-Patent Document 1 and Non-Patent Document 2. (P1), (P2), (P3), and (P4) are the results of the present method. (P1) shows the results when only the domain discrimination loss calculated by the domain discrimination loss calculation unit 112 is used. (P2) shows the results of the comparative method (C2) when the integrated pseudo-label self-learning loss calculated by the first self-learning loss calculation unit 118 is added. (P3) shows the results when the domain discrimination loss calculated by the domain discrimination loss calculation unit 112 is used, and (P4) shows the results when the manifold synthesis self-learning loss calculated by the second self-learning loss calculation unit 119 is also added. As shown in Figure 21, the performance of both the present method and the conventional method is improved. Note that the open circles in Figure 21 represent the implementation of the comparative method, and the closed circles represent the implementation of the present method. P2 and P3 in Figure 21 have the same combination of functions, but for Global and Local, P2 is an implementation of the comparison method, while P3 is an implementation of the method of the present application. Comparing P2 and P3, it can be seen that P3 has a better mAP than P2.

[0158] The above examples are merely illustrative, and the third embodiment is not limited thereto. For example, a detection network may be installed in the evaluation device 7. Alternatively, the object detection device 2 may be installed with an acquisition unit 710, an evaluation unit 720, a selection unit 730, and an output unit 740.

[0159] As described above, the deep learning network to be evaluated is not limited to a detection network, and the third embodiment can be applied to a deep learning network that performs any task, such as image generation, speech recognition, anomaly detection, etc. Accordingly, the content and data format of the new domain data and existing domain data may be changed.

[0160] As described above, the learning support system 3 according to the third embodiment includes a history storage device 6 and an evaluation device 7. The history storage device 6 stores a plurality of networks that execute a predetermined task and that have been trained in a plurality of learning situations with different learning data and / or learning conditions, in association with the plurality of learning situations. The evaluation device 7 applies evaluation data to the plurality of networks to generate a plurality of evaluation results, and displays the plurality of evaluation results and the plurality of learning situations side by side on a display device 705.

[0161] According to the above configuration, when there are multiple networks for the same task trained under different learning conditions, by applying the multiple networks to common evaluation data and comparing the multiple evaluation results and learning conditions obtained, it becomes possible to determine whether the learning conditions are appropriate or inappropriate for constructing a network for that task, thereby improving the performance of the network for that task and improving the accuracy of that task.

[0162] Thus, according to the above embodiment, it is possible to improve the performance of object detection using a detection network obtained by domain adaptation.

[0163] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0164] 1...Learning device, 2...Object detection device, 3...Learning support system, 4...Existing domain data storage device, 5...New domain data storage device, 6...History storage device, 7...Evaluation device, 101...Processing circuit, 102...Storage device, 103...Input device, 104...Communication device, 105...Display device, 110...Learning unit, 111...Detection task loss calculation unit, 112...Domain identification loss calculation unit, 113...Self-learning loss calculation unit, 114...Overall loss calculation unit, 115...Update unit, 116...Pseudo label generation unit, 117...learning control unit, 118...first self-learning loss calculation unit, 119...second self-learning loss calculation unit, 120...acquisition unit, 130...output unit, 201...processing circuit, 202...storage device, 203...input device, 204...communication device, 205...display device, 210...acquisition unit, 220...processing unit, 230...output unit, 701...processing circuit, 702...storage device, 703...input device, 704...communication device, 705...display device, 710...acquisition unit, 720...evaluation unit, 730...selection unit, 740...output unit.

Claims

1. A learning unit that generates a detection network for detecting objects from images belonging to a target domain, The learning unit a detection task loss calculation unit that calculates a detection task loss, which is a loss related to detection position and class identification, by supervised learning of the detection network based on images belonging to the target domain; a domain discrimination loss calculation unit that calculates a domain discrimination loss, which is a loss related to discrimination between the target domain and the source domain, by adversarial learning of the detection network based on an image belonging to the target domain and an image belonging to the source domain; a self-learning loss calculation unit that calculates a self-learning loss, which is a loss related to class identification by self-learning of the detection network based on images belonging to the target domain; an overall loss calculation unit that calculates an overall loss based on a weighted sum of the domain discrimination loss, the detection task loss, and / or the self-learning loss; an update unit that updates learning parameters of the detection network so as to minimize the overall loss; A learning device, comprising: the self-learning loss calculation unit includes a first self-learning loss calculation unit and / or a second self-learning loss calculation unit, the first self-learning loss calculation unit calculates, as the self-learning loss, a first self-learning loss related to class classification by self-learning using an integrated pseudo label obtained by integrating a plurality of pseudo labels generated based on a plurality of transformed images generated by applying a plurality of types of image transformation to an image belonging to the target domain; the second self-learning loss calculation unit calculates, as the self-learning loss, a second self-learning loss related to class classification by self-learning using a manifold synthesis pseudo label based on a linear combination of intermediate features output from a feature extraction layer of the detection network when the image belonging to the target domain is forward propagated through the detection network to obtain the pseudo label; the total loss calculation unit uses the first self-learning loss and / or the second self-learning loss as the self-learning loss; Learning device.

2. The learning unit further includes a control unit that determines whether a current repetition unit is a repetition unit for performing the self-learning; the self-learning loss calculation unit calculates the self-learning loss when it is determined that the current iteration unit is an iteration unit for performing the self-learning. The learning device according to claim 1 .

3. a pseudo label generation unit that performs a plurality of types of image transformation on an image belonging to the target domain to generate a plurality of transformed images, applies the plurality of transformed images to the detection network to calculate a plurality of detection labels, and generates the integrated pseudo label based on an ensemble of the plurality of detection labels; the first self-learning loss calculation unit calculates the first self-learning loss based on the images belonging to the target domain with the integrated pseudo-labels; The learning device according to claim 1 .

4. a pseudo label generation unit that, in each iteration unit, applies a first image and a second image belonging to the target domain to the detection network, respectively, to calculate a first intermediate output and a second intermediate output, and calculates the manifold synthesis pseudo label based on a linear combination of the first intermediate output and the second intermediate output; the second self-learning loss calculation unit calculates the second self-learning loss in the current iteration unit based on a manifold synthesis pseudo label in the current iteration unit and a manifold synthesis pseudo label in a previous iteration unit; The learning device according to claim 1 .

5. The total loss calculation unit If it is determined that the current iteration unit is not the iteration unit for performing the self-training, calculate the overall loss based on a weighted sum of the domain discrimination loss and the detection task loss; If it is determined that the current iteration unit is the iteration unit that performs the self-training, calculate the overall loss based on a weighted sum of the domain discrimination loss, the detection task loss, and / or the self-training loss. The learning device according to claim 2.

6. a classification network for identifying a domain to which an input image belongs is connected to the detection network via a gradient inversion layer that inverts the sign of the gradient; the domain discrimination loss calculation unit calculates the domain discrimination loss based on an output from the discrimination network; The learning device according to claim 1 .

7. the detection network includes a local feature extraction layer that outputs a feature vector representing a local image feature, and a global feature extraction layer that outputs a feature vector representing a global image feature; The discrimination network includes a convolution layer connected to the local feature extraction layer via a first gradient inversion layer and a domain discrimination layer connected to the global feature extraction layer via a second gradient inversion layer; the domain discrimination loss calculation unit calculates the domain discrimination loss based on a first domain discrimination loss related to the domain prediction map output from the convolution layer and a second domain discrimination loss related to the domain discrimination value output from the domain discrimination layer; The learning device according to claim 6.

8. The learning device according to claim 1 , wherein the detection task loss calculation unit calculates the detection task loss based on images belonging to the target domain with teaching labels or pseudo-labels.

9. The learning device according to claim 1 , wherein the detection network is a one-stage object detector.

10. The learning device according to claim 1 , wherein the detection network is a deep learning network that has been trained based on images belonging to the source domain.

11. an acquisition unit that acquires a processing target image that belongs to a target domain; a processing unit that outputs a detected position and a class name of an object included in the processing target image based on the processing target image and the detection network according to any one of claims 1 to 10; An object detection device comprising:

12. a history storage device that stores a plurality of networks that execute a predetermined task and that have been trained in a plurality of learning situations in which learning data and / or learning conditions are different from one another, in association with the plurality of learning situations, wherein each of the plurality of networks is a detection network that detects an object from an image, and the detection network is trained by the learning device according to any one of claims 1 to 10; an evaluation device that applies evaluation data to the plurality of networks to generate a plurality of evaluation results, and displays the plurality of evaluation results and the plurality of learning situations side by side on a display device, wherein each of the plurality of evaluation results is an image in which a rectangle representing the detected position of the object is drawn; A learning support system equipped with the above.

13. 13. The learning support system according to claim 12, wherein the evaluation device selects, from the plurality of learning situations, a learning situation that is useful for training the network that executes the predetermined task, based on the plurality of evaluation results.

14. the plurality of networks are a plurality of detection networks that detect objects from an image, The evaluation device displaying, as the plurality of evaluation results, a plurality of evaluation images in which rectangles representing the detected positions of the objects are drawn; Identifying an evaluation image that correctly detects the object from among the plurality of evaluation images; Identifying a detection network used to generate the identified evaluation image from the plurality of detection networks; selecting a learning situation associated with the identified detection network from the plurality of learning situations as the useful learning situation; The learning support system according to claim 13.

15. a learning data storage device for storing a plurality of pieces of learning data among the plurality of learning situations; a learning device that learns the plurality of networks based on the plurality of learning situations, the history storage device stores the plurality of learning situations as a result of the plurality of networks by the learning device; The learning support system according to claim 12.

16. The computer has a learning process for generating a detection network for detecting objects from images belonging to a target domain, The learning step includes: Calculate a detection task loss, which is a loss related to detection location and class identification, by supervised learning of the detection network based on images belonging to the target domain; Calculating a domain discrimination loss, which is a loss related to discrimination between the target domain and the source domain, by adversarial learning of the detection network based on the image belonging to the target domain and the image belonging to the source domain; Calculating a self-training loss, which is a loss related to class identification by self-training the detection network based on images belonging to the target domain; calculating an overall loss based on a weighted sum of the domain discrimination loss and the detection task loss and / or the self-training loss; updating learning parameters of the detection network to minimize the overall loss. A learning method comprising: the step of calculating the self-learning loss includes a first self-learning loss calculation step and / or a second self-learning loss calculation step; The first self-learning loss calculation step calculates, as the self-learning loss, a first self-learning loss related to class classification by self-learning using an integrated pseudo label obtained by integrating a plurality of pseudo labels generated based on a plurality of transformed images generated by applying a plurality of types of image transformation to an image belonging to the target domain; the second self-learning loss calculation step calculates, as the self-learning loss, a second self-learning loss related to class classification by self-learning using a manifold synthesis pseudo label based on a linear combination of intermediate features output from a feature extraction layer of the detection network when the image belonging to the target domain is forward propagated through the detection network to obtain the pseudo label; the step of calculating the overall loss uses the first self-learning loss and / or the second self-learning loss as the self-learning loss; How to learn.

17. Obtain an image to be processed that belongs to the target domain; outputting a detected position and a class name of an object included in the processing target image based on the processing target image and the detection network according to any one of claims 1 to 10; An object detection method comprising:

18. storing a plurality of networks that perform a predetermined task and that have been trained in a plurality of learning situations, each having different learning data and / or learning conditions, in a storage device in association with the plurality of learning situations; applying the evaluation data to the plurality of networks to generate a plurality of evaluation results, and displaying the plurality of evaluation results and the plurality of learning situations side by side on a display device; A learning support method comprising: Each of the plurality of networks is a detection network for detecting an object from an image, the detection network being trained by the learning device according to any one of claims 1 to 10; A learning support method, wherein each of the plurality of evaluation results is an image in which a rectangle representing the detected position of the object is drawn.

19. On the computer, A learning function is realized to generate a detection network for detecting objects from images belonging to a target domain, The learning function is Calculate a detection task loss, which is a loss related to detection location and class identification, by supervised learning of the detection network based on images belonging to the target domain; Calculating a domain discrimination loss, which is a loss related to discrimination between the target domain and the source domain, by adversarial learning of the detection network based on the image belonging to the target domain and the image belonging to the source domain; Calculating a self-training loss, which is a loss related to class identification by self-training the detection network based on images belonging to the target domain; calculating an overall loss based on a weighted sum of the domain discrimination loss and the detection task loss and / or the self-training loss; updating learning parameters of the detection network to minimize the overall loss. A learning program comprising: the function for calculating the self-learning loss includes a first self-learning loss calculation function and / or a second self-learning loss calculation function; The first self-learning loss calculation function calculates, as the self-learning loss, a first self-learning loss related to class classification by self-learning using an integrated pseudo label obtained by integrating a plurality of pseudo labels generated based on a plurality of transformed images generated by applying a plurality of types of image transformation to an image belonging to the target domain; the second self-learning loss calculation function calculates, as the self-learning loss, a second self-learning loss related to class classification by self-learning using a manifold synthesis pseudo label based on a linear combination of intermediate features output from a feature extraction layer of the detection network when the image belonging to the target domain is forward propagated through the detection network to obtain the pseudo label; the function for calculating the overall loss uses the first self-learning loss and / or the second self-learning loss as the self-learning loss; Learning program.

20. On the computer, A function for acquiring an image to be processed that belongs to a target domain; a function of outputting a detected position and a class name of an object included in the processing target image based on the processing target image and the detection network according to any one of claims 1 to 10; An object detection program that achieves this.

21. On the computer, a function of storing in a storage device a plurality of networks that are trained in a plurality of learning situations, each of which has different learning data and / or learning conditions, and that execute a predetermined task, in association with the plurality of learning situations; a function of applying the evaluation data to the plurality of networks to generate a plurality of evaluation results, and displaying the plurality of evaluation results and the plurality of learning situations side by side on a display device; A learning support program that realizes the following: Each of the plurality of networks is a detection network for detecting an object from an image, the detection network being trained by the learning device according to any one of claims 1 to 10; a learning assistance program, wherein each of the plurality of evaluation results is an image in which a rectangle representing the detected position of the object is drawn.

Images