Learning device, detection device, learning system, learning method, learning program, detection method, and detection program

Abstract

To provide a learning apparatus, a learning method, a learning system, and a learning program for improving object detection accuracy by learning using a smaller amount of learning data.SOLUTION: A learning apparatus 10 is provided with a first learning unit 20. The first learning unit 20 has a first supervised learning unit 22 and a first self-supervised learning unit 24. The first supervised learning unit 22 uses learning data 40 to learn the first object detection network 30 so that a first loss between an output of a first object detection network 30 for detecting an object from target image data and teacher data 40B can be reduced. The first self-supervised learning unit 24 uses image data 40A and self-supervised data generated from the image data 40A to learn the first object detection network 30 so that a second loss of a characteristic amount of the image data 40A and a candidate area P corresponding to the self-supervised data derived by the first object detection network 30 can be reduced.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a learning device, a detection device, a learning system, a learning method, a learning program, a detection method, and a detection program. [Background technology]

[0002] In recent years, object detection methods using CNN (Convolutional Neural Network) have made great advances in detection accuracy. However, to achieve good performance, a wealth of data on which the learning subject is trained is required. Therefore, learning using a small amount of data has been investigated. For example, a method has been disclosed for efficiently learning new classes using a small amount of data by utilizing knowledge learned using a wealth of data (see, for example, Patent Document 1, Non-Patent Document 1, and Non-Patent Document 2, etc.).

[0003] Patent Document 1 discloses multitask learning that estimates the conversion contents for each image conversion type by self-supervised learning without using teacher data. However, the technology of Patent Document 1 is applied only to classification tasks, and there is no verification as to whether it is a useful task for object detection. Non-Patent Document 1 discloses a technology that quickly adapts to a new class by conditioning by multiplication of a feature vector for each class extracted from a small data set and a feature obtained from an object detection network. Non-Patent Document 2 shows that fine tuning of only classification and regression at the latter stage of the detection network is effective in learning with a small amount of data. However, in the technologies of Non-Patent Document 1 and Non-Patent Document 2, the teacher data learned in advance contains almost no information on the new class. For this reason, the technologies of Non-Patent Document 1 and Non-Patent Document 2 lack the expressive ability to detect new classes. In other words, in the conventional technology, it was difficult to improve the object detection accuracy by learning using a smaller amount of learning data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] WO2021 / 059388 publication [Non-patent literature]

[0005] [Non-Patent Document 1] Xiaopeng Yan and 7 others, "Meta R-CNN: Towards General Solver for Instance-level Low-shot Learning", [online], ICCV2019, Internet (URL: https: / / arxiv.org / pdf / 1909.13032.pdf) [Non-Patent Document 2] Xin Wang and 4 others, "Frustratingly Simple Few-Shot Object Detection", [online], ICML2020, Internet (URL: https: / / arxiv.org / pdf / 2003.06957.pdf) Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the above, and aims to provide a learning device, a detection device, a learning system, a learning method, a learning program, a detection method, and a detection program that can improve object detection accuracy by learning using a smaller amount of training data. [Means for solving the problem]

[0007] A learning device according to an embodiment includes a first learning unit. The first learning unit includes a first supervised learning unit, a first self-supervised learning unit, and a first learning unit. The first supervised learning unit uses learning data including image data and supervised data including a class representing a correct object detection result of an object area included in the image data and position information of the object area in the image data to train the object detection network so as to reduce a first loss between an output of a first object detection network for detecting an object from target image data and the supervised data. The first self-supervised learning unit uses the image data and self-supervised data generated from the image data to train the first object detection network so as to reduce a second loss of a feature amount of a candidate area corresponding to the image data and the self-supervised data, which is derived by the first object detection network. The first loss is a loss of a class included in a detection result output from the first object detection network by inputting the image data to the first object detection network, relative to a class representing the correct object detection result included in the teacher data corresponding to the image data, and the second loss is a loss of a feature of the corresponding candidate region in the self-teacher data relative to the feature of the candidate region in the image data, which is derived by the first object detection network by inputting the image data and the self-teacher data to the first object detection network. [Brief description of the drawings]

[0008] [Figure 1] FIG. [Figure 2A] FIG. 1 is an explanatory diagram of identifying a candidate region. [Figure 2B] Schematic diagram of self-supervised data. [Diagram 3] 1 is a flowchart showing the flow of information processing. [Figure 4] FIG. [Diagram 5] 1 is a flowchart showing the flow of information processing. [Figure 6] Schematic diagram of a detection device. [Figure 7] 1 is a flowchart showing the flow of information processing. [Figure 8] Schematic diagram of the learning system. [Figure 9] FIG. [Figure 10] 1 is a flowchart showing the flow of information processing. [Figure 11] Hardware configuration diagram. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The learning device, the detection device, the learning system, the learning method, the learning program, the detection method, and the detection program will be described in detail below with reference to the accompanying drawings.

[0010] (First embodiment) FIG. 1 is a block diagram showing an example of the configuration of a learning device 10 according to this embodiment.

[0011] The learning device 10 is an information processing device that learns an object detection network for detecting objects included in image data.

[0012] The learning device 10 of this embodiment is preferably applied to learning an object detection network used, for example, for detecting people in footage captured by a security camera and for detecting vehicles in footage captured by an in-vehicle camera.

[0013] The learning device 10 of this embodiment includes a first learning unit 20. The first learning unit 20 learns a first object detection network 30. The first object detection network 30 is an example of an object detection network.

[0014] The first object detection network 30 is a neural network for detecting an object included in the target image data of the object detection target. For example, the first object detection network 30 is a neural network that receives image data as input and outputs a class representing an object detection result of an object area included in the image data and position information of the object area.

[0015] First object detection network 30 may be any neural network for performing object detection, and the detection method is not limited.

[0016] For example, the first object detection network 30 may use a CNN (Convolution Neural Network) such as VGG (Non-Patent Document 3) or ResNet (Non-Patent Document 4) as a backbone. The first object detection network 30 may use a method of directly classifying the target object and regressing the area for each pixel of a feature map to estimate the position information of the object area and the class identification of the object area. Examples of this method include a one-stage detector, such as Single Shot Multibox Detector (SSD) (Non-Patent Document 5) or Fully Convolutional One-Stage Object Detection (FCOS) (Non-Patent Document 6).

[0017] A two-stage detector that extracts object candidate regions and then performs class classification and regression of the object region may be used for the first object detection network 30. An example of a two-stage detector is Faster R-CNN (Non-Patent Document 5).

[0018] A detection method based on correlation with a feature vector for each class may be used in the first object detection network 30. Examples of such detection methods include Meta R-CNN (Non-Patent Document 1).

[0019] Non-Patent Document 3: Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014). ·Non-patent document 4: He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. ·Non-patent document 5: Liu Wei, et al. "SSD: Single shot multibox detector." European conference on computer vision. Springer, Cham, 2016. Non-Patent Document 6: Zhi Tian, ​​et al. "Fcos: Fully convolutional one-stage object detection." Proceedings of the IEEE / CVF international conference on computer vision. 2019. ·Non-patent document 7: Ren, Shaoqing, et al. "Faster r-cnn: Towards real-time object detection with region proposal networks." Advances in neural information processing systems. 2015.

[0020] The first learning unit 20 uses learning data 40 to learn the first object detection network 30 .

[0021] The learning data 40 includes image data 40A and teacher data 40B.

[0022] Image data 40A is image data used for training first object detection network 30. Image data 40A is image data to which teaching data 40B has not been added.

[0023] The teacher data 40B is data that directly or indirectly represents correct data that should be output from the first object detection network 30 when the image data 40A is input to the first object detection network 30 during learning. In this embodiment, the teacher data 40B includes a class that represents a correct object detection result of an object region included in the image data 40A, and position information of the object region in the image data 40A. The object region is represented, for example, as a rectangular region on the image of the image data 40A. The position information of the object region is represented, for example, as information that represents the position of the object region on the image of the image data 40A.

[0024] The first learning unit 20 has a first supervised learning unit 22, a first self-supervised learning unit 24, and an update unit 26. The first supervised learning unit 22 has an input unit 22A and a first loss calculation unit 22B. The first self-supervised learning unit 24 has a first self-supervised data generation unit 24A and a first self-supervised learning loss calculation unit 24B.

[0025] The first supervised learning unit 22, the first self-supervised learning unit 24, the update unit 26, the input unit 22A, the first loss calculation unit 22B, the first self-supervised data generation unit 24A, and the first self-supervised learning loss calculation unit 24B are realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC, that is, by hardware. Each of the above units may be realized by using both software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0026] The learning data 40 and the first object detection network 30 may be stored in a storage unit provided outside the learning device 10. In addition, the storage unit and at least one of the multiple functional units included in the first learning unit 20 may be mounted on an external information processing device communicatively connected to the learning device 10 via a network or the like.

[0027] The first supervised learning unit 22 trains the first object detection network 30 using the training data 40. That is, the first supervised learning unit 22 trains the first object detection network 30 using supervised data, which is image data 40A to which supervised data 40B has been added.

[0028] The first supervised learning unit 22 learns the first object detection network 30 using the learning data 40 so as to reduce a first loss between the output of the first object detection network 30 and the teacher data 40B. The first supervised learning unit 22 has an input unit 22A and a first loss calculation unit 22B.

[0029] The input unit 22A acquires training data 40 of an arbitrary mini-batch size from a training data set 41 including a plurality of training data 40, and inputs image data 40A included in the training data 40 to the first object detection network 30.

[0030] First loss calculation unit 22B acquires a detection result including a class of an object region and positional information of the object region, which is output from first object detection network 30 when image data 40A is input to first object detection network 30 by input unit 22A. First loss calculation unit 22B calculates the loss of the acquired detection result with respect to teacher data 40B corresponding to image data 40A, as a first loss.

[0031] For example, assume that first object detection network 30 is a one-stage detector using SSD described in the above Non-Patent Document 5. In this case, for example, first loss calculation unit 22B calculates the loss for class classification of the detection target and the loss function for position identification using the following formula (1).

[0032]

number

[0033] In formula (1), L conf represents the loss for classification, and L loc represents the loss for position estimation. In addition, in formula (1), x is a constant representing whether the object region corresponds to the correct object region, which is a correct rectangle. x indicates 1 when the object region F corresponds to the correct object region, and indicates 0 when the object region does not correspond to the correct object region. c represents the class confidence. l represents the predicted rectangle. g represents the correct rectangle. The rectangle means the object region, which is a rectangular region. α represents a coefficient for adjusting the weight of the loss.

[0034] In this case, first loss calculation unit 22B may calculate the first loss by calculating the loss and the loss function in the same manner as in Non-Patent Document 5. Note that first loss calculation unit 22B may use a loss function according to the object detection method of first object detection network 30, and is not limited to the method using the above formula (1).

[0035] The first loss calculated by the first loss calculation unit 22B is output to the update unit .

[0036] The update unit 26 updates the parameters of the first object detection network 30 so as to reduce the first loss (described in detail later). Note that the process of updating the parameters of the first object detection network 30 in accordance with the first loss may be executed by the first supervised learning unit 22. That is, each of the first supervised learning unit 22 and a first self-supervised learning unit 24 described later may include the update unit 26.

[0037] Furthermore, the update unit 26 may update the parameters of the first object detection network 30 using the first loss calculated by the first supervised learning unit 22 and a second loss described later after the second loss is calculated by the first self-supervised learning unit 24 described later. In this embodiment, a form will be described as an example in which the update unit 26 updates the parameters of the first object detection network 30 using the first loss calculated by the first supervised learning unit 22 and a second loss described later after the second loss is calculated by the first self-supervised learning unit 24 described later.

[0038] The first self-supervised learning unit 24 uses image data 40A and self-supervised data generated from the image data 40A to train the first object detection network 30 so as to reduce a second loss, which is the difference between the features of corresponding candidate regions between the image data 40A and the self-supervised data derived by the first object detection network 30.

[0039] The first self-supervised learning unit 24 has a first self-supervised data generation unit 24A and a first self-supervised learning loss calculation unit 24B.

[0040] The first self-teacher data generation unit 24A generates self-teacher data, which is image data after image conversion of the image data 40A. The first self-teacher data generation unit 24A also identifies one or more pairs of candidate regions corresponding to each other between the image data 40A and the self-teacher data, from each of the image data 40A and the self-teacher data.

[0041] 2A is an explanatory diagram of an example of specifying a candidate region P. The first self-teacher data generation unit 24A specifies one or more rectangular regions from the image data 40A as candidate regions P for feature extraction.

[0042] The first self-teacher data generation unit 24A identifies, as a candidate region P, a region that is randomly identified in the image data 40A or a region that is identified by a foreground extraction method that extracts a region that is likely to be an object.

[0043] When identifying the candidate region P by the foreground extraction method, the first self-supervised data generation unit 24A may identify an object-like region as the candidate region P by using, for example, Selective Search or the like as described in Non-Patent Document 8.

[0044] Non-Patent Document 8: JRRUijlings, et al. "Selective search for object recognition." International journal of computer vision 104.2 (2013): 154-171.

[0045] FIG. 2A illustrates, as an example, a scene in which the first self-teacher data generation unit 24A identifies candidate region Pa′ and candidate region Pb′ as candidate region P.

[0046] The first self-teacher data generation unit 24A identifies a candidate region P from the image data 40A, and thus a candidate region P including at least a portion of a non-overlapping region with the object region F is identified.

[0047] Here, for example, it is assumed that the teacher data 40B corresponding to the image data 40A specifies the class Ca of the object region Fa included in the image data 40A and the position information of the object region Fa. The object region Fa is an example of the object region F included in the image data 40A. The class Ca is an example of the class C of the object region F. That is, it is assumed that the image data 40A includes the object region Fa as the object region F that has been taught the class C.

[0048] The first self-supervised data generation unit 24A identifies, from the image data 40A, a randomly identified area or an object-like area as a candidate area P, so that the identified candidate area P includes areas other than the object area F included in the image data 40A. That is, the first self-supervised data generation unit 24A identifies, as a candidate area P, an area included in the image data 40A, including a background area that is not taught class C by the supervised data 40B.

[0049] The first self-supervisory data generation unit 24A may specify, as the candidate region P, a region that is at least partially non-overlapping with the object region F specified by the teacher data 40B, among the multiple candidate regions P specified from the image data 40A by the above method. Furthermore, the first self-supervisory data generation unit 24A may select a predetermined number of candidate regions P randomly or in order of object resemblance, among the multiple candidate regions P specified from the image data 40A by the above method, and specify the selected regions as the candidate regions P.

[0050] The first self-teacher data generation unit 24A performs a generation process of identifying the candidate region P and generating self-teacher data from the image data 40A.

[0051] The self-teaching data is image data obtained by converting the image data 40A.

[0052] FIG. 2B is a schematic diagram of an example of self-supervised data 40C generated from image data 40A.

[0053] The first self-teaching data generating unit 24A performs at least one image transformation, including luminance transformation, color tone transformation, contrast transformation, inversion, rotation, and cropping, on the image data 40A to generate one or more self-teaching data 40C from one image data 40A. FIG. 2B shows an example of the self-teaching data 40C generated by inverting the image data 40A.

[0054] The first self-teacher data generation unit 24A identifies, for the generated self-teacher data 40C, candidate regions P corresponding to each of one or more candidate regions P in the image data 40A before image conversion of the self-teacher data 40C from which the self-teacher data 40C was generated.

[0055] The candidate region P in the image data 40A and the candidate region P corresponding to the candidate region P in the self-teaching data 40C are the same region. In other words, the candidate region P in the image data 40A and the corresponding candidate region P in the self-teaching data 40C are the same region before and after image conversion.

[0056] FIG. 2B shows a state in which the first self-supervisory data generation unit 24A has identified, from the self-supervisory data 40C, a candidate region Pa corresponding to the candidate region Pa′ in the image data 40A and a candidate region Pb corresponding to the candidate region Pb′ in the image data 40A.

[0057] For example, the first self-teacher data generation unit 24A identifies an area in the self-teacher data 40C that is the same position and range as the candidate area P identified in the image data 40A, which is the image data before the image conversion of the self-teacher data 40C, as the corresponding candidate area P of the self-teacher data 40C. Note that the first self-teacher data generation unit 24A may generate the self-teacher data 40C by performing image conversion including coordinate conversion that affects the coordinate position, such as inversion, rotation, and cropping. In this case, the first self-teacher data generation unit 24A may identify the candidate area P, which is the same corresponding area in the self-teacher data 40C, by performing the same coordinate conversion on the identified candidate area P in the image data 40A.

[0058] Through these processes, the first self-teacher data generation unit 24A generates self-teacher data 40C, which is image data after image conversion of the image data 40A. In addition, the first self-teacher data generation unit 24A identifies one or more pairs of candidate regions P, which are the same region corresponding to each other between the image data 40A and the self-teacher data 40C, from each of the image data 40A and the self-teacher data 40C.

[0059] Returning to Figure 1, we will continue with the explanation.

[0060] The first self-supervisory data generation unit 24A inputs, to the first object detection network 30, the image data 40A and the self-supervisory data 40C generated from the image data 40A.

[0061] The first self-supervised learning loss calculation unit 24B calculates a second loss for the feature quantity of the corresponding candidate region P in the self-supervised data 40C, which is derived by the first object detection network 30 based on the input of the image data 40A and the self-supervised data 40C.

[0062] The feature amount is output as an array from an intermediate layer or a final layer of the first object detection network 30 by processing each of the image data 40A and the self-supervised data 40C input to the first object detection network 30 according to parameters in the first object detection network 30. The feature amount is represented, for example, as a vector of a group of feature values, that is, a feature vector.

[0063] For example, the first self-supervisory data generation unit 24A inputs image data 40A, self-supervisory data 40C generated from the image data 40A, and information representing pairs of corresponding candidate regions P of the image data 40A and the self-supervisory data 40C to the first object detection network 30.

[0064] Then, the first self-supervised learning loss calculation unit 24B extracts feature amounts of a candidate region P, which is an identical region between the image data 40A and the self-supervised data 40C generated from the image data 40A. For example, the first self-supervised learning loss calculation unit 24B may extract feature amounts of the corresponding candidate region P from each of the image data 40A and the self-supervised data 40C using ROIAlign shown in Non-Patent Document 9 for the feature map of the intermediate layer of the first object detection network 30.

[0065] Non-Patent Document 9: Kaiming He, et al. "Mask r-cnn." Proceedings of the IEEE international conference on computer vision. 2017.

[0066] Then, the first self-supervised learning loss calculation unit 24B calculates a second loss, which is a loss function of the corresponding candidate region P, which is the same region, between the image data 40A and the self-supervised data 40C generated from the image data 40A, by the following formula (2). In addition, as this loss function, Mean Squared Error (MSE) or InfoNCE (see Non-Patent Document 10), etc. may be used.

[0067]

number

[0068] In formula (2), p i represents the feature vector of the candidate region P in the image data 40A, and P j represents the feature vector of the candidate region P corresponding to the candidate region P in the self-teaching data 40C. unsup represents the loss function. (pi, pj) represents a pair of feature vectors of the corresponding candidate region P.

[0069] Non-Patent Document 10: Aaron van den Oord, et al, "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).

[0070] When the first object detection network 30 is a method based on feature vectors for each class such as Meta R-CNN (see Non-Patent Document 1), the following method may be used. In this case, the first self-supervised learning loss calculation unit 24B may calculate the second loss using the loss function shown in the above formula (1) based on the feature amount of the candidate region P in the self-supervised data 40C so as to detect a corresponding candidate region P that is the same region in the image data 40A before the image conversion of the self-supervised data 40C.

[0071] Then, the first self-supervised learning loss calculation unit 24B may calculate the above loss function as the second loss.

[0072] Furthermore, the first self-supervised data generation unit 24A may input the image data 40A and the self-supervised data 40C generated from the image data 40A to the first object detection network 30. Then, the first self-supervised data generation unit 24A may output information representing pairs of corresponding candidate regions P of the image data 40A and the self-supervised data 40C to the first self-supervised learning loss calculation unit 24B.

[0073] In this case, the first self-supervised learning loss calculation unit 24B extracts features of a candidate region P identified by information representing a pair of candidate regions P received from the first self-supervised data generation unit 24A, from features output as an array from the intermediate layer or the final layer by processing each of the image data 40A and the self-supervised data 40C according to the parameters of the first object detection network 30. Through these processes, the first self-supervised learning loss calculation unit 24B extracts features of the candidate region P, which is the same region, in the image data 40A and the self-supervised data 40C generated from the image data 40A, and calculates the second loss in the same manner as described above.

[0074] The second loss calculated by the first self-supervised learning loss calculation unit 24B is output to the update unit .

[0075] The update unit 26 updates the parameters of the first object detection network 30 so as to reduce the second loss. That is, the update unit 26 updates the parameters of the first object detection network 30 so as to reduce both the first loss received from the first loss calculation unit 22B and the second loss received from the first self-supervised learning loss calculation unit 24B.

[0076] Specifically, the update unit 26 updates the parameters of the first object detection network 30 by back-propagating the first loss received from the first loss calculation unit 22B and the second loss received from the first self-supervised learning loss calculation unit 24B to the first object detection network 30.

[0077] The process of updating the parameters of the first object detection network 30 in response to the second loss may be executed by the first self-supervised learning unit 24. That is, each of the first supervised learning unit 22 and the first self-supervised learning unit 24 may include an update unit 26.

[0078] Furthermore, the first learning unit 20 may be configured to include a first object detection network 30 for the first supervised learning unit 22 and a first object detection network 30 for the first self-supervised learning unit 24 .

[0079] In this case, the update unit 26 updates the parameters of the first object detection network 30 for the first supervised learning unit 22 so as to reduce the first loss received from the first loss calculation unit 22B.

[0080] Then, after the first supervised learning unit 22 has finished learning of the first object detection network 30 for the first supervised learning unit 22, the update unit 26 may gradually reflect the parameters of the first object detection network 30 for the first supervised learning unit 22 in the first object detection network 30 for the first self-supervised learning unit 24. Furthermore, the update unit 26 may gradually reflect the parameters of the first object detection network 30 for the first supervised learning unit 22 in the first object detection network 30 for the first self-supervised learning unit 24 while the first supervised learning unit 22 is learning the first object detection network 30 for the first supervised learning unit 22.

[0081] Then, after the first supervised learning unit 22 has completed learning of the first object detection network 30 for the first supervised learning unit 22, the update unit 26 may update the parameters of the first object detection network 30 for the first self-supervised learning unit 24 so as to reduce the second loss received from the first self-supervised learning loss calculation unit 24B.

[0082] Next, an example of the flow of information processing executed by the learning device 10 of this embodiment will be described.

[0083] FIG. 3 is a flowchart showing an example of the flow of information processing executed by the learning device 10 of this embodiment.

[0084] The input unit 22A of the first supervised learning unit 22 acquires training data 40 of an arbitrary mini-batch size from a training dataset 41 containing multiple training data 40, and inputs image data 40A contained in the training data 40 to the first object detection network 30 (step S100).

[0085] The first loss calculation unit 22B calculates the loss of the detection result including the class C of the object region F and the position information of the object region F output from the first object detection network 30 by the processing of step S100, relative to the teacher data 40B corresponding to the image data 40A, as the first loss (step S102).

[0086] The first self-supervisory data generation unit 24A generates self-supervisory data 40C from the image data 40A input to the first object detection network 30 in step S100 (step S104).

[0087] In addition, the first self-teacher data generation unit 24A identifies a corresponding candidate region P for each of the image data 40A input to the first object detection network 30 in step S100 and the self-teacher data 40C generated in step S104 (step S106).

[0088] The first self-supervisory data generation unit 24A inputs the self-supervisory data 40C generated in step S104 and the image data 40A used to generate the self-supervisory data 40C to the first object detection network 30 (step S108).

[0089] The first self-supervised learning loss calculation unit 24B calculates a second loss of the feature amount derived by the first object detection network 30 for the corresponding candidate region P, which is the same region, of the self-supervised data 40C generated in step S104 and the image data 40A used to generate the self-supervised data 40C (step S110). In detail, the first self-supervised learning loss calculation unit 24B calculates a second loss of the feature amount of the corresponding candidate region P in the self-supervised data 40C for the feature amount of the candidate region P in the image data 40A, which is derived by the first object detection network 30 by inputting the image data 40A and the self-supervised data 40C.

[0090] Updater 26 updates the parameters of first object detection network 30 so as to reduce both the first loss calculated in step S102 and the second loss calculated in step S110 (step S112).

[0091] Next, the first learning unit 20 determines whether or not a learning end condition for the first object detection network 30 is satisfied (step S114). For example, the first learning unit 20 makes the determination in step S114 by determining whether or not the number of times the series of processes from step S100 to step S112 has been repeated is equal to or greater than a predetermined threshold. If the determination in step S114 is negative (step S114: No), the process returns to step S100. If the determination in step S114 is positive (step S114: Yes), this routine ends.

[0092] As described above, the learning device 10 of this embodiment includes the first learning unit 20. The first learning unit 20 has a first supervised learning unit 22 and a first self-supervised learning unit 24. The learning data 40 includes image data 40A and teacher data 40B including a class C representing a correct object detection result for an object region F included in the image data 40A and position information of the object region F in the image data 40A.

[0093] The first supervised learning unit 22 uses the learning data 40 to train the first object detection network 30 so as to reduce a first loss between the output of the first object detection network 30 for detecting an object from target image data and the teacher data 40B. The first self-supervised learning unit 24 uses the image data 40A and self-supervised data 40C generated from the image data 40A to train the first object detection network 30 so as to reduce a second loss in the feature amount of the candidate region P corresponding to the image data 40A and the self-supervised data 40C, which is derived by the first object detection network 30.

[0094] The first supervised learning unit 22 of the learning device 10 of this embodiment trains the first object detection network 30 using the teacher data 40B. Furthermore, in the learning device 10 of this embodiment, the first object detection network 30 is trained so as to reduce a second loss of the feature amount of the candidate region P corresponding to the image data 40A and the self-supervised data 40C.

[0095] That is, in the learning device 10 of this embodiment, the first object detection network 30 is trained by self-supervised learning so that for a candidate region P including a background region, which is a region not taught class C, the same candidate region P has the same features between the image data 40A and the self-supervised data 40C generated from the image data 40A.

[0096] Therefore, the learning device 10 of this embodiment can learn the first object detection network 30 capable of performing object detection with high accuracy even in a candidate area P of class C that is not taught by the teacher data 40B. In other words, the learning device 10 of this embodiment can learn the first object detection network 30 capable of performing object detection with high accuracy using a small amount of image data 40A that includes a new candidate area P of class C that is not taught as the teacher data 40B.

[0097] Furthermore, the learning device 10 of this embodiment trains the first object detection network 30 using self-supervised data 40C generated from image data 40A. Therefore, the learning device 10 of this embodiment can train the first object detection network 30 with a smaller amount of training data 40. That is, the learning device 10 of this embodiment can improve the object detection accuracy of a new class C that is not shown as the supervised data 40B with a smaller amount of training data 40.

[0098] Therefore, the learning device 10 of this embodiment can improve the object detection accuracy by learning using a smaller amount of learning data 40.

[0099] Second Embodiment In this embodiment, an example of a learning device that can efficiently handle a small amount of new image data will be described by using the trained first object detection network 30 trained by the first learning unit 20 of the above embodiment. Note that in this embodiment, the same components as those in the above embodiment are given the same reference numerals, and detailed descriptions thereof will be omitted.

[0100] FIG. 4 is a block diagram showing an example of the configuration of the learning device 12 of this embodiment.

[0101] The learning device 12 includes a first learning unit 20 and a second learning unit 21. The first learning unit 20 is similar to that in the first embodiment.

[0102] The second learning unit 21 learns the second object detection network 32 using new learning data 42 different from the learning data 40 and the first object detection network 30 learned by the first learning unit 20.

[0103] Similar to the first object detection network 30, the second object detection network 32 is a neural network for detecting an object included in the target image data of the object detection target. The second object detection network 32 is similar to the first object detection network 30, except that the second object detection network 32 is an object detection network trained by the second learning unit 21. The object detection method by the second object detection network 32 may be the same as or different from that of the first object detection network 30. A specific example of the detection method by the second object detection network 32 is similar to the above-mentioned detection method by the first object detection network 30, and therefore will not be described here.

[0104] The second learning unit 21 has an additional learning initialization unit 28, a second supervised learning unit 23, a second self-supervised learning unit 25, and an update unit 27. The second supervised learning unit 23 includes an input unit 23A and a second loss calculation unit 23B. The second self-supervised learning unit 25 includes a second self-supervised data generation unit 25A and a second self-supervised learning loss calculation unit 25B. The additional learning initialization unit 28, the second supervised learning unit 23, the input unit 23A, the second loss calculation unit 23B, the second self-supervised learning unit 25, the second self-supervised data generation unit 25A, the second self-supervised learning loss calculation unit 25B, and the update unit 27 are realized by, for example, one or more processors.

[0105] The additional learning initialization unit 28 initializes the second object detection network 32 using the first object detection network 30 trained by the first learning unit 20 .

[0106] In detail, the additional learning initialization unit 28 applies parameters of at least a part of the tasks set in the first object detection network 30 to the second object detection network 32. Furthermore, the additional learning initialization unit 28 initializes the parameters of the new class with random numbers. For example, it is assumed that the second object detection network 32 is an object detection network based on correlation with the feature vector of each class, such as Meta R-CNN (Non-Patent Document 1). In this case, the additional learning initialization unit 28 may use, as the feature vector of the new class, features of the teaching region of the new class of the new learning data 42 extracted by ROIAlign (Non-Patent Document 9).

[0107] The additional learning initialization unit 28 may apply all parameters of each of the multiple tasks included in the first object detection network 30 to the second object detection network 32. Furthermore, the additional learning initialization unit 28 may apply parameters of a predetermined task included in the first object detection network 30 as parameters of a corresponding task in the second object detection network 32. Which task parameters are to be applied to the second object detection network 32 may be set in advance, for example, by an operational instruction by the user. Furthermore, the task to be applied may be changeable as appropriate by an operational instruction by the user.

[0108] Additionally, additional learning initialization unit 28 may set tasks to be learned in second object detection network 32 so as to learn the same task parameters as those of first object detection network 30. Additionally, additional learning initialization unit 28 may set some of the tasks of the parameters learned in first object detection network 30 as the learning targets. Additionally, the tasks to be learned may be changeable as appropriate by an operational instruction by the user, etc.

[0109] The new learning data 42 includes new image data 42A and new teacher data 42B.

[0110] The new image data 42A is image data newly prepared for learning, separate from the image data 40A used when the first learning unit 20 learned the first object detection network 30. For example, the new image data 42A is image data different from the image data 40A. Like the image data 40A, the new image data 42A is image data to which the new teacher data 42B has not been added.

[0111] Similar to the teacher data 40B, the new teacher data 42B is data that directly or indirectly represents correct data that should be output from the second object detection network 32 when new image data 42A is input to the second object detection network 32 during learning. In this embodiment, the new teacher data 42B includes a class C that represents the correct object detection result of an object region F included in the new image data 42A, and position information of the object region F in the new image data 42A. The object region F and the position information are the same as those in the above embodiment.

[0112] Note that new training data set 43 including the plurality of new training data 42 and second object detection network 32 may be stored in a storage unit provided outside learning device 12. Also, at least one of the storage unit and the plurality of functional units included in second learning unit 21 may be mounted on an external information processing device communicatively connected to learning device 12 via a network or the like.

[0113] The second supervised learning unit 23 is similar to the first supervised learning unit 22 of the first learning unit 20, except that new learning data 42 is used instead of the learning data 40. That is, the input unit 23A and the second loss calculation unit 23B of the second supervised learning unit 23 are similar to the input unit 22A and the first loss calculation unit 22B of the first supervised learning unit 22, respectively, except that new learning data 42 is used instead of the learning data 40. In this embodiment, the loss calculated by the second loss calculation unit 23B will be described as a third loss.

[0114] The second self-supervised learning unit 25 is similar to the first self-supervised learning unit 24 of the first learning unit 20, except that new image data 42A is used instead of image data 40A. That is, the second self-supervised data generation unit 25A and the second self-supervised learning loss calculation unit 25B of the second self-supervised learning unit 25 are similar to the first self-supervised data generation unit 24A and the first self-supervised learning loss calculation unit 24B of the first self-supervised learning unit 24, respectively, except that new image data 42A is used instead of image data 40A. In this embodiment, the loss calculated by the second self-supervised learning loss calculation unit 25B will be described as a fourth loss.

[0115] The update unit 27 receives a third loss from the second loss calculation unit 23B in place of the first loss received from the first loss calculation unit 22B. Furthermore, the update unit 27 receives a fourth loss from the second self-supervised learning loss calculation unit 25B in place of the second loss received from the first self-supervised learning loss calculation unit 24B. Then, the update unit 27 uses the third loss and the fourth loss to update the parameters of the second object detection network 32. Except for these points, the update unit 27 updates the parameters of the second object detection network 32 in the same manner as the update unit 26.

[0116] Next, an example of the flow of information processing executed by the learning device 12 of this embodiment will be described.

[0117] FIG. 5 is a flowchart showing an example of the flow of information processing executed by the learning device 12 of this embodiment.

[0118] The first learning unit 20 executes a learning process for the first object detection network 30 using the learning data 40 (step S200). The process of step S200 is similar to the processes of steps S100 to S114 in the above embodiment (see FIG. 3).

[0119] Next, the additional learning initialization unit 28 of the second learning unit 21 initializes the second object detection network 32 by using the first object detection network 30 learned by the first learning unit 20 in step S200 (step S202).

[0120] Next, the input unit 23A of the second supervised learning unit 23 acquires new training data 42 of an arbitrary mini-batch size from a new training dataset 43 containing multiple new training data 42, and inputs new image data 42A contained in the new training data 42 to the second object detection network 32 (step S204).

[0121] The second loss calculation unit 23B calculates as a third loss the loss of the detection result including the class C of the object region F and the position information of the object region F output from the second object detection network 32 by the processing of step S204, relative to the new teacher data 42B corresponding to the new image data 42A (step S206).

[0122] The second self-supervised data generation unit 25A of the second self-supervised learning unit 25 generates new self-supervised data from the new image data 42A input to the second object detection network 32 in step S204 (step S208).

[0123] In addition, the second self-teacher data generation unit 25A identifies candidate regions P that are the same corresponding regions for each of the new self-teacher data generated in step S208 and the new image data 42A used to generate the new self-teacher data (step S210).

[0124] The second self-supervisory data generation unit 25A inputs the new self-supervisory data generated in step S208 and the new image data 42A used to generate the new self-supervisory data to the second object detection network 32 (step S212).

[0125] The second self-supervised learning loss calculation unit 25B calculates a fourth loss of the feature amount derived by the second object detection network 32 for the corresponding candidate region P, which is the same region, of the new self-supervised data generated in step S208 and the new image data 42A used to generate the new self-supervised data (step S214). The second self-supervised learning loss calculation unit 25B calculates a fourth loss of the feature amount of the corresponding candidate region P in the new self-supervised data relative to the feature amount of the candidate region P in the new image data 42A, which is derived by the second object detection network 32 by inputting the new image data 42A and the new self-supervised data.

[0126] Updater 27 updates the parameters of second object detection network 32 so as to reduce both the third loss calculated in step S206 and the fourth loss calculated in step S214 (step S216).

[0127] Next, the second learning unit 21 determines whether or not a learning end condition for the second object detection network 32 is satisfied (step S218). For example, the second learning unit 21 makes the determination in step S218 by determining whether or not the number of times the series of processes from step S204 to step 216 has been repeated is equal to or greater than a predetermined threshold. If the determination in step S218 is negative (step S218: No), the process returns to step S204. If the determination in step S218 is positive (step S218: Yes), the present routine is terminated.

[0128] As described above, the learning device 12 of this embodiment includes the first learning unit 20 and the second learning unit 21. The second learning unit 21 learns the second object detection network 32 using new learning data 42 different from the learning data 40, and the first object detection network 30 learned by the first learning unit 20.

[0129] That is, the second learning unit 21 of the learning device 12 of this embodiment learns the second object detection network 32 using a learned model that is the first object detection network 30 learned by the first learning unit 20.

[0130] Therefore, the learning device 12 of the present embodiment can learn the second object detection network 32 that can quickly adapt to target objects that have been taught only a small amount of data, for example. In other words, the learning device 12 of the present embodiment can use a small amount of new training data 42 to learn, in a shorter time, the second object detection network 32 that can output class C, which is an object detection result, for an area where class C included in the new training data 42 has not been taught.

[0131] Therefore, in addition to the effects of the above-described embodiments, the learning device 12 of this embodiment can learn the second object detection network 32 that can quickly adapt to a small amount of new learning data .

[0132] (Third embodiment) In this embodiment, a detection device using at least one of the first object detection network 30 and the second object detection network 32 trained in the above embodiment will be described. In this embodiment, the same components as those in the above embodiment will be given the same reference numerals, and detailed description will be omitted.

[0133] FIG. 6 is a schematic diagram of an example of a detection device 50 of this embodiment.

[0134] The detection device 50 includes an image processing unit 50A. The image processing unit 50A is realized by, for example, one or more processors.

[0135] The image processing unit 50A inputs target image data 44 of an object detection target to the object detection network 34. The target image data 44 is image data of an object detection target. The image processing unit 50A derives, as output from the object detection network 34, a class C representing the object detection result contained in the target image data 44 and position information of the object in the target image data 44.

[0136] The object detection network 34 is at least one of the first object detection network 30 trained by the first learning unit 20 of the above embodiment and the second object detection network 32 trained by the second learning unit 21 of the above embodiment.

[0137] Next, an example of the flow of information processing executed by the detection device 50 of this embodiment will be described.

[0138] FIG. 7 is a flowchart showing an example of the flow of information processing executed by the detection device 50 of this embodiment.

[0139] The image processing unit 50A acquires the target image data 44, and shapes the acquired target image data 44 to an input size for the object detection network 34 (step S300).

[0140] Then, the image processing unit 50A inputs the shaped target image data 44 to the object detection network 34 (step S302).

[0141] The image processing unit 50A obtains a rectangular region representing the object region F for each class C that represents the object detection result of the object region F output from the object detection network 34 by inputting the target image data 44 to the object detection network 34 in step S302. Then, the image processing unit 50A removes overlapping regions of these object regions F from the object region F for each class C (step S304).

[0142] The rectangular regions that are the object regions F for each class C output from the object detection network 34 may be detected as overlapping regions. For this reason, the image processing unit 50A eliminates overlapping rectangular regions that have low detection scores and overlap by NMS (Non Maximum Suppression) through the process of step S304. It is preferable that the image processing unit 50A pre-sets a threshold value for the reliability for each class C. It is preferable that the image processing unit 50A reduces the number of rectangular regions detected for each class C by eliminating rectangular regions with reliability equal to or lower than the threshold value set for each class C. Through this process, the image processing unit 50A can selectively detect a desired object from the target image data 44.

[0143] Then, the image processing unit 50A derives the class C of the object region F after removing the overlapping regions in step S304, and the position information of the object region F (step S306), and then ends this routine.

[0144] As described above, the image processing unit 50A of the detection device 50 of this embodiment inputs the target image data 44 of the object detection target to the object detection network 34. The object detection network 34 is at least one of the first object detection network 30 trained by the first learning unit 20 of the above embodiment and the second object detection network 32 trained by the second learning unit 21 of the above embodiment. Then, the image processing unit 50A derives, as output from the object detection network 34, a class C representing the object detection result contained in the target image data 44 and position information of the object (object region F) in the target image data 44.

[0145] As described above, the first object detection network 30 and the second object detection network 32 are the object detection network 34 that achieves improved object detection accuracy.

[0146] Therefore, by inputting the target image data 44 to the object detection network 34, the image processing unit 50A can derive with high accuracy a class C representing the object detection result and position information of the object (object area F) in the target image data 44 as output from the image processing unit 50A.

[0147] Therefore, the detection device 50 of this embodiment can improve the object detection accuracy in addition to the effects of the above-described embodiments.

[0148] The application of the detection device 50 of the present embodiment is not limited. The detection device 50 of the present embodiment is preferably applied to, for example, person detection in a video captured by a security camera, vehicle detection in a video captured by an in-vehicle camera, and the like.

[0149] (Fourth embodiment) In this embodiment, an example of a learning system including the learning device 12 and the detection device 50 of the above embodiment will be described. In this embodiment, the same components as those in the above embodiment will be given the same reference numerals, and detailed description thereof will be omitted.

[0150] FIG. 8 is a schematic diagram of an example of the learning system 1 of this embodiment.

[0151] The learning system 1 includes a learning device 12, a trained model storage unit 52, a detection device 50, an evaluation unit 54, a history storage unit 56, an output control unit 58, and a display unit 60. The learning device 12, the trained model storage unit 52, the detection device 50, the evaluation unit 54, the history storage unit 56, the output control unit 58, and the display unit 60 are communicatively connected. The first learning unit 20, the second learning unit 21, the image processing unit 50A, the evaluation unit 54, and the output control unit 58 are realized by, for example, one or more processors.

[0152] The learning device 12 is similar to the learning device 12 in the above embodiment. The learning device 12 includes a first learning unit 20 and a second learning unit 21. The first learning unit 20 and the second learning unit 21 are similar to those in the above embodiment.

[0153] The trained model storage unit 52 stores the object detection network 34. As in the above embodiment, the object detection network 34 is at least one of the first object detection network 30 and the second object detection network 32. That is, the trained model storage unit 52 stores the trained first object detection network 30 and the trained second object detection network 32 trained by the learning device 12.

[0154] As in the above embodiment, additional learning initialization unit 28 of second learning unit 21 initializes second object detection network 32 using first object detection network 30 trained by first learning unit 20. Then, second learning unit 21 trains second object detection network 32 using new training data 42. Second learning unit 21 updates second object detection network 32 in trained model storage unit 52 at the end of training or each time training is performed using new training data 42 with a number of data of an arbitrary mini-batch size.

[0155] The detection device 50 includes an image processing unit 50A. The detection device 50 and the image processing unit 50A are the same as those in the above embodiment. Except for using evaluation data 46 instead of the target image data 44, the present embodiment is the same as the above embodiment.

[0156] The evaluation data 46 is image data and teacher data used for evaluating the object detection network 34. In detail, the evaluation data 46 includes evaluation image data 46A and evaluation teacher data 46B.

[0157] The evaluation image data 46A may be image data to which no teaching data has been added. The evaluation image data 46A may be the same image data as the image data 40A or the new image data 42A, or may be different image data.

[0158] Similar to the teacher data 40B and the new teacher data 42B, the evaluation teacher data 46B is data that directly or indirectly represents correct data that should be output from the object detection network 34 when the evaluation image data 46A is input to the object detection network 34. In this embodiment, the evaluation teacher data 46B includes a class C that represents the correct object detection result of the object region F included in the evaluation image data 46A, and position information of the object region F in the evaluation image data 46A. The object region F and the position information are the same as in the above embodiment.

[0159] In this embodiment, the image processing unit 50A inputs the evaluation image data 46A to the object detection network 34 instead of the target image data 44. Note that in this embodiment, a form in which one evaluation image data 46A, i.e., the same one evaluation image data 46A is always input to the image processing unit 50A, will be described as an example. The image processing unit 50A derives, as output from the object detection network 34, a class C representing the object detection result contained in the evaluation image data 46A and position information of the object in the evaluation image data 46A.

[0160] The evaluation unit 54 evaluates the detection results that are output from the object detection network 34.

[0161] The evaluation unit 54 uses the detection result including the class C, which is the object detection result output from the object detection network 34, and the position information, and the evaluation teacher data 46B to evaluate the detection accuracy of the detection result.

[0162] Then, the evaluation unit 54 associates the evaluation image data 46A used in the evaluation with the detection result and the evaluation result, and stores them as history information in the history storage unit 56. Note that the evaluation unit 54 may also store other information related to the object detection network 34 used in the evaluation in association with each other in the history storage unit 56. The other information may include, for example, information related to parameters of the object detection network 34 used in the evaluation, and the learning data 40 and new learning data 42 used to train the object detection network 34.

[0163] The output control unit 58 outputs the learning result including at least one of the evaluation result of the evaluation by the evaluation unit 54 and the detection result to the display unit 60. The display unit 60 is, for example, a display.

[0164] FIG. 9 is a schematic diagram of an example of a display screen 62 that the output control unit 58 displays on the display unit 60. As shown in FIG.

[0165] For example, the output control unit 58 outputs a display screen 62 including the learning results 64 of each of the first object detection network 30 and the second object detection network 32 to the display unit 60.

[0166] The learning results 64 include evaluation image data 46A used to evaluate the object detection network 34, detection results 66 of the object detection network 34 using the evaluation image data 46A, and evaluation results 68 of the detection results 66.

[0167] Specifically, the display screen 62 includes a learning result 64A and a learning result 64B as learning results 64.

[0168] Learning result 64A is an example of learning result 64 by first object detection network 30. Learning result 64A includes evaluation image data 46A used to evaluate first object detection network 30, detection result 66A, and evaluation result 68A.

[0169] Position information of object region F included in detection result 66A is represented, for example, by displaying a rectangular frame representing object region F on evaluation image data 46A. Fig. 9 shows a rectangular frame of object region Fa and class Ca as object region F and class C detected from evaluation image data 46A by first object detection network 30. Note that text information representing class C of object region F included in detection result 66A is displayed, for example, in a display field of evaluation result 68A.

[0170] The display field of the evaluation result 68A includes, for example, identification information of the learning dataset 41 used in training the first object detection network 30, and the detection accuracy of the detection result using the evaluation image data 46A by the first object detection network 30. Fig. 9 shows "dataset A" as the identification information of the learning dataset 41 used in training the first object detection network 30. Fig. 9 also shows, as the detection accuracy of the detection result using the evaluation image data 46A by the first object detection network 30, the "base class" which is the class Ca detected from the evaluation image data 46A, and the detection accuracy of the class Ca of "80.5%".

[0171] Learning result 64B is an example of learning result 64 by second object detection network 32. Learning result 64B includes evaluation image data 46A used to evaluate second object detection network 32, detection result 66B, and evaluation result 68B.

[0172] Position information of object region F included in detection result 66B is represented, for example, by displaying a rectangular frame representing object region F on evaluation image data 46A. Fig. 9 shows a rectangular frame and class Ca of object region Fa, and a rectangular frame and class Cb of object region Fb, as object region F and class C detected from evaluation image data 46A by second object detection network 32. Note that text information representing class C of object region F included in detection result 66B is displayed in a column of evaluation result 68B, etc.

[0173] The display column of the evaluation result 68B includes, for example, identification information of the new training data set 43 used in training the second object detection network 32, and the detection accuracy of the detection result using the evaluation image data 46A by the second object detection network 32. FIG. 9 shows "data set B" as the identification information of the new training data set 43 used in training the second object detection network 32. FIG. 9 also shows, as the detection accuracy of the detection result using the evaluation image data 46A by the second object detection network 32, the "base class" which is the class Ca detected from the evaluation image data 46A and the detection accuracy of the class Ca of "79.3%", and the "new class" which is the detected class Cb and the detection accuracy of the class Cb of "50.4%".

[0174] Thus, in this embodiment, the output control unit 58 outputs to the display unit 60 a display screen 62 including the learning results 64 of each of the first object detection network 30 and the second object detection network 32. In addition, the output control unit 58 outputs to the display unit 60 the learning results 64 of each of the different first object detection networks 30 and second object detection networks 32 for the same evaluation image data 46A.

[0175] Therefore, the learning system 1 of this embodiment can provide a list of changes in the learning result 64 in an easily checkable manner.

[0176] Each time the second learning unit 21 acquires new training data 42 of a mini-batch size and learns the second object detection network 32, the evaluation unit 54 may evaluate the detection result 66 of the second object detection network 32 for the evaluation image data 46A. Then, each time the evaluation unit 54 evaluates the detection result 66 of the second object detection network 32, the output control unit 58 may output to the display unit 60 a display screen 62 to which a learning result 64 including an evaluation result 68 of the new evaluation has been further added.

[0177] In this case, learning system 1 of the present embodiment can provide an easily checkable list of changes in learning result 64 according to the progress of learning of second object detection network 32.

[0178] Next, an example of the flow of information processing executed by the learning system 1 of this embodiment will be described.

[0179] FIG. 10 is a flowchart showing an example of the flow of information processing executed by the learning system 1 of this embodiment.

[0180] The first learning unit 20 executes a learning process for the first object detection network 30 using the learning data 40 (step S400). The process of step S400 is similar to the processes of steps S100 to S114 in the above embodiment (see FIG. 3).

[0181] Next, the additional learning initialization unit 28 of the second learning unit 21 initializes the second object detection network 32 by using the first object detection network 30 learned by the first learning unit 20 in step S400 (step S402).

[0182] Next, second learning unit 21 executes a learning process for second object detection network 32 (step S404). The process in step S404 is similar to steps S204 to S218 in the above embodiment (see FIG. 5).

[0183] Next, image processing unit 50A inputs the same evaluation image data 46A to each of first object detection network 30 trained by first learning unit 20 and second object detection network 32 trained by second learning unit 21 (step S406).

[0184] The evaluation unit 54 uses the detection results 66, which include class C and position information, which are object detection results output from each of the first object detection network 30 and the second object detection network 32, and the evaluation teacher data 46B to evaluate the detection accuracy of each detection result 66 (step S408).

[0185] Then, the evaluation unit 54 associates the evaluation image data 46A used in the evaluation with the detection result 66 and the evaluation result 68, and stores them as history information in the history storage unit 56 (step S410).

[0186] The output control unit 58 outputs the learning result 64 based on the history information stored in step S410 and the evaluation result 68 in step S408 to the display unit 60 (step S412).

[0187] Next, the learning system 1 determines whether new learning data 42 has been added (step S414). If the determination in step S414 is affirmative (step S414: Yes), the process returns to step S404, where the second object detection network 32 is trained using the newly added new learning data 42. On the other hand, if the determination in step S414 is negative (step S414: No), this routine ends.

[0188] As described above, the learning system 1 of this embodiment includes the learning device 12, the detection device 50, the evaluation unit 54, and the output control unit 58. The evaluation unit 54 evaluates the detection result 66 that is output from the object detection network 34, which is at least one of the first object detection network 30 and the second object detection network 32. The output control unit 58 outputs the learning result 64 that includes at least one of the detection result 66 and the evaluation result 68 of the evaluation.

[0189] In this way, learning system 1 of the present embodiment detects objects from evaluation image data 46A using object detection network 34, which is a trained model trained by learning device 12, and derives detection result 66 including class C representing the object detection result of object region F and position information of object region F. Then, learning system 1 outputs learning result 64 including at least one of detection result 66 of at least one of first object detection network 30 and second object detection network 32 included in object detection network 34 and evaluation result 68 of detection result 66 to display unit 60 or the like.

[0190] Therefore, in addition to the effects of the above embodiments, the learning system 1 of this embodiment can easily provide the user with the learning status of the object detection network 34 and evaluation results 68 of the object detection accuracy by the object detection network 34.

[0191] Furthermore, learning system 1 of the present embodiment displays display screen 62 including learning results 64 of first object detection network 30 and second object detection network 32 on display unit 60. Therefore, learning system 1 of the present embodiment can provide multiple learning results 64 to the user in an easily verifiable manner.

[0192] Next, an example of the hardware configuration of the learning device 10, the learning device 12, the detection device 50, and the learning system 1 of the above embodiment will be described.

[0193] FIG. 11 is a hardware configuration diagram of an example of the learning device 10, the learning device 12, the detection device 50, and the learning system 1 of the above embodiment.

[0194] The learning device 10, learning device 12, detection device 50, and learning system 1 of the above embodiments have a hardware configuration utilizing a conventional computer, in which a CPU (Central Processing Unit) 81, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 83, a communication I / F 84, etc. are interconnected via a bus 85.

[0195] The CPU 81 is a calculation device that controls the learning device 10, the learning device 12, the detection device 50, and the learning system 1 of the above-mentioned embodiment. The ROM 82 stores programs and the like that realize various processes by the CPU 81. Although a CPU is used for the explanation here, a GPU (Graphics Processing Unit) may be used as the calculation device that controls the learning device 10, the learning device 12, the detection device 50, and the learning system 1. The RAM 83 stores data necessary for various processes by the CPU 81. The communication I / F 84 is an interface that is connected to the display unit 60, etc., and is used to transmit and receive data.

[0196] In the learning device 10, learning device 12, detection device 50, and learning system 1 of the above-mentioned embodiments, the CPU 81 reads a program from the ROM 82 onto the RAM 83 and executes it, thereby realizing each of the above-mentioned functions on the computer.

[0197] The programs for executing the above-mentioned processes executed by the learning device 10, learning device 12, detection device 50, and learning system 1 of the above-mentioned embodiments may be stored in an HDD (hard disk drive). Also, the programs for executing the above-mentioned processes executed by the learning device 10, learning device 12, detection device 50, and learning system 1 of the above-mentioned embodiments may be provided by being pre-installed in ROM 82.

[0198] The programs for executing the above processes executed by the learning device 10, learning device 12, detection device 50, and learning system 1 of the above embodiments may be stored in a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) in an installable or executable format file and provided as a computer program product. The programs for executing the above processes executed by the learning device 10, learning device 12, detection device 50, and learning system 1 of the above embodiments may be stored on a computer connected to a network such as the Internet and provided by downloading the programs via the network. The programs for executing the above processes executed by the learning device 10, learning device 12, detection device 50, and learning system 1 of the above embodiments may be provided or distributed via a network such as the Internet.

[0199] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0200] 1. Learning System 10, 12 Learning device 20 First Study Section 21 Second Study Section 22 First Supervised Learning Unit 22A Input section 22B 1st loss calculation section 24 First Self-Supervised Learning Unit 24A First self-teaching data generation unit 24B First self-supervised learning loss calculation unit 30 First Object Detection Network 32 Second Object Detection Network 34 Object Detection Network 50 Detection Device 50A Image processing unit 54 Evaluation Section 58 Output control section 60 Display section

Claims

1. a first supervised learning unit that uses learning data including image data and supervised data including a class representing a correct object detection result for an object area included in the image data and position information of the object area in the image data to train the first object detection network so as to reduce a first loss between an output of the first object detection network for detecting an object from target image data and the supervised data; A first self-supervised learning unit that learns the first object detection network using the image data and self-supervised data generated from the image data so as to reduce a second loss of feature amounts of corresponding candidate regions between the image data and the self-supervised data derived by the first object detection network; A first learning unit having Equipped with the first loss is a loss of a class included in a detection result output from the first object detection network by inputting the image data to the first object detection network, for a class representing the correct object detection result included in the teacher data corresponding to the image data; The second loss is a loss of a feature amount of the candidate region in the self-supervised data corresponding to a feature amount of the candidate region in the image data, which is derived by the first object detection network by inputting the image data and the self-supervised data to the first object detection network. Learning device.

2. The first self-supervised learning unit is A first self-teacher data generation unit that generates the self-teacher data, which is image data after image conversion of the image data, and identifies the corresponding candidate area from each of the image data and the self-teacher data; a first self-supervised learning loss calculation unit that inputs the image data and the self-supervised data to the first object detection network and calculates the second loss of a feature amount of the candidate region in the self-supervised data corresponding to a feature amount of the candidate region in the image data derived by the first object detection network; The learning device according to claim 1 , further comprising:

3. The first self-teacher data generation unit is Identifying, as the candidate region, a region that is randomly identified from the image data and the self-teaching data, or a region that is identified by a foreground extraction method; The learning device according to claim 2 .

4. The first self-teacher data generation unit is Identifying the candidate region, at least a part of which includes a region that does not overlap with the object region, from each of the image data and the self-supervised data. The learning device according to claim 2 or 3.

5. The first self-teacher data generation unit is The self-teaching data is generated by performing at least one image transformation of luminance conversion, color tone conversion, contrast conversion, inversion, rotation, and cropping on the image data. The learning device according to any one of claims 2 to 4.

6. a second learning unit that learns a second object detection network by using new learning data different from the learning data and the first object detection network learned by the first learning unit; The learning device according to any one of claims 1 to 5, comprising:

7. an image processing unit that inputs target image data of an object detection target to an object detection network that is at least one of the first object detection network trained by the first learning unit included in the learning device according to claim 6 and the second object detection network trained by the second learning unit included in the learning device according to claim 6, and derives, as outputs from the object detection network, a class representing an object detection result included in the target image data and position information of an object in the target image data; A detection device comprising:

8. A learning device according to claim 6 ; A detection device according to claim 7; an evaluation unit that evaluates a detection result that is an output from at least one of the first object detection network and the second object detection network; an output control unit that outputs a learning result including at least one of the detection result and the evaluation result of the evaluation; A learning system comprising:

9. The output control unit is outputting a display screen including the learning results of the first object detection network and the second object detection network to a display unit; The learning system according to claim 8.

10. a first supervised learning step of learning the first object detection network using learning data including image data and supervised data including a class representing a correct object detection result for an object area included in the image data and position information of the object area in the image data, so as to reduce a first loss between an output of the first object detection network for detecting an object from target image data and the supervised data; a first self-supervised learning step of learning the first object detection network using the image data and self-supervised data generated from the image data so as to reduce a second loss of feature amounts of corresponding candidate regions between the image data and the self-supervised data derived by the first object detection network; a first learning step having Including, the first loss is a loss of a class included in a detection result output from the first object detection network by inputting the image data to the first object detection network, for a class representing the correct object detection result included in the teacher data corresponding to the image data; The second loss is a loss of a feature amount of the candidate region in the self-supervised data corresponding to a feature amount of the candidate region in the image data, which is derived by the first object detection network by inputting the image data and the self-supervised data to the first object detection network. How to learn.

11. The first self-supervised learning step includes: A first self-teacher data generation step of generating the self-teacher data which is image data after image conversion of the image data, and identifying the corresponding candidate area from each of the image data and the self-teacher data; a first self-supervised learning loss calculation step of inputting the image data and the self-supervised data to the first object detection network and calculating the second loss of a feature amount of the candidate region in the self-supervised data corresponding to a feature amount of the candidate region in the image data, the feature amount being derived by the first object detection network; The method of claim 10, comprising:

12. The first self-teacher data generation step includes: Identifying, as the candidate region, a region that is randomly identified from the image data and the self-teaching data, or a region that is identified by a foreground extraction method; The learning method according to claim 11.

13. The first self-teacher data generation step includes: Identifying the candidate region, at least a part of which includes a region that does not overlap with the object region, from each of the image data and the self-supervised data. The learning method according to claim 11 or 12.

14. The first self-teacher data generation step includes: The self-teaching data is generated by performing at least one image transformation of luminance conversion, color tone conversion, contrast conversion, inversion, rotation, and cropping on the image data. The learning method according to any one of claims 11 to 13.

15. a second learning step of learning a second object detection network using new learning data different from the learning data and the first object detection network learned in the first learning step; The learning method according to any one of claims 10 to 14.

16. A learning program for causing a computer to execute the program, a first supervised learning step of learning the first object detection network using learning data including image data and supervised data including a class representing a correct object detection result for an object area included in the image data and position information of the object area in the image data, so as to reduce a first loss between an output of the first object detection network for detecting an object from target image data and the supervised data; a first self-supervised learning step of learning the first object detection network using the image data and self-supervised data generated from the image data so as to reduce a second loss of feature amounts of corresponding candidate regions between the image data and the self-supervised data derived by the first object detection network; a first learning step having Including, the first loss is a loss of a class included in a detection result output from the first object detection network by inputting the image data to the first object detection network, for a class representing the correct object detection result included in the teacher data corresponding to the image data; The second loss is a loss of a feature amount of the candidate region in the self-supervised data corresponding to a feature amount of the candidate region in the image data, which is derived by the first object detection network by inputting the image data and the self-supervised data to the first object detection network. Study program.

17. The first self-supervised learning step includes: A first self-teacher data generation step of generating the self-teacher data which is image data after image conversion of the image data, and identifying the corresponding candidate area from each of the image data and the self-teacher data; a first self-supervised learning loss calculation step of inputting the image data and the self-supervised data to the first object detection network and calculating the second loss of a feature amount of the candidate region in the self-supervised data corresponding to a feature amount of the candidate region in the image data, the feature amount being derived by the first object detection network; The learning program according to claim 16 , comprising:

18. The first self-teacher data generation step includes: Identifying, as the candidate region, a region that is randomly identified from the image data and the self-teaching data, or a region that is identified by a foreground extraction method; 18. The learning program according to claim 17.

19. The first self-teacher data generation step includes: Identifying the candidate region, at least a part of which includes a region that does not overlap with the object region, from each of the image data and the self-supervised data.

18. The learning program according to claim 17.

20. The first self-teacher data generation step includes: The learning program according to claim 17 or 18, further comprising: performing at least one of image transformations including luminance transformation, color tone transformation, contrast transformation, inversion, rotation, and cropping on the image data to generate the self-teaching data.

21. a second learning step of learning a second object detection network using new learning data different from the learning data and the first object detection network learned in the first learning step; The learning program according to any one of claims 16 to 20.

22. an image processing step of inputting target image data of an object detection target to an object detection network which is at least one of the first object detection network trained by the first learning unit included in the learning device according to claim 6 and the second object detection network trained by the second learning unit included in the learning device according to claim 6, and deriving, as an output from the object detection network, a class representing the object detection result included in the target image data and position information of the object in the target image data; A detection method comprising:

23. an image processing step of inputting target image data of an object detection target to an object detection network which is at least one of the first object detection network trained by the first learning unit included in the learning device according to claim 6 and the second object detection network trained by the second learning unit included in the learning device according to claim 6, and deriving, as an output from the object detection network, a class representing the object detection result included in the target image data and position information of the object in the target image data; A detection program for causing a computer to execute the following.

Images