Learning method, learning device, learning program, and learning data generation method

Abstract

To provide a technology capable of generating an AI model that can accurately detect an object regardless of the image size of the object.SOLUTION: An exemplary learning method performs learning of a machine learning model by using unsupervised data that does not applied with a right-answer label. The exemplary learning method generates pseudo labels by inputting the unsupervised data to the machine learning model, selects a pseudo label for learning, from the generated pseudo labels, on the basis of the reliability of the pseudo labels and an image size of the object to which the pseudo labels are imparted, and performs the learning by using the pseudo label for learning.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a machine learning technique using unsupervised data. [Background technology]

[0002] In supervised learning, it is necessary to attach a correct label to all data used in learning, which results in high costs for creating a dataset. To address this issue, a technique has been known in the past that associates pseudo-labels with non-correct data in order to generate a highly accurate detection model (AI model) from a small amount of correct data (supervised data) (see, for example, Patent Document 1).

[0003] In addition, there is a known technology that generates highly accurate AI models while reducing the cost of creating datasets by combining a small amount of labeled data (supervised data) with a large amount of unlabeled data (unsupervised data) for learning. This type of learning method is called semi-supervised learning. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2022 / 185899 Summary of the Invention [Problem to be solved by the invention]

[0005] To obtain a highly accurate AI model, highly reliable pseudo labels are used for learning when machine learning is performed. However, when pseudo labels are simply selected based on their reliability, it has been found that pseudo labels for small-sized object images tend to be judged to be less reliable than pseudo labels for large-sized object images, and thus the pseudo labels for small-sized object images tend to be discarded. In more detail, what is used for learning is not just the pseudo labels, but both the pseudo labels and the images (image data) corresponding to those pseudo labels. When pseudo labels are discarded by the pseudo label selection process as described above, the images corresponding to those pseudo labels are also excluded from learning together with the pseudo labels.

[0006] In conventional methods, when selecting pseudo labels to use in training, many pseudo labels for small-sized object images tend to be discarded, while many pseudo labels for large-sized object images remain. This creates an imbalance in object size in the training data, raising concerns that the detection accuracy of the AI ​​model obtained after training for small object images may decrease.

[0007] In view of the above, an object of the present invention is to provide a technology that enables the generation of an AI model that can detect objects with high accuracy regardless of the image size of the object. [Means for solving the problem]

[0008] An exemplary learning method of the present invention is a learning method for training a machine learning model using unsupervised data that does not have a correct answer label, in which the unsupervised data is input into the machine learning model to generate pseudo labels, training pseudo labels are selected from the generated pseudo labels based on the reliability of the pseudo labels and the image size of the object to which the pseudo label is assigned, and the learning is performed using the training pseudo labels. [Effects of the Invention]

[0009] According to the exemplary embodiment of the present invention, the selection of pseudo labels to be used for learning is performed taking into consideration not only the reliability of the pseudo labels but also the size of the object images to which the pseudo labels are assigned. This prevents the selection of pseudo labels from being biased toward pseudo labels of small object images and resulting in the generation of an AI model that can accurately detect objects regardless of the size of the object images. [Brief explanation of the drawings]

[0010] [Figure 1] Block diagram showing the hardware configuration of the learning device [Figure 2] A block diagram showing an overview of the functional units of the learning device. [Figure 3] Schematic diagram of mini-batch structure [Figure 4] Block diagram showing the detailed functional configuration of the label selection unit [Figure 5] FIG. 10 is a diagram showing an overview of a first selection performed by a first selection unit; [Figure 6] 1 is a flowchart illustrating the flow of a learning method executed by a learning device. [Figure 7] 7 is a flowchart illustrating the detailed processing of step S2 in FIG. 6. [Figure 8] 7 is a flowchart illustrating the detailed processing of step S3 in FIG. 6. [Figure 9] 7 is a flowchart illustrating the detailed processing of step S4 in FIG. 6. [Figure 10] FIG. 10 is a block diagram showing a detailed functional configuration of a label selection unit according to a first modified example. [Figure 11] 10 is a flowchart illustrating processing using a student model of unsupervised data in a learning device according to a first modification; [Figure 12] 10 is a flowchart showing the flow of a training data generation method according to a second modification. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0012] <1. Learning device> [1-1. Overview] 1 is a block diagram showing the hardware configuration of a learning device 100 according to an embodiment of the present invention. The learning device 100 is a device that trains a machine learning model (AI model). In this embodiment, the machine learning model is a neural network model, and more specifically, an object detection model that detects objects based on input image data.

[0013] The data structure and learning algorithm of the object detection model are not particularly limited. The object detection algorithm in the object detection model may be, for example, R-CNN, Fast R-CNN, Faster R-CNN, YOLO, SSD, or the like.

[0014] Furthermore, the object detection model (trained model) learned (trained) by the learning device 100 is mounted on, for example, a vehicle. As a detailed example, the object detection model is applied to an in-vehicle periphery monitoring device that monitors the situation around the vehicle. The in-vehicle periphery monitoring device processes images of the vehicle's surroundings input from an in-vehicle camera using the object detection model to detect automobiles, motorcycles, people, traffic lights, guide signs, and the like that exist around the vehicle.

[0015] Learning device 100 is a computer device, and as shown in Figure 1, includes a controller 1 and a memory 2. Learning device 100 may also include an input device such as a keyboard and an output device such as a display.

[0016] The controller 1 is configured to include an arithmetic circuit that performs arithmetic processing. More specifically, the controller 1 includes a processor that performs arithmetic processing and the like. The processor is configured to include, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The controller 1 may be configured with one processor or multiple processors. When configured with multiple processors, the processors only need to be connected to each other so that they can communicate with each other.

[0017] The memory 2 includes a volatile memory and a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM). The non-volatile memory may include, for example, a read-only memory (ROM), a flash memory, or a hard disk drive. The non-volatile memory stores computer-readable programs and data. In this embodiment, the memory 2 also stores the structure and parameters of the machine learning model, as well as code instructions for executing the machine learning model.

[0018] The program stored in memory 2 is a computer program that causes a computer to realize the functions of controller 1. Such a computer program may be provided, for example, by a computer-readable nonvolatile recording medium. The nonvolatile recording medium may be, for example, the nonvolatile memory described above, as well as an optical recording medium (for example, an optical disk), a magneto-optical recording medium (for example, a magneto-optical disk), a USB memory, or an SD card. As another example, the computer program may be provided from a program providing server via a communication line such as the Internet (provided by so-called download).

[0019] The learning device 100 is a device that uses unsupervised data that does not have a correct answer label to train a machine learning model. The learning method performed by the learning device 100 is a learning method that uses unsupervised data that does not have a correct answer label to train a machine learning model. A program that causes the learning device 100 to execute this learning method corresponds to a learning program.

[0020] In detail, the learning device 100 trains the machine learning model by so-called semi-supervised learning. Semi-supervised learning is a type of machine learning that combines supervised learning and unsupervised learning. In semi-supervised learning, learning is performed using both image data with a correct answer label (supervised image data) and image data without a correct answer label (unsupervised image data). In semi-supervised learning, learning is typically performed using a large amount of unsupervised image data and a small amount of supervised image data. Hereinafter, unsupervised image data may be simply referred to as unsupervised data, and supervised image data may be simply referred to as supervised data.

[0021] The learning device 100 performs a function of learning a machine learning model through semi-supervised learning by having a processor included in the controller 1 execute arithmetic processing in accordance with a learning program stored in the memory 2. As shown in FIG. 1, the learning device 100 acquires learning data required for learning the machine learning model. The learning device 100 uses the acquired learning data to learn the machine learning model through semi-supervised learning.

[0022] The training data may be provided, for example, by a computer-readable nonvolatile recording medium. Alternatively, the training data may be provided from a training data server via a communication line such as the Internet. The training device 100 stores the acquired training data in the memory 2 as appropriate.

[0023] [1-2. Functional parts of the learning device] Fig. 2 is a block diagram showing an overview of the functional units included in the learning device 100. The functional units shown in Fig. 2 are realized by a processor included in the controller 1 executing arithmetic processing in accordance with a learning program stored in the memory 2. The functional units included in the learning device 100 include a student model 11, a teacher model 12, a mini-batch generation unit 13, a first data augmentation unit 14, a second data augmentation unit 15, a supervised loss calculation unit 16, a label selection unit 17, an unsupervised loss calculation unit 18, and an update unit 19.

[0024] The student model 11 is a machine learning model that is the subject of learning (training), and more specifically, an object detection model that is the subject of learning. The student model 11 configured as an object detection model performs inference when an image is input and detects an object in the image. When the student model 11 detects an object in the image, it identifies the type, position, and size of the object. Note that the student model 11 is the subject of learning, and the object detection accuracy is low, at least in the initial stage of learning. The structure and parameters of the student model 11, as well as code instructions for executing the student model, are stored in memory 2. The student model 11 may be configured to have been trained in advance using a small amount of supervised data. However, such prior learning is not required.

[0025] The teacher model 12 is a machine learning model provided as a means for achieving semi-supervised learning. The teacher model 12, configured as an object detection model, performs inference by receiving an image as input and detects an object in the image. When the teacher model 12 detects an object in the image, it identifies the type, position, and size of the object and outputs the result as an inference result. Note that the teacher model 12 is an example of a machine learning model separate from the student model of the present invention. In this embodiment, as a semi-supervised learning method, a consistency-based method is used, which expects the model output (inference result) to be the same even when images obtained by adding different perturbations to the same image are input. Note that the perturbation referred to here includes data augmentation, dropout regularization, etc. Semi-supervised learning is achieved by calculating the consistency loss between the outputs of the student model 11 and the teacher model 12 for unlabeled data (unsupervised data). That is, in this embodiment, the teacher model 12 is configured to train so that such consistency loss can be calculated.

[0026] In detail, the weights (parameters) of the teacher model 12 are defined by the exponential moving average (EMA) of the weights of the student model 11. That is, a method is adopted in which a target value is generated by the teacher model 12 having an intermediate representation to calculate the consistency loss. Note that the dashed arrow extending from the student model 11 to the teacher model 12 in FIG. 2 indicates that the teacher model 12 is defined by the EMA of the weights of the student model 11. The structure and parameters of the teacher model 12, as well as code instructions for executing the teacher model, are stored in memory 2. Furthermore, this embodiment is a method that uses so-called Mean Teacher, but instead, for example, a Π-model, Temporal Ensembling, etc. may be used.

[0027] The mini-batch generation unit 13 generates mini-batches by sampling (extracting) data from previously prepared unsupervised data and supervised data according to predetermined conditions. Generating mini-batches enables mini-batch learning, in which parameters of a machine learning model to be learned are updated not on a sample-by-sample basis, but on a group basis of a small number of samples. The previously prepared unsupervised data and supervised data may be, for example, data already stored in the memory 2, or data input from outside via a computer-readable non-volatile recording medium or the like.

[0028] FIG. 3 is a diagram schematically illustrating the configuration of a mini-batch 3. As shown in FIG. 3, the mini-batch 3 includes an unsupervised data group 31, which is a collection of unsupervised data without a correct answer label, and a supervised data group 32, which is a collection of supervised data with a correct answer label. The amount of unsupervised data and supervised data included in the mini-batch 3 is determined according to the above-mentioned predetermined condition. The predetermined condition is set so that the amount of unsupervised data and the amount of supervised data included in the mini-batch 3 are in a certain ratio. The certain ratio may be, for example, 80% for the amount of unsupervised data and 20% for the amount of supervised data.

[0029] The mini-batch generation unit 13 may be configured to be provided in a device separate from the learning device 100. In other words, an externally generated mini-batch (a collection of data) may be input to the learning device 100.

[0030] The first data augmentation unit 14 and the second data augmentation unit 15 (see FIG. 2 ) are provided as means for applying different perturbations to the same image as described above. Specifically, the first data augmentation unit 14 and the second data augmentation unit 15 perform data augmentation on unsupervised data. The first data augmentation unit 14 is a data augmentation unit for the student model 11 and outputs data after data augmentation to the student model 11. The second data augmentation unit 15 is a data augmentation unit for the teacher model 12 and outputs data after data augmentation to the teacher model 12. The first data augmentation unit 14 performs weaker data augmentation than the second data augmentation unit 15. Hereinafter, data augmentation by the first data augmentation unit 14 may be referred to as weak data augmentation, and data augmentation by the second data augmentation unit 15 may be referred to as strong data augmentation. When strong data augmentation is performed, the degree of change to the original data before data augmentation is greater than when weak data augmentation is performed.

[0031] The data augmentation may be, for example, a color tone transformation or an affine transformation of image data. The color tone transformation may include, for example, a color transformation, a brightness transformation, a contrast transformation, or at least two of these transformations. The affine transformation may include, for example, a rotation, a left-right flip, an enlargement, a reduction, a translation, or at least two of these transformations. The data augmentation may include both a color tone transformation and an affine transformation.

[0032] The supervised loss calculation unit 16 calculates the supervised loss Ls. The supervised loss Ls is the loss (error) between the inference result obtained by inputting supervised data with a correct label into the student model 11 and the correct label. The supervised loss Ls may be calculated using a known method, such as mean square error or cross entropy.

[0033] The label selection unit 17 performs a selection process for pseudo labels obtained as a result of inference by the student model 11 on unsupervised data (specifically, data on which weak data augmentation has been performed). A pseudo label is a label provisionally assigned to unsupervised data in accordance with the inference result of the student model 11 on the unsupervised data. In this embodiment, the pseudo label includes information such as the type (class) of the detected object, its position in the image, and its image size. The position and size of an object in the image are given by a bounding box. Note that the number of pseudo labels obtained from one piece of unsupervised data may be one or more. In some cases, no pseudo label may be obtained from one piece of unsupervised data.

[0034] Highly reliable labels and low-reliable labels are mixed in the pseudo labels obtained as the inference results of the student model 11. The label selection unit 17 selects a relatively highly reliable pseudo label from among the multiple pseudo labels that are a mixture of high-reliability and low-reliability labels.

[0035] In detail, the label selection unit 17 performs a pseudo label selection process for each mini-batch. The mini-batch 3 includes multiple pieces of unsupervised data, and multiple pseudo labels are generated by inputting the multiple pieces of unsupervised data to the student model 11. The label selection unit 17 performs a selection process for multiple pseudo labels obtained by inference using the student model 11. The label selection unit 17 selects a learning pseudo label to be used for learning from the multiple pseudo labels based on the reliability of the pseudo label and the size of the object image to which the pseudo label is assigned. Note that among the pseudo labels, pseudo labels not selected as learning pseudo labels are excluded from the learning targets together with the image data corresponding to the pseudo label. The learning pseudo label is considered to be a correct label and is used for learning.

[0036] This configuration allows the selection of pseudo labels used for learning to be performed taking into account not only the reliability of the pseudo labels but also the image size of the object to which the pseudo labels are assigned. In other words, when selecting pseudo labels, it is possible to prevent the discarding of pseudo labels that are biased toward pseudo labels of small object images. As a result, it is possible to generate an object detection model (AI model) that can accurately detect objects regardless of the size of the object image.

[0037] In this embodiment, the configuration is such that pseudo labels for learning are selected, but it is also possible to select pseudo labels that are not to be used as learning pseudo labels as targets for exclusion (discard), and use the remaining pseudo labels that are not to be excluded as learning pseudo labels.

[0038] The selection process of the label selection unit 17 will be described in further detail with reference to Fig. 4 etc. Fig. 4 is a block diagram showing a detailed functional configuration of the label selection unit 17. As shown in Fig. 4, the label selection unit 17 includes a first selection unit 171, a second selection unit 172, and an integration unit 173.

[0039] The first selection unit 171 performs a first selection to select pseudo labels for learning based on reliability from among a plurality of pseudo labels generated by the student model 11 using unsupervised data. In detail, in the first selection, the first selection unit 171 divides the plurality of pseudo labels into a high-reliability group having high reliability and a low-reliability group having low reliability. The reliability of the pseudo labels may be classified using a known clustering method. The reliability of the pseudo labels may be classified using, for example, a Gaussian Mixture Model (GMM) or k-means. The following description will be given assuming that GMM is used for reliability classification.

[0040] In this embodiment, the reliability classification using GMM is performed based on the score of the pseudo label. The score indicates the probability that the area enclosed by the bounding box (rectangular box) contains an object, and is obtained as an inference result of the student model 11. The score is a number between "0" and "1." The closer the score is to 0, the more the content of the bounding box is "background," and the closer the score is to 1, the more the content of the bounding box is "object."

[0041] Fig. 5 is a diagram showing an overview of the first selection performed by the first selection unit 171. The one-dimensional scatter plot shown in the upper part of Fig. 5 shows the distribution of scores of multiple pseudo labels obtained by processing the unsupervised data included in mini-batch 3 by student model 11. The one-dimensional scatter plot shown in the lower part of Fig. 5 shows the results of clustering by GMM using the score distribution shown in the upper diagram. In the one-dimensional scatter plot shown in the lower part of Fig. 5, circles with diagonal hatching indicate pseudo labels belonging to a high-confidence group, and open circles indicate pseudo labels belonging to a low-confidence group.

[0042] Furthermore, in the first selection, the first selection unit 171 selects training pseudo labels from the high-reliability group. That is, the first selection allows highly reliable pseudo labels to be left as training pseudo labels. In detail, the first selection unit 171 selects some of the pseudo labels in the high-reliability group as training pseudo labels using statistical processing. Although all of the pseudo labels belonging to the high-reliability group may be selected as training pseudo labels, leaving only some of the pseudo labels belonging to the high-reliability group as training pseudo labels in this manner enables training using more reliable pseudo labels.

[0043] Various methods can be used to determine, by statistical processing, some pseudo labels to be retained as training pseudo labels from the high-reliability group. In the example shown in FIG. 5, the score that maximizes the logarithmic likelihood among the pseudo labels in the high-reliability group is determined as a threshold, and pseudo labels having scores equal to or greater than the threshold are retained (selected) as training pseudo labels. However, the method is not limited to this. For example, among the pseudo labels in the high-reliability group, pseudo labels having scores equal to or greater than the median or average value of the group may be retained as training pseudo labels. Furthermore, for example, in a Gaussian distribution (normal distribution) that maximizes the logarithmic likelihood in the high-reliability group, pseudo labels having scores within ±3σ (σ: standard deviation) of the average score may be retained as training pseudo labels.

[0044] The second selection unit 172 performs a second selection to select a pseudo label for learning based on the image size of the object to which the pseudo label is assigned from among the pseudo labels that were not selected as a pseudo label for learning and are to be discarded in the first selection by the first selection unit 171. Note that the image size of the object may be determined, for example, according to the size of the bounding box, and as an example, the image size of the object may be calculated from the area of ​​the bounding box.

[0045] In mini-batch 3, the image sizes of objects to which pseudo labels are assigned include a variety of sizes. Smaller objects tend to have lower pseudo-label scores compared to larger objects, due to factors such as the smaller number of anchors assigned. For this reason, pseudo labels of larger objects, which tend to have relatively higher scores, tend to be determined to have high reliability, while pseudo labels of smaller objects, which tend to have relatively lower scores, tend to be determined to have low reliability. In other words, in a configuration that performs only the first selection based on the reliability described above, pseudo labels of larger object images tend to remain disproportionately as training pseudo labels, which can lead to imbalances in the image sizes of objects used for training and reduced detection accuracy for small object images. In this regard, in the present embodiment, a second selection is performed to select training pseudo labels based on the image sizes of the objects, thereby reducing the possibility of imbalances in the image sizes of objects used for training.

[0046] Specifically, in the second selection, the second selection unit 172 extracts some pseudo labels from the pseudo labels to be discarded in the first selection based on the image size of the object. More specifically, in extracting the pseudo labels, the second selection unit 172 extracts some pseudo labels in order from the smallest image size of the object. This makes it possible to retain pseudo labels of objects with small image sizes as pseudo labels for learning. For example, from the pseudo labels to be discarded in the first selection, the top N% pseudo labels are extracted in order from the smallest size. Note that, here, the smaller the size of the object, the higher the ranking. The value N in the top N% may be determined appropriately through experiments or the like, and may be, for example, N%=50%.

[0047] Furthermore, the second selection unit 172 selects pseudo labels for learning based on reliability from among the pseudo labels extracted in ascending order of image size in the second selection. This allows pseudo labels with relatively high reliability to be left as pseudo labels for learning when pseudo labels of objects with small image sizes are left, enabling appropriate learning.

[0048] The method for selecting the training pseudo labels based on the reliability in the second selection may be the same as the method for selecting the training pseudo labels based on the reliability in the first selection. This prevents the processing for selecting the training pseudo labels from becoming complicated. In detail, the second selection unit 172 groups the pseudo labels extracted because the image size of the object is small into a high-reliability group and a low-reliability group based on the score. Then, the second selection unit 172 selects some pseudo labels from the high-reliability group as training pseudo labels using statistical processing.

[0049] In the second selection, grouping based on the scores may be performed using GMM, as in the first selection. In addition, as in the first selection, a method may be adopted in which the score that maximizes the logarithmic likelihood is used as a threshold to determine, by statistical processing, some pseudo labels to be left as learning pseudo labels from the high-confidence group.

[0050] The integrating unit 173 integrates the selection result of the first selecting unit 171 and the selection result of the second selecting unit 172. Specifically, both the pseudo labels selected as the learning pseudo labels by the first selecting unit 171 and the pseudo labels selected as the learning pseudo labels by the second selecting unit 172 are finalized as the learning pseudo labels. After the finalization, the finalized learning pseudo labels are used for learning, and the pseudo labels that are not selected as the learning pseudo labels are discarded. Note that the pseudo labels to be discarded here are excluded from learning together with the image data corresponding to the pseudo labels.

[0051] The unsupervised loss calculation unit 18 (see FIG. 2) calculates the unsupervised loss Lu. The unsupervised loss Lu is the same as the consistency loss described above. The unsupervised loss (consistency loss) Lu is the loss (error) between the inference result obtained by inputting unsupervised data (more specifically, data that has been subjected to strong data augmentation) into the teacher model 12 and the learning pseudo labels obtained using the student model 11. The unsupervised loss Lu is calculated by comparing the labels (inference results) of the teacher model 12 and the student model 11 for images with the same original image. For this reason, among the labels obtained as the inference result of the teacher model 12, labels that correspond to pseudo labels that were discarded without being adopted as learning pseudo labels in the student model 11 are discarded in the same way as in the case of the student model 11 and are not used as learning targets.

[0052] The unsupervised loss Lu may be calculated by a known method, such as mean square error or KL divergence. The smaller the value of the unsupervised loss Lu, the more robust the student model 11 is against perturbations. By training the student model 11 to be robust against perturbations, it is possible to acquire more abstract invariance for similar input data.

[0053] The update unit 19 updates the parameters of the student model 11 based on the unsupervised loss Lu and the supervised loss Ls. In detail, the update unit 19 calculates a weighted sum of the supervised loss Ls and the unsupervised loss Ls (Ls + λuLu; λu is a weighting coefficient). Then, the update unit 19 updates the parameters (weights) of the student model 11 using backpropagation so that the calculated weighted sum (total loss) is minimized. Note that the parameters of the teacher model 12 are also updated in accordance with the update of the parameters of the student model 11.

[0054] As can be seen from the above, the learning device 100 performs different processing depending on whether or not the input data has a label (correct label). In the case of labeled data (supervised data), inference is performed by the student model 11, and the supervised loss Ls is calculated using the inference result (see the dashed-dotted arrow in FIG. 2). In the case of unlabeled data (unsupervised data), different processes are performed on the unlabeled data to generate first unsupervised data and second unsupervised data. In the example shown in FIG. 2, the first unsupervised data is data that has undergone weak data augmentation. By inputting the first unsupervised data to the student model 11, pseudo labels for training are generated (see the dashed-dotted arrow in FIG. 2). The second supervised data is data that has undergone strong data augmentation. The inference result obtained by inputting the second unsupervised data to the supervised model 12 is used to calculate the unsupervised loss Lu, which is the loss for the pseudo labels for training (see the solid-line arrow in FIG. 2).

[0055] In the above description, the selection of learning pseudo labels for a plurality of images is performed collectively, but this is merely an example. For example, the selection of learning pseudo labels may be performed sequentially for each image. In this case, a threshold may be set in advance to enable selection of whether or not to adopt the obtained pseudo label as the learning pseudo label for each image. Such a threshold may be calculated based on, for example, statistical processing or empirical rules.

[0056] <2. Learning Methods> Next, a learning method executed by the learning device 100 for learning a machine learning model (student model 11) using unsupervised data will be described.

[0057] [2-1. Overall flow of learning method] FIG. 6 is a flowchart illustrating the flow of a learning method executed by the learning device 100 (controller 1) according to an embodiment of the present invention. The learning method (learning flow) shown in FIG. 6 is realized by the controller 1 (more specifically, the processor) executing arithmetic processing in accordance with a learning program stored in memory 2. The learning method shown in FIG. 6 starts when the preparation of learning data is complete. The learning data includes supervised data and unsupervised data. The learning data may be already stored in memory 2 of the learning device 100, or may be available for external acquisition.

[0058] In step S1, the mini-batch generation unit 13 generates mini-batches 3 (see FIG. 3) that contain a mixture of supervised data and unsupervised data. When the generation of mini-batches 3 is complete, the process proceeds to the next step S2.

[0059] In step S2, processing is performed using student model 11 on the supervised data included in mini-batch 3. Details of the processing in step S2 will be described later. Note that there are multiple pieces of supervised data included in mini-batch 3, and processing is performed using student model 11 on all of the multiple pieces of supervised data. When the processing in step S2 using student model 11 is completed, the process proceeds to step S3.

[0060] In step S3, processing is performed using the teacher model 12 on the unsupervised data included in mini-batch 3. Details of the processing in step S3 will be described later. Note that there are multiple pieces of unsupervised data included in mini-batch 3, and processing is performed using the teacher model 12 on all of the multiple pieces of unsupervised data. The number of pieces of unsupervised data may be the same as or different from the number of supervised data. When the processing in step S3 is completed, the processing proceeds to step S4.

[0061] In step S4, processing is performed using the student model 11 on the unsupervised data included in mini-batch 3. Details of the processing in step S4 will be described later. As described above, there are multiple pieces of unsupervised data included in mini-batch 3, and processing is performed using the student model 11 on all of the multiple pieces of unsupervised data. When the processing in step S4 is completed, the process proceeds to step S5.

[0062] The order of steps S2, S3, and S4 is not limited to the order in this embodiment, and may be arbitrarily changed. In some cases, the configuration may be such that, while determining whether the data extracted from mini-batch 3 is supervised data or unsupervised data, one of the processes by student model 11 and / or teacher model 12 is executed for each data as needed, and the number of processes is the same for each data included in mini-batch 3.

[0063] In step S5, the unsupervised loss calculation unit 18 calculates the unsupervised loss Lu. The unsupervised loss Lu is calculated using the inference result of the teacher model 12 obtained in step S3 and the learning pseudo-labels obtained in step S4. Once the unsupervised loss Lu is calculated, the process proceeds to step S6.

[0064] In step S6, the update unit 19 calculates the total loss. Specifically, the total loss is the weighted sum of the supervised loss Ls obtained in step S2 and the unsupervised loss Lu obtained in step S5. Once the total loss is calculated, the process proceeds to the next step S7.

[0065] In step S7, the update unit 19 updates the parameters of the student model 11 using the backpropagation algorithm so as to minimize the overall loss. The parameters of the teacher model 12 are also updated using the exponential moving average of the updated parameters of the student model 11. When the parameter update process is completed, the process proceeds to the next step S8.

[0066] In step S8, the mini-batch generation unit 13 determines whether a predetermined number of learning times (the number of times mini-batches have been generated) has been reached. The predetermined number of learning times is determined appropriately depending on the amount of prepared training data. If the predetermined number of learning times has been reached (Yes in step S8), the learning process shown in FIG. 6 is terminated. If the predetermined number of learning times has not been reached (No in step S8), the process returns to step S1, and the processes from step S1 onwards are executed. Note that the process shown in FIG. 6 may be repeated a predetermined number of times (number of epochs) to complete the learning.

[0067] The teacher model 12, for which learning has been completed, is applied as a trained object detection model (AI model) to, for example, the above-mentioned in-vehicle periphery monitoring device. The trained AI model is loaded into the in-vehicle device, for example, by reading it from a recording medium or downloading it from a server device. Note that, although this embodiment is configured to use the teacher model 12 as the trained object detection model, it may also be configured to use the student model 11 as the trained object detection model.

[0068] [2-2. Supervised Data Processing] Fig. 7 is a flowchart illustrating detailed processing of step S2 in Fig. 6. That is, Fig. 7 is a flowchart illustrating detailed processing using student model 11 of supervised data.

[0069] In step S21, the student model 11 sequentially performs inference on the multiple supervised data included in the mini-batch 3. When the student model 11 has performed inference on all the supervised data included in the mini-batch 3, the process proceeds to the next step S22. Note that the supervised data input to the student model 11 may be configured to be subjected to data augmentation before being input to the student model 11. Furthermore, one piece of supervised data may be expanded into multiple pieces of supervised data by using data augmentation.

[0070] In step S22, the supervised loss calculation unit 16 calculates the supervised loss Ls using all the inference results calculated in step S21 and the correct labels of all the supervised data. The calculated supervised loss Ls is stored in the memory 2. This completes the processing of the supervised data using the student model 11 (the processing of step S2).

[0071] [2-3. Processing unsupervised data using a supervised model] Fig. 8 is a flowchart illustrating the detailed processing of step S3 in Fig. 6. That is, Fig. 8 is a flowchart illustrating the details of the processing using the teacher model 12 for unsupervised data.

[0072] In step S31, the second data augmentation unit 15 sequentially performs strong data augmentation on the plurality of unsupervised data included in the mini-batch 3. When the data augmentation process by the second data augmentation unit 15 has been performed on all of the unsupervised data included in the mini-batch 3, the process proceeds to the next step S32.

[0073] In step S32, the teacher model 12 sequentially performs inference on all unsupervised data (second unsupervised data) that has undergone strong data augmentation. The results obtained by the inference (inference results) are stored in memory 2. When the teacher model 12 has performed inference on all unsupervised data that has undergone strong data augmentation, processing using the teacher model 12 on the unsupervised data (processing of step S3) is completed.

[0074] [2-4. Processing unsupervised data using student models] Fig. 9 is a flowchart illustrating detailed processing of step S4 in Fig. 6. That is, Fig. 9 is a flowchart illustrating detailed processing using the student model 11 of unsupervised data.

[0075] In step S41, the first data augmentation unit 14 sequentially performs weak data augmentation on the plurality of unsupervised data included in the mini-batch 3. When the first data augmentation unit 14 has performed the data augmentation process on all of the unsupervised data included in the mini-batch 3, the process proceeds to the next step S42.

[0076] In step S42, the student model 11 sequentially performs inference on all of the unsupervised data (first unsupervised data) that has undergone weak data augmentation. As a result of the inference, pseudo labels are generated. The generated pseudo labels are stored in memory 2. Once the student model 11 has performed inference on all of the unsupervised data that has undergone weak data augmentation, the process proceeds to the next step S43.

[0077] In step S43, the label selection unit 17 (more specifically, the first selection unit 171) performs reliability determination using GMM using each score of all the generated pseudo labels. As a result of this reliability determination, the generated pseudo labels are divided into a high-reliability group and a low-reliability group, as described above. When the grouping based on the reliability determination is completed, the process proceeds to the next step S44.

[0078] In step S44, the label selection unit 17 (more specifically, the first selection unit 171) performs a process of selecting pseudo labels for training (first selection process) using the result of the reliability determination in step S43. In the first selection process, highly reliable pseudo labels are selected as pseudo labels for training from all the generated pseudo labels. More specifically, in the first selection process, some pseudo labels belonging to a high reliability group are selected as pseudo labels for training. For example, among the pseudo labels belonging to the high reliability group, a score that maximizes the logarithmic likelihood is set as a threshold, and pseudo labels having a score equal to or greater than this threshold are selected as pseudo labels for training. Pseudo labels that are not selected as pseudo labels for training by the first selection process are temporarily discarded. Discarding means that they will not be used for training. When the first selection process is completed, the process proceeds to the next step S45.

[0079] In step S45, the label selection unit 17 (more specifically, the second selection unit 172) extracts some pseudo labels from the pseudo labels that were discarded in the first selection process based on the image sizes of the objects to which the pseudo labels are assigned. Specifically, the top N% (N is an arbitrary value) of pseudo labels with smallest object image sizes are extracted. The image size of the object may be determined according to the size of the bounding box, and for example, the area of ​​the bounding box may be used. Once the extraction of pseudo labels for small object images is complete, the process proceeds to the next step S46.

[0080] In step S46, the label selection unit 17 (more specifically, the second selection unit 172) performs reliability determination using GMM with the scores of the pseudo labels extracted based on the image sizes of the objects, similar to step S43. The reliability determination classifies the pseudo labels extracted based on the image sizes of the objects into a high-reliability group and a low-reliability group. When the grouping based on the reliability determination is completed, the process proceeds to the next step S47.

[0081] In step S47, the label selection unit 17 (more specifically, the second selection unit 172) performs a process of selecting pseudo labels for learning (second selection process) using the result of the reliability determination in step S46. In the second selection process, highly reliable pseudo labels are selected as pseudo labels for learning from among the pseudo labels extracted based on the image size of the object (pseudo labels of the small object image). More specifically, in the second selection process, some pseudo labels belonging to the high reliability group are selected as pseudo labels for learning using the same method as in the first selection process. When the selection of pseudo labels for learning by the second selection process is completed, the process proceeds to the next step S48.

[0082] In step S48, the label selection unit 17 (more specifically, the integration unit 173) integrates the selection results of the first and second selection processes. Specifically, the learning pseudo labels selected in the first and second selection processes are determined as pseudo labels to be used for learning. Furthermore, among all pseudo labels obtained as inference results of the student model 11, the pseudo labels that were not selected as learning pseudo labels in the first and second selection processes are determined to be discarded. By determining the learning pseudo labels and the pseudo labels to be discarded, the processing of the unsupervised data using the student model 11 (the processing of step S4) is completed.

[0083] As can be seen from the above explanation, according to the learning method of this embodiment, pseudo labels with small object sizes (image sizes) that are likely to be discarded in the first selection process based solely on reliability are more likely to be added to the learning target by the additional second selection process. This prevents learning that is biased toward objects with large image sizes, and is expected to enable appropriate learning of objects with small image sizes. As a result, it is possible to generate an object detection model that can appropriately detect objects regardless of the image size of the object to be detected.

[0084] <3. Modifications> [3-1. First Modification] The present invention is configured to select a training pseudo label from a plurality of pseudo labels generated by a machine learning model (student model 11) based on the reliability of the pseudo label and the image size of the object to which the pseudo label is assigned. In the first modification, the plurality of pseudo labels are classified into a plurality of groups based on the image size of the object, and a training pseudo label is selected for each of the plurality of groups based on the reliability of the pseudo label.

[0085] In the configuration of this modification, as in the above-described embodiment, the selection of pseudo labels to be used for learning can be performed taking into consideration not only the reliability of the pseudo labels but also the image size of the object to which the pseudo labels are assigned. In other words, when selecting pseudo labels, it is possible to prevent the discarding of pseudo labels that are biased toward pseudo labels of small object images. As a result, it is possible to generate an object detection model (AI model) that can accurately detect objects regardless of the image size of the object.

[0086] Fig. 10 is a block diagram showing a detailed functional configuration of label selection unit 17A according to Modification 1. Fig. 11 is a flowchart illustrating processing using student model 11 of unsupervised data in learning device 100 according to Modification 1. The flowchart shown in Fig. 11 corresponds to a modification of the flowchart shown in Fig. 9 described above.

[0087] 10, the label selecting unit 17A includes a classifying unit 170, a first selecting unit 171A, a second selecting unit 172A, and an integrating unit 173A. The functions of these units will be described below together with the flowchart shown in FIG.

[0088] The processes in steps S41A and S42A are similar to those in steps S41 and S42 described above, and therefore a description of these processes will be omitted.

[0089] In step S43A, the classification unit 170 classifies the multiple pseudo labels generated by the student model 11 into multiple groups based on the image size of the object. The image size of the object is given, for example, by the area of ​​a bounding box. In detail, the multiple groups are a large object group for objects with large image sizes and a small object group for objects with small image sizes. Using a preset threshold value for the image size of the object as a criterion, the classification unit 170 classifies the pseudo labels of objects with image sizes equal to or larger than the threshold value into the large object group, and classifies the pseudo labels of objects with image sizes smaller than the threshold value into the small object group. Once the classification based on the image size of the object is complete, the process proceeds to the next step S44A.

[0090] In this modification, the number of groups classified by object image size is two. However, the number of groups classified by object image size may be three or more. However, by setting the number of groups to two as in this modification, it is possible to suppress an increase in processing load while making it easier for pseudo labels of objects with small image sizes to remain as pseudo labels for learning. In the following description, one of the large object group and the small object group classified by the classification unit 170 will be referred to as the first classification, and the other will be referred to as the second classification.

[0091] In step S44A, the first selection unit 171A performs a reliability determination using GMM using each score of the pseudo labels of the first classification. The reliability determination divides the pseudo labels of the first classification into a high reliability group and a low reliability group. When the grouping of the first classification based on the reliability determination is completed, the process proceeds to the next step S45A.

[0092] In step S45A, the first selection unit 171A performs a process of selecting pseudo labels for training (first selection process) using the result of the reliability determination in step S44A. In the first selection process, highly reliable pseudo labels are selected as pseudo labels for training from among the pseudo labels of the first classification. Specifically, in the first selection process, some pseudo labels belonging to a high reliability group are selected as pseudo labels for training. As in the above-described embodiment, the selection method may be a method in which the score that maximizes the logarithmic likelihood is set as a threshold. When the first selection process is completed, the process proceeds to the next step S46A.

[0093] In step S46A, the second selection unit 172A performs a reliability determination using GMM using each score of the pseudo labels of the second classification. The reliability determination separates the pseudo labels of the second classification into a high-reliability group and a low-reliability group. When the grouping of the second classification based on the reliability determination is completed, the process proceeds to the next step S47A.

[0094] In step S47A, the second selection unit 172A performs a process of selecting pseudo labels for training (second selection process) using the result of the reliability determination in step S46A. In the second selection process, highly reliable pseudo labels are selected as pseudo labels for training from among the pseudo labels of the second classification. Specifically, in the second selection process, some pseudo labels belonging to a high reliability group are selected as pseudo labels for training. The selection method may be, as in the first selection process in step S45A, a method in which the score that maximizes the logarithmic likelihood is used as a threshold. When the second selection process is completed, the process proceeds to the next step S48A.

[0095] In step S48A, the integration unit 173A integrates the selection results of the first and second selection processes. Specifically, the integration unit 173A determines the learning pseudo labels selected in the first and second selection processes as the pseudo labels to be used for learning. Furthermore, among all the pseudo labels obtained as the inference results of the student model 11, the integration unit 173A determines that the pseudo labels that were not selected as learning pseudo labels in the first and second selection processes are to be discarded. By determining the learning pseudo labels and the pseudo labels to be discarded, processing of the unsupervised data using the student model 11 is completed.

[0096] As can be seen from the above explanation, in this modified example, for each of the large object group and the small object group, the multiple pseudo labels within the group are divided into a high-reliability group and a low-reliability group based on their reliability, and a process is performed to select pseudo labels for learning from the high-reliability group.

[0097] This allows the selection of highly reliable pseudo labels for training from among the pseudo labels for large object images and the pseudo labels for small object images, thereby preventing training that is biased toward object image sizes and generating an object detection model that can appropriately detect objects regardless of the image size of the target object.

[0098] [3-2. Second Modification] In the above, the characteristic configuration of the present invention has been described as being used in part of semi-unsupervised learning, but the characteristic configuration of the present invention may also be used as a method for generating training data. The second modification discloses a method for generating supervised data with a correct answer label using a machine learning model from unsupervised data without a correct answer label.

[0099] In the training data generation method of the second modification, a plurality of unsupervised data are input to a machine learning model to generate a plurality of pseudo labels. Next, some pseudo labels are selected from the plurality of pseudo labels based on the reliability of the pseudo labels and the image size of the object to which the pseudo labels are assigned. Next, supervised data is generated using the selected pseudo labels as correct labels. A specific example will be described below.

[0100] Fig. 12 is a flowchart showing the flow of a training data generation method according to the second modification. The training data generation method shown in Fig. 12 is realized by a controller (more specifically, a processor included in the controller) of a training data generation device (not shown) executing arithmetic processing in accordance with a training data generation program stored in memory. The training data generation method shown in Fig. 12 starts when multiple image data without a correct answer label (training data candidate images that serve as supervised data) are prepared. Note that the training data candidate images may be already stored in the memory of the training data generation device or may be available for external acquisition.

[0101] In step S101, the machine learning model performs inference on all the training data candidate images in order. The machine learning model is an object detection model and is a trained model. As a result of the inference by the machine learning model, pseudo labels are generated. The generated pseudo labels are stored in memory. Once the inference by the machine learning model has been performed on all the training data candidate images, the process proceeds to the next step S102.

[0102] In step S102, a reliability determination is performed using GMM using the scores of all the generated pseudo labels. The generated pseudo labels are divided into a high-reliability group and a low-reliability group by this reliability determination. Once the grouping by the reliability determination is completed, the process proceeds to the next step S103.

[0103] In step S103, a first selection process of pseudo labels is performed using the result of the reliability determination in step S102. In the first selection process, some pseudo labels that belong to a high reliability group are selected from all the generated pseudo labels. For example, among the pseudo labels that belong to the high reliability group, the score that maximizes the logarithmic likelihood is set as a threshold, and pseudo labels that have a score equal to or greater than this threshold are selected as pseudo labels for training. Pseudo labels that remain unselected in the first selection process are temporarily discarded. When the first selection process is completed, the process proceeds to the next step S104.

[0104] In step S104, some pseudo labels are extracted from the pseudo labels discarded in the first selection process based on the image sizes of the objects to which the pseudo labels are assigned. Specifically, the top N% (N is an arbitrary value) of pseudo labels with smallest object image sizes are extracted. The image size of the object may be determined based on the size of the bounding box, for example, the area of ​​the bounding box. Once the extraction of pseudo labels for small object images is complete, the process proceeds to the next step S105.

[0105] In step S105, similar to step S102, the reliability of the pseudo labels extracted based on the image size of the object is determined by GMM using the pseudo label scores. The reliability determination divides the pseudo labels extracted based on the image size of the object into a high-reliability group and a low-reliability group. Once the grouping based on the reliability determination is complete, the process proceeds to the next step S106.

[0106] In step S106, a second selection process of pseudo labels is performed using the result of the reliability determination in step S105. In the second selection process, some pseudo labels belonging to a high reliability group are selected from the pseudo labels extracted based on the image size of the object (pseudo labels of the small object image). The selection method may be, as in step S103, a method in which the score that maximizes the logarithmic likelihood is used as a threshold. When the second selection process is completed, the process proceeds to the next step S107.

[0107] In step S107, the labels to be used in learning are determined. That is, learning data is generated. In detail, the pseudo labels selected in the first selection process and the second selection process are determined as correct labels. The image data to which the determined correct labels are attached is determined as supervised data. Note that the pseudo labels not selected in the first selection process and the second selection process are discarded and are not used as correct labels.

[0108] The image data determined as supervised data is stored in a memory as training data. The supervised data generated in this manner is used as training data as appropriate in a learning device that executes learning of a machine learning model.

[0109] As can be seen from the above explanation, according to the training data generation method of this example, pseudo labels of small-sized objects that are likely to be discarded in the first selection process based only on reliability are more likely to be added to the correct labels in the additional second selection process. For this reason, by using supervised data generated by the training data generation method, it is expected that learning biased toward large-sized objects will be suppressed and that learning of small-sized objects will also be performed appropriately.

[0110] <4. Things to keep in mind> The various technical features disclosed in the detailed description of the present invention may be modified in various ways without departing from the spirit of the technical creation. For example, at least a portion of the software-implemented configuration described above may be implemented by hardware. Furthermore, the multiple embodiments and modifications disclosed in the detailed description of the present invention may be combined to the extent possible. [Explanation of symbols]

[0111] 11. Student model (machine learning model) 12. Teacher model (another machine learning model) 100...Learning Device Ls: Supervised loss Lu···Unsupervised loss

Claims

1. A learning method for learning a machine learning model using unsupervised data that does not have a correct answer label, inputting the unsupervised data into the machine learning model to generate pseudo-labels; selecting a pseudo label for training from the generated pseudo labels based on the reliability of the pseudo label and the image size of the object to which the pseudo label is assigned; The learning method includes performing the learning using the learning pseudo labels.

2. performing a first selection of the training pseudo labels based on the confidence scores from the pseudo labels; The learning method according to claim 1 , further comprising: performing a second selection to select the learning pseudo label from the pseudo labels that were not selected as the learning pseudo label in the first selection and are to be discarded based on an image size of the object.

3. In the first selection, Dividing the pseudo labels into a high-confidence group and a low-confidence group; The learning method according to claim 2 , wherein the pseudo labels for learning are selected from the high confidence group.

4. The learning method according to claim 3 , wherein some of the pseudo labels in the high-reliability group are selected as the learning pseudo labels using statistical processing.

5. In the second selection, extracting a portion of the pseudo labels from the discarded pseudo labels based on an image size of the object; The learning method according to claim 2 , further comprising selecting the learning pseudo labels from the extracted pseudo labels based on the reliability.

6. The learning method according to claim 5 , wherein in the extraction of the pseudo labels, some of the pseudo labels are extracted in ascending order of image size of the object.

7. The learning method according to claim 5 , wherein a method for selecting the training pseudo labels based on the reliability in the second selection is the same as a method for selecting the training pseudo labels based on the reliability in the first selection.

8. classifying the plurality of pseudo labels into a plurality of groups based on image sizes of the objects; The learning method according to claim 1 , wherein the pseudo labels for learning are selected for each of the plurality of groups based on the reliability of the pseudo labels.

9. The learning method according to claim 8 , wherein the plurality of groups are a large object group in which the image size of the object is large and a small object group in which the image size of the object is small.

10. In each of the large object group and the small object group, Dividing the plurality of pseudo labels in the group into a high-confidence group having high confidence and a low-confidence group having low confidence based on the confidence; The method of claim 9 , wherein the pseudo labels for training are selected from the high confidence group.

11. performing different processes on the unsupervised data to generate first unsupervised data and second unsupervised data; inputting the first unsupervised data into the machine learning model to generate the training pseudo labels; calculating an unsupervised loss, which is a loss between an inference result obtained by inputting the second unsupervised data into a machine learning model other than the machine learning model, and the training pseudo label; The learning method according to claim 1 , further comprising updating parameters of the machine learning model based on the unsupervised loss.

12. Calculating a supervised loss, which is the loss between an inference result obtained by inputting supervised data with the correct label into the machine learning model and the correct label; The learning method according to claim 11 , wherein the parameters are updated based on the unsupervised loss and the supervised loss.

13. A learning device that learns a machine learning model using unsupervised data that does not have a correct answer label, inputting the unsupervised data into the machine learning model to generate pseudo-labels; selecting a pseudo label for training from the generated pseudo labels based on the reliability of the pseudo label and the image size of the object to which the pseudo label is assigned; A learning device that performs the learning using the learning pseudo labels.

14. A program that causes a learning device to execute a learning method for learning a machine learning model using unsupervised data that does not have a correct answer label, The learning device inputting the unsupervised data into the machine learning model to generate pseudo-labels; selecting a pseudo label for training from the generated pseudo labels based on the reliability of the pseudo label and the image size of the object to which the pseudo label is assigned; performing the learning using the learning pseudo labels; A learning program that serves as a means to implement the above.

15. A method for generating supervised image data with a correct answer label from unsupervised image data without a correct answer label using a machine learning model, comprising: inputting a plurality of the unsupervised image data into the machine learning model to generate a plurality of pseudo labels; selecting some of the pseudo labels from the plurality of pseudo labels based on the reliability of the pseudo labels and the image size of the object to which the pseudo labels are assigned; The learning data generation method generates the supervised image data using the selected portion of pseudo labels as the correct labels.

Images