Learning device, learning method, and program

Abstract

In this training device, an order setting means sets a use order for unlabeled data to be used in the training of a model. A data augmentation means sets a data augmentation corresponding to the use order and applies the set data augmentation to the unlabeled data. A training means trains the model using the data obtained by the data augmentation.

Description

Learning device, learning method, and recording medium

[0001] The present disclosure relates to techniques that can be used to train machine learning models.

[0002] As a method for training a machine learning model, for example, a method called semi-supervised learning has been proposed, in which a machine learning model is trained using a small amount of data to which a label indicating a correct answer is assigned and a large amount of data to which the label is not assigned. Also, for example, Patent Literature 1 discloses a method corresponding to the semi-supervised learning described above.

[0003] Specifically, Patent Document 1 discloses a method of acquiring a trained model by performing training using supervised images in which a class corresponding to the type of object to be identified is assigned to each pixel, generating a new supervised image by assigning a class to each pixel of an image selected from unsupervised images to which no class has been assigned, and retraining the trained model using the new supervised image.

[0004] Japanese Patent Application Laid-Open No. 2020-154602

[0005] However, the method disclosed in Patent Document 1 has a problem in that, for example, the accuracy of the classes assigned to unsupervised images is low, which may result in a decrease in the inference accuracy of the trained model.

[0006] In other words, the method disclosed in Patent Document 1 poses a problem corresponding to the aforementioned issue that, when a trained model is used in practical systems such as biometric authentication, object recognition, and automatic control, the operating accuracy may be reduced.

[0007] One object of the present disclosure is to provide a learning device that can improve the operating accuracy when a trained model is used in a practical system.

[0008] In one aspect of the present disclosure, a learning device includes an order setting means for setting an order of use of unlabeled data to be used for model training, a data augmentation means for setting data augmentation according to the order of use and applying the set data augmentation to the unlabeled data, and a learning means for training the model using data obtained by the data augmentation.

[0009] In another aspect of the present disclosure, a learning device includes an order setting means for setting an order of use of unlabeled data to be used for model training, a data augmentation means for setting data augmentation according to the order of use and applying the set data augmentation to the unlabeled data, a display screen generation means for generating a display screen that prompts a user to set the data augmentation, and a learning means for training the model using data obtained by the data augmentation that corresponds to the setting content set on the display screen.

[0010] In yet another aspect of the present disclosure, a training method includes setting an order of use of unlabeled data to be used in training a model, setting data augmentation according to the order of use, applying the set data augmentation to the unlabeled data, and training the model using the data obtained by the data augmentation.

[0011] In yet another aspect of the present disclosure, a training method includes setting an order of use of unlabeled data to be used in training a model, setting data augmentation according to the order of use, applying the set data augmentation to the unlabeled data, generating a display screen that prompts a user to set the data augmentation, and training the model using data obtained by the data augmentation that corresponds to the settings set on the display screen.

[0012] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute a process of setting an order of use of unlabeled data to be used for training a model, setting data augmentation according to the order of use, applying the set data augmentation to the unlabeled data, and training the model using data obtained by the data augmentation.

[0013] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute a process of setting an order of use of unlabeled data to be used for model training, setting data augmentation according to the order of use, applying the set data augmentation to the unlabeled data, generating a display screen that prompts a user to set the data augmentation, and training the model using data obtained by the data augmentation that corresponds to the settings set on the display screen.

[0014] According to the present disclosure, it is possible to improve the operational accuracy when using a trained model in a practical system.

[0015] 1 is a diagram showing a schematic configuration of a learning device according to the present disclosure. FIG. 1 is a block diagram showing an example of a hardware configuration of a learning device according to the present disclosure. FIG. 2 is a diagram showing an overview of machine learning performed by a learning device according to the present disclosure. FIG. 3 is a block diagram showing an example of a functional configuration of a learning device according to the present disclosure. FIG. 4 is a diagram showing an example of a distance matrix. FIG. 5 is a diagram showing an example of a method for setting the order of use of unlabeled data. FIG. 6 is a diagram showing another example of a method for setting the order of use of unlabeled data. FIG. 7 is a diagram showing another example of a method for setting the order of use of unlabeled data. FIG. 8 is a diagram showing another example of a method for setting the order of use of unlabeled data. FIG. 9 is a diagram showing another example of a data augmentation. FIG. 10 is a diagram showing another example of data augmentation. A flowchart showing an example of processing performed in a learning device according to the present disclosure. A diagram for explaining an example of the configuration of a data augmentation unit. A diagram showing an example of a display screen that prompts a user to make settings related to data augmentation. A block diagram showing another example of the functional configuration of a learning device according to the present disclosure. A flowchart showing another example of processing performed in a learning device according to the present disclosure.

[0016] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0017] First Embodiment [Overall Configuration] FIG. 1 is a diagram illustrating the overall configuration of a learning device according to the present disclosure. The learning device 100 learns a machine learning model based on input learning data. The learning data includes a small amount of data to which a label indicating a correct answer is assigned and a large amount of data to which the label is not assigned. Hereinafter, data including images to which a label indicating a correct answer is assigned will also be referred to as labeled data. Hereinafter, data including images to which a label indicating a correct answer is not assigned will also be referred to as unlabeled data. The learning device 100 can train a machine learning model using a small amount of labeled data and a large amount of unlabeled data. The learning device 100 can also train a machine learning model by using a large amount of unlabeled data in order of "closeness" to the labeled data. The learning device 100 can also perform data augmentation suitable for learning on unlabeled data for which a usage order has been set.

[0018] [Hardware Configuration] Fig. 2 is a block diagram showing an example of the hardware configuration of a learning device 100 according to the present disclosure. As shown in Fig. 2, the learning device 100 includes an interface (IF) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0019] The IF 111 inputs and outputs data to and from an external device. For example, labeled data and unlabeled data used for training a machine learning model are input to the training device 100 via the IF 111. Also, for example, a trained machine learning model obtained by performing training, which will be described later, is output to an external device via the IF 111.

[0020] Labeled data corresponds to, for example, data in which an image including a recognition target is assigned a label indicating the type of the recognition target as a correct answer, and attribute information indicating the attributes of the image. Unlabeled data corresponds to, for example, data in which attribute information indicating the attributes of an image is assigned to an image including a recognition target. The attribute information may include at least one piece of information indicating the attributes of the image, such as the position of the camera when the recognition target was captured, the position of the recognition target, the angle of the recognition target, the type of camera, the time of capture, the time period during which the recognition target was captured, the image quality, and the zoom magnification.

[0021] The processor 112 is a computer such as a CPU (Central Processing Unit) and controls the entire learning device 100 by executing a pre-prepared program. Specifically, the processor 112 performs learning of a machine learning model using, for example, labeled data and unlabeled data. The processor 112 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array).

[0022] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0023] Recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from learning device 100. Recording medium 114 records various programs executed by processor 112. When learning device 100 executes various processes, the programs recorded on recording medium 114 are loaded into memory 113 and executed by processor 112.

[0024] The DB 115 stores, for example, data input via the IF 111 and processing results obtained by processing by the processor 112. Specifically, the DB 115 stores the learning progress status, which will be described later.

[0025] [Overview] FIG. 3 is a diagram illustrating an overview of machine learning performed by a learning device according to the present disclosure. Note that the inner ellipse represents labeled data, and the outer ellipse represents unlabeled data. The size of the ellipse represents the amount of data. The larger the amount of data, the larger the size of the ellipse. In the example of FIG. 3 , the learning device 100 performs training of a machine learning model in n steps, from the first step to the nth step (n≧2). In the first step, the learning device 100 performs training using labeled data and some of the unlabeled data. In the kth step (2≦k≦n), the learning device 100 performs training using each of the unlabeled data used in training up to the (k−1)th step and at least some of the remaining unlabeled data. In other words, the learning device 100 gradually increases the amount of unlabeled data used in training the machine learning model from the first step to the nth step. In addition, the learning device 100 sets the order in which unlabeled data is used in each step in order of proximity to the labeled data. Furthermore, the learning device 100 performs data extension on unlabeled data for which a usage order has been set, in accordance with the usage order.

[0026] [Functional Configuration] Figure 4 is a block diagram showing an example of the functional configuration of a learning device according to the present disclosure. The learning device 100 includes a learning control unit 31, a data expansion unit 32, an inference unit 33, a loss calculation unit 34, a loss calculation unit 35, a parameter acquisition unit 36, and a learning progress status storage unit 37. The learning control unit 31 also includes a distance calculation unit 31A, a curriculum setting unit 31B, and a learning data setting unit 31C. The inference unit 33 also includes an inference model 33A constructed as a machine learning model.

[0027] The learning control unit 31 functions as an order setting unit. Labeled data LD and unlabeled data UD are input to the learning control unit 31. The learning control unit 31 sets a use order UJ of the unlabeled data UD used to train the inference model 33A based on the distance between the labeled data LD and the unlabeled data UD. The learning control unit 31 identifies at least one piece of data from each piece of unlabeled data UD to be used in training the pth (1≦p≦n) step in the use order UJ, and sets the identified data as unlabeled data UDE. The learning control unit 31 outputs the use order UJ and the unlabeled data UDE to the data extension unit 32.

[0028] The labeled data LD corresponds to data in which a label LB indicating the type of the recognition target as a correct answer and attribute information LZ indicating the attributes of the image LG are assigned to an image LG including the recognition target. The attribute information LZ may include at least one piece of information indicating the attributes of the image LG, such as the position of the camera when the recognition target was photographed, the position of the recognition target, the angle of the recognition target, the type of camera, the time of photographing, the time period during which the recognition target was photographed, the image quality, and the zoom magnification.

[0029] The unlabeled data UD corresponds to data in which attribute information UZ indicating the attributes of an image UG including the target is added to the image UG. The attribute information UZ may include at least one piece of information indicating the attributes of the image UG, such as the position of the camera when the target was photographed, the position of the target, the angle of the target, the type of camera, the time of photographing, the time period during which the target was photographed, the image quality, and the zoom magnification.

[0030] The distance calculation unit 31A classifies the labeled data LD into classes. Then, the distance calculation unit 31A calculates the distance between a data group of the labeled data LD for each class and the unlabeled data UD. Note that, hereinafter, the data group of the labeled data LD is also referred to as a labeled data set LDS.

[0031] Specifically, the distance calculation unit 31A calculates the distance between the labeled data set LDS and the unlabeled data UD using, for example, any of the methods of feature point matching, cosine similarity, and softmax score.

[0032] For example, when feature point matching is used, the distance calculation unit 31A detects feature points in the labeled data LD and feature points in the unlabeled data UD and calculates feature amounts from the feature points.The distance calculation unit 31A then matches feature points with high similarity.The distance calculation unit 31A calculates the distance between the labeled data set LDS and the unlabeled data UD based on the similarity and the distance between the feature points.

[0033] When using cosine similarity, the distance calculation unit 31A calculates a feature vector of the labeled data LD and a feature vector of the unlabeled data UD, and obtains cosine similarity from the two feature vectors. The distance calculation unit 31A uses the calculated cosine similarity as the distance between the labeled data set LDS and the unlabeled data UD.

[0034] Furthermore, when the softmax score is used, the distance calculation unit 31A calculates the softmax score by performing inference on the unlabeled data UD using a model that has been trained on the labeled data LD. The distance calculation unit 31A uses the calculated softmax score as the distance between the labeled data set LDS and the unlabeled data UD.

[0035] When calculating distances by feature point matching or cosine similarity, the distance calculation unit 31A sets points to be used for calculating distances from the labeled data set LDS, and calculates the distances between points in the labeled data set LDS and points in the unlabeled data UD.

[0036] For example, the distance calculation unit 31A may calculate the distance between the average of the feature points of the plurality of labeled data LD included in the labeled data set LDS and the feature point of the unlabeled data UD. Alternatively, the distance calculation unit 31A may select a feature point from the feature points of the plurality of labeled data LD included in the labeled data set LDS that has the smallest distance from the feature point of the unlabeled data UD, and calculate the distance. Alternatively, the distance calculation unit 31A may calculate the distance between a representative point obtained by approximating the feature points of the plurality of labeled data LD included in the labeled data set LDS with a Gaussian distribution and the feature point of the unlabeled data UD. The representative point is generated based on, for example, a representative value such as the mean or median used in statistical analysis.

[0037] The distance calculation unit 31A calculates the distance between the labeled data set LDS of each class and the unlabeled data UD, and generates a distance matrix DM.

[0038] FIG. 5 is a diagram illustrating an example of a distance matrix. In FIG. 5, the distance between the labeled dataset LDS and the unlabeled data UD is represented by the softmax score. The larger the value of the softmax score, the closer the distance between the labeled dataset LDS and the unlabeled data UD. For example, in FIG. 5, the distance between the labeled dataset of class A and the unlabeled data a is "0.7," the distance between the labeled dataset of class B and the unlabeled data a is "0.2," and the distance between the labeled dataset of class C and the unlabeled data a is "0.1." Note that, according to this embodiment, for example, in the early stages of model learning, it is desirable that only the labeled data LD is included in the labeled dataset LDS. Furthermore, according to this embodiment, for example, when model learning has progressed to a certain extent, some or all of the unlabeled data UD used in learning may be added to the labeled dataset LDS.

[0039] The curriculum setting unit 31B sets the order UJ of use of the unlabeled data UD used to train the inference model 33A based on the distance matrix DM generated by the distance calculation unit 31A.

[0040] 6 to 8 are diagrams showing specific examples of methods for setting the order of use of unlabeled data. In Fig. 6 to Fig. 8, the distance between the labeled data set LDS and the unlabeled data UD is shown as a softmax score.

[0041] 6 shows an example of setting the use order UJ in descending order of distance from the entire distance matrix DM. For example, in FIG. 6, the distance between the labeled data set of class B and unlabeled data c is the closest at "0.9." The distance between the labeled data set of class A and unlabeled data e is the second closest at "0.8." The distance between the labeled data set of class A and unlabeled data a is the third closest at "0.7." Therefore, in the example shown in FIG. 6, the curriculum setting unit 31B sets the use order of the unlabeled data in the following order: "unlabeled data c" → "unlabeled data e" → "unlabeled data a."

[0042] 7 shows an example of extracting the closest unlabeled data UD for each labeled data set LDS of each class and setting the use order UJ. First, the curriculum setting unit 31B extracts one piece of closest unlabeled data for each labeled data set of each class. Next, the curriculum setting unit 31B extracts the closest unlabeled data for each labeled data set of each class from the remaining unlabeled data after excluding the extracted unlabeled data.

[0043] For example, in the first iteration of Fig. 7, the curriculum setting unit 31B extracts "unlabeled data e" with the shortest distance from among the combinations of the labeled data set and each unlabeled data of class A. The curriculum setting unit 31B also extracts "unlabeled data c" with the shortest distance from among the combinations of the labeled data set and each unlabeled data of class B. The curriculum setting unit 31B also extracts "unlabeled data d" with the shortest distance from among the combinations of the labeled data set and each unlabeled data of class C.

[0044] Then, in the second round of processing shown in FIG. 7 , the curriculum setting unit 31B extracts the closest unlabeled data for each labeled data set of each class from the remaining unlabeled data excluding "unlabeled data e," "unlabeled data c," and "unlabeled data d." Specifically, the curriculum setting unit 31B extracts the closest "unlabeled data a" from among the combinations of the labeled data set of class A and the remaining unlabeled data. The curriculum setting unit 31B also extracts the closest "unlabeled data b" from among the combinations of the labeled data set of class B and the remaining unlabeled data. The curriculum setting unit 31B also extracts the closest "unlabeled data f" from among the combinations of the labeled data set of class C and the remaining unlabeled data.

[0045] The curriculum setting unit 31B performs the above-described processing and sets the order of use of the unlabeled data in the order in which the unlabeled data is extracted.

[0046] 8 shows an example of extracting unlabeled data UD for each labeled data set LDS of each class in order of proximity and setting the order of use UJ. Note that in the method of FIG. 8, the same unlabeled data UD will be used multiple times.

[0047] For example, in the first iteration of FIG. 8 , the curriculum setting unit 31B extracts the "unlabeled data e" with the shortest distance from among the combinations of the labeled data set of class A and each unlabeled data. The curriculum setting unit 31B also extracts the "unlabeled data c" with the shortest distance from among the combinations of the unlabeled data of class B and each unlabeled data. The curriculum setting unit 31B also extracts the "unlabeled data d" with the shortest distance from among the combinations of the labeled data set of class C and each unlabeled data.

[0048] Next, in the second iteration of Figure 8, the curriculum setting unit 31B extracts "unlabeled data a" with the second closest distance from among the combinations of the labeled data set and each unlabeled data of class A. The curriculum setting unit 31B also extracts "unlabeled data b" with the second closest distance from among the combinations of the labeled data set and each unlabeled data of class B. The curriculum setting unit 31B also extracts "unlabeled data f" with the second closest distance from among the combinations of the labeled data set and each unlabeled data of class C.

[0049] 8, the curriculum setting unit 31B extracts "unlabeled data f" with the third closest distance from the combinations of the labeled data set and each unlabeled data of class A. The curriculum setting unit 31B also extracts "unlabeled data d" with the third closest distance from the combinations of the labeled data set and each unlabeled data of class B. The curriculum setting unit 31B also extracts "unlabeled data e" with the third closest distance from the combinations of the labeled data set and each unlabeled data of class C.

[0050] The curriculum setting unit 31B performs the above-described processing and sets the order of use of the unlabeled data in the order in which the unlabeled data is extracted.

[0051] The learning data setting unit 31C acquires, from each unlabeled data UD, unlabeled data UDE to be used in learning for the pth step in the use order UJ. The learning data setting unit 31C outputs the use order UJ and the unlabeled data UDE to the data extension unit 32. The learning data setting unit 31C acquires, from each unlabeled data UD, unlabeled data UDF to be used in learning for the (p+1)th step in the use order UJ. The learning data setting unit 31C outputs the unlabeled data UDF to the data extension unit 32 at an output timing set based on the learning progress status LSJ (described below) for the pth step. The learning data setting unit 31C does not perform processing related to outputting the unlabeled data UDF in the nth step of learning (p=n).

[0052] The learning data setting unit 31C can set the output timing of the unlabeled data for the (p+1)th step based on, for example, the pseudo label PL assignment rate PGR (described later) included in the learning progress status LSJ for the pth step. Specifically, the learning data setting unit 31C can set, for example, the timing when the pseudo label PL assignment rate PGR in the pth step learning becomes equal to or greater than a predetermined threshold as the output timing of the unlabeled data UDF. Alternatively, the learning data setting unit 31C can set, for example, the timing when the pseudo label PL assignment rate PGR in the pth step learning falls within a predetermined range as the output timing of the unlabeled data UDF.

[0053] The learning data setting unit 31C can set the output timing of the unlabeled data UDF based on, for example, a learning time LT (described later) included in the learning progress status LSJ of the pth step. Specifically, the learning data setting unit 31C can set, for example, the timing when the learning time LT of the pth step becomes equal to or longer than a predetermined time as the output timing of the unlabeled data UDF.

[0054] The learning data setting unit 31C can set the output timing of the unlabeled data UDF based on, for example, the number of parameter updates AN (described later) included in the learning progress status LSJ of the pth step. Specifically, the learning data setting unit 31C can set, for example, the timing when the number of parameter updates AN in the pth step learning becomes equal to or greater than a predetermined number as the output timing of the unlabeled data UDF.

[0055] The training data setting unit 31C can set the output timing of the unlabeled data UDF based on, for example, at least one of the labeled loss LL and the unlabeled loss UL (both of which will be described later) included in the training progress status LSJ of the pth step. Specifically, the training data setting unit 31C can set, for example, the timing when the change in the labeled loss LL in the training of the pth step falls within a predetermined range as the output timing of the unlabeled data UDF. Alternatively, the training data setting unit 31C can set, for example, the timing when the change in the unlabeled loss UL in the training of the pth step falls within a predetermined range as the output timing of the unlabeled data UDF. Alternatively, the training data setting unit 31C can set, for example, the timing when the change in the weighted sum of the labeled loss LL in the training of the pth step and the unlabeled loss UL in the training of the pth step falls within a predetermined range as the output timing of the unlabeled data UDF.

[0056] The data extension unit 32 functions as a data extension means. The data extension unit 32 sets data extension according to the use order UJ. The data extension unit 32 applies the data extension set as described above to the labeled data LD and / or the unlabeled data UDE to generate at least one unlabeled data UDQ as additional data to be used in the p-th step of learning in the use order UJ. The data extension unit 32 sequentially outputs the unlabeled data UDE and UDQ to the inference unit 33. The data extension unit 32 does not perform processing related to the generation and output of unlabeled data UDQ in the n-th step of learning (p=n).

[0057] Here, we will explain a specific example of the processing performed by the data extension unit 32. Note that in the following specific example, we will explain that in the first step, learning is performed using labeled data, and then learning is performed using unlabeled data.

[0058] First, the process performed by the data extension unit 32 in the first step of learning (when p=1) will be described.

[0059] The data extension unit 32 acquires labeled data LD1 to be used in the first step of learning from each labeled data LD. Furthermore, the data extension unit 32 acquires unlabeled data UDE1 to be used immediately after learning using labeled data LD1 from each unlabeled data UD or each unlabeled data UDE based on the use order UJ. The unlabeled data UDE1 corresponds to the data to be used first in the first step of learning in the use order UJ.

[0060] The data expansion unit 32 sets the element EL1 to be changed by data expansion based on attribute information LZ1 indicating the attributes of image LG1 contained in labeled data LD1 and attribute information UZE1 indicating the attributes of image UGE1 contained in unlabeled data UDE1.

[0061] Specifically, for example, if the attribute included in both the attribute information LZ1 and UZE1 is the position of the camera when the recognition target was photographed, the data expansion unit 32 sets the viewpoint position as the element EL1. Furthermore, for example, if the attribute included in both the attribute information LZ1 and UZE1 is the angle of the recognition target when the recognition target was photographed, the data expansion unit 32 sets the rotation angle of the recognition target as the element EL1. Furthermore, for example, if the attribute included in both the attribute information LZ1 and UZE1 is the shooting time or shooting time period when the recognition target was photographed, the data expansion unit 32 sets brightness as the element EL1. Furthermore, for example, if the attribute included in both the attribute information LZ1 and UZE1 is the image quality when the recognition target was photographed, the data expansion unit 32 sets resolution as the element EL1. Furthermore, for example, if the attribute included in both the attribute information LZ1 and UZE1 is the zoom magnification when the recognition target was photographed, the data expansion unit 32 sets the size of the recognition target as the element EL1. In addition, if there are multiple attributes included in both the attribute information LZ1 and UZE1, the data expansion unit 32 can set multiple elements as the element EL1. For example, if the attributes included in both the attribute information LZ1 and UZE1 are the shooting time and image quality when the recognition target was photographed, the data expansion unit 32 can set brightness and resolution as the element EL1. In addition, if there are multiple attributes included in both the attribute information LZ1 and UZE1, the data expansion unit 32 can set one of the multiple attributes as the element EL1. For example, the data expansion unit 32 can set one attribute of the multiple attributes included in both the attribute information LZ1 and UZE1 that has the smallest entropy in class classification after data expansion as the element EL1.

[0062] According to the above-described process, the data extension unit 32 can set the element EL1 according to the attributes included in both the attribute information LZ1 and UZE1.

[0063] The data expansion unit 32 may set the element EL1 based on the image LG1 and the image UGE1 instead of the attribute information LZ1 and UZE1. Specifically, the data expansion unit 32 may set, for example, an element where the distance determined by the attributes and / or features extracted from the image LG1 and the attributes and / or features extracted from the image UGE1 is equal to or less than a predetermined value as the element EL1. Furthermore, the data expansion unit 32 may use, for example, the following method for calculating the distance between the attributes extracted from the image LG1 and the attributes extracted from the image UGE1.

[0064] The data expansion unit 32 may calculate an inter-attribute distance based on the difference between one attribute included in the attribute information LZ1 and the attribute included in the attribute information UZE1, and set the attribute with the smallest calculated inter-attribute distance as the element EL1. The data expansion unit 32 may also calculate the inter-attribute distance based on, for example, the difference in the positions of the image capture devices, the sum of squares of the differences in the positions of the image capture devices, or the weighted sum of squares of the differences in the positions of the image capture devices. The data expansion unit 32 may also calculate the inter-attribute distance based on, for example, the difference in the angles of the recognition objects, the sum of squares of the differences in the angles of the recognition objects, or the weighted sum of squares of the differences in the angles of the recognition objects. The data expansion unit 32 may also calculate the inter-attribute distance based on, for example, the difference in the image capture times. It is desirable for the data expansion unit 32 to use, as a method for calculating the inter-attribute distance, a method capable of converting multiple parameters expressed in different units into distances.

[0065] The data expansion unit 32 sets the strength ST1 of data expansion to be applied to the image LG1 based on how the attributes used to set the element EL1 have changed from the image LG1 to the image UGE1.

[0066] Specifically, for example, if the position of the imaging device used to set element EL1 changes from the left side to the right side of the recognition target between images LG1 and UGE1, the data expansion unit 32 sets the intensity ST1 so that the viewpoint is on a diagonal side or in front of the recognition target. Furthermore, for example, if the angle of the recognition target used to set element EL1 changes from 0 degrees to 30 degrees between images LG1 and UGE1, the data expansion unit 32 sets the intensity ST1 so that the rotation angle of the recognition target is greater than 0 degrees and less than 30 degrees. Furthermore, for example, if the shooting time period used to set element EL1 changes from morning to noon between images LG1 and UGE1, the data expansion unit 32 sets the intensity ST1 so that the increase in brightness of the entire image falls within a predetermined range. Furthermore, for example, if the zoom factor used to set element EL1 changes from 1x to 2x between image LG1 and image UGE1, the data expansion unit 32 sets strength ST1 so that the size of the recognition target is greater than 1x and less than 2x. Note that, for example, if element EL1 is set using multiple attributes, the data expansion unit 32 may set multiple strengths ST1 corresponding to each of the multiple attributes. For example, the data expansion unit 32 may input image LG1 to a model trained to change element EL1 contained in an image to obtain an output image, and set strength ST1 based on the output image. For example, the data expansion unit 32 may set strength ST1 based on the difference between the feature value associated with element EL1 extracted from image LG1 and the feature value associated with element EL1 extracted from image UGE1.

[0067] According to the above-described process, the data extension unit 32 can set the intensity ST1 such that an image is generated that represents the intermediate state of the change from image LG1 to image UGE1.

[0068] The data expansion unit 32 may set the intensity ST1 based on the attribute information LZ1 and UZE1 instead of the image LG1 and the image UGE1. Specifically, the data expansion unit 32 may set, for example, one intensity that falls within a predetermined range that is set according to the types of attributes included in both the attribute information LZ1 and UZE1 as the intensity ST1.

[0069] FIG. 9 is a diagram illustrating an example of data augmentation. The data augmentation unit 32 applies data augmentation to the image LG1, changing the element EL1 by strength ST1, thereby generating at least one labeled data LDQ1 as additional data to be used in the first step of learning (see FIG. 9). When generating multiple labeled data LDQ1, the data augmentation unit 32 preferably performs data augmentation such that the element EL1 is changed by multiple intensities ST1 that are different from each other. Furthermore, the data augmentation unit 32 may perform processing corresponding to either the "strong augmentation" or "weak augmentation" disclosed in, for example, Kihyuk Sohn, et al., "FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence" (hereinafter referred to as "NPD").

[0070] The data expansion unit 32 outputs the labeled data LDQ1 to the inference unit 33, and then outputs the unlabeled data UDE1 to the inference unit 33 (see FIG. 9). The labeled data LDQ1 preferably includes, for example, an image LGQ1 obtained by changing the element EL1 of the image LG1, attribute information LZ1, and a label LB1. Furthermore, it is desirable that the label LB1 be the same as the label included in the labeled data LD1.

[0071] The data extension unit 32 acquires unlabeled data UDE2 to be used at the end of the first step of learning from each unlabeled data UD or each unlabeled data UDE based on the use order UJ. Also, the data extension unit 32 acquires unlabeled data UDF2 to be used at the beginning of the second step of learning from each unlabeled data UD based on the use order UJ.

[0072] The data expansion unit 32 sets the element EL2 to be changed by data expansion based on attribute information UZE2 indicating the attributes of image UGE2 contained in label-less data UDE2 and attribute information UZF2 indicating the attributes of image UGF2 contained in label-less data UDF2.

[0073] Specifically, for example, if the attribute included in both the attribute information UZE2 and UZF2 is the position of the camera when the target was photographed, the data expansion unit 32 sets the viewpoint position as the element EL2. Furthermore, for example, if the attribute included in both the attribute information UZE2 and UZF2 is the angle of the target when it was photographed, the data expansion unit 32 sets the rotation angle of the target as the element EL2. Furthermore, for example, if the attribute included in both the attribute information UZE2 and UZF2 is the time or time period when the target was photographed, the data expansion unit 32 sets brightness as the element EL2. Furthermore, for example, if the attribute included in both the attribute information UZE2 and UZF2 is the image quality when the target was photographed, the data expansion unit 32 sets resolution as the element EL2. Furthermore, for example, if the attribute included in both the attribute information UZE2 and UZF2 is the zoom magnification when the target was photographed, the data expansion unit 32 sets the size of the target as the element EL2. If there are multiple attributes included in both the attribute information UZE2 and UZF2, the data expansion unit 32 can set multiple elements as the element EL2. For example, if the attributes included in both the attribute information UZE2 and UZF2 are the shooting time and image quality when the recognition target was photographed, the data expansion unit 32 can set brightness and resolution as the element EL2.

[0074] According to the above-described process, the data extension unit 32 can set the element EL2 based on the attributes of the unlabeled data UDE2 and UDF2 that are adjacent in the use order UJ. Furthermore, according to the above-described process, the data extension unit 32 can set the element EL2 according to the attributes included in both the attribute information UZE2 and UZF2.

[0075] The data expansion unit 32 may set the element EL2 based on the images UGE2 and UGF2 instead of the attribute information UZE2 and UZF2. Specifically, the data expansion unit 32 may set, as the element EL2, an element for which the distance determined by the attributes and / or features extracted from the image UGE2 and the attributes and / or features extracted from the image UGF2 is equal to or less than a predetermined value. Furthermore, the data expansion unit 32 may use, for example, the following method for calculating the distance between attributes as a method for calculating the distance between the attributes extracted from the image UGE2 and the attributes extracted from the image UGF2.

[0076] The data expansion unit 32 may calculate an inter-attribute distance corresponding to the distance based on the difference between one attribute included in the attribute information UZE2 and the attribute included in the attribute information UZF2, and set the attribute with the smallest calculated inter-attribute distance as the element EL2. The data expansion unit 32 may also calculate the inter-attribute distance based on, for example, the difference in position of the image capture device, the sum of squares of the differences in position of the image capture device, or the weighted sum of squares of the differences in position of the image capture device. The data expansion unit 32 may also calculate the inter-attribute distance based on, for example, the difference in angle of the recognition object, the sum of squares of the differences in angle of the recognition object, or the weighted sum of squares of the differences in angle of the recognition object. The data expansion unit 32 may also calculate the inter-attribute distance based on, for example, the difference in image capture time. It is desirable for the data expansion unit 32 to use, as a method for calculating the inter-attribute distance, a method capable of converting multiple parameters expressed in different units into distances.

[0077] The data expansion unit 32 sets the strength ST2 of data expansion to be applied to the image UGE2 based on how the attributes used to set the element EL2 have changed from the image UGE2 to the image UGF2.

[0078] Specifically, for example, if the position of the imaging device used to set element EL2 changes from the left side to the right side of the recognition target between images UGE2 and UGF2, the data expansion unit 32 sets the intensity ST2 so that the viewpoint is on a diagonal side or in front of the recognition target. Furthermore, for example, if the angle of the recognition target used to set element EL2 changes from 0 degrees to 30 degrees between images UGE2 and UGF2, the data expansion unit 32 sets the intensity ST2 so that the rotation angle of the recognition target is greater than 0 degrees and less than 30 degrees. Furthermore, for example, if the imaging time zone used to set element EL2 changes from morning to daytime between images UGE2 and UGF2, the data expansion unit 32 sets the intensity ST2 so that the increase in brightness of the entire image falls within a predetermined range. Furthermore, for example, if the zoom factor used to set element EL2 changes from 1x to 2x between image UGE2 and image UGF2, the data expansion unit 32 sets strength ST2 so that the size of the recognition target is greater than 1x and less than 2x. Note that, for example, if element EL2 is set using multiple attributes, the data expansion unit 32 may set multiple strengths ST2 corresponding to each of the multiple attributes. For example, the data expansion unit 32 may input image UGE2 to a model trained to change element EL2 included in an image to obtain an output image, and set strength ST2 based on the output image. For example, the data expansion unit 32 may set strength ST2 based on the difference between the feature amount related to element EL2 extracted from image UGE2 and the feature amount related to EL2 extracted from image UGF2.

[0079] According to the processing described above, the data extension unit 32 can set the strength ST2 based on how the attribute used to set the element EL2 has changed from the unlabeled data UDE2, which is relatively earlier in the usage order UJ, to the unlabeled data UDF2, which is relatively later in the usage order UJ. Furthermore, according to the processing described above, the data extension unit 32 can set the strength ST2 so as to generate an image that represents an intermediate state of change from image UGE2 to image UGF2.

[0080] The data extension unit 32 may set the intensity ST2 based on the attribute information UZE2 and UZF2 instead of the images UGE2 and UGF2. Specifically, the data extension unit 32 may set, for example, one intensity within a predetermined range set according to the types of attributes included in both the attribute information UZE2 and UZF2 as the intensity ST2.

[0081] FIG. 10 is a diagram showing another example of data extension. The data extension unit 32 applies data extension to the image UGE2, changing the element EL2 by strength ST2, to generate at least one unlabeled data UDQ2 as additional data to be used in the first-step learning (see FIG. 10 ). Note that, when generating multiple unlabeled data UDQ2, the data extension unit 32 preferably performs data extension such that the element EL2 is changed by multiple intensities ST2 that are different from each other. Furthermore, the data extension unit 32 may perform processing corresponding to either "strong extension" or "weak extension" as disclosed in the non-patent document NPD, for example, as the data extension described above.

[0082] The data expansion unit 32 sequentially outputs the unlabeled data UDE to the inference unit 33 in accordance with the use order UJ. After outputting the unlabeled data UDE2 to the inference unit 33, the data expansion unit 32 outputs the unlabeled data UDQ2 to the inference unit 33 (see FIG. 10). The unlabeled data UDQ2 preferably includes, for example, an image UGQ2 obtained by changing the element EL2 of the image UGE2, and attribute information UZE2.

[0083] Next, the processing performed by the data extension unit 32 in learning at each step from the second step to the (n-1)th step (when 2≦p≦n-1) will be described.

[0084] The data extension unit 32 acquires unlabeled data UDE3 to be used at the end of the p-th step of learning from each unlabeled data UD or each unlabeled data UDE based on the use order UJ. Also, the data extension unit 32 acquires unlabeled data UDF3 to be used at the beginning of the (p+1)-th step of learning from each unlabeled data UD based on the use order UJ.

[0085] The data expansion unit 32 sets the element EL3 to be changed by data expansion based on attribute information UZE3 indicating the attributes of image UGE3 included in unlabeled data UDE3 and attribute information UZF3 indicating the attributes of image UGF3 included in unlabeled data UDF3. The data expansion unit 32 can set the element EL3, for example, by performing a process similar to the process described above for setting element EL2. Therefore, the data expansion unit 32 can set the element EL3 based on the attributes of unlabeled data UDE3 and UDF3, which are adjacent in the usage order UJ. Furthermore, the data expansion unit 32 can set the element EL3 according to the attributes included in both attribute information UZE3 and UZF3.

[0086] The data expansion unit 32 sets the strength ST3 of data expansion to be applied to image UGE3 based on how the attributes used to set element EL3 changed from image UGE3 to image UGF3. The data expansion unit 32 can set the strength ST3 by, for example, performing a process similar to the process described above for setting strength ST2. Therefore, the data expansion unit 32 can set the strength ST3 based on how the attributes used to set element EL3 changed from unlabeled data UDE3, which is relatively earlier in the use order UJ, to unlabeled data UDF3, which is relatively later in the use order UJ. Furthermore, the data expansion unit 32 can set the strength ST3 so that an image representing an intermediate state of change from image UGE3 to image UGF3 is generated.

[0087] FIG. 11 is a diagram showing another example of data extension. The data extension unit 32 applies data extension to the image UGE3, changing the element EL3 by strength ST3, to generate at least one unlabeled data UDQ3 as additional data to be used in the pth step of learning (see FIG. 11). Note that, when generating multiple unlabeled data UDQ3, it is desirable for the data extension unit 32 to perform data extension such that the element EL3 is changed by multiple intensities ST3 that are different from each other. Furthermore, the data extension unit 32 may perform processing corresponding to either "strong extension" or "weak extension" as disclosed in the non-patent document NPD, for example, as the data extension described above.

[0088] The data expansion unit 32 may perform data expansion in which a predetermined element ELX is changed by a predetermined strength STX in each step from the first step to the (n-1)th step. When performing such data expansion, the data expansion unit 32 may, for example, perform a process of setting one of the elements EL1, EL2, and EL3 as the predetermined element ELX before the learning of the first step begins. Furthermore, when performing the above-described data expansion, the data expansion unit 32 may, for example, perform a process of calculating an average value of the strength of each step in the data expansion corresponding to the predetermined element ELX before the learning of the first step begins, and setting the calculated average value as the predetermined strength STX.

[0089] The data expansion unit 32 sequentially outputs the unlabeled data UDE to the inference unit 33 in accordance with the use order UJ. After outputting the unlabeled data UDE3 to the inference unit 33, the data expansion unit 32 outputs the unlabeled data UDQ3 to the inference unit 33 (see FIG. 11 ). The unlabeled data UDQ3 preferably includes, for example, an image UGQ3 obtained by changing the element EL3 of the image UGE3, and attribute information UZE3.

[0090] The inference unit 33 obtains an inference result LR by inputting the image LG included in the labeled data LD into the inference model 33A, and outputs the obtained inference result LR to the loss calculation unit 34. The inference unit 33 can obtain, as the inference result LR, for example, an inference score indicating the likelihood of the label LB.

[0091] The inference unit 33 obtains an inference result UR by inputting an image UG included in the unlabeled data UD into the inference model 33A, generates a pseudo label PL based on the inference result UR, and assigns the generated pseudo label PL to the image UG. The inference unit 33 also outputs the inference result UR and the pseudo label PL to the loss calculation unit 35. The inference unit 33 can obtain, as the inference result UR, for example, an inference score indicating the likelihood of each class. The inference unit 33 can also generate, for example, a pseudo label PL indicating one class corresponding to the largest inference score MS included in the inference result UR.

[0092] Note that the inference unit 33 does not generate or assign a pseudo label PL if the inference score MS does not exceed a predetermined threshold. Furthermore, the inference unit 33 does not output to the loss calculation unit 35 an inference result UR for unlabeled data UD whose inference score MS does not exceed a predetermined threshold. The inference unit 33 also calculates an assignment rate PGR corresponding to the ratio of the number of data items to which the pseudo label PL is assigned to the number of unlabeled data items UD input to the inference model 33A, and outputs the calculated assignment rate PGR to the learning progress status storage unit 37. Furthermore, when new parameters PM for inference are obtained by the parameter acquisition unit 36, the inference unit 33 applies the parameters PM to the inference model 33A. Furthermore, the inference unit 33 measures the elapsed time from the start of learning for each step and outputs the measured elapsed time as a learning time LT to the learning progress status storage unit 37.

[0093] The inference unit 33 can perform processing similar to that described above using the unlabeled data UDE and UDQ acquired from the data expansion unit 32. Therefore, the inference unit 33 can train the inference model 33A using the unlabeled data UDQ2 and UDQ3 acquired by data expansion by the data expansion unit 32. Furthermore, the inference unit 33 can train the inference model 33A using the labeled data LDQ1 before training using the unlabeled data UDE1 set first in the usage order UJ. Furthermore, the inference unit 33 can train using the unlabeled data UDQ2 after training using the unlabeled data UDE2. Furthermore, the inference unit 33 can train using the unlabeled data UDF2 after training using the unlabeled data UDQ2. Furthermore, the inference unit 33 can train using the unlabeled data UDF3 after training using the unlabeled data UDE3. Furthermore, the inference unit 33 can train using the unlabeled data UDF3 after training using the unlabeled data UDQ3.

[0094] The loss calculation unit 34 calculates a labeled loss LL corresponding to the loss of the labeled data LD based on the labeled data LD and the inference result LR. The loss calculation unit 34 also outputs the labeled loss LL to the parameter acquisition unit 36 ​​and the learning progress status storage unit 37.

[0095] The loss calculation unit 35 calculates an unlabeled loss UL corresponding to the loss of the unlabeled data UD to which the pseudo label PL is assigned, based on the inference result UR and the pseudo label PL. The loss calculation unit 35 also outputs the unlabeled loss UL to the parameter acquisition unit 36 ​​and the learning progress status storage unit 37.

[0096] The parameter acquisition unit 36 ​​acquires new parameters PM for inference based on the labeled loss LL and the unlabeled loss UL, and outputs the acquired parameters PM to the inference unit 33. The parameter acquisition unit 36 ​​also outputs the number of parameter updates AN of the inference model 33A, which corresponds to the number of times the parameters PM have been acquired, to the learning progress status storage unit 37.

[0097] The learning progress status storage unit 37 stores a learning progress status LSJ including the assignment rate PGR, the learning time LT, the labeled loss LL, the unlabeled loss UL, and the number of parameter updates AN. Each piece of information included in the learning progress status LSJ can be used, for example, to set the output timing of the unlabeled data UD from the learning control unit 31 to the data extension unit 32.

[0098] In this embodiment, it is desirable that the inference unit 33 functions as a learning unit. Alternatively, in this embodiment, the function of the learning unit may be realized by each unit including the inference unit 33. Specifically, for example, the function of the learning unit may be realized by the inference unit 33, the loss calculation unit 34, the loss calculation unit 35, and the parameter acquisition unit 36.

[0099] [Processing Flow] Next, a description will be given of the flow of processing performed in the learning device 100. Fig. 12 is a flowchart showing an example of processing performed in the learning device according to the present disclosure.

[0100] First, the learning device 100 sets the order of use of unlabeled data to be used for learning based on the distance between labeled data and unlabeled data (step S11).

[0101] Next, the learning device 100 acquires unlabeled data corresponding to the use order set in step S11 (step S12). For example, the learning device 100 acquires unlabeled data UDE2 and UDF2 in step S12.

[0102] Next, the learning device 100 sets elements to be modified by data extension based on the image or attribute information included in the unlabeled data acquired in step S12 (step S13). For example, in step S13, the learning device 100 sets element EL2 according to attributes included in both attribute information UZE2 and UZF2.

[0103] Next, the learning device 100 sets the strength for data augmentation for the element set in step S13 based on the image or attribute information included in the unlabeled data acquired in step S12 (step S14). For example, in step S14, the learning device 100 sets the strength ST2 so that an image representing an intermediate state of change from image UGE2 to image UGF2 is generated.

[0104] Next, the learning device 100 applies data augmentation to the unlabeled data corresponding to the use order set in step S11, modifying the element set in step S13 by the strength set in step S14 (step S15). For example, in step S15, the learning device 100 applies data augmentation to image UGE2, modifying element EL2 by strength ST2. Furthermore, according to the processing of step S15, the learning device 100 can generate, for example, unlabeled data UDQ2 as unlabeled data to be added to the middle of the use order set in step S11.

[0105] Next, the learning device 100 learns the inference model 33A using the unlabeled data corresponding to the usage order set in step S11 and the unlabeled data generated by data expansion in step S15 (step S16).

[0106] Next, the learning device 100 determines whether the learning of the inference model 33A is complete based on the order of use set in step S11 (step S17). If the learning of the inference model 33A is complete (step S17: YES), the learning device 100 ends the series of processes in Fig. 12. If the learning of the inference model 33A is not complete (step S17: NO), the learning device 100 performs the processes from step S12 onwards again.

[0107] As described above, according to this embodiment, the use order of unlabeled data used in training a machine learning model can be set, and data augmentation can be performed according to the set use order, thereby generating unlabeled data to be added to the middle of the use order. Furthermore, according to this embodiment, elements to be changed by data augmentation can be set based on the attributes of adjacent unlabeled data in the use order of unlabeled data used in training a machine learning model. Furthermore, according to this embodiment, the strength of data augmentation can be set based on changes between adjacent unlabeled data in the use order of unlabeled data used in training a machine learning model. Therefore, according to this embodiment, it is possible to improve the accuracy of pseudo labels assigned to unlabeled data by a machine learning model currently being trained, and to improve the inference accuracy of a trained machine learning model. Therefore, according to this embodiment, it is possible to improve the operating accuracy when a trained model is used in a practical system.

[0108] According to this embodiment, the learning device 100 can train the inference model 33A to suit any of the following applications, for example, product recognition, behavior classification, person detection, object detection, or segmentation.

[0109] For example, in a product recognition application, the learning device 100 may train the inference model 33A using labeled data LD to which an image containing a product to be recognized is assigned a label indicating the name of the product as a correct answer and attribute information indicating the attributes of the image. Also, in a product recognition application, the learning device 100 may train the inference model 33A using unlabeled data UD to which an image containing a product to be recognized is assigned attribute information indicating the attributes of the image.

[0110] For example, in the case of behavior classification, the learning device 100 may train the inference model 33A using labeled data LD to which an image including a person to be recognized is assigned a label indicating the type of behavior of the person as a correct answer and attribute information indicating the attributes of the image. Also, in the case of behavior classification, the learning device 100 may train the inference model 33A using unlabeled data UD to which an image including a person to be recognized is assigned attribute information indicating the attributes of the image.

[0111] For example, in the case of person detection, the learning device 100 may train the inference model 33A using labeled data LD to which an image containing a person to be recognized is assigned a label indicating the position of the person in the image as a correct answer and attribute information indicating the attributes of the image. Also, in the case of person detection, the learning device 100 may train the inference model 33A using unlabeled data UD to which an image containing a person to be recognized is assigned attribute information indicating the attributes of the image.

[0112] For example, in the case of object detection, the learning device 100 may train the inference model 33A using labeled data LD to which an image containing an object to be recognized is assigned, the labeled data LD being a label indicating the position of the object in the image and the correct class of the object, and attribute information indicating the attributes of the image. Also, in the case of object detection, the learning device 100 may train the inference model 33A to which an image containing an object to be recognized is assigned, the unlabeled data UD being a label indicating the attributes of the image.

[0113] For example, in segmentation applications, the learning device 100 may train the inference model 33A using labeled data LD, which is assigned a label indicating the type of object at each pixel in an image as a correct answer and attribute information indicating the attributes of the image, for an image containing an object to be recognized. Also, in segmentation applications, the learning device 100 may train the inference model 33A using unlabeled data UD, which is assigned attribute information indicating the attributes of the image, for an image containing an object to be recognized.

[0114] [Modifications] Modifications of the present embodiment will be described below. For simplicity, specific descriptions of parts to which the above-described processes can be applied will be omitted as appropriate.

[0115] (Modification 1) Fig. 13 is a diagram illustrating an example of the configuration of the data expansion unit 32. The data expansion unit 32 may include a display screen generation unit 32A, for example, as shown in Fig. 13.

[0116] The display screen generation unit 32A generates a display screen HG that prompts the user to make settings related to the data expansion performed in the data expansion unit 32, and outputs the generated display screen HG to a display device 200 such as a monitor. In addition, the display screen generation unit 32A acquires the setting contents set on the display screen HG based on an instruction signal SS output from an input device 300 such as a pointing device.

[0117] Here, a specific example of processing related to the generation of the display screen HG will be described. Note that the following description will be given taking as an example a case where data extension is performed on the image UGE2.

[0118] The data extension unit 32 sets a plurality of different strengths STX as the strength of data extension to be applied to image UGE2 based on how the attributes used to set element EL2 changed between image UGE2 and image UGF2. Note that it is desirable that the data extension unit 32 sets strength STX using, for example, a method similar to the method for setting strength ST2. Furthermore, the data extension unit 32 applies data extension to image UGE2, changing element EL2 with each of the plurality of strengths STX, thereby generating a plurality of candidate images CGX corresponding to each of the plurality of strengths STX.

[0119] The display screen generation unit 32A generates a display screen HG that prompts the user to select at least one candidate image CGX from among multiple candidate images CGX to be used as an image after data extension has been applied to the image UGE2. According to this processing, for example, a display screen HG such as that shown in Fig. 14 can be displayed on the display device 200. Fig. 14 is a diagram showing an example of a display screen that prompts the user to make settings related to data extension.

[0120] The display screen HG in Fig. 14 includes images UGE2 and UGF2, four candidate images CGX1 to CGX4, a selection complete button SCB, and a re-apply button RAB. Candidate image CGX1 corresponds to an image generated by applying data augmentation to image UGE2, changing element EL2 with strength STX1. Candidate image CGX2 corresponds to an image generated by applying data augmentation to image UGE2, changing element EL2 with strength STX2. Candidate image CGX3 corresponds to an image generated by applying data augmentation to image UGE2, changing element EL2 with strength STX3. Candidate image CGX4 corresponds to an image generated by applying data augmentation to image UGE2, changing element EL2 with strength STX4.

[0121] According to the display image HG of FIG. 14 , the user can refer to images UGE2 and UGF2 and select at least one candidate image CGX from among the candidate images CGX1 to CGX4 to be used as the image after data extension has been applied to image UGE2. In the display image HG of FIG. 14 , the candidate images CGX1, CGX3, and CGX4 selected by the user are highlighted. Furthermore, according to the display image HG of FIG. 14 , the user can instruct the use of candidate images CGX1, CGX3, and CGX4 as the image after data extension has been applied to image UGE2 by pressing a selection completion button SCB. Furthermore, when, for example, a re-application button RAB included in the display image HG of FIG. 14 is pressed, the data extension unit 32 resets the intensities STX1 to STX4 and performs data extension that changes element EL2 with each of the reset intensities STX1 to STX4, thereby generating multiple candidate images different from candidate images CGX1 to CGX4. In the display image HG of FIG. 14, for example, if the user has not selected any candidate image CGX, pressing the selection complete button SCB may be disabled.

[0122] The display screen generation unit 32A acquires the setting contents corresponding to the user's instruction based on the instruction signal SS output from the input device 300 in response to the pressing of the selection completion button SCB. Then, the data expansion unit 32 acquires, as unlabeled data UDQ2, data in which attribute information UZE2 is assigned to candidate image CGX1, data in which attribute information UZE2 is assigned to candidate image CGX3, and data in which attribute information UZE2 is assigned to candidate image CGX4, in accordance with the setting contents acquired by the display screen generation unit 32A. The data expansion unit 32 outputs the unlabeled data UDE2 to the inference unit 33, and then outputs the unlabeled data UDQ2 to the inference unit 33. Therefore, the inference unit 33 can train the inference model 33A using the unlabeled data UDQ2 acquired by data expansion corresponding to the setting contents set on the display screen HG.

[0123] The display screen generation unit 32A may generate a display image that includes, for example, at least one image extracted from each unlabeled data UDE, at least one image extracted from each unlabeled data UDF, a plurality of candidate images CGX, a selection complete button SCB, and a re-apply button RAB, instead of the display image HG. In such a case, the display screen generation unit 32A may generate a display screen that does not include the images UGE2 and / or UGF2.

[0124] As described above, according to this modification, the strength of unlabeled data generated by data augmentation can be set to a strength desired by the user.

[0125] (Variation 2) The data extension unit 32 may, for example, apply a known data extension to the unlabeled data UDF to generate at least one unlabeled data UDR as additional data to be used in learning in the (p+1)th step in the use order UJ. Furthermore, the inference unit 33 may, for example, train the inference model 33A using the unlabeled data UDQ instead of the unlabeled data UDE, and then train the inference model 33A using the unlabeled data UDR instead of the unlabeled data UDF.

[0126] Second Embodiment FIG. 15 is a block diagram showing another example of the functional configuration of a learning device according to the present disclosure.

[0127] The learning device 500 has the same hardware configuration as the learning device 100. The learning device 500 also includes an order setting unit 511, a data extension unit 512, and a learning unit 513.

[0128] The order setting means 511 can be realized, for example, by using the function of the learning control unit 31. The data extension means 512 can be realized, for example, by using the function of the data extension unit 32. The learning means 513 can be realized, for example, by using the function of the inference unit 33.

[0129] FIG. 16 is a flowchart illustrating another example of processing performed in the learning device according to the present disclosure.

[0130] The order setting means 511 sets the order in which unlabeled data to be used for model training will be used (step S51).

[0131] The data extension means 512 sets data extension according to the order of use, and applies the set data extension to the unlabeled data (step S52).

[0132] The learning means 513 uses the data obtained by data expansion to learn the model 600 (step S53).

[0133] According to this embodiment, it is possible to improve the operating accuracy when a trained model is used in a practical system.

[0134] Third Embodiment FIG. 17 is a block diagram showing another example of the functional configuration of a learning device according to the present disclosure.

[0135] The learning device 700 has the same hardware configuration as the learning device 100. The learning device 700 also has an order setting unit 711, a data extension unit 712, a display image generation unit 713, and a learning unit 714.

[0136] The order setting means 711 can be realized, for example, by using the functions of the learning control unit 31. The data expansion means 712 can be realized, for example, by using the functions of the data expansion unit 32. The display image generation means 713 can be realized, for example, by using the functions of the display screen generation unit 32A. The learning means 714 can be realized, for example, by using the functions of the inference unit 33.

[0137] FIG. 18 is a flowchart illustrating another example of processing performed in the learning device according to the present disclosure.

[0138] The order setting means 711 sets the order in which unlabeled data to be used for model training will be used (step S71).

[0139] The data extension means 712 sets data extension according to the order of use, and applies the set data extension to the unlabeled data (step S72).

[0140] The display image generating means 713 generates a display screen that prompts the user to make settings related to data expansion (step S73).

[0141] The learning means 714 learns the model 800 using the data obtained by data expansion corresponding to the settings made on the display screen (step S74).

[0142] According to this embodiment, it is possible to improve the operating accuracy when a trained model is used in a practical system.

[0143] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0144] (Supplementary Note 1) A learning device comprising: an order setting means for setting an order of use of unlabeled data to be used for model learning; a data augmentation means for setting data augmentation according to the order of use and applying the set data augmentation to the unlabeled data; and a learning means for learning the model using data obtained by the data augmentation.

[0145] (Supplementary Note 2) The learning device according to Supplementary Note 1, wherein the data expansion means sets elements to be changed by the data expansion based on attributes of first unlabeled data and second unlabeled data that are adjacent in the order of use.

[0146] (Supplementary Note 3) The learning device according to Supplementary Note 2, wherein the data extension means sets the element according to an attribute included in both the first unlabeled data and the second unlabeled data.

[0147] (Supplementary Note 4) The learning device of Supplementary Note 2, wherein the data augmentation means sets the elements based on images included in the first unlabeled data and the second unlabeled data, or attribute information included in the first unlabeled data and the second unlabeled data.

[0148] (Supplementary Note 5) The learning device of Supplementary Note 4, wherein the data expansion means sets, as the elements, elements for which a distance determined by attributes and / or features between each attribute and / or feature extracted from an image included in the first unlabeled data and each attribute and / or feature extracted from an image included in the second unlabeled data is equal to or less than a predetermined value.

[0149] (Supplementary Note 6) The learning device of Supplementary Note 2, wherein the data augmentation means sets the strength of the data augmentation based on how an attribute used to set the element has changed from the first unlabeled data, which is relatively earlier in the order of use, to the second unlabeled data, which is relatively later in the order of use.

[0150] (Supplementary Note 7) The learning device of Supplementary Note 6, wherein the data augmentation means applies data augmentation to the first unlabeled data, modifying the element by the intensity, to generate at least one unlabeled data as additional data to be used in training the model.

[0151] (Supplementary Note 8) The learning device of Supplementary Note 7, wherein the learning means performs learning using the first unlabeled data and then performs learning using the additional data, and performs learning using the second unlabeled data after performing learning using the additional data.

[0152] (Supplementary Note 9) The learning device of Supplementary Note 6, wherein the data augmentation means sets the strength based on an image included in the first unlabeled data and the second unlabeled data, or attribute information included in the first unlabeled data and the second unlabeled data.

[0153] (Supplementary Note 10) The learning device of Supplementary Note 1, wherein the data augmentation means applies data augmentation to labeled data used for training the model, and the learning means trains the model using data obtained by data augmentation of the labeled data before training using the unlabeled data set first in the usage order.

[0154] (Supplementary Note 11) The learning device according to Supplementary Note 1, wherein the order setting means sets the order of use in order of proximity between the labeled data and the unlabeled data used for learning the model.

[0155] (Supplementary Note 12) The learning device of Supplementary Note 1, wherein the order setting means acquires a labeled data set corresponding to a data group obtained by classifying labeled data used for learning the model, calculates a distance between the acquired labeled data set and the unlabeled data for each class, and sets the use order for each class in descending order of the distance.

[0156] (Supplementary Note 13) The learning device according to Supplementary Note 12, wherein the order setting means sets the order of use so that unlabeled data used as unlabeled data of one class is not used as unlabeled data of another class.

[0157] (Supplementary Note 14) The learning device of Supplementary Note 1, wherein unlabeled data used in training the model is added to a labeled data set corresponding to a data group obtained by classifying labeled data used in training the model.

[0158] (Supplementary Note 15) The learning device of Supplementary Note 1, wherein the order setting means acquires a labeled dataset corresponding to a data group obtained by classifying labeled data used for training the model, calculates a distance between an average of feature points of a plurality of labeled data included in the labeled dataset and a feature point of the unlabeled data, and sets the use order based on the distance.

[0159] (Supplementary Note 16) The learning device of Supplementary Note 1, wherein the order setting means acquires a labeled dataset corresponding to a data group obtained by classifying labeled data used for training the model, calculates the smallest distance among multiple distances between feature points of multiple labeled data included in the labeled dataset and feature points of the unlabeled data, and sets the use order based on the calculated distance.

[0160] (Supplementary Note 17) A learning device comprising: an order setting means for setting an order of use of unlabeled data used for model learning; a data augmentation means for setting data augmentation according to the order of use and applying the set data augmentation to the unlabeled data; a display screen generation means for generating a display screen that prompts a user to set the data augmentation; and a learning means for learning the model using data obtained by the data augmentation that corresponds to the setting content set on the display screen.

[0161] (Supplementary Note 18) A learning method comprising: setting an order of use of unlabeled data to be used for model training; setting data augmentation according to the order of use; applying the set data augmentation to the unlabeled data; and training the model using data obtained by the data augmentation.

[0162] (Supplementary Note 19) A learning method comprising: setting an order of use of unlabeled data to be used for model training; setting data augmentation according to the order of use; applying the set data augmentation to the unlabeled data; generating a display screen that prompts a user to set the data augmentation; and training the model using data obtained by the data augmentation that corresponds to the settings set on the display screen.

[0163] (Supplementary Note 20) A recording medium having recorded thereon a program that causes a computer to execute a process of setting an order of use of unlabeled data to be used for model training, setting data augmentation according to the order of use, applying the set data augmentation to the unlabeled data, and training the model using data obtained by the data augmentation.

[0164] (Supplementary Note 21) A recording medium having recorded thereon a program that causes a computer to execute the following processes: setting an order of use of unlabeled data to be used for model training; setting data augmentation according to the order of use; applying the set data augmentation to the unlabeled data; generating a display screen that prompts a user to set the data augmentation; and training the model using data obtained by the data augmentation that corresponds to the settings set on the display screen.

[0165] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0166] 31 Learning control unit 32 Data expansion unit 32A Display screen generation unit 33 Inference unit 33A Inference model 100 Learning device

Claims

1. An ordering mechanism for setting the order in which unlabeled data used for training a model is used, A data extension means that sets data extension according to the usage order and applies the set data extension to the unlabeled data, A learning means for training the model using the data obtained through the aforementioned data augmentation, A learning device equipped with the following features.

2. The learning device according to claim 1, wherein the data augmentation means sets the elements to be modified by the data augmentation based on the attributes of the first unlabeled data and the second unlabeled data adjacent to each other in the usage order.

3. The learning device according to claim 2, wherein the data augmentation means sets the elements according to the attributes contained in both the first unlabeled data and the second unlabeled data.

4. The learning device according to claim 2, wherein the data augmentation means sets the elements based on images included in the first unlabeled data and the second unlabeled data, or attribute information included in the first unlabeled data and the second unlabeled data.

5. The learning device according to claim 4, wherein the data augmentation means sets elements such that the distance determined by the attributes and / or features is less than or equal to a predetermined value, between each attribute and / or feature extracted from an image contained in the first unlabeled data and each attribute and / or feature extracted from an image contained in the second unlabeled data.

6. An ordering mechanism for setting the order in which unlabeled data used for training a model is used, A data extension means that sets data extension according to the usage order and applies the set data extension to the unlabeled data, A display screen generation means that generates a display screen that prompts the user to configure the settings related to the aforementioned data expansion, A learning means for training the model using the data obtained by the data augmentation corresponding to the settings set on the display screen, A learning device equipped with the following features.

7. A computer, Set the order in which to use the unlabeled data to train the model. Set data extensions according to the usage order, and apply the set data extensions to the unlabeled data. A learning method for training the model using the data obtained through the aforementioned data augmentation.

8. A computer, Set the order in which to use the unlabeled data to train the model. Set data extensions according to the usage order, and apply the set data extensions to the unlabeled data. A display screen is generated that prompts the user to configure the settings related to the aforementioned data expansion. A learning method that performs learning of the model using data obtained by the data augmentation corresponding to the settings set on the display screen.

9. Set the order in which to use the unlabeled data to train the model. Set data extensions according to the usage order, and apply the set data extensions to the unlabeled data. A program that causes a computer to perform a process of training the model using the data obtained through the aforementioned data augmentation.

10. Set the order in which to use the unlabeled data to train the model. Set data extensions according to the usage order, and apply the set data extensions to the unlabeled data. A display screen is generated that prompts the user to configure the settings related to the aforementioned data expansion. A program that causes a computer to perform a process of training the model using the data obtained by the data augmentation corresponding to the settings set on the display screen.