Learning device, learning method, program, and pre-trained model

Abstract

To provide a learning device capable of improving the inference performance of a pre-trained model while reducing learning costs, a learning method, a program, and a pre-trained model.SOLUTION: A disclosed embodiment of a learning device includes: a generation unit that generates a first training data; an inference unit that performs inference on the first training data using the pre-trained model and outputs a first score that indicates the reliability of the inference; and a learning unit that retrains the pre-trained model using the first training data when the first score is less than a first threshold.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a learning method, a program, and a trained model. [Background technology]

[0002] In object detection and recognition using machine learning, large amounts of training data are used to improve inference performance. For this reason, data augmentation is used to expand the training data. For example, when the training data is images, data augmentation can be performed using filtering or synthesis, which combines new objects into existing images.

[0003] Patent Document 1 discloses an evaluation support device that supports the construction of a sophisticated training dataset. The evaluation support device in Patent Document 1 evaluates evaluation data using a trained model used to evaluate the training dataset, and if the evaluation data has a high degree of confidence but a low degree of stability, generates data similar to the evaluation data and adds it to the training dataset. Note that data similar to the evaluation data is generated by a method such as slightly changing the feature quantities of the evaluation data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-104464 Summary of the Invention [Problem to be solved by the invention]

[0005] Consider a situation where a trained model that has already been generated is retrained in order to improve its inference performance. If the method of Patent Document 1 described above is used in this case, the model will be retrained using data similar to evaluation data with a high degree of confidence. This may result in training using training data that does not contribute to improving the inference performance of the trained model, such as training data that has similar features to training data that has already been trained, which may increase training costs.

[0006] The present disclosure has been made in consideration of the above circumstances, and contributes to providing a learning device, a learning method, a program, and a trained model that can reduce learning costs and improve the inference performance of trained models. [Means for solving the problem]

[0007] One aspect of the learning device of the present disclosure includes a generation unit that generates first learning data, an inference unit that performs inference on the first learning data using a trained model and outputs a first score that indicates the likelihood of the inference, and a learning unit that re-trains the trained model using the first learning data if the first score is less than a first threshold.

[0008] In one aspect of the learning method of the present disclosure, a generation unit generates first learning data, an inference unit performs inference on the first learning data using a trained model and outputs a first score indicating the likelihood of the inference, and if the first score is less than a first threshold, a learning unit re-trains the trained model using the first learning data.

[0009] One aspect of the program disclosed herein is for causing a computer to execute a generation step of generating first training data, an inference step of performing inference on the first training data using a trained model and outputting a first score indicating the likelihood of the inference, and a training step of re-training the trained model using the first training data if the first score is less than a first threshold.

[0010] One aspect of the trained model of the present disclosure is generated by the above device, the above method, or the above program. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide a learning device, a learning method, a program, and a trained model that can reduce learning costs and improve the inference performance of a trained model. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of a learning device according to this embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the functional configuration of the learning device of this embodiment. [Figure 3] FIG. 3 is an explanatory diagram showing an example of generating an image that is a candidate for relearning by the synthesis process of this embodiment. [Figure 4] FIG. 4 is an explanatory diagram showing an example of generating an image that is a candidate for relearning by the generation AI of this embodiment. [Figure 5] FIG. 5 is an explanatory diagram showing an example of generating an image that is a candidate for relearning by the filter processing of this embodiment. [Figure 6] FIG. 6 is a diagram showing an example of information stored in the parameter storage unit of this embodiment. [Figure 7] FIG. 7 is a diagram showing another example of information stored in the parameter storage unit of the present embodiment. [Figure 8] FIG. 8 is an explanatory diagram illustrating an example of the relationship between the distribution of features of trained images used when training the trained model of this embodiment and the features of images that are candidates for retraining the trained model. [Figure 9] FIG. 9 is a diagram showing an example of a parameter list screen according to this embodiment. [Figure 10] FIG. 10 is a diagram showing an example of screen transition between a distribution screen relating to the parameter "position" and a list screen of images for relearning in this embodiment. [Figure 11]FIG. 11 is a diagram showing an example of screen transition between a distribution screen relating to the parameter "size" and a list screen of images for relearning in this embodiment. [Figure 12] FIG. 12 is a diagram showing an example of screen transitions between a list screen of parameters related to the parameter "concealment" of this embodiment, a distribution screen, and a list screen of images for re-learning. [Figure 13] FIG. 13 is a diagram showing an example of screen transitions between a filter type distribution screen, a filter strength distribution screen, and a list screen of images for relearning, relating to the parameter "filter processing" of this embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of the re-learning process performed by the learning device of this embodiment. [Figure 15] FIG. 15 is a flowchart showing an example of a display process performed by the learning device of this embodiment. [Figure 16] FIG. 16 is an explanatory diagram showing an example of generating images that are candidates for relearning in Modification 1. In FIG. [Figure 17] FIG. 17 is an explanatory diagram showing another example of generating images that are candidates for relearning in Modification 1. In FIG. [Figure 18] FIG. 18 is a block diagram showing an example of the configuration of a learning device according to the second modification. [Figure 19] FIG. 19 is a block diagram showing an example of the configuration of a learning device according to the third modification. [Figure 20] FIG. 20 is a block diagram showing an example of the configuration of a learning device according to the fourth modification. [Figure 21] FIG. 21 is an explanatory diagram showing an example of whether or not an image that is a candidate for relearning in Modification 4 is adopted as an image for relearning, comparing before and after calibration. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present disclosure (hereinafter simply referred to as "the present embodiment") will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the following embodiment. Furthermore, the following embodiment and modified examples can be combined as appropriate.

[0014] The learning device of this embodiment evaluates training data generated by data augmentation with a trained model, and if there is a high possibility that the generated training data will contribute to improving the inference performance of the trained model, re-trains the trained model using the generated training data. In this way, the learning device of this embodiment can prevent re-training using training data that does not contribute to improving the inference performance of the trained model, thereby reducing training costs and improving the inference performance of the trained model.

[0015] Furthermore, the learning device of this embodiment displays information about the parameters used to generate the training data used for relearning. Therefore, the learning device of this embodiment makes it possible to objectively grasp which information about the parameters used to generate the training data is likely to contribute to improving the inference performance of the trained model. This makes it possible to provide a well-founded explanation of the learning process of the trained model even to users of the trained model who are not familiar with the trained model.

[0016] In the following, an example will be described in which the trained model obtained after training by the learning device of this embodiment is adopted in an on-board camera for an AEB (Autonomous Emergency Braking) function or the like, and detects pedestrians and other detection targets from video captured by the on-board camera. However, the trained model of this embodiment is not limited to this, and may be, for example, a model for confirming the use of protective equipment at a manufacturing site or a model for inspecting the appearance of products at a manufacturing site. Note that the trained model of this embodiment can be any model generated by machine learning, such as an object recognition model or a person recognition model. In addition, in the following, an example will be described in which the training data of this embodiment is images for training, but this is not limited to this.

[0017] First, the configuration of the learning device of this embodiment will be described.

[0018] Fig. 1 is a block diagram showing an example of the hardware configuration of a learning device 10 of this embodiment. As shown in Fig. 1, learning device 10 includes a control device 11, a main memory device 13, an auxiliary memory device 15, a display device 17, an input device 19, a communication device 21, and various buses 23. Control device 11, main memory device 13, auxiliary memory device 15, display device 17, input device 19, and communication device 21 are connected via various buses 23. As such, learning device 10 of this embodiment has an existing hardware configuration that utilizes an existing computer.

[0019] The control device 11 controls the overall operation of the learning device 10. Examples of the control device 11 include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), but are not limited to these. There may be any number of CPUs or GPUs as long as they are one or more, and they may be single-core or multi-core.

[0020] Examples of the main memory device 13 include, but are not limited to, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores various programs, such as a program for controlling the learning device 10 and the machine learning program of this embodiment. The RAM is used as a work area when the control device 11 performs various controls based on the programs stored in the ROM.

[0021] The auxiliary storage device 15 stores various data, such as the various programs described above and data for machine learning in this embodiment. The various programs described above may be stored in at least one of the main storage device 13 and the auxiliary storage device 15. Examples of the auxiliary storage device 15 include at least one of existing storage devices capable of magnetic, electrical, or optical storage, such as a hard disk drive (HDD), a solid state drive (SSD), and a digital versatile disc (DVD). However, the auxiliary storage device 15 is not limited to these. The auxiliary storage device 15 may be built into the learning device 10 or externally connected to the learning device 10 via an interface such as a universal serial bus (USB). The auxiliary storage device 15 may also be a network-attached storage (NAS) connected via a network, such as a local area network (LAN) or a wide area network (WAN).

[0022] Display device 17 displays various screens used during machine learning by learning device 10 and serves as a user interface with the user (operator). Examples of display device 17 include, but are not limited to, various displays such as a liquid crystal display, an organic electroluminescence (EL) display, and a touch panel display. Display device 17 may be an internal display built into learning device 10, or an external display connected to learning device 10 via a display interface such as HDMI (registered trademark).

[0023] The input device 19 is used for various inputs, selections, and specifications used during machine learning by the learning device 10, and serves as a user interface with the user (operator). Examples of the input device 19 include, but are not limited to, a keyboard, a mouse, and a touch panel. The input device 19 may be built into the learning device 10 or may be externally attached to the learning device 10 via an interface such as a USB.

[0024] Examples of the communication device 21 include, but are not limited to, a communication device for a wired LAN and a wireless communication device for a wireless LAN. The communication device 21 may be used to externally acquire the machine learning program and data of this embodiment, or to externally output the trained model generated by the learning device 10.

[0025] In addition to the above configuration, the learning device 10 may further include hardwired circuits such as an IC (Integrated Circuit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable Gate Array) specific to the learning device 10 in order to realize the learning function.

[0026] 2 is a block diagram showing an example of the functional configuration of the learning device 10 of this embodiment. As shown in FIG. 2, the learning device 10 includes a trained model storage unit 101, a generation unit 103, a parameter storage unit 105, an inference unit 107, an evaluation unit 109, a rejection data storage unit 111, an adoption data storage unit 113, an association unit 115, a learning unit 117, an input reception unit 119, and a display control unit 121.

[0027] The generation unit 103, the inference unit 107, the evaluation unit 109, the association unit 115, the learning unit 117, the input reception unit 119, and the display control unit 121 can be realized, for example, by the control device 11 and the main memory device 13 described in FIG. 1.

[0028] For example, the control device 11 reads a machine learning program stored in the main memory device 13 (ROM) or the auxiliary memory device 15, or a machine learning program acquired from an external device via the communication device 21, and loads the read program into the main memory device 13 (RAM). The control device 11 executes various processes in accordance with the loaded program, thereby realizing each of the above-described functional units. Here, the above-described functional units are implemented as software, but at least some of the above-described functional units may be implemented as hardware. In this case, the functional units implemented as hardware may be implemented, for example, by the above-described hardwired circuit. Furthermore, any of the above-described functional units may be implemented by a combination of software and hardware.

[0029] The trained model storage unit 101, the parameter storage unit 105, the rejected data storage unit 111, and the adopted data storage unit 113 can be realized, for example, by at least one of the main storage unit 13 and the auxiliary storage unit 15 described in FIG.

[0030] The trained model storage unit 101 stores a trained model trained by a learning unit 117, which will be described later. The trained model is a model that performs inference on input data and outputs an inference result and a score indicating the reliability of the inference. For example, when an image is input, the trained model infers whether or not a detection object such as a pedestrian is included in the image, and outputs an inference result and a score indicating the reliability of the inference. However, the trained model is not limited to this. Furthermore, in this embodiment, an example will be described in which the trained model is a neural network (NN). However, the trained model is not limited to this, and may be any model or network used in machine learning.

[0031] The generation unit 103 generates training data (an example of first training data). Specifically, the generation unit 103 generates training data that is a candidate for retraining the trained model stored in the trained model storage unit 101. For example, the generation unit 103 generates, as the training data, an image that shows a detection object such as a pedestrian. The training data (image) may include an identification ID that identifies the training data, or the identification ID may be added to the training data. Furthermore, the training data (image) may include label information indicating attributes of the detection object such as a pedestrian included in the training data, or the label information may be added as an annotation.

[0032] In this embodiment, the generation unit 103 generates images that are candidates for relearning by data augmentation. Examples of methods for generating images by data augmentation include, but are not limited to, a synthesis process that synthesizes a new object into an existing image, a filter process, and a generation process using artificial intelligence (AI). The generation unit 103 may use a plurality of these generation methods to generate images that are candidates for relearning for each generation method, or may use any one of these generation methods to generate images that are candidates for relearning.

[0033] FIG. 3 is an explanatory diagram showing an example of generating an image that is a candidate for re-learning by the synthesis process of this embodiment. In the example shown in FIG. 3, image 201 shows a pedestrian 203 walking on a road, and image 211 shows an umbrella 213. Image 201 is, for example, a trained image used when training a trained model. The umbrella 213 is, for example, an object used as an occlusion that conceals a part of the pedestrian 203. When image 201 is a trained image, an image with variations on the pedestrian 203 is considered useful for re-learning the trained model. For this reason, the generation unit 103 synthesizes the umbrella 213 shown in image 211 with image 201 to make the pedestrian 203 walking on the road hold the umbrella 213, thereby generating image 221 in which a part of the pedestrian 203 is concealed by the umbrella 213, as a candidate image for re-learning.

[0034] When the umbrella 213 is superimposed on the image 201, information such as the position, size, and rotation angle of the umbrella 213 when superimposing the umbrella 213 is used as parameters. The parameters may be input and set by a developer of the trained model, or may be set according to a setting file prepared by the developer. The setting file may include, for example, information that defines a range in which an umbrella can be placed relative to the position of a pedestrian, information that defines a settable umbrella size relative to the size of a pedestrian, and information that defines a settable umbrella rotation angle relative to the rotation angle of a pedestrian. However, the information included in the setting file is not limited to these.

[0035] 4 is an explanatory diagram showing an example of an image generated by the generation AI of this embodiment as a candidate for relearning. In the example shown in Fig. 4, the generation AI 104 has a pedestrian 203 walking on a road hold an umbrella 213, and generates an image 221 in which the pedestrian 203 is partly concealed by the umbrella 213 as a candidate image for relearning.

[0036] The generation AI 104 may generate the image 221 based on the image 201 described in Figure 3 in accordance with a prompt input by a developer of the trained model, etc. In this case, the prompt may be, for example, a written description of the parameters used when combining the umbrella 213 described in Figure 3 with the image 201, but is not limited to this. The generation AI 104 may also generate the image 221 from scratch in accordance with a prompt input by a developer of the trained model, etc.

[0037] FIG. 5 is an explanatory diagram showing an example of generating an image that is a candidate for re-learning by the filter processing of this embodiment. In the example shown in FIG. 5, a pedestrian 203 is walking down a road while holding an umbrella 213, and is partially obscured by the umbrella 213. The image 231 is, for example, a trained image that was used when training a trained model. When the image 231 is a trained image, it is considered useful to provide variation to the image 231 in re-learning the trained model. For this reason, the generation unit 103 performs a filter processing on the image 231 to generate the image 241 as a candidate for re-learning.

[0038] As a variation, for example, blurring objects such as pedestrian 203 and umbrella 213 appearing in the image in anticipation of shake of the in-vehicle camera, etc., can be mentioned, but is not limited to this. In the example shown in Fig. 5, generation unit 103 performs blurring processing as a filter process on image 231 to generate image 241 in which objects such as pedestrian 203 and umbrella 213 appearing in the image are blurred.

[0039] Furthermore, when blurring the image 231, information such as filter strength (blur strength) is used as a parameter. The parameters may be input and set by a developer of the trained model, or may be set according to a setting file prepared by the developer. The setting file may include information that defines, for example, the range of settable filter strengths.

[0040] Other variations that can be added to the image 231 include, for example, slightly changing at least one of the position, size, and angle of the pedestrian 203 or the umbrella 213, or inverting the pedestrian 203 or the umbrella 213. Furthermore, the filter processing is not limited to the above-mentioned blur processing, and may be, for example, at least one of inversion processing, rotation processing, and transformation processing.

[0041] The generation unit 103 also associates the identification ID of the generated training data with parameter information about the parameters used to generate the training data and outputs the associated information to the parameter storage unit 105. Examples of the parameters include at least one of parameters related to a detection target, such as a pedestrian 203, parameters related to a concealing object, such as an umbrella 213 that conceals the detection target, parameters related to the surrounding background, and parameters related to filtering. However, the parameters are not limited to these. Examples of the parameters related to the detection target include at least one of the position, size, rotation angle, posture, clothing, and texture of the detection target, but are not limited to these. Examples of the parameters related to the concealing object include at least one of the type, position, size, rotation angle, texture, and relative position of the concealing object with respect to the detection target, but are not limited to these. Examples of the parameters related to filtering include at least one of the type and strength of filtering, but are not limited to these. Examples of the parameter information include, but are not limited to, a combination of a parameter type and a parameter value.

[0042] The parameter storage unit 105 stores the identification ID of the learning data generated by the generation unit 103 and the parameter information of the parameters used to generate the learning data in association with each other.

[0043] Fig. 6 is a diagram showing an example of information stored in the parameter storage unit 105 of this embodiment. In the example shown in Fig. 6, the identification ID "ID001" of the image 221 described in Figs. 3 and 4 and the parameter information (parameter type and parameter value) of the image 221 are associated with each other and stored in the parameter storage unit 105.

[0044] For example, the position of the pedestrian 203 is indicated by a parameter type "position" and a parameter value "(100, 300)." The parameter value "(100, 300)" is an x- and y-coordinate value with the center of the image 221 set as the origin (0, 0). For example, the type of the umbrella 213 is indicated by a parameter type "occlusion (object type)" and a parameter value "umbrella." For example, the position of the umbrella 213 is indicated by a parameter type "occlusion (object position)" and a parameter value "(300, 400)." The parameter value "(300, 400)" is an x- and y-coordinate value on the image 221. For example, the size of the umbrella 213 is indicated by a parameter type "occlusion (object size)" and a parameter value "10."

[0045] Fig. 7 is a diagram showing another example of information stored in the parameter storage unit 105 of this embodiment. In the example shown in Fig. 7, the identification ID "ID002" of the image 241 described in Fig. 5 and the parameter information (parameter type and parameter value) of the image 241 are associated with each other and stored in the parameter storage unit 105.

[0046] For example, the position of the pedestrian 203 is indicated by the parameter type "position" and the parameter value "(100,300)." Also, for example, the type of blurring is indicated by the parameter type "filtering (filter type)" and the parameter value "blur." Also, for example, the strength of the blurring is indicated by the parameter type "filtering (filter strength)" and the parameter value "4."

[0047] The inference unit 107 uses the trained model stored in the trained model storage unit 101 to perform inference on the training data generated by the generation unit 103, and outputs the inference result and a first score indicating the likelihood of the inference. For example, when an image is input, the trained model of this embodiment infers whether or not a detection target, such as a pedestrian, is included in the image. For example, the trained model of this embodiment outputs, as the inference result, whether or not an object that is thought to be a detection target is detected in the image by inference, and outputs the likelihood of the inference as the first score. Therefore, the inference unit 107 of this embodiment inputs an image that is a candidate for re-training, generated by the generation unit 103, to the trained model of this embodiment, and thereby obtains the inference result of whether or not a detection target, such as a pedestrian, is included in the image, and a first score indicating the likelihood of the inference.

[0048] It should be noted that the trained model of this embodiment is assumed to be a neural network consisting of, for example, an input layer, one or more intermediate layers, and an output layer. In this case, when an image (for example, the pixel values ​​of each pixel constituting the image) is input to the neural network, each of the input layer and one or more intermediate layers multiplies the input value by a weight and outputs the result to the next layer, so that the output layer finally outputs the above-mentioned first score.

[0049] The evaluation unit 109 uses the first score output by the inference unit 107 to evaluate whether the training data generated by the generation unit 103 is likely to contribute to improving the inference performance of the trained model stored in the trained model storage unit 101. Specifically, if the first score is equal to or greater than a first threshold, the evaluation unit 109 determines that the training data is unlikely to contribute to improving the inference performance of the trained model, and outputs the training data whose first score is equal to or greater than the first threshold together with the identification ID to the rejected data storage unit 111. On the other hand, if the first score is less than the first threshold, the evaluation unit 109 determines that the training data is likely to contribute to improving the inference performance of the trained model, and outputs the training data whose first score is less than the first threshold together with the identification ID to the adopted data storage unit 113.

[0050] Here, we will explain why a first score indicating the likelihood of inference that is less than a first threshold is likely to contribute to improving the inference performance of a trained model. Figure 8 is an explanatory diagram of an example of the relationship between a distribution 251 of features of trained images used when training the trained model of this embodiment and the features of images 253 and 255 that are candidates for retraining the trained model. The example shown in Figure 8 illustrates a two-dimensional feature space of image feature s and image feature t. Distribution 251 schematically illustrates the distribution of features of trained images, showing that the number of trained images is greater toward the center of the distribution and decreases as the distribution extends outward from the center.

[0051] The feature values ​​of image 253, which is a candidate for re-learning, are located near the center of distribution 251. As described above, there are many trained images near the center of distribution 251, and many trained images with feature values ​​similar to those of image 253 exist, so the accuracy of inference by the trained model for image 253 tends to be high, and the first score tends to be high. For this reason, it is highly likely that images with feature values ​​similar to those of image 253 have already been trained, and evaluation unit 109 determines that re-learning image 253 is unlikely to contribute to improving the inference performance of the trained model (for example, the detection performance of the detection target object).

[0052] On the other hand, the feature values ​​of image 255, which is a candidate for re-learning, are located near the edge of distribution 251. As described above, the further away from the center of distribution 251 is, the fewer trained images there are, and the fewer trained images there are with feature values ​​similar to those of image 255. Therefore, the accuracy of inference by the trained model for image 255 tends to be low, and the first score tends to be low. For this reason, it is unlikely that an image with feature values ​​similar to those of image 255 has already been trained, and evaluation unit 109 determines that re-learning image 255 is likely to contribute to improving the inference performance of the trained model (for example, the detection performance of the detection target object).

[0053] In this embodiment, as described above, the generation unit 103 generates training data that are candidates for relearning by data augmentation. Therefore, in this embodiment, it is unlikely that an image that is completely unrelated to an image that shows a detection target such as a pedestrian is generated as a candidate image for relearning. As such, in this embodiment, it is unlikely that an image with features that deviate significantly from distribution 251, such as an image with an extremely low first score, is generated. Therefore, in this embodiment, if the first score is less than the first threshold, it can be evaluated as being highly likely to contribute to improving the inference performance of the trained model.

[0054] The rejection data storage unit 111 stores the learning data whose first score is equal to or greater than the first threshold together with the identification ID. Note that the learning data stored in the rejection data storage unit 111 is unlikely to contribute to improving inference performance, and is therefore not used for association with parameters or relearning, which will be described later.

[0055] The adopted data storage unit 113 stores the learning data whose first score is less than the first threshold together with the identification ID. Note that the learning data stored in the adopted data storage unit 113 is highly likely to contribute to improving inference performance, and is therefore adopted for association with parameters, relearning, etc., which will be described later.

[0056] The associating unit 115 associates, for each type of parameter, the value of the parameter used to generate each piece of learning data for re-learning. The associating unit 115 also associates, with the parameter value, learning data generated using the parameter value.

[0057] For example, the associating unit 115 acquires learning data for relearning and an identification ID from the adopted data storage unit 113. Furthermore, the associating unit 115 acquires parameter information (parameter type and parameter value) associated with the acquired identification ID from the parameter storage unit 105. The associating unit 115 generates association information in which the learning data of the parameter value and the identification ID are associated with each other using the acquired parameter type as a key, and outputs the association information to the display control unit 121.

[0058] When the first score output by the inference unit 107 is less than the first threshold, the learning unit 117 re-trains the trained model stored in the trained model storage unit 101 using the training data for which the first score is determined to be less than the first threshold. Specifically, when the evaluation unit 109 determines that the first score is less than the first threshold, the learning unit 117 re-trains the trained model stored in the trained model storage unit 101 using the training data for re-training stored in the adopted data storage unit 113. For example, the learning unit 117 receives the training data (images) as input, and re-trains the trained model using label information included in or added to the training data as the correct answer for an object (a detected object such as a pedestrian) appearing in the training data.

[0059] In detail, the learning unit 117 updates the dataset by adding the learning data for re-learning stored in the adopted data storage unit 113 to the dataset of learned learning data used when learning the trained model. The learning unit 117 re-learns the trained model stored in the trained model storage unit 101 using the updated dataset.

[0060] The retraining method for the trained model may be the same as the training method for the trained model. For example, if the trained model has been trained using a supervised learning method such as deep learning (DL), the trained model may be trained using a similar supervised learning method. However, the training method for the trained model is not limited to this, and any existing machine learning method may be used.

[0061] The input receiving unit 119 receives operation inputs made by a developer, a user, or the like of a trained model using the input device 19. Examples of operation inputs include, but are not limited to, an operation input instructing the display of training data used for relearning, an operation input instructing the display of parameter information used to generate the training data, and the like.

[0062] When the first score output by the inference unit 107 is less than the first threshold, the display control unit 121 displays information about the parameters used to generate the learning data for which the first score is determined to be less than the first threshold on the display device 17. Specifically, the display control unit 121 displays a list of parameters on the display device 17 using the association information acquired from the association unit 115. Furthermore, when a predetermined parameter is selected on the displayed list of parameters, the display control unit 121 displays the distribution of the values ​​of the predetermined parameter on the display device 17. Furthermore, when a predetermined value is selected on the distribution, the display control unit 121 displays the learning data associated with the predetermined value on the display device 17.

[0063] For example, when the input receiving unit 119 receives an operation input instructing display of a parameter list screen, the display control unit 121 displays the parameter list screen on the display device 17. FIG. 9 is a diagram showing an example of a parameter list screen 300 according to this embodiment. The list screen 300 shown in FIG. 9 displays, as a list of parameters, a position 301, a size 311, a rotation angle 321, a posture 331, clothing 341, occlusion 351, a surrounding background 371, and a filter processing 381, which are indicated by parameter type. Note that the position 301, the size 311, the rotation angle 321, the posture 331, and the clothing 341 are parameters related to a detected object such as a pedestrian 203. The occlusion 351 is a parameter related to an obscuring object such as an umbrella 213 that obscures the detected object.

[0064] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting a position 301 on a parameter list screen 300, the display control unit 121 displays a distribution screen showing the distribution of the positions of the pedestrians 203 on the display device 17. FIG. 10 is a diagram showing an example of screen transitions between a distribution screen 303 related to the parameter "position" and a list screen 305 of images for relearning in this embodiment. The distribution screen 303 shows the distribution of the positions of the pedestrians 203 on the images for relearning in two-dimensional x-y coordinates with the center of the screen as the origin (0,0). Note that the images for relearning are images stored in the adopted data storage unit 113. This makes it possible to see at a glance which positions of the pedestrians 203 in the images have been evaluated as having a high probability of contributing to improving the inference performance of the trained model and have been adopted for relearning.

[0065] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting the position coordinates 304 on the distribution screen 303, the display control unit 121 displays on the display device 17 a list screen 305 of images for relearning associated with the value of a parameter indicating the position coordinates 304. Since the value of the parameter indicating the position coordinates 304 is associated with the image for relearning 261 in which the pedestrian 203 is located at the position of the position coordinates 304, the list screen 305 displays a list of the images 261. This allows the user to actually understand what kind of image the pedestrian 203 is located at the position indicated on the distribution screen 303 is, and allows the user to understand details of the image that has been evaluated as having a high possibility of contributing to improving the inference performance of the trained model and has been adopted for relearning.

[0066] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting size 311 on the parameter list screen 300, the display control unit 121 displays a distribution screen showing the distribution of sizes of pedestrians 203 on the display device 17. FIG. 11 is a diagram showing an example of screen transitions between a distribution screen 313 related to the parameter "size" and a list screen 315 of images for relearning in this embodiment. The distribution screen 313 shows the size distribution of pedestrians 203 on the images for relearning in a histogram with the vertical axis representing the number of images adopted for relearning and the horizontal axis representing the size of the pedestrians 203. This makes it possible to understand at a glance what size of images of pedestrians 203 were evaluated as having a high probability of contributing to improving the inference performance of the trained model and were therefore adopted for relearning.

[0067] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting class 314 on the distribution screen 313, the display control unit 121 displays on the display device 17 a list screen 315 of images for re-learning associated with the value of the parameter indicating class 314. Since the value of the parameter indicating class 314 is associated with images for re-learning 221 and 271 that depict pedestrians 203 of the size of class 314, the list screen 315 displays a list of images 221 and 271. This allows the user to actually understand what kind of image an image depicting a pedestrian 203 of the size indicated on the distribution screen 313 is, and allows the user to understand details of the image that has been evaluated as having a high possibility of contributing to improving the inference performance of the trained model and has been adopted for re-learning.

[0068] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting occlusion 351 on the parameter list screen 300, the display control unit 121 displays a list screen of parameters of occlusion objects that occlude the pedestrian 203 on the display device 17. Fig. 12 is a diagram showing an example of screen transitions between a parameter list screen 355 related to the parameter "occlusion" of this embodiment, a distribution screen 361, and a list screen 365 of images for relearning. The list screen 355 displays, as a list of parameters, an object type 356 indicated by the parameter type, an object position 357, an object size 358, and an object rotation angle 359.

[0069] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting object type 356 on the occluding object parameter list screen 355, the display control unit 121 displays a distribution screen 361 showing the distribution of the object types of occluding objects on the display device 17. The distribution screen 361 shows the distribution of occluding objects of the pedestrian 203 on the images for relearning in a histogram with the vertical axis representing the number of images adopted for relearning and the horizontal axis representing the object types of occluding objects. This makes it possible to see at a glance which type of occluding object has been evaluated as having a high probability of contributing to improving the inference performance of the trained model and has been adopted for relearning.

[0070] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting class 362 on the distribution screen 361, the display control unit 121 displays on the display device 17 a list screen 365 of images for relearning associated with the parameter value "bag" indicating class 362. The parameter value "bag" is associated with image for relearning 281, which shows a pedestrian 203 obscured by a bag 215, and image for relearning 286, which shows a pedestrian 203 obscured by a bag 216. Therefore, images 281 and 286 are displayed in a list on the list screen 365. This allows the user to actually understand what kind of image an image showing pedestrian 203 obscured by an obscuring object of the type indicated on the distribution screen 361 is, and allows the user to understand details of images adopted for relearning that are evaluated as having a high possibility of contributing to improving the inference performance of the trained model.

[0071] Furthermore, for example, when the input receiving unit 119 receives an operation input selecting filter processing 381 on the parameter list screen 300, the display control unit 121 displays a distribution screen showing the distribution of filter types for the filter processing on the display device 17. FIG. 13 is a diagram showing an example of screen transitions between a filter type distribution screen 386, a filter strength distribution screen 391, and a list screen 395 for images for relearning, related to the parameter "filter processing" of this embodiment. The filter type distribution screen 386 shows the distribution of filter types for images for relearning in a histogram with the vertical axis representing the number of images adopted for relearning and the horizontal axis representing filter types. This allows users to see at a glance which filter types of images have been evaluated as having a high probability of contributing to improving the inference performance of the trained model and have been adopted for relearning.

[0072] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting class 387 on the filter type distribution screen 386, the display control unit 121 displays a filter strength distribution screen 391 for the parameter value "blur" indicating class 387 on the display device 17. The filter strength distribution screen 391 shows the distribution of filter strengths for the images for re-learning in a histogram with the vertical axis representing the number of images adopted for re-learning and the horizontal axis representing filter strength. This makes it possible to see at a glance which images with which filter strengths have been evaluated as having a high probability of contributing to improving the inference performance of the trained model and have been adopted for re-learning.

[0073] Furthermore, for example, when the input receiving unit 119 receives an operation input for selecting class 392 on the distribution screen 391, the display control unit 121 displays on the display device 17 a list screen 395 of images for re-learning associated with the value of a parameter indicating class 392. The parameter value indicating class 392 is associated with images for re-learning 241 that have been blurred with a filter strength corresponding to the parameter value. Therefore, the list screen 395 displays a list of images 241. This allows the user to actually understand what kind of image has been blurred with the filter strength indicated on the distribution screen 391, and allows the user to understand details of the image that has been evaluated as having a high probability of contributing to improving the inference performance of the trained model and has been adopted for re-learning.

[0074] Next, the flow of processing by the learning device of this embodiment will be described.

[0075] 14 is a flowchart showing an example of the re-learning process by the learning device 10 of this embodiment. Note that the order of the processes shown below is merely an example, and any order may be used as long as the effects of the present disclosure can be obtained.

[0076] First, the generation unit 103 generates candidate images for retraining the trained model stored in the trained model storage unit 101 (step S101). The generation unit 103 also associates the identification ID of the generated image with parameter information (parameter type and parameter value) of the parameters used to generate the image, outputs the associated information to the parameter storage unit 105, and stores the associated information.

[0077] Next, the inference unit 107 uses the trained model stored in the trained model storage unit 101 to infer whether or not a detection target such as a pedestrian is included in the image that is a candidate for re-learning and that is generated by the generation unit 103 (step S103). The inference unit 107 outputs the inference result and a first score that indicates the likelihood of the inference.

[0078] Next, the evaluation unit 109 evaluates the inference result using the first score output by the inference unit 107 (step S105).

[0079] If the first score is less than the first threshold (Yes in step S107), the evaluation unit 109 evaluates that the image to be a candidate for relearning generated by the generation unit 103 is highly likely to contribute to improving the inference performance of the trained model. In order to adopt the image to be a candidate for relearning generated by the generation unit 103 for relearning, the evaluation unit 109 outputs the image together with an identification ID to the adopted data storage unit 113 and stores the image (step S109).

[0080] On the other hand, if the first score is equal to or greater than the first threshold (No in step S107), the evaluation unit 109 evaluates that the image to be a candidate for re-learning generated by the generation unit 103 is unlikely to contribute to improving the inference performance of the trained model. Since the evaluation unit 109 does not adopt the image generated by the generation unit 103 for re-learning, the evaluation unit 109 outputs the image together with the identification ID to the rejected data storage unit 111 and stores them (step S111).

[0081] Next, the processes of steps S103 to S111 are repeated until all images generated by generating unit 103 have been processed (No in step S113), and once all images have been processed (Yes in step S113), the process proceeds to step S115.

[0082] 14, the description is given assuming a case in which the generation unit 103 generates all images that are candidates for relearning in step S101 and then proceeds to step S103, but the present invention is not limited to this. For example, each time the generation unit 103 generates an image that is a candidate for relearning in step S101, the process may proceed to step S103 and perform the processes of steps S103 to S111. In this case, if the generation of all images that are candidates for relearning has not been completed in step S113, the process may return to step S101.

[0083] Next, the associating unit 115 performs an associating process for each type of parameter to associate the parameter value used to generate each image for re-learning with the image for re-learning generated using the parameter value (step S115).

[0084] Next, the learning unit 117 re-learns the trained model stored in the trained model storage unit 101 using the image for re-learning stored in the adopted data storage unit 113 (step S117).

[0085] Fig. 15 is a flowchart showing an example of display processing by the learning device 10 of this embodiment. The display processing shown in Fig. 15 may be performed at any timing after the association processing described in step S115 of the flowchart shown in Fig. 14. For example, it may be performed either before or after the relearning described in step S117 of the flowchart shown in Fig. 14. The order of processing shown below is merely an example, and any order may be used as long as the effects of the present disclosure can be obtained.

[0086] First, when the input receiving unit 119 receives an operation input instructing to display a parameter list screen (step S201), the display control unit 121 displays the parameter list screen on the display device 17 (step S203).

[0087] Next, when the input receiving unit 119 receives an operation input instructing the selection of a parameter on the parameter list screen 300 (step S205), the display control unit 121 displays a distribution screen showing the distribution of the values ​​of the selected parameter on the display device 17 (step S207).

[0088] Next, when the input receiving unit 119 receives an operation input instructing the selection of a value on the parameter value distribution screen (step S209), the display control unit 121 displays a list screen of images for re-learning associated with the selected value on the display device 17 (step S211).

[0089] As described above, the learning device of this embodiment uses a trained model to infer training data generated by data augmentation. If the first score indicating the likelihood of inference is less than a first threshold, the learning device of this embodiment evaluates that the generated training data is likely to contribute to improving the inference performance of the trained model, and retrains the trained model using the training data. In this way, the learning device of this embodiment can prevent retraining using training data that does not contribute to improving the inference performance of the trained model, thereby reducing training costs and improving the inference performance of the trained model.

[0090] Furthermore, the learning device of this embodiment displays information about the parameters used to generate the training data used for relearning. Therefore, the learning device of this embodiment makes it possible to objectively grasp which information about the parameters used to generate the training data is likely to contribute to improving the inference performance of the trained model. This makes it possible to provide a well-founded explanation of the learning process of the trained model even to users of the trained model who are not familiar with the trained model.

[0091] (Variation 1) In the above embodiment, an example has been described in which the generation unit 103 generates one image that is a candidate for relearning using one generation method. However, the use of the generation method when generating an image that is a candidate for relearning is not limited to this. In Modification 1, an example will be described in which a plurality of generation methods are used to generate one image that is a candidate for relearning.

[0092] FIG. 16 is an explanatory diagram showing an example of generating an image that is a candidate for relearning in Modification Example 1. In the example shown in FIG. 16, the generation unit 103 generates an image that is a candidate for relearning by using a plurality of generation AIs and a synthesis process in combination. For example, in a first step, the generation AI 404A generates an image 201 that shows a pedestrian 203 walking on a road, and the generation AI 404B generates an image 211 that shows an umbrella 213. Subsequently, in a second step, the generation unit 103 synthesizes the umbrella 213 that appears in the image 211 with the image 201. As a result, the generation unit 103 has the pedestrian 203 walking on the road hold up the umbrella 213, and generates an image 221 in which the pedestrian 203 is partially concealed by the umbrella 213, as an image that is a candidate for relearning.

[0093] FIG. 17 is an explanatory diagram showing another example of generating an image that is a candidate for relearning in Modification Example 1. In the example shown in FIG. 17, the generation unit 103 generates an image that is a candidate for relearning by using a generation AI, a filter process, and a composition process in combination. For example, in a first step, the generation AI 404A generates an image 201 that shows a pedestrian 203 walking on a road, and the generation unit 103 generates an image 411 by applying a filter process to an image 211 that shows an umbrella 213. Subsequently, in a second step, the generation unit 103 composites the filtered umbrella 413 that appears in the image 411 with the image 201. As a result, the generation unit 103 has the pedestrian 203 walking on the road hold up the umbrella 413, and generates an image 421 in which the pedestrian 203 is partially concealed by the umbrella 413 as a candidate for relearning.

[0094] In this way, in Modification 1, the generation unit 103 uses a plurality of generation methods in combination to generate training data (an example of first training data) that are candidates for relearning. According to Modification 1, multiple generation processes are used in combination to generate images that are candidates for relearning in a plurality of steps, which makes it easier to generate images that are candidates for relearning as intended by the developer of the generative model.

[0095] (Variation 2) In Modification 2, an example will be described in which trial training data is used to identify a training data generation method that is likely to contribute to improving the inference performance of a trained model, and the identified generation method is used to generate training data that is a candidate for re-training. Below, differences from the above embodiment will be mainly described, and components that have the same functions as the above embodiment will be given the same names and symbols as the above embodiment, and their description will be omitted.

[0096] Fig. 18 is a block diagram showing an example of the configuration of a learning device 510 of Modification 2. As shown in Fig. 18, the learning device 510 of Modification 2 differs from the learning device 10 of the above embodiment in that it has a generation unit 603 and an inference unit 607. The learning device 510 of Modification 2 also differs from the learning device 10 of the above embodiment in that it further includes an identification unit 623.

[0097] The generation unit 603 generates trial training data (an example of second training data). The trial training data is data used to identify a generation method that is likely to contribute to improving the inference performance of a trained model. Therefore, a smaller amount of trial training data is sufficient compared to re-training training data. In Modification 2, an example will be described in which the generation unit 603 generates small amounts of trial training data using each of the multiple generation methods described in the above embodiment, but the present invention is not limited to this. Furthermore, in Modification 2, an example will be described in which the trial training data is an image that shows a detection target such as a pedestrian, similar to the re-training training data, but the present invention is not limited to this.

[0098] The inference unit 607 uses the trained model stored in the trained model memory unit 101 to perform inference on each trial training data generated by the generation unit 603, and outputs the inference result and a second score indicating the likelihood of the inference.

[0099] The identification unit 623 aggregates the trial learning data for which the second score output by the inference unit 607 is less than the first threshold value for each generation method, and identifies the generation method with the largest aggregate value. In this way, the identification unit 623 identifies the generation method that was most successful in generating trial learning data that is most likely to contribute to improving the inference performance of the trained model. Note that, in addition to the largest aggregate value, an aggregate value equal to or greater than a predetermined value may also be used.

[0100] The generating unit 603 generates training data (an example of first training data) that are candidates for re-learning, using the generation method identified by the identifying unit 623. Specifically, the generating unit 603 generates a large amount of training data that are candidates for re-learning, using the generation method identified by the identifying unit 623, in order to build a dataset of training data for re-learning.

[0101] As described above, in Modification 2, training data that are candidates for re-training are generated using the generation method that was most successful in generating trial training data that is likely to contribute to improving the inference performance of the trained model. Therefore, according to Modification 2, it is expected that the training data that are candidates for re-training will also be more likely to contribute to improving the inference performance of the trained model, and it is expected that the generation of training data for re-training will be more efficient.

[0102] (Variation 3) In Variation 3, an example will be described in which a generative model is tuned so that it can generate training data that is likely to contribute to improving the inference performance of a trained model, and the tuned generative model is used to generate training data that is a candidate for re-training. Below, differences from the above embodiment will be mainly described, and components that have the same functions as the above embodiment will be given the same names and symbols as the above embodiment, and their description will be omitted.

[0103] Fig. 19 is a block diagram showing an example of the configuration of a learning device 710 of Modification 3. As shown in Fig. 19, the learning device 710 of Modification 3 differs from the learning device 10 of the above embodiment in that it has a generation unit 803 and an inference unit 807. The learning device 710 of Modification 3 also differs from the learning device 10 of the above embodiment in that it further includes an update unit 825.

[0104] The generation unit 803 generates trial training data (an example of third training data) using a generation AI (an example of a generation model). Examples of the generation AI include, but are not limited to, the generation AI described in the above embodiment and Modification 1. The trial training data is data used to tune the generation AI so that the generation AI can generate training data that is likely to contribute to improving the inference performance of a trained model. Therefore, a smaller amount of trial training data is sufficient than training data for re-learning. In Modification 2, an example will be described in which the trial training data is, like the training data for re-learning, images that include detection targets such as pedestrians, but the present invention is not limited to this.

[0105] The inference unit 807 uses the trained model stored in the trained model memory unit 101 to perform inference on each trial training data generated by the generation unit 803, and outputs the inference result and a third score indicating the likelihood of the inference.

[0106] The update unit 825 updates the generation AI used by the generation unit 803 by using the trial learning data whose third score is less than the first threshold and output by the inference unit 807. For example, the update unit 825 fine-tunes the generation AI used by the generation unit 803 by using the trial learning data whose third score is less than the first threshold. In this way, the update unit 825 tunes the generation AI by using the trial learning data that is likely to contribute to improving the inference performance of the trained model.

[0107] The generation unit 803 generates training data (an example of first training data) that are candidates for re-learning, using the generation AI updated by the update unit 825. Specifically, the generation unit 803 generates a large amount of training data that are candidates for re-learning, using the generation AI updated by the update unit 825, in order to build a dataset of training data for re-learning.

[0108] As described above, in Modification 3, the generation AI is fine-tuned using trial training data that is likely to contribute to improving the inference performance of the trained model, and the tuned generation AI is used to generate training data that are candidates for re-training. Therefore, according to Modification 3, it is expected that the training data that are candidates for re-training will also be more likely to contribute to improving the inference performance of the trained model, and it is expected that the generation of training data for re-training will be more efficient.

[0109] (Variation 4) In Modification 4, an example will be described in which the score output by the inference unit 107 is calibrated. The following mainly describes the differences from the above embodiment, and components having the same functions as those in the above embodiment will be given the same names and symbols as those in the above embodiment, and their description will be omitted.

[0110] Fig. 20 is a block diagram showing an example of the configuration of a learning device 910 of Modification 4. As shown in Fig. 20, the learning device 910 of Modification 4 differs from the learning device 10 of the above embodiment in that an evaluation unit 1009 includes a calibration unit 1027 and in that a learning unit 1017 is included.

[0111] The calibration unit 1027 performs calibration on the first score output by the inference unit 107. For example, the calibration unit 1027 performs an Out Of Distribution (OOD) determination as the calibration. The calibration clarifies whether the training data generated by the generation unit 103 is located within or outside the distribution of the trained training data used to train the trained model stored in the trained model storage unit 101.

[0112] The evaluation unit 1009 uses the first score calibrated by the calibration unit 1027 to evaluate whether the training data that is a candidate for relearning and generated by the generation unit 103 is likely to contribute to improving the inference performance of the trained model. For example, the evaluation unit 1009 determines whether the calibrated first score is less than a first threshold and whether the training data that is a candidate for relearning and generated by the generation unit 103 is outside the distribution of trained training data used to train the trained model. If the first score is less than the first threshold and the training data that is a candidate for relearning is outside the distribution of trained training data, the evaluation unit 1009 determines that the training data is likely to contribute to improving the inference performance of the trained model, and outputs the result to the adopted data storage unit 113. On the other hand, if the first score is less than the first threshold and the training data that is a candidate for relearning is not outside the distribution of trained training data, the evaluation unit 1009 determines that the training data is unlikely to contribute to improving the inference performance of the trained model, and outputs the result to the rejected data storage unit 111.

[0113] 21 is an explanatory diagram showing an example of whether or not an image that is a candidate for relearning in Modification Example 4 is adopted as an image for relearning, comparing before and after calibration. Graph 1110 shows whether or not an image that is a candidate for relearning before calibration is adopted as an image for relearning. Graph 1120 shows whether or not an image that is a candidate for relearning after calibration is adopted as an image for relearning. In graphs 1110 and 1120, plotted dots indicate images that are candidates for relearning.

[0114] Boundary 1101 indicates whether the first score output by inference unit 107 is less than or equal to a first threshold. Boundary 1101 indicates, for example, whether inference unit 107 has been able to detect a likely detection target in an image through inference. The area in the direction of arrow 1103 (the area above boundary 1101) based on boundary 1101 indicates that the first score of an image candidate for re-learning is less than the first threshold (for example, an object likely to be a detection target, such as a pedestrian, could not be detected in the image). The area in the direction of arrow 1105 based on boundary 1101 (the area below boundary 1101) indicates that the first score of an image candidate for re-learning is greater than or equal to the first threshold (for example, an object likely to be a detection target, such as a pedestrian, was detected in the image). Therefore, in graph 1110, of the 11 images candidate for re-learning, seven images located in the area in the direction of arrow 1103 are adopted as images for re-learning.

[0115] Meanwhile, distribution 1107 in graph 1120 schematically shows the distribution of trained images used in training the trained model. Four images that are candidates for re-training and are located within distribution 1107 and in the area indicated by arrow 1103 are actually untrained but have been determined to have already been trained. This can occur, for example, when the trained model erroneously infers that a detected object (pedestrian) in an image that is a candidate for re-training is actually a different object and has nearly the same appearance, or when the trained model lacks expressive power. The former is referred to as data uncertainty, and for example, a case can occur in which a pedestrian and a pole are inferred to have the same appearance due to insufficient resolution. The latter is referred to as model uncertainty, and so on. Data uncertainty and model uncertainty are fundamentally difficult to improve even if the candidate images for re-training are tweaked, so they are not used as images for re-training and are excluded. Therefore, in graph 1120, of the 11 images that are candidates for re-learning, three images that are outside distribution 1107 and in the area in the direction of arrow 1103 will be adopted as images for re-learning.

[0116] If the calibrated first score is less than the first threshold and the candidate training data for re-learning generated by the generation unit 103 is outside the distribution of trained training data, the training unit 1017 re-trains the trained model using the candidate training data for re-learning.

[0117] As described above, according to variant example 4, calibration allows training data that is unlikely to contribute to improving the inference performance of the trained model, even if the first score is less than the first threshold, to be excluded from the images used for re-training, thereby making it possible to further reduce training costs.

[0118] (Variation 5) In the above embodiment, an example has been described in which, when the first score is less than the first threshold, the training data that is a candidate for relearning is evaluated as having a high probability of contributing to improving the inference performance of the trained model and is adopted as the training data for relearning. In Modification 5, when the first score is less than the first threshold and equal to or greater than the second threshold, the training data that is a candidate for relearning may be evaluated as having a high probability of contributing to improving the inference performance of the trained model and may be adopted as the training data for relearning. Note that the second threshold is a threshold that is less than the first threshold. In this way, even if the first score is extremely low and training data with features that deviate significantly from the distribution 251 shown in FIG. 8 is generated, the training data can be excluded and not adopted as images for relearning.

[0119] (Variation 6) In the above embodiment, the training data stored in the rejection data storage unit 111 is not associated with parameters, but it may be associated with parameters and displayed together with the parameters. In this way, it is possible to objectively grasp training data and parameter information that is unlikely to contribute to improving the inference performance of the trained model.

[0120] (program) The programs executed by the learning devices of the above embodiments and each of the above variations are provided as files in an installable or executable format stored on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD).

[0121] The programs executed by the learning devices of the above embodiments and each of the above modifications may be stored on a computer connected to a network such as the Internet and provided by downloading them via the network. The programs executed by the learning devices of the above embodiments and each of the above modifications may be provided or distributed via a network such as the Internet. The programs executed by the learning devices of the above embodiments and each of the above modifications may be provided by being pre-installed in a ROM or the like.

[0122] The programs executed by the learning devices of the above-described embodiments and modifications have a modular structure for implementing the above-described units on a computer. In terms of actual hardware, for example, the CPU reads the learning program from a hard disk drive onto a RAM and executes it, thereby implementing the above-described units on a computer.

[0123] As described above, according to the above embodiment and each of the above modified examples, it is possible to reduce the learning cost and improve the inference performance of the trained model.

[0124] The above-described embodiment and each of the above-described modifications merely illustrate examples of specific embodiments of the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these. For example, the present disclosure can be implemented in various forms without departing from the gist or main features thereof. For example, the above-described embodiment and each of the above-described modifications may be appropriately combined in their respective constituent units. Furthermore, for example, some components may be deleted from all components in the above-described embodiment and each of the above-described modifications.

[0125] In the above description, the notation "... part" used for each component may be replaced with other notations such as "... assembly," "... circuit," "... device," "... unit," or "... module."

[0126] The present disclosure includes the following aspects.

[0127] (1) a generation unit that generates first training data; an inference unit that performs inference on the first training data using a trained model and outputs a first score that indicates the likelihood of the inference; a learning unit that re-learns the trained model using the first training data when the first score is less than a first threshold; A learning device comprising:

[0128] (2) The computer further includes a display control unit that displays information about parameters used to generate the first training data when the first score is less than the first threshold. The learning device according to (1) above.

[0129] (3) The method further includes an associating unit that associates, for each type of parameter, a value of the parameter used in generating each piece of re-learning learning data including the first learning data, and the display control unit, when a predetermined parameter is selected on the displayed list of parameters, displays a distribution of values ​​of the predetermined parameter. The learning device according to (2) above.

[0130] (4) The associating unit further associates, with the value of the parameter, learning data generated using the value of the parameter; when a predetermined value is selected on the distribution, the display control unit displays the learning data associated with the predetermined value. The learning device according to (3) above.

[0131] (5) The generation unit generates the first training data by using a plurality of generation methods in combination. The learning device according to (1) above.

[0132] (6) The generation unit generates second trial training data using each of a plurality of generation methods; the inference unit performs inference on each second training data using the trained model and outputs a second score indicating the likelihood of the inference; an identification unit that aggregates, for each generation method, the second training data whose second score is less than the first threshold, and identifies the generation method with the largest aggregated score; the generation unit generates the first training data using the identified generation method. The learning device according to (1) above.

[0133] (7) the generation unit generates third trial training data using the generative model; the inference unit performs inference on the third training data using the trained model and outputs a third score indicating the likelihood of the inference; an update unit that updates the generative model using third training data in which the third score is less than the first threshold; the generation unit generates the first training data using the updated generative model. The learning device according to (1) above.

[0134] (8) The method further includes a calibration unit that calibrates the first score output by the inference unit. The learning device according to (1) above.

[0135] (9) When the calibrated first score is less than the first threshold and the first training data is outside a distribution of trained training data used to train the trained model, the training unit retrains the trained model using the first training data. The learning device according to (8) above.

[0136] (10) A generation unit generates first training data; an inference unit performs inference on the first training data using the trained model and outputs a first score indicating the likelihood of the inference; a learning unit that, when the first score is less than a first threshold, re-learns the trained model using the first training data; How to learn.

[0137] (11) The display control unit further displays information about parameters used to generate the first learning data when the first score is less than the first threshold. The learning method described in (10) above.

[0138] (12) The associating unit further associates, for each type of the parameter, a value of the parameter used to generate each piece of re-learning learning data including the first learning data; when a predetermined parameter is selected on the displayed list of parameters, the display control unit displays a distribution of values ​​of the predetermined parameter. The learning method described in (11) above.

[0139] (13) The associating unit further associates, with the value of the parameter, learning data generated using the value of the parameter, the display control unit, when a predetermined value is selected on the distribution, displays the learning data associated with the predetermined value. The learning method described in (12) above.

[0140] (14) The generation unit generates the first training data by using a plurality of generation methods in combination. The learning method described in (10) above.

[0141] (15) The generation unit generates second trial training data using each of a plurality of generation methods; the inference unit performs inference on each second training data using the trained model and outputs a second score indicating the likelihood of the inference; further, the identification unit aggregates the second training data for which the second score is less than the first threshold value for each generation method, and identifies the generation method for which the aggregate value is the largest; the generation unit generates the first training data using the identified generation method. The learning method described in (10) above.

[0142] (16) The generation unit generates third trial training data using the generative model; the inference unit performs inference on the third training data using the trained model and outputs a third score indicating the likelihood of the inference; further, an update unit updates the generative model using third training data in which the third score is less than the first threshold; the generation unit generates the first training data using the updated generative model. The learning method described in (10) above.

[0143] (17) Furthermore, a calibration unit calibrates the first score output by the inference unit. The learning method described in (10) above.

[0144] (18) The learning unit retrains the trained model using the first training data when the calibrated first score is less than the first threshold and the first training data is outside a distribution of trained training data used to train the trained model. The learning method described in (17) above.

[0145] (19) a generation step of generating first learning data; an inference step of performing inference on the first training data using the trained model and outputting a first score indicating the likelihood of the inference; a learning step of re-learning the trained model using the first training data if the first score is less than a first threshold; A program that causes a computer to execute the following.

[0146] (20) A trained model generated by the device according to any one of (1) to (9), the method according to any one of (10) to (18), or the program according to (19). [Explanation of symbols]

[0147] 10, 510, 710, 910 Learning Device 11 Control device 13 Main memory 15 Auxiliary storage 17 Display device 19 Input Devices 21 Communication equipment 23 Various buses 101 Trained model memory unit 103, 603, 803 generation section 105 Parameter storage unit 107, 607, 807 Reasoning section 109, 1009 Evaluation Department 111 Rejected Data Storage Unit 113 Recruitment Data Storage Unit 115 Association section 117, 1017 Learning Department 119 Input reception section 121 Display control unit 623 Specific part 825 Update Department 1027 Calibration Section

Claims

1. a generation unit that generates first learning data; an inference unit that performs inference on the first training data using a trained model and outputs a first score that indicates the likelihood of the inference; a learning unit that re-learns the trained model using the first training data when the first score is less than a first threshold; A learning device comprising:

2. a display control unit that displays information about parameters used to generate the first learning data when the first score is less than the first threshold value; The learning device according to claim 1 .

3. an associating unit that associates, for each type of parameter, a value of the parameter used in generating each piece of re-learning learning data including the first learning data; the display control unit, when a predetermined parameter is selected on the displayed list of parameters, displays a distribution of values ​​of the predetermined parameter. The learning device according to claim 2 .

4. the associating unit further associates, with the value of the parameter, learning data generated using the value of the parameter; when a predetermined value is selected on the distribution, the display control unit displays the learning data associated with the predetermined value. The learning device according to claim 3 .

5. the generation unit generates the first training data by using a plurality of generation methods in combination. The learning device according to claim 1 .

6. the generation unit generates trial second training data using each of a plurality of generation methods; the inference unit performs inference on each second training data using the trained model and outputs a second score indicating the likelihood of the inference; an identification unit that aggregates the second learning data for which the second score is less than the first threshold value for each generation method and identifies the generation method for which the aggregate value is the largest; the generation unit generates the first training data by using the identified generation method. The learning device according to claim 1 .

7. the generation unit generates third trial training data using the generative model; the inference unit performs inference on the third training data using the trained model and outputs a third score indicating the likelihood of the inference; an update unit that updates the generative model using third training data in which the third score is less than the first threshold; the generation unit generates the first training data using the updated generative model. The learning device according to claim 1 .

8. further comprising a calibration unit that calibrates the first score output by the inference unit, The learning device according to claim 1 .

9. the learning unit retrains the trained model using the first training data when the calibrated first score is less than the first threshold and the first training data is outside a distribution of trained training data used to train the trained model; The learning device according to claim 8 .

10. a generating unit generating first training data; an inference unit performs inference on the first training data using the trained model and outputs a first score indicating the likelihood of the inference; a learning unit that, when the first score is less than a first threshold, re-learns the trained model using the first training data; How to learn.

11. and a display control unit that, when the first score is less than the first threshold, displays information about parameters used to generate the first learning data. The learning method according to claim 10.

12. the associating unit associates, for each type of parameter, a value of the parameter used in generating each piece of re-learning learning data including the first learning data; when a predetermined parameter is selected on the displayed list of parameters, the display control unit displays a distribution of values ​​of the predetermined parameter. The learning method according to claim 11.

13. the associating unit further associates, with the value of the parameter, learning data generated using the value of the parameter; the display control unit, when a predetermined value is selected on the distribution, displays the learning data associated with the predetermined value. The learning method according to claim 12.

14. the generation unit generates the first training data by using a plurality of generation methods in combination. The learning method according to claim 10.

15. the generation unit generates trial second training data using each of a plurality of generation methods; the inference unit performs inference on each second training data using the trained model and outputs a second score indicating the likelihood of the inference; the identification unit aggregates the second learning data for which the second score is less than the first threshold value for each generation method, and identifies the generation method for which the aggregate value is the largest; the generation unit generates the first training data using the identified generation method. The learning method according to claim 10.

16. the generation unit generates third trial training data using the generative model; the inference unit performs inference on the third training data using the trained model and outputs a third score indicating the likelihood of the inference; further, an update unit updates the generative model using third training data in which the third score is less than the first threshold; the generation unit generates the first training data using the updated generative model. The learning method according to claim 10.

17. Furthermore, a calibration unit calibrates the first score output by the inference unit. The learning method according to claim 10.

18. the learning unit, when the calibrated first score is less than the first threshold and the first learning data is outside a distribution of learned learning data used to learn the learned model, re-learning the learned model using the first learning data; The learning method according to claim 17.

19. a generating step of generating first learning data; an inference step of performing inference on the first training data using the trained model and outputting a first score indicating the likelihood of the inference; a learning step of re-learning the trained model using the first training data if the first score is less than a first threshold; A program that causes a computer to execute the following.

20. A trained model generated by the device according to any one of claims 1 to 9, the method according to any one of claims 10 to 18, or the program according to claim 19.

Images