Retraining device, retraining method, and program

Abstract

To provide a re-learning apparatus, a re-learning method, and a program configured to suppress increase in learning cost.SOLUTION: A re-learning apparatus 10 re-learns a re-learning dataset and updates a learning model. The re-learning apparatus 10 includes: a storage unit 11 whish stores a learned dataset including data indicating a learned image, a score output by performing image determination processing based on the learning model on the learned image, and a target region of the image determination processing; a region acquisition unit 13 which re-acquires, based on an instruction of a user, in the learned dataset, a target region of the image determination processing in an image of which the score is equal to or less than a first threshold; and an unlearned dataset acquisition unit 14 which acquires an unlearned dataset including data indicating an unlearned image which has not been learned, a score output by performing the image determination processing based on the learning model on the unlearned image, and the target region of the image determination processing.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a relearning device, a relearning method, and a program.

Background Art

[0002] The inspection device disclosed in Patent Document 1 stores the estimation result of the state of the inspection object and the reliability of the estimation result in association with the data related to the inspection object. The inspection device extracts additional learning data from the stored data. The inspection device instructs the machine learning device to perform relearning using the learning data created based at least on the extracted additional learning data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The inventor of the present application has discovered the following technical problems. In such an inspection device, the learning data used for initial learning may include data that is not included during subsequent inspections. Therefore, when such an inspection device performs relearning, there is a risk that the cost of creating the learning data used for relearning and the learning cost due to an increase in the total amount of this learning data will increase.

[0005] The present disclosure has been made in view of the above problems, and provides a relearning device, a relearning method, and a program capable of suppressing the risk of increasing learning costs.

Means for Solving the Problems

[0006] In the first aspect of this disclosure, a retraining device updates a learning model by retraining a retraining dataset, A storage unit that stores a trained dataset including data showing trained images, scores output by performing image classification processing on the trained images based on the trained model, and regions targeted for the image classification processing. A region acquisition unit that, based on user instructions, reacquires the region targeted for the image judgment process in the trained dataset where the score was below a first threshold in the image, An untrained dataset acquisition unit acquires an untrained dataset that includes data showing untrained images, a score output by performing image classification processing on the untrained images based on the learning model, and the region targeted for the image classification processing. A relearning device is provided, comprising a generation unit that generates a relearning dataset including the trained dataset, data showing an image including the region targeted for the reacquired image determination process, and the untrained dataset.

[0007] In the second aspect of this disclosure, a retraining method is performed by a retraining device that retrains a retraining dataset to update a learning model, A trained dataset is stored which includes data showing trained images, scores output by performing image classification processing on the trained images based on the trained model, and the regions targeted for the image classification processing. Based on user instructions, the region targeted for the image classification process in the trained dataset where the score was below the first threshold is reacquired. An untrained dataset is obtained that includes data showing untrained images, scores output by performing image classification processing on the untrained images based on the learning model, and the regions targeted by the image classification processing. A relearning method is provided in which a relearning device performs a process to generate a relearning dataset that includes the trained dataset, data showing an image including the region targeted for the image determination process that has been reacquired, and the untrained dataset.

[0008] In the third aspect of this disclosure, a program is to be executed by a retraining device that retrains a retraining dataset and updates the learning model, A trained dataset is stored which includes data showing trained images, scores output by performing image classification processing on the trained images based on the trained model, and the regions targeted for the image classification processing. Based on user instructions, the region targeted for the image classification process in the trained dataset where the score was below the first threshold is reacquired. An untrained dataset is obtained that includes data showing untrained images, scores output by performing image classification processing on the untrained images based on the learning model, and the regions targeted by the image classification processing. A program is provided that causes a relearning device to execute a process to generate a relearning dataset that includes the trained dataset, data showing an image containing the region targeted for the image determination process that has been reacquired, and the untrained dataset. [Effects of the Invention]

[0009] According to this disclosure, the risk of increased learning costs can be suppressed. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an example of the configuration of a relearning device according to an embodiment. [Figure 2] Figure 2 is a flowchart showing an example of the processing of the relearning device according to the embodiment. [Figure 3] Figure 3 is a schematic diagram showing the data hierarchy of an example initial training dataset. [Figure 4] Figure 4 is a schematic diagram showing an example of a learning image according to the embodiment. [Figure 5] Figure 5 is a graph showing the frequency of each mode in an example of each dataset. [Figure 6]Figure 6 shows an example of the hardware configuration of an information processing device according to the embodiment. [Modes for carrying out the invention]

[0011] The principles of this disclosure will be described with reference to several exemplary embodiments. These embodiments are described for illustrative purposes only and should be understood as helping those skilled in the art to understand and implement this disclosure without implying any limitation on the scope of this disclosure. The disclosures described herein may be implemented in various ways other than those described below. In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those generally understood by those skilled in the art to which this disclosure belongs. Embodiments of this disclosure will be described below with reference to the drawings. Specific embodiments of the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited to the following embodiments. Also, for clarity of explanation, the following description and drawings have been simplified as appropriate.

[0012] <Structure> Referring to Figure 1, the configuration of the relearning device 10 according to the embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the relearning device 10 according to the embodiment. In the example of Figure 1, the relearning device 10 has a storage unit 11, an image determination processing unit 12, a region acquisition unit 13, an unlearned data set acquisition unit 14, a generation unit 15, and a learning unit 16. Each of these units may be realized through the cooperation of one or more programs installed in the relearning device 10 and hardware such as the processor and memory of the relearning device 10.

[0013] The storage unit 11 stores the learned dataset. The learned dataset includes data indicating the learned image, a score output by performing image determination processing on the learned image based on the learning model, and the region targeted for the image determination processing. The score and the region targeted for the image determination processing may be associated with each learned image. The learned dataset is, for example, an initial learning dataset or a relearning dataset that has already been relearned. Note that the learned dataset may include data indicating the date and time when the learned image was captured or the order.

[0014] A camera device (not shown) captures an image and generates data indicating the captured image. The camera device outputs the generated data to the storage unit 11. The camera device may also output to the storage unit 11 the date and time or the order when the camera device captured the image. The camera device may be an external device of the relearning device 10 or may be included in the relearning device 10.

[0015] The image determination processing unit 12 performs image determination processing on the images in the unlearned dataset and the learned dataset based on the learning model to detect the object of image determination. The image determination processing unit 12 may output the area and score of the object of image determination by the image determination processing. The image determination processing technology can utilize, for example, YOLO known as a real-time object detection algorithm. The score indicates the accuracy of the image determination processing. Based on the score, the result of the image determination processing may be "OK" if the object of the image determination processing can be detected from the image, "NG" if the object of the image determination processing cannot be determined from the image, and "KP (KEEP)" if the determination is postponed. The object of the image determination processing is, for example, a workpiece. The workpiece may be, for example, a long object. The area of the object of the image determination processing may be, for example, a bounding box that is a rectangle (rectangle or square) surrounding the object of the image determination processing shown in the image. When the object of the image determination processing is a long object, the area of the object of the image determination processing is a rectangle, and the aspect ratio of the rectangle may be greater than 0 (zero) and less than or equal to 0.5, or 2.0 or more. Thereby, suitable image determination processing can be performed on the long object that is the object of the image determination processing. Also, the information indicating the area of the image determination processing mode in the image may be, for example, segmentation data that is a pixel group of the object of the image determination shown in the image.

[0016] Based on the user's instruction, the area acquisition unit 13 reacquires the area of the object of the image determination processing in the image in the learned dataset where the score is less than or equal to the first threshold. The image where the score is less than or equal to the first threshold may be, for example, the result of the image determination processing is "NG".

[0017] The untrained dataset acquisition unit 14 acquires an untrained dataset. The untrained dataset includes data representing untrained images that have not been trained, scores output by performing image judgment processing on the untrained images based on the trained model, and the regions targeted for image judgment processing. For example, the untrained dataset includes data representing images taken by a camera device (not shown), the regions targeted for image judgment processing output by the image judgment processing unit 12, and the scores.

[0018] The generation unit 15 generates a retraining dataset that includes the trained dataset stored in the storage unit 11, data indicating images containing the target region for image judgment processing reacquired by the region acquisition unit 13, and the untrained dataset acquired by the untrained dataset acquisition unit 14.

[0019] The learning unit 16 updates the learning model by retraining on the retraining dataset generated by the domain acquisition unit 13. The learning unit 16 may also repeat this update. Alternatively, the learning unit 16 may create a learning model by training on the initial training dataset. When the learning unit 16 updates the learning model for the first time, the retraining dataset may be the initial training dataset. When the learning unit 16 updates the learning model for the second time or later, the retraining dataset may be a retrained retraining dataset.

[0020] <Processing> Next, an example of the processing of the retraining device 10 according to the embodiment will be described with reference to Figures 2 to 5. Figure 2 is a flowchart of an example of the processing of the retraining device according to the embodiment. Figure 3 is a schematic diagram showing the data hierarchy of an example of an initial training dataset. Figure 4 is a schematic diagram showing an example of training images according to the embodiment. Figure 5 is a graph showing the frequency in each mode of an example of each dataset.

[0021] Each process in Figure 2 may be executed, for example, when a predetermined operation is performed by the user (operator of the relearning device 10). The relearning device 10 may also repeatedly execute each process in Figure 2 until it has performed the number of relearning sessions specified by the user. Furthermore, each process in Figure 2 may be executed in a different order, as long as it is not contradictory.

[0022] The initial training dataset is obtained (Step ST1). Specifically, the region acquisition unit 13 obtains the initial training dataset stored in the storage unit 11. The initial training dataset includes data indicating images that have been trained in the initial training, data indicating the score output after image classification processing of the said images, and data indicating the region targeted by the image classification processing, and these data are associated with each other. The score output after image classification processing of the said images and the region targeted by the image classification processing may be provided in advance by the user.

[0023] Image IM1, shown in Figure 3, is an example of an image that makes up the initial training dataset. Image IM1 shows OB1 and OB2, which are targets for image classification processing. Targets OB1 and OB2 are workpieces, and these workpieces are long. A bounty box BB1, which is the region of target OB1 for image classification processing, is set. Bounty box BB1 is a rectangle with width W1 and height H1, centered at a defined center point (Xc, Yc) in a Cartesian coordinate system. The aspect ratio of this rectangle is the ratio of width W1 to height H1. In the example shown in Figure 3, the score (Score ZZZ) of target OB1 for image classification processing is indicated.

[0024] The images that make up the initial training dataset may be intentionally created by the user. For example, the images that make up the initial training dataset may be intentionally created by the user so that the image judgment processing unit 12 of the retraining device 10 is more difficult to judge compared to the images that make up the retraining dataset. For example, if the target of the image judgment processing is a workpiece, the image may be intentionally created by the user so that the workpiece is positioned close to the wall of the basket. Also, the image may be intentionally created by the user so that multiple workpieces overlap, the background area ratio is higher than that of the workpiece, or the intensity of reflected light is higher. Furthermore, in the initial training dataset, the region of the image to be judged may be acquired by the user's instruction. The number of data showing images in the initial training dataset should be smaller than the number of data showing images in the untrained dataset acquired in step ST4. For example, the initial training dataset consists of sample data.

[0025] An example of an initial training dataset shown in Figure 4 may have data hierarchies such as the first, second, and third layers, and be stored in the storage unit 11. The first, second, and third layers each contain images to be classified, the results of the classification, and the images themselves. Multiple classification results are grouped for each image to be classified. Multiple images are grouped for each image to be classified. Examples of images to be classified in the first layer include "First Product" and "Second Product". Examples of classification results in the second layer include "OK", "NG", and "KP". Examples of images in the third layer include "First Image", "Second Image", "nth Image", and "Image". For example, "First Image", "Second Image", and "nth Image" are grouped into group L1, which is "First Product" in the first layer and "OK" in the second layer. Each image in the third layer may be associated with data such as score, frequency, date and time of capture, and order.

[0026] Figure 5 shows the frequency distribution for each mode in an example of each dataset. The examples of datasets shown in Figure 5 are the initial training dataset SD1, the untrained dataset UD1, and the retraining dataset RD1. The frequency distribution for SD1, an example of the initial training dataset, is mainly distributed in modes DT1 to DT3. The initial training dataset SD1 consists of sample data.

[0027] Next, the initial training dataset is trained to generate the trained model (Step ST2).

[0028] The learning unit 16 learns, for example, the initial training dataset SD1. Then, the data in modes DT1 and DT2 are frequently observed and are therefore well learned by the relearning device 10. On the other hand, the data in mode DT3 appears infrequently and is therefore only partially learned by the relearning device 10. Furthermore, the data in DT4 appears very rarely and is therefore hardly learned by the relearning device 10. Thus, the learning model generated in step ST2 may have sufficient learning for the data in modes DT1 and DT2, but insufficient learning for the data in mode DT3. The learning for the data in mode DT4 is also insufficient.

[0029] Next, the region to be processed for image recognition is reacquired (Step ST3). Specifically, based on user instructions, the region to be processed for image recognition is reacquired for images in the initial training dataset whose score was below the first threshold. Images whose score was below the first threshold are, for example, those for which the image recognition result was "NG," meaning that the recognition could not be performed.

[0030] Next, the untrained dataset is acquired (step ST4). Specifically, the untrained dataset acquisition unit 14 acquires the untrained dataset. The untrained dataset includes data indicating untrained images that have not been trained, scores output by the image judgment processing unit 12 based on the trained model after image judgment processing of the untrained images, and the regions targeted for image judgment processing. These data are associated with each other. The untrained dataset is, for example, data generated in places where the retraining device 10 is actually used, regardless of the user's intentions. The untrained dataset may also be, for example, actual flow data generated in production processes or inspection processes in a factory. The untrained dataset should ideally have a larger amount of data compared to the initial training data. The scores output by image judgment processing of the images and the regions targeted for image judgment processing may be provided in advance by the user as appropriate. Note that the untrained dataset UD1 shown in Figure 5 is mainly distributed in modes DT2 to DT4.

[0031] Next, a retraining dataset is generated (step ST5). Specifically, the generation unit 15 generates the retraining dataset. The retraining dataset includes the initial training dataset stored in step ST1, data indicating images containing the target region for image classification processing, which are reacquired in step ST3, and the untrained dataset acquired in step ST4.

[0032] More specifically, the generation unit 15 may identify data from the untrained dataset that represent untrained images whose score was below a second threshold or whose frequency was below a third threshold. Furthermore, the generation unit 15 may generate a retraining dataset that includes this identified data and data that represents images containing the region targeted for image judgment processing, which were reacquired in step ST3. Since untrained images whose score was below the second threshold or whose frequency was below the third threshold are unlikely to be overtrained, it is possible to train on untrained images that are not overtrained. Therefore, the total amount of data in the retraining dataset can be reduced.

[0033] More specifically, the generation unit 15 may further select data from the initial training dataset and the identified data that represent trained and untrained images based on a function or random number whose score or frequency is a coefficient. The generation unit 15 may also generate a retraining dataset that includes the selected data and data representing images that include the region targeted by the image judgment process, which was reacquired in step ST3. The total amount of data in this generated retraining dataset should be less than or equal to a specified value. This limits the total amount of data in the retraining dataset, thereby suppressing the risk of increased training costs.

[0034] Furthermore, the generation unit 15 may generate the retraining dataset generated in step ST5 such that the ratio of the amount of data in the initial training dataset to the untrained dataset remains constant. By keeping this ratio constant, it is possible to prevent an imbalance between untrained training data and trained training data in the retraining dataset, and to perform retraining appropriately.

[0035] Furthermore, the generation unit 15 may select data indicating trained or untrained images based on the date and time, or order, in which the trained or untrained images were taken. For example, the generation unit 15 may select data indicating trained or untrained images in order of newest date and time, or in order of latest order. The generation unit 15 may generate a retraining dataset that includes this selected data and data indicating images that include the region targeted for image judgment processing, which has been reacquired. This allows retraining to be performed using a retraining dataset that contains a large amount of data indicating recently taken trained or untrained images.

[0036] Furthermore, the generation unit 15 may generate a retraining dataset containing N trained images from groups L1 to L3, as shown in Figure 4. Specifically, the generation unit 15 selects X trained images from the trained images of group L2. The generation unit 15 also selects Y trained images from the trained images of group L3. Furthermore, the generation unit 15 randomly selects NXY trained images from the trained images of group L1. For the X trained images selected from the trained images of group L2, data indicating the region to be processed for image recognition is reacquired. For the Y trained images selected from the trained images of group L3 and the NXY trained images from the trained images of group L1, it is not necessary to reacquire data indicating the region to be processed for image recognition. Finally, the generation unit 15 may generate a retraining dataset containing data indicating these selected N trained images.

[0037] Finally, update the trained model by training it with a new dataset (Step ST5).

[0038] The frequency of the first example in the relearning dataset is the sum of the frequency of the untrained dataset UD1 shown in Figure 5 and the frequency of the trained initial training dataset SD1, and is distributed across modes DT1 to DT4. The learning unit 16 learns this first example in the relearning dataset. As a result, the data in modes DT1, DT2, and DT4 are frequently observed and are therefore well learned by the relearning device 10. On the other hand, the data in mode DT3 is extremely frequently observed and, although learned by the relearning device 10, there is a risk of overfitting. Therefore, the updated learning model may have sufficient training for the data in modes DT1 and DT2, but may have excessive training for the data in mode DT3.

[0039] A second example of a relearning dataset is the relearning dataset RD1 shown in Figure 5, which is distributed in modes DT1 to DT4. For example, the learning unit 16 learns the relearning dataset RD1 shown in Figure 5. Then, since the data in modes DT1 to DT4 of the relearning dataset RD1 are frequently occurring, the relearning device 10 learns them well. In addition, since the data in mode DT3 of the relearning dataset RD1 is less than that of the first example of a relearning dataset, the occurrence of overfitting can be suppressed. Therefore, the learning model updated in step ST5 has sufficient learning on the data in modes DT1 to DT4. Moreover, there is little risk of overfitting on the data in mode DT3.

[0040] Based on the above, the learning model can be updated.

[0041] Furthermore, after step ST5 is completed, you may accumulate more real-world flow data and use that accumulated real-world flow data to retrain the retraining dataset and further update the learning model. In other words, you can return to step ST3 and repeat steps ST3 to ST5. In these repeated steps ST3 to ST5, you can update the learning model by retraining the already retrained retraining dataset instead of using the initial training data.

[0042] Furthermore, in the retraining method described above, retraining is performed using the target region for image recognition processing that was reacquired in step ST3. Since the user does not need to specify the target region for image recognition processing, the increase in retraining costs can be suppressed by eliminating user input.

[0043] <Hardware Configuration> Figure 6 shows an example of the hardware configuration of a relearning device 10 according to an embodiment. In the example in Figure 6, the relearning device 10 (computer 100) includes a processor 101, memory 102, and a communication interface 103. These components may be connected by a bus or the like. The memory 102 stores at least a portion of the program 104. The communication interface 103 includes an interface necessary for communication with other network elements.

[0044] When program 104 is executed in cooperation with the processor 101 and memory 102, etc., the computer 100 performs at least some of the processing of embodiments of this disclosure. Memory 102 may be of any type. Memory 102 may, in non-limiting examples, be a non-temporary computer-readable storage medium. Memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. Although only one memory 102 is shown for computer 100, computer 100 may have several physically different memory modules. Processor 101 may be of any type. Processor 101 may include one or more general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), and, in non-limiting examples, processors based on multicore processor architectures. Computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-dependent to a clock that synchronizes the main processor.

[0045] Embodiments of the present disclosure may be implemented in hardware or in dedicated circuitry, software, logic, or any combination thereof. Some embodiments may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device.

[0046] This disclosure also provides at least one computer program product tangibly stored on a non-temporary computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions contained in a program module, and is executed on a device on a target real or virtual processor to perform the processes or methods of this disclosure. The program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The functionality of the program module may be combined or divided among the program module as desired in various embodiments. The machine-executable instructions of the program module can be executed on a local or distributed device. On a distributed device, the program module can reside on both local and remote storage media.

[0047] Program code for performing the methods of this disclosure may be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing device. When the program code is executed by the processor or controller, the functions / operations in the flowchart and / or block diagrams it implements are performed. The program code may run entirely on a machine, partially on a machine, partially as a standalone software package, partially on a machine, partially on a remote machine, or entirely on a remote machine or server.

[0048] Programs can be stored and supplied to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media, magneto-optical recording media, optical disc media, and semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Optical disc media include, for example, Blu-ray discs, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), and CD-RW (ReWritable). Semiconductor memory includes, for example, solid-state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memory). Programs may also be supplied to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can supply programs to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0049] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. Furthermore, the present invention may be implemented by combining the above embodiments or examples thereof as appropriate. [Explanation of symbols]

[0050] 10 Retraining device 11 Storage section 12 Image Recognition Processing Unit 13 Area acquisition part 14. Unit for acquiring untrained datasets 15 Generation part 16. Learning Department

Claims

1. A retraining device that updates a learning model by retraining it on a retraining dataset, A storage unit that stores a trained dataset including data showing trained images, scores output by performing image classification processing on the trained images based on the trained model, and regions targeted for the image classification processing. A region acquisition unit that, based on user instructions, reacquires the region targeted for the image judgment process in the trained dataset for images whose score was below a first threshold, An untrained dataset acquisition unit acquires an untrained dataset that includes data showing untrained images, a score output by performing image classification processing on the untrained images based on the learning model, and the region targeted for the image classification processing. The system comprises a generation unit that generates a retraining dataset including the trained dataset, data showing an image including the region targeted for the image determination process that has been reacquired, and the untrained dataset. The generation unit identifies data from the untrained dataset that represent untrained images whose score is below the second threshold or whose frequency is below the third threshold. The generation unit generates a retraining dataset that includes the identified data, the score corresponding to the identified data, the region targeted for the image judgment processing, and data showing an image including the reacquired region targeted for the image judgment processing. A learning device.

2. The generation unit selects data from the trained dataset and the identified data that represent trained images and untrained images based on a function or random number whose score or frequency is a coefficient. The generation unit generates a retraining dataset which includes the selected data, the score corresponding to the selected data, the region to be processed by the image determination process, and data showing an image which includes the reacquired region to be processed by the image determination process. The total amount of data in the aforementioned retraining dataset is less than or equal to the specified value. The relearning device according to claim 1.

3. A retraining method executed by a retraining device that updates a learning model by retraining a retraining dataset, A trained dataset is stored which includes data showing trained images, scores output by performing image classification processing on the trained images based on the trained model, and the regions targeted for the image classification processing. Based on user instructions, the region targeted for the image classification process in the trained dataset where the score was below the first threshold is reacquired. An untrained dataset is obtained that includes data showing untrained images, scores output by performing image classification processing on the untrained images based on the learning model, and the regions targeted by the image classification processing. A retraining dataset is generated, which includes the trained dataset, data showing the image including the region targeted for the image classification process that was reacquired, and the untrained dataset. In generating the aforementioned retraining dataset, From the aforementioned untrained dataset, data representing untrained images whose score was below the second threshold, or whose frequency was below the third threshold, are identified. A retraining dataset is generated, which includes the identified data, the score corresponding to the identified data, the region targeted for the image judgment processing, and data showing an image including the reacquired region targeted for the image judgment processing. A relearning method in which a relearning device performs the processing.

4. A program to be executed by a retraining device that retrains a dataset for retraining and updates the learning model, A trained dataset is stored which includes data showing trained images, scores output by performing image classification processing on the trained images based on the trained model, and the regions targeted for the image classification processing. Based on user instructions, the region targeted for the image classification process in the trained dataset where the score was below the first threshold is reacquired. An untrained dataset is obtained that includes data showing untrained images, scores output by performing image classification processing on the untrained images based on the learning model, and the regions targeted by the image classification processing. A retraining dataset is generated, which includes the trained dataset, data showing the image including the region targeted for the image classification process that was reacquired, and the untrained dataset. In generating the aforementioned retraining dataset, From the aforementioned untrained dataset, data representing untrained images whose score was below the second threshold, or whose frequency was below the third threshold, are identified. A retraining dataset is generated, which includes the identified data, the score corresponding to the identified data, the region targeted for the image judgment processing, and data showing an image including the reacquired region targeted for the image judgment processing. A program that causes a relearning device to execute a process.

Images