Learning device, learning method, and program

Abstract

The learning device updates the model by executing one or more active learning processes. Each of the one or more active learning processes includes: selecting one or more evaluation images from a group of images that are not used for learning; and repeatedly executing the re-learning process until the evaluation result of the model using the one or more evaluation images satisfies the end reference. In at least one active learning process, selecting one or more evaluation images includes: calculating, for each image of the image group, an evaluation value indicating a degree of discrimination reliability based on the latest model; and selecting the image having relatively low reliability as one or more evaluation images. As a result, a model having high attribute discrimination accuracy is generated in a short time.

Description

Technical Field

[0001] The present disclosure relates to a learning device, a learning method, and a program. Background Art

[0002] In recent years, in the context of manufacturing products, the development of technology for automatically determining the appearance-related attributes of products based on the obtained images by photographing the manufactured products with a photographing device has been progressing. Specifically, the attributes of the products are determined using a model obtained by machine learning using multiple images of products with known attributes.

[0003] The accuracy of attribute discrimination based on the model depends on the images used in machine learning. As a method for selecting images that are effective in improving discrimination accuracy, active learning is known (see "B. Settles, "Active Learning Literature Survey", Computer Sciences Technical Report 1648, University of Wisconsin, 2010" (Non-Patent Literature 1)). In the known active learning, a model is generated by performing machine learning using a training data set consisting of a small number of images arbitrarily selected from multiple images. Then, until the evaluation results of the model using all the remaining images other than the images used for learning in the multiple images meet a predetermined benchmark, the following process is repeated: the training data set is updated in a manner that includes images in the remaining images that are difficult to discriminate by the model, and the model is re-learned using the updated training data set.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-Patent Literature 1: B. Settles, "Active Learning Literature Survey", Computer Sciences Technical Report 1648. University of Wisconsin, 2010, [online], [retrieved on June 14, 2010], Internet 〈URL: https: / / burrsettles.com / pub / settles.activelearning.pdf〉

[0007] Non-patent document 2: Tatsuro Yamane and Takashi Zenbo, "Detection of cracks on concrete surface using semantic segmentation based on deep learning", Proceedings of Structural Engineering, Vol. 65A (2019), [online], [retrieved on June 14, 2019], Internet<https: / / www.jstage.jst.go.jp / article / structcivil / 65A / 0 / 65A_130 / _pdf> Summary of the invention

[0008] Problems to be solved by the invention

[0009] In the known active learning method, the time required for the re-learning process that is repeatedly performed becomes long.

[0010] The present disclosure has been made in view of such actual circumstances, and an object of the present disclosure is to provide a learning device, a learning method, and a program that can generate a model with high attribute discrimination accuracy in a short time.

[0011] Means for solving problems

[0012] A learning device according to one aspect of the present disclosure comprises: an acquisition unit that acquires a model used to discriminate the attributes of an object captured in an input image by performing machine learning using a training data set; and an active learning unit that updates the model by performing active learning processing one or more times. The training data set includes one or more learning images selected from a plurality of images to which labels representing the attributes of the object are assigned. The one or more active learning processings respectively include: selecting one or more evaluation images from a group of images not used for learning among the plurality of images; and repeatedly performing relearning processing until the evaluation result of the model using the one or more evaluation images satisfies an end criterion. The relearning processing includes: updating the training data set in a manner that includes one or more additional images selected from the one or more evaluation images; and relearning the model using the updated training data set. In at least one of more than one active learning processes, selecting one or more evaluation images includes: calculating a first evaluation value representing the reliability of discrimination of an attribute based on a latest model for each image included in an image group; and selecting, based on the first evaluation value, images with relatively low reliability from the image group as one or more evaluation images.

[0013] According to the present disclosure, images with relatively low reliability are selected from an image group as evaluation images. Therefore, by relearning the model using additional images selected from one or more evaluation images, the discrimination accuracy of the model can be easily improved. Furthermore, in each of more than one active learning processes, the model is evaluated each time the relearning process is performed. The model can be evaluated by inputting one or more evaluation images selected from an image group into the model. Therefore, compared with a method of evaluating a model by inputting all images of an image group that are not used for learning from a plurality of images into the model, the time required for evaluating the model can be shortened. In this way, according to the above disclosure, a model with high discrimination accuracy of attributes can be generated in a short time.

[0014] In the above disclosure, the relearning process also includes: for one or more evaluation images, respectively calculating a second evaluation value representing the reliability of the discrimination of the attribute based on the latest model; and based on the second evaluation value, selecting an evaluation image with relatively low reliability from one or more evaluation images as one or more additional images.

[0015] According to the present disclosure, an image that is more difficult to distinguish among one or more evaluation images is selected as an additional image, thereby making it easier to improve the discrimination accuracy of the model.

[0016] In the above disclosure, the model outputs a feature quantity representing a feature of an input image. The feature quantity is compared with a discrimination boundary for distinguishing an attribute. The first evaluation value for a target image included in the image group represents the distance between the feature quantity calculated from the target image and the discrimination boundary.

[0017] According to the present disclosure, it is easy to select an image that is difficult to distinguish as an evaluation image, thereby improving the discrimination accuracy of the model in a shorter time.

[0018] In the above disclosure, the model outputs a feature quantity representing a feature of an input image. The first evaluation value for a target image included in the image group represents the distance between the feature quantity calculated based on the target image and the feature quantity calculated based on an image used for learning the latest model and assigned the same label as the target image.

[0019] According to the present disclosure, the feature distribution of one or more selected evaluation images is broadened, thereby making it possible to generate a model with high discrimination accuracy in a short time even when there are multiple features of images with labels indicating the same attribute.

[0020] In the above disclosure, the one or more active learning processes include M active learning processes and N active learning processes that are executed after the M active learning processes and correspond to at least one active learning process. M and N are integers greater than 1. In the M active learning processes, the step of selecting one or more evaluation images includes: selecting one or more evaluation images from the image group randomly or according to a predetermined rule without using the first evaluation value.

[0021] When the number of learning images is small, the discrimination accuracy of the model acquired by the acquisition unit is low. According to the present disclosure, in M ​​active learning processes, the first evaluation value is not used, and one or more evaluation images are selected randomly or according to a predetermined rule. Therefore, an image with multiple features is selected as the evaluation image. Thus, through M active learning processes, it is possible to use evaluation images with multiple features so that the discrimination accuracy of the model converges to a certain degree of high state. Moreover, in N active learning processes, an image suitable for learning is selected as the evaluation image based on the first evaluation value obtained by applying a model whose discrimination accuracy is improved to a certain extent by using M active learning processes. As a result, a model with high discrimination accuracy can be generated in a shorter time.

[0022] In the above disclosure, one or more active learning processes include the M+N+1th active learning process performed after the Nth active learning process. In the M+N+1th active learning process, selecting one or more evaluation images includes: selecting all images included in the image group as one or more evaluation images.

[0023] According to the present disclosure, in the M+N+1th active learning process, the discrimination accuracy of the model can be evaluated more accurately.

[0024] In the above disclosure, when N is an integer greater than or equal to 2, in the N active learning processes, at least one of the number of evaluation images selected from the image group, the type of the first evaluation value, and the end criterion may be different from each other.

[0025] In the above disclosure, when M is an integer greater than 2, in M ​​active learning processes, at least one of the number of evaluation images selected from the image group, the method of selecting one or more evaluation images, and the end criterion may be different from each other.

[0026] In the above disclosure, the label includes a first label indicating that the object is a qualified product and a second label indicating that the object is a defective product. The plurality of images include a plurality of qualified product images assigned with the first label and one or more defective product images assigned with the second label. One or more learning images are selected from the plurality of qualified product images. One or more evaluation images include one or more qualified product images and one or more defective product images selected from the image group. A first evaluation value is calculated for each qualified product image included in the image group.

[0027] According to the present disclosure, even when the number of defective product images to which the second label is assigned is small, a model with high discrimination accuracy can be generated in a short time based on only the conforming product images.

[0028] A learning method according to one aspect of the present disclosure comprises the following steps: a computer acquires a model used for distinguishing the attributes of an object captured in an input image by performing machine learning using a training data set; and the computer updates the model by performing active learning processing one or more times. The training data set includes one or more learning images selected from a plurality of images to which labels representing the attributes of the object are assigned. The one or more active learning processings respectively include: selecting one or more evaluation images from a group of images not used for learning among the plurality of images; and repeatedly performing relearning processing until the evaluation result of the model using the one or more evaluation images satisfies an end criterion. The relearning processing includes: updating the training data set in a manner including one or more additional images selected from the one or more evaluation images; and relearning the model using the updated training data set. In at least one of the one or more active learning processings, selecting one or more evaluation images includes: calculating a first evaluation value representing the reliability of distinguishing the attributes based on the latest model for each image included in the image group; and selecting images with relatively low reliability as one or more evaluation images from the image group based on the first evaluation value.

[0029] The program of one aspect of the present disclosure causes a computer to execute the above-mentioned learning method. According to these disclosures, a model with high attribute discrimination accuracy can also be generated in a short time.

[0030] Effects of the Invention

[0031] According to the present disclosure, a model with high attribute discrimination accuracy can be generated in a short time. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a diagram showing an example of a learning method according to an embodiment.

[0033] Figure 2It is a schematic diagram showing the overall configuration of a system including a learning device according to an embodiment.

[0034] Figure 3 Yes means Figure 2 Schematic diagram of an example of the hardware configuration of the learning device shown.

[0035] Figure 4 Yes means Figure 2 The figure is a schematic diagram of an example of the hardware configuration of the identification device shown.

[0036] Figure 5 This is a diagram schematically showing an example of the software configuration of the learning device.

[0037] Figure 6 This is a diagram showing an example of a user interface screen provided by the label adding unit.

[0038] Figure 7 This is a diagram showing another example of the user interface screen provided by the label adding unit.

[0039] Figure 8 This is a diagram showing an example of a user interface screen provided by the acquisition unit.

[0040] Fig. 9 This is a diagram for explaining a method of obtaining a model.

[0041] Fig.10 This is a diagram showing an example of the types of evaluation values ​​E1.

[0042] Fig.11 This is a diagram showing another example of the types of evaluation values ​​E1.

[0043] Fig.12 This is a flowchart showing an example of a processing flow of the learning device according to the embodiment.

[0044] Fig.13 Yes means Fig.12 Flowchart of the subroutine flow of the reference example of step S5 shown in FIG.

[0045] Fig.14 Yes Instructions to follow Fig.13 FIG. 5 is a flow chart of the active learning process shown in FIG.

[0046] Fig.15 This is a diagram showing an example of a screen for setting conditions for active learning processing.

[0047] Fig.16 Yes means Fig.12 Flow chart of the subroutine procedure of the first embodiment of step S5 shown.

[0048] Fig.17 Yes Instructions to follow Fig.16 FIG. 5 is a flow chart of the active learning process shown in FIG.

[0049] Fig.18 Yes means Fig.12 Flow chart of the subroutine flow of the second embodiment of step S5 shown.

[0050] Fig.19 Yes means Fig.18 Flowchart of the subroutine flow of step S21 shown.

[0051] Fig. 20 Yes Instructions to follow Fig.19 FIG. 5 is a flow chart of the active learning process shown in FIG.

[0052] Fig.21 Yes means Fig.12 Flow chart of the subroutine flow of the third embodiment of step S5 shown.

[0053] Fig. 22 Yes means Fig.12 Flow chart of the subroutine flow of the fourth embodiment of step S5 shown.

[0054] Fig.23 Yes means Fig. 22 Flowchart of the subroutine procedure of step S23 shown in FIG.

[0055] Fig.24 This is a diagram showing an example of a screen showing a result of active learning processing.

[0056] Fig.25 It is a diagram showing another example of a screen showing a result of the active learning process.

[0057] Fig.26 This is a diagram for explaining a method of acquiring a model in Modification Example 1.

[0058] Fig. 27 This is a diagram for explaining the active learning process in Modification Example 1.

[0059] Fig.28 This is a diagram showing an example of evaluation values ​​E1 and E2 in Modification Example 1.

[0060] Fig.29 This is a diagram showing another example of the evaluation values ​​E1 and E2 in Modification 1. DETAILED DESCRIPTION

[0061] Embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the same or corresponding parts in the drawings are denoted by the same reference numerals and their description will not be repeated.

[0062] §1 Application Examples

[0063] First, use Figure 1 , an example of a scenario in which the present invention is applied is described. Figure 1 is a diagram showing an example of a learning method of an implementation method. Figure 1 As shown, a plurality of images 3 of the object 2 are prepared in advance. A label indicating an attribute of the object 2 captured by the image 3 is assigned to each of the plurality of images 3 .

[0064] The learning method comprises the following steps (1): a computer performs machine learning using a training data set 6 including one or more learning images selected from a plurality of images 3, thereby obtaining a model 7 used for determining the attributes of an object 2 captured by an input image. Figure 1 In the example shown, image 3a is selected as the learning image.

[0065] The learning method further includes the following step (2): the computer updates the model 7 by executing the active learning process one or more times.

[0066] The active learning process of one or more times includes: step (2-1), selecting one or more evaluation images 4 from the image group 8 not used for learning among the plurality of images 3; and step (2-2), repeatedly performing the relearning process until the evaluation result of the model 7 using the one or more evaluation images 4 meets the end criterion. The number of evaluation images 4 selected from the image group 8 is predetermined. Figure 1 In the example shown, five evaluation images 4 are selected.

[0067] The relearning process includes: step (2-2-1) of updating the training data set 6 to include one or more additional images selected from one or more evaluation images 4; and step (2-2-2) of relearning the model 7 using the updated training data set 6. The evaluation images 4 that are difficult to be distinguished by the model 7 are selected as additional images from the one or more evaluation images 4. Figure 1 In the example shown, the evaluation image 4a is selected as the additional image.

[0068] In at least one of the more than one active learning processes, step (2-1) of selecting one or more evaluation images 4 includes steps (2-1-1) and (2-1-2). Step (2-1-1) is a step of calculating an evaluation value E1 representing the reliability of discrimination of the attribute based on the latest model 7 for the images included in the image group 8. Step (2-1-2) is a step of selecting an image 3 with relatively low reliability as one or more evaluation images 4 from the image group 8 based on the evaluation value E1.

[0069] According to step (2-1-2), the image 3 with relatively low reliability is selected from the image group 8 as the evaluation image 4. Therefore, by relearning the model 7 using the additional image selected from one or more evaluation images 4, the discrimination accuracy of the model 7 is easily improved.

[0070] In one or more active learning processes, each time the re-learning process is performed, the model 7 is evaluated according to step (2-2). According to the learning method of this embodiment, the model 7 can be evaluated by inputting one or more evaluation images 4 selected from the image group 8 not used for learning among the plurality of images 3 into the model 7. Therefore, compared with the conventional active learning method in which the model 7 is evaluated by inputting all the images of the image group 8 not used for learning among the plurality of images 3 into the model 7, the time required for evaluating the model 7 can be shortened.

[0071] As described above, according to the learning method of this embodiment, it is possible to generate the model 7 with high attribute discrimination accuracy in a short time.

[0072] §2Concrete examples

[0073] <System Structure>

[0074] Figure 2 It is a schematic diagram showing the overall configuration of a system including a learning device according to an embodiment. Figure 2 The illustrated system 1 is assembled in a production line, etc., and uses an image of an object 2 as a product to determine the attributes of the object 2. The attributes of the object 2 include, for example, the quality of the appearance, the presence or absence of defects, the type of defects, etc. Figure 2 As shown, the system 1 includes a learning device 100 and a determination device 200 .

[0075] The learning device 100 is a computer configured to generate a model 7 for determining the attributes of the object 2 captured by the input image. The learning device 100 is connected to a display device 600 and an input device 700. The display device 600 is typically composed of a liquid crystal display and displays various screens. The input device 700 includes, for example, a keyboard and a mouse. In addition, the display device 600 and the input device 700 may also be integrated into a touch panel.

[0076] The discrimination device 200 is a computer configured to discriminate the attribute of the object 2 using the model 7 transmitted from the learning device 100. The discrimination device 200 is connected to the camera 300. The camera 300 is set so that the object 2 is included in the field of view. Thus, the discrimination device 200 obtains an image of the object 2 from the camera 300. The discrimination device 200 discriminates the attribute of the object 2 using the output information of the model 7 when the obtained image is input.

[0077] For example, the determination device 200 determines whether the object 2 is a good product or a bad product based on whether the appearance of the object 2 contains defects. Defects may be scratches, stains, cracks, dents, burrs, color unevenness, foreign matter, etc.

[0078] Figure 2 The illustrated learning device 100 and the identification device 200 are connected to each other via a network. The type of network can be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, and the like.

[0079] The method of exchanging data between the learning device 100 and the discrimination device 200 may not be limited to such an example, and may be appropriately selected according to the implementation method. For example, data may be exchanged between the learning device 100 and the discrimination device 200 using a storage medium. In addition, in the present embodiment, the learning device 100 and the discrimination device 200 are independent computers. However, the structure of the system 1 may not be limited to such an example, and may be appropriately determined according to the implementation method. For example, the learning device 100 and the discrimination device 200 may also be an integrated computer. And for example, at least one of the learning device 100 and the discrimination device 200 may be composed of multiple computers.

[0080] <Hardware Structure of Learning Device>

[0081] The learning device 100 is typically a computer with a general-purpose architecture, and executes various processes of the present embodiment by executing a pre-installed program (command code). Typically, such a program is circulated in a state stored in various recording media, or installed in the learning device 100 via a network.

[0082] When such a general-purpose computer is used, in addition to the application for executing various processes of the present embodiment, an OS (Operating System) for executing basic processes of the computer may be installed. In this case, the program of the present embodiment may also execute processes by calling necessary modules among program modules provided as part of the OS in a prescribed arrangement at a prescribed timing. That is, the program of the present embodiment itself may not include the above-mentioned modules, but may execute processes in cooperation with the OS. The program of the present embodiment may also be in a form that does not include such a part of the modules.

[0083] Furthermore, the program of this embodiment can also be provided by being incorporated into a part of other programs. In this case, the program itself does not include the modules included in the other programs of the above combination, but cooperates with the other programs to perform processing. That is, the program of this embodiment can also be incorporated into such other programs. In addition, part or all of the functions provided by the execution of the program can also be installed as a dedicated hardware circuit.

[0084] Figure 3 Yes means Figure 2 A schematic diagram of an example of the hardware structure of the learning device shown in FIG. Figure 3 As shown, the learning device 100 includes a CPU (Central Processing Unit) 110, a RAM (Random Access Memory) 111, a ROM (Read Only Memory) 112, a communication interface 113, an input interface 114, a display controller 115, a driver 116, and a memory 120. These components are connected via a bus so as to be able to communicate data with each other.

[0085] The CPU 110 develops the program (code) installed in the memory 120 in the RAM 111 and executes them in a predetermined order to perform various operations. The RAM 111 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0086] The communication interface 113 is an interface for performing wired or wireless communication via a network, such as a wired LAN (Local Area Network) module or a wireless LAN module. The learning device 100 can perform data communication with other information processing devices (eg, the determination device 200) by using the communication interface 113.

[0087] The input interface 114 mediates data transmission between the CPU 110 and the input device 700. In other words, the input interface 114 receives input information input to the input device 700 by the user.

[0088] The display controller 115 is connected to the display device 600 , and controls the screen of the display device 600 to notify the user of the processing results and the like in the CPU 110 .

[0089] The memory 120 is composed of, for example, a hard disk drive, a solid-state drive, etc. The memory 120 stores a plurality of images 3 in which an object 2 is captured. Furthermore, the memory 120 stores a learning program 122 and a labeling program 124. The learning program 122 is a program for causing the learning device 100 to perform a machine learning process for generating a model 7. The labeling program 124 is a program for causing the learning device 100 to perform a process of assigning labels to each of the plurality of images 3, the labels representing the attributes of the object 2 captured in the image 3. The learning program 122 and the labeling program 124 each include a series of instructions for information processing. Furthermore, the memory 120 stores the model 7 obtained by executing the learning program 122.

[0090] The drive 116 is, for example, a CD drive or a DVD drive, and is a drive device for reading a program stored in the storage medium 130. The type of the drive 116 can be appropriately selected according to the type of the storage medium 130. At least one of the learning program 122 and the labeling program 124 may also be stored in the storage medium 130.

[0091] The storage medium 130 is a medium that stores information such as programs by electrical, magnetic, optical, mechanical or chemical action in a manner that allows a computer, other device, machine, etc. to read the recorded information such as programs. The learning device 100 can also obtain at least one of the learning program 122 and the labeling program 124 from the storage medium 130.

[0092] Here, in Figure 2 In the embodiment, a disk-type storage medium such as a CD or a DVD is illustrated as an example of the storage medium 130. However, the type of the storage medium 130 is not limited to the disk type, and may be a type other than the disk type. As a storage medium other than the disk type, for example, a semiconductor memory such as a flash memory can be cited.

[0093] In addition, regarding the specific hardware structure of the learning device 100, it is possible to appropriately omit, replace, and add structural elements according to the implementation method. For example, the learning device 100 may also include multiple hardware processors. The hardware processor may be composed of a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), etc. At least one of the communication interface 113 and the driver 116 may also be omitted. The learning device 100 may also have a controller, for example, which is connected to an output device other than the display device 600 such as a speaker and controls the output device. The learning device 100 may also be composed of multiple computers. In this case, the hardware structure of each computer may be consistent or inconsistent. In addition, in addition to being designed as an information processing device dedicated to the service provided, the learning device 100 may also be a general-purpose server device, a general-purpose PC (Personal Computer), etc.

[0094] <Hardware Structure of Determination Device>

[0095] Figure 4 Yes means Figure 2 A schematic diagram of an example of the hardware structure of the discrimination device shown in FIG. Figure 4 As shown, the determination device 200 includes a CPU 210, a RAM 211, a ROM 212, a communication interface 213, an external interface 214, an input interface 215, a display controller 216, a driver 217, and a memory 220. These components are connected via a bus so as to be able to communicate data with each other.

[0096] The CPU 210 performs various operations by developing the program (code) installed in the memory 220 in the RAM 211 and executing them in a predetermined order. The RAM 211 is typically a volatile storage device such as a DRAM.

[0097] The communication interface 213 is, for example, a wired LAN module or a wireless LAN module, and is an interface for performing wired or wireless communication via a network. The determination device 200 can perform data communication with other information processing devices (eg, the learning device 100 ) by using the communication interface 213 .

[0098] The external interface 214 is, for example, a USB (Universal Serial Bus) port, a dedicated port, etc., and is an interface for connecting to an external device. The type and number of the external interface 214 can be appropriately selected according to the type and number of the external devices connected. In this embodiment, the identification device 200 is connected to the camera 300 via the external interface 214.

[0099] The camera 300 is used to obtain an image 3 of an object 2 that is the object of appearance inspection. The type and configuration location of the camera 300 are not particularly limited and can be appropriately determined according to the implementation method. The camera 300 can use, for example, a common digital camera, a depth camera, an infrared camera, etc. In addition, the camera 300 can be appropriately configured to be able to observe the object 2 conveyed by the production line. The camera 300 can be configured, for example, near the production line that conveys the object 2. In addition, when the camera 300 has a communication interface, the determination device 200 can also be connected to the camera 300 via the communication interface 213 instead of the external interface 214.

[0100] The input interface 215 mediates data transmission between the CPU 210 and an input device (not shown). In other words, the input interface 215 receives input information inputted by the user into the input device.

[0101] The display controller 216 is connected to a display device (not shown) and controls the screen of the display device so as to notify the user of the processing results of the CPU 210 and the like.

[0102] The memory 220 is constituted by, for example, a hard disk drive, a solid state drive, etc. The memory 220 stores various information such as a determination program 221 and the model 7 .

[0103] The discrimination program 221 is a program for causing the discrimination device 200 to perform information processing for discriminating the attributes of the object 2 captured in the image 3 using the model 7 generated by the learning device 100. In the present embodiment, the information processing for discriminating the attributes of the object 2 is information processing for discriminating the quality of the object 2. The discrimination program 221 includes a series of instructions for the information processing.

[0104] The drive 217 is, for example, a CD drive, a DVD drive, etc., and is a drive device for reading a program stored in a storage medium (not shown). At least one of the discrimination program 221 and the model 7 may also be stored in the storage medium. In addition, the discrimination device 200 may also obtain at least one of the discrimination program 221 and the model 7 from the storage medium.

[0105] <Software Structure of Learning Device>

[0106] Figure 5 FIG. 1 is a diagram schematically showing an example of the software structure of a learning device. Figure 5 As shown in FIG. 1 , the learning device 100 includes a storage unit 10, a label assigning unit 11, an acquiring unit 12, and an active learning unit 13. The storage unit 10 is composed of Figure 3The memory 120 and the RAM 111 are shown as implemented. The labeling unit 11 is implemented by the CPU 110 interpreting and executing commands included in the labeling program 124. The acquisition unit 12 and the active learning unit 13 are implemented by the CPU 110 interpreting and executing commands included in the learning program 122.

[0107] The storage unit 10 stores a plurality of images 3 obtained by imaging by the camera 300. The object 2 is captured in each of the plurality of images 3.

[0108] The label assigning unit 11 assigns labels to the plurality of images 3 stored in the storage unit 10 according to user operations. The labels indicate the attributes of the object 2 captured in the image 3. In the present embodiment, the label assigning unit 11 assigns the label "qualified product" to the image 3 in which the object 2 (non-defective object 2) is captured. The label assigning unit 11 assigns the label "defective product" to the image 3 in which the object 2 having defects is captured.

[0109] Figure 6 This is a diagram showing an example of a user interface screen provided by the label adding unit. Figure 6 The user interface screen 70 shown is displayed on Figure 3 A display device 600 is shown.

[0110] like Figure 6 As shown, the user interface screen 70 includes an image list 71 , a good product image list 72 , a bad product image list 73 , an area 74 , add buttons 75 a and 75 b , a move button 76 , and a button 78 .

[0111] The image list 71 is a list of a plurality of images 3 stored in the storage unit 10. The qualified product image list 72 shows a list of images 3 in which the qualified product object 2 is photographed among the plurality of images 3. The defective product image list 73 shows a list of images 3 in which the defective product object 2 is photographed among the plurality of images 3. In the area 74, one image 3 selected from the image list 71, the qualified product image list 72, and the defective product image list 73 is displayed.

[0112] The add button 75a is a button for adding the image 3 selected from the image list 71 to the qualified product image list 72. The user selects the image 3 in which the qualified product object 2 is photographed from the image list 71 while observing the area 74, and operates the add button 75a. When the add button 75a is operated, the label assigning unit 11 adds the image 3 selected from the image list 71 to the qualified product image list 72. As a result, the qualified product image list 72 is also updated.

[0113] The add button 75b is a button for adding the image 3 selected from the image list 71 to the defective product image list 73. The user selects the image 3 of the object 2 photographed with defective products from the image list 71 while observing the area 74, and operates the add button 75b. When the add button 75b is operated, the label assigning unit 11 adds the image 3 selected from the image list 71 to the defective product image list 73. As a result, the defective product image list 73 is also updated.

[0114] The move button 76 is a button for accepting an operation of moving one image 3 selected from one of the qualified product image list 72 and the unqualified product image list 73 to the other. For example, when the user confirms that the unqualified product object 2 is captured in the image 3 included in the qualified product image list 72 while observing the area 74, the user selects the image 3 and operates the move button 76. Alternatively, when the user confirms that the qualified product object 2 is captured in the image 3 included in the unqualified product image list 73 while observing the area 74, the user selects the image 3 and operates the move button 76.

[0115] When the move button 76 is operated in a state where one image 3 in the qualified product image list 72 is selected, the label assigning unit 11 deletes the selected image 3 from the qualified product image list 72 and adds the selected image 3 to the unqualified product image list 73. When the move button 76 is operated in a state where one image 3 in the unqualified product image list 73 is selected, the label assigning unit 11 deletes the selected image 3 from the unqualified product image list 73 and adds the selected image 3 to the qualified product image list 72.

[0116] After the user classifies the plurality of images 3 into either the acceptable product image list 72 or the unacceptable product image list 73, the user presses the button 78. When the button 78 is pressed, the labeling unit 11 labels the acceptable product image list 72 and the unacceptable product image list 73.

[0117] Each image 3 included in the list 72 is given a label of "conforming product", and each image included in the list 73 of defective products is given a label of "conforming product".

[0118] 3. Label it as “non-conforming product”.

[0119] Furthermore, the label assigning unit 11 specifies the region where the defect is captured for the image 3 to which the label “defective product” is assigned, based on the user operation.

[0120] Figure 7 This is a diagram showing another example of the user interface screen provided by the label adding unit. Figure 7 The user interface screen 80 shown is displayed on Figure 3 A display device 600 is shown.

[0121] like Figure 7 As shown, the user interface screen 80 includes a defective product image list 81 and an area 83. The defective product image list 81 shows an overview of the images 3 to which the label "defective product" is assigned. The image 3 selected in the defective product image list 81 is displayed in the area 83. While confirming the area 83, the user adjusts the size and position of the frame line 84 in a manner to surround the defect D. The label assigning unit 11 records the area surrounded by the frame line 84 as an area where the defect is photographed. The label assigning unit 11 assigns information representing the recorded area (hereinafter referred to as "defect position information") to each image 3 included in the defective product image list 81.

[0122] Figure 5 The acquisition unit 12 shown acquires a model 7 used for distinguishing the attributes of the object 2 captured in the input image by performing machine learning using a training data set 6 including one or more learning images selected from the plurality of images 3. The acquisition unit 12 selects one or more learning images from the plurality of images 3 according to a user operation.

[0123] Figure 8 This is a diagram showing an example of a user interface screen provided by the acquisition unit. Figure 8 The user interface screen 86 shown is displayed on Figure 3 A display device 600 is shown.

[0124] like Figure 8 As shown, the user interface screen 86 includes a defective product image list 81 and a good product image list 82. The good product image list 82 shows a list of images 3 labeled "good product". The user adds a mark 87 to an image used as a learning image in each of the defective product image list 81 and the good product image list 82. Figure 8 In the example shown, the image 3 with the file name "Img003.bmp" is marked with a mark 87 in the defective product image list 81, and the image 3 with the file name "Img001.bmp" is marked with a mark 87 in the conforming product image list 82. The acquisition unit 12 selects the image 3 with the mark 87 as a learning image.

[0125] The acquisition unit 12 generates a model 7 by performing machine learning using a training data set 6 containing one or more learning images. The acquisition unit 12 generates an optimal model 7 based on the learning images contained in the training data set 6 and the labels assigned to the learning images. The acquisition unit 12 may also perform machine learning using defect position information assigned to the learning images in which the defective object 2 is photographed. By learning the defect position together with the learning image, the defect position is determined and the inspection accuracy is improved. As a method for performing machine learning on the defect position together with the image, a well-known technique is used (for example, Tatsuro Yamane, Quan Bangding, "Detecting Cracks on Concrete Surfaces Using Semantic Segmentation Based on Deep Learning", Proceedings of Structural Engineering, Vol. 65A (2019) (Non-Patent Literature 2)). As a machine learning algorithm, a well-known algorithm (for example, linear recognition, Bayesian recognition, logistic regression, support vector machine, decision tree, K nearest neighbor method, neural network, local outlier factor method, mixed normal distribution method, etc.) can be used.

[0126] Fig. 9 This is a diagram for explaining a method of obtaining a model. Fig. 9 The model 7 shown calculates a feature quantity on a feature space from an input image, and determines the attribute of an object 2 captured in the input image based on the positional relationship between the calculated feature quantity and a discrimination boundary 60. The acquisition unit 12 generates a model 7 using a training data set 6, and the model 7 defines a feature space and a discrimination boundary 60 where the distance between a point 63 corresponding to a feature quantity calculated from a learning image with a label "qualified product" and a point 64 corresponding to a feature quantity calculated from a learning image with a label "defective product" is the largest. The discrimination boundary 60 is a boundary between an area 61 in the feature space where there are points corresponding to a feature quantity calculated from a learning image with a label "qualified product" and an area 62 where there are points corresponding to a feature quantity calculated from a learning image with a label "defective product".

[0127] In addition, use Figure 8 The number of selected learning images is small. Therefore, the model 7 (initial model 7) acquired by the acquisition unit 12 has low discrimination accuracy.

[0128] Figure 5 The active learning unit 13 shown in FIG. 1 updates the model 7 by performing active learning processing once or more in order to improve the discrimination accuracy of the model 7. Figure 1As described, the one or more active learning processes respectively include: selecting one or more evaluation images 4 from the image group 8 not used for learning among the multiple images 3; and repeatedly performing relearning processes until the evaluation results of the model 7 using the one or more evaluation images 4 meet the predetermined criteria. The relearning process includes: updating the training data set 6 in a manner that includes one or more additional images selected from the one or more evaluation images 4; and relearning the model 7 using the updated training data set 6. In at least one of the one or more active learning processes, selecting one or more evaluation images 4 includes: calculating an evaluation value E1 representing the discrimination reliability of the attributes based on the latest model 7 for the images 3 included in the image group 8 not used for learning among the multiple images 3; and selecting an image 3 with relatively low reliability from the image group 8 as one or more evaluation images 4 based on the evaluation value E1.

[0129] The active learning unit 13 may display a screen showing the result of the active learning process on the display device 600 during or after the active learning process is executed.

[0130] Fig.10 is a diagram showing an example of the type of evaluation value E1. In the following, among the multiple images 3, the image 3 labeled "qualified product" is referred to as a "qualified product image", and the image 3 labeled "defective product" is referred to as a "defective product image". Fig.10 In FIG. 1 , the evaluation value E1 calculated for each image 3 is represented by the length of the arrow. That is, Fig.10 The evaluation value E1 shown represents the distance between the point corresponding to the feature quantity calculated based on each image 3 and the discrimination boundary 60 in the feature space. In addition, when the point corresponding to the feature quantity calculated based on the qualified product image exists in the region 61 (the region where the feature quantity calculated based on the learning image to which the label "qualified product" is assigned exists), the evaluation value E1 takes a positive value. When the point corresponding to the feature quantity calculated based on the qualified product image exists in the region 62 (the region where the feature quantity calculated based on the learning image to which the label "defective product" is assigned exists), the evaluation value E1 takes a negative value. When the point corresponding to the feature quantity calculated based on the defective product image exists in the region 62, the evaluation value E1 takes a positive value. When the point corresponding to the feature quantity calculated based on the defective product image exists in the region 61, the evaluation value E1 takes a negative value. Fig.10 In the case of the evaluation value E1 shown, the smaller the evaluation value E1 is, the lower the attribute determination reliability is.

[0131] Fig.11 is a diagram showing another example of the types of evaluation values ​​E1. Fig.11 In FIG. 1 , the evaluation value E1 calculated for each image 3 is represented by the length of the arrow. That is, Fig.11 The evaluation value E1 shown represents the shortest distance between a point corresponding to a feature quantity calculated from each image 3 and a point corresponding to a feature quantity calculated from a learning image to which the same label as that of the image 3 is assigned in the feature space. Fig.11 In the case of the evaluation value E1 shown, the larger the evaluation value E1 is, the lower the attribute determination reliability is.

[0132] In addition, when using Fig.10 In the case of the evaluation value E1 shown in FIG. 1 , although the image 3 which is difficult to distinguish is easily selected as the evaluation image 4, the distribution of the feature quantity calculated from the image 3 included in the training data set 6 tends to be biased toward a narrow range. Fig.11 In the case of the evaluation value E1 shown in FIG. 1 , although the distribution of the feature quantity calculated based on the selected one or more evaluation images 4 becomes wider, the image 3 which is the most difficult to distinguish may not be selected as the evaluation image 4. Therefore, the type of evaluation value E1 is preferably selected appropriately according to the defect that may occur in the object 2. For example, when the deviation of the feature quantity calculated from the qualified product image is large, Fig.11 The evaluation value E1 shown is suitable. When the deviation is small, Fig.10 The evaluation value E1 shown is appropriate.

[0133] <Processing flow of learning device>

[0134] Fig.12 : is a flowchart showing an example of a processing flow of a learning device according to an embodiment. Fig.12 As shown, the CPU 110 of the learning device 100 obtains a plurality of images 3 (step S1). The plurality of images 3 are obtained by photographing with the camera 300. The learning device 100 may receive the plurality of images 3 from the camera 300 or from the identification device 200.

[0135] Next, the CPU 110 assigns a label indicating the attribute of the object 2 captured by the image 3 to each of the plurality of images 3 according to the user operation (step S2). In step S2, the CPU 110 records the defect position for each image 3 assigned the label "defective product" according to the user operation.

[0136] Next, the CPU 110 determines an initial training data set 6 including one or more learning images selected from the plurality of images 3 according to a user operation (step S3). The number of learning images included in the initial training data set 6 may be small. For example, the CPU 110 selects one qualified product image and one unqualified product image as learning images.

[0137] Next, the CPU 110 performs machine learning using the initial training data set 6 to obtain the model 7 used to discriminate the attribute of the object 2 captured in the input image (step S4 ).

[0138] Since the number of learning images included in the initial training data set 6 is small, the discrimination accuracy based on the attributes of the model 7 acquired in step S4 is low. Therefore, the CPU 110 updates the model 7 by executing the active learning process once or more (step S5).

[0139] (Reference example of step S5)

[0140] Reference Fig.13 and Fig.14 , a reference example of step S5 is explained. Fig.13 Yes means Fig.12 Flowchart of the subroutine flow of the reference example of step S5 shown in FIG. Fig.13 The flow of steps S101 to S105 is shown in FIG. Fig.14 Yes Instructions to follow Fig.13 FIG. 1 is a flow chart of the active learning process shown in FIG. Fig.14 As shown, the plurality of images 3 include a plurality of images 30 of conforming products to which a label “conforming products” is assigned and a plurality of images 32 of defective products to which a label “defective products” is assigned.

[0141] like Fig.13 As shown, in step S101, all images 3 that are not used for learning are selected as evaluation images. Fig.14 In the example shown, the qualified product image 30a is selected as the learning image from the plurality of qualified product images 30, and the defective product image 32a is selected as the learning image from the plurality of defective product images 32. Therefore, all the qualified product images 30 except the qualified product image 30a and all the defective product images 32 except the defective product image 32a are selected as the evaluation images.

[0142] Next, in step S102, the latest model 7 is evaluated using all the selected evaluation images. Specifically, the discrimination accuracy of the model 7 and the evaluation value E2 of the discrimination reliability of the attribute represented by each evaluation image are calculated. As the discrimination accuracy, for example, the error rate is calculated by dividing the sum of the number of images of qualified products judged as "defective products" and the number of images of unqualified products judged as "qualified products" of the attribute of the object 2 by the total number of evaluation images. The evaluation value E2 is calculated by the same method as the above-mentioned evaluation value E1. That is, the evaluation value E1 of each evaluation image, for example, represents the distance between the point in the feature space corresponding to the feature quantity calculated based on the evaluation image and the discrimination boundary 60 (refer to Fig.10Alternatively, the evaluation value E2 of each evaluation image may also represent the shortest distance between a point corresponding to the feature quantity calculated based on the evaluation image and a point corresponding to the feature quantity calculated based on a learning image to which the same label as that of the evaluation image is assigned (see Fig.11 ).exist Fig.14 In the example shown, an evaluation value E2 is calculated, which represents the shortest distance between a point corresponding to a feature quantity calculated from an evaluation image and a point corresponding to a feature quantity calculated from a learning image to which the same label as the evaluation image is assigned. Therefore, the larger the evaluation value E2, the lower the reliability of attribute discrimination.

[0143] Next, in step S103, it is determined whether the evaluation result satisfies the end criterion. The end criterion includes a first criterion that the error rate is less than a predetermined first threshold value. The end criterion may also include a second criterion that the proportion of images 3 included in the training data set 6 among the plurality of images 3 exceeds a second threshold value. In the case where the end criterion includes the first criterion and the second criterion, the end criterion is satisfied when at least one of the first criterion and the second criterion is satisfied. In the case where the end criterion is satisfied (step S103: yes), step S5 ends.

[0144] When the end criterion is not satisfied (step S103: No), an evaluation image with relatively low reliability is selected from all evaluation images as an additional image based on the evaluation value E2 (step S104). Fig.14 In the example shown, the qualified product image 30b with the largest evaluation value E2 is selected as the additional image from the evaluation images labeled "qualified product". Also, the defective product image 32b with the largest evaluation value E2 is selected as the additional image from the evaluation images labeled "defective product".

[0145] In step S105 after step S104, the training data set 6 is updated to include the additional image, and the model 7 is relearned using the updated training data set 6. By performing relearning using the training data set 6 including the additional image with relatively low reliability, the discrimination accuracy based on the attribute of the model 7 is improved. After step S105, the process returns to step S102. In this way, steps S102 to S105 are repeatedly performed until the evaluation result of the model 7 meets the end criterion.

[0146] exist Fig.13 and Fig.14 In the reference example shown, all images 3 not used for learning among the plurality of images 3 are selected as evaluation images, so the time required to execute step S102 becomes longer. Step S102 is repeatedly executed until the evaluation result of the model 7 satisfies the end criterion. Therefore, the time required to execute step S5 of the reference example becomes longer.

[0147] (Example 1 of Step S5)

[0148] Reference Figure 15 to Figure 17 , Example 1 of step S5 is described. Fig.15 This is a diagram showing an example of a screen for setting conditions for active learning processing. Fig.16 Yes means Fig.12 Flow chart of the subroutine procedure of the first embodiment of step S5 shown. Fig.17 Yes Instructions to follow Fig.16 FIG. 5 is a flow chart of the active learning process shown in FIG.

[0149] In the first embodiment, the CPU 110 of the learning device 100 performs Fig.15 The input of the screen 90 shown in FIG. 1 sets the conditions for active learning processing in advance. Fig.15 As shown, screen 90 includes input fields 91 to 93 .

[0150] The number of evaluation images is input in the input field 91. The CPU 110 sets the number of evaluation images to be selected from the plurality of images 3 according to the input in the input field 91.

[0151] A first threshold value (error rate threshold value) defining a first criterion included in the end criterion is input in the input field 92. The CPU 110 sets the first threshold value in accordance with the input in the input field 93.

[0152] A second threshold value (maximum scale of the learning image) defining a second criterion that can be included in the end criterion is input in the input field 93. The CPU 110 sets the second threshold value in accordance with the input to the input field 93.

[0153] like Fig.16 As shown in FIG. 1 , the CPU 110 of the learning device 100 calculates an evaluation value E1 indicating the reliability of the determination of the attribute based on the model 7 for each image 3 not used for learning among the plurality of images 3 (step S11). Fig.17In the example shown, the plurality of images 3 include a plurality of images 30 of qualified products to which the label "qualified products" is assigned and a plurality of images 32 of unqualified products to which the label "unqualified products" is assigned. Then, the qualified product image 30a is selected as the learning image from the plurality of qualified product images 30, and the unqualified product image 32a is selected as the learning image from the plurality of unqualified product images 32. Therefore, the CPU 110 applies the model 7 to the image group 8g of the qualified product images 30 other than the qualified product image 30a, and calculates the evaluation value E1 for each qualified product image 30 included in the image group 8g. Furthermore, the CPU 110 applies the model 7 to the image group 8ng of the unqualified product images 32 other than the unqualified product image 32a, and calculates the evaluation value E1 for each unqualified product image 32 included in the image group 8ng.

[0154] Next, the CPU 110 selects an image 3 with relatively low reliability as one or more evaluation images 4 from the image group not used for learning among the plurality of images 3 based on the evaluation value E1 (step S12). Fig.17 In the example shown, a qualified product image 30 and a defective product image 32 with relatively low reliability are selected from the image groups 8g and 8ng, respectively, as evaluation images 40 and 42. Evaluation image 40 is image 3 with a label "qualified product". Evaluation image 42 is image 3 with a label "defective product". CPU 110 selects a preset number of evaluation images. Fig.17 In the example shown, the CPU 110 selects the evaluation image group 9g consisting of six evaluation images 40 and the evaluation image group 9ng consisting of four evaluation images 42. The ratio of the number of evaluation images 40 to the number of evaluation images 42 is predetermined.

[0155] Next, the CPU 110 uses one or more evaluation images 4 (evaluation images 40, 42) to evaluate the latest model 7 (step S13). Specifically, the CPU 110 calculates the error rate as the discrimination accuracy of the model 7 in the same manner as step S102, and calculates the evaluation value E2 representing the discrimination reliability of the attribute for each evaluation image 4. Fig.17 In the example shown, an evaluation value E2 is calculated, which represents the shortest distance between a point corresponding to a feature quantity calculated from each evaluation image and a point corresponding to a feature quantity calculated from a learning image to which the same label as the evaluation image is assigned. Therefore, the larger the evaluation value E2, the lower the reliability of attribute discrimination.

[0156] Next, the CPU 110 determines whether the evaluation result satisfies the end criterion (step S14). The end criterion includes a first criterion that the error rate is less than a predetermined first threshold value. The end criterion may also include a second criterion that the proportion of images 3 included in the training data set in the plurality of images 3 exceeds a second threshold value. In the case where the end criterion includes the first criterion and the second criterion, the end criterion is satisfied when at least one of the first criterion and the second criterion is satisfied. Fig.15 The first criterion and the second criterion are set in advance on the screen 90 shown. When the end criterion is satisfied (step S14: Yes), step S5 ends. By including the second criterion in the end criterion, when the model 7 converges to an inappropriate state and does not satisfy the first criterion even after repeated relearning, step S5 can be ended.

[0157] Furthermore, when the end criterion includes the second criterion, the CPU 110 calculates the ratio a of the qualified product images 30 included in the training data set 6 among the plurality of qualified product images 30 and the ratio b of the defective product images 32 included in the training data set 6 among the plurality of defective product images 32. The CPU 110 determines that the second criterion is satisfied when both the ratio a and the ratio b exceed the second threshold value, and determines that the second criterion is not satisfied when at least one of the ratio a and the ratio b is below the second threshold value.

[0158] When the end criterion is not satisfied (step S14: No), the CPU 110 selects the evaluation image 4 with relatively low reliability as an additional image based on the evaluation value E2 (step S15). Fig.17 In the example shown, the evaluation image 40a having the highest evaluation value E2 is selected as an additional image from the evaluation image group 9g, and the evaluation image 42a having the highest evaluation value E2 is selected as an additional image from the evaluation image group 9ng.

[0159] Furthermore, when the end criterion includes the second criterion, the CPU 110 selects an additional image from the evaluation image group 9g when the above-mentioned ratio a is less than the second threshold value, and selects an additional image from the evaluation image group 9ng when the above-mentioned ratio b is less than the second threshold value. In other words, the CPU 110 does not select an additional image from the evaluation image group 9g when the ratio a exceeds the second threshold value, and does not select an additional image from the evaluation image group 9ng when the ratio b exceeds the second threshold value.

[0160] After step S15, the CPU 110 updates the training data set 6 in a manner including the additional image, and relearns the model 7 using the updated training data set 6 (step S16). By performing relearning using the training data set 6 including the additional image with relatively low reliability, the discrimination accuracy based on the attribute of the model 7 is improved. After step S16, the process returns to step S13. In this way, steps S13 to S16 are repeatedly performed until the evaluation result of the model 7 meets the end criterion.

[0161] exist Fig.16 and Fig.17 In the embodiment 1 shown, a portion of the images 3 in the image groups 8g and 8ng not used for learning among the plurality of images 3 are selected as evaluation images. Therefore, the time required for executing step S13 is shorter than that required for performing step S20. Fig.13 The time required to execute step S102 shown is shortened. As a result, the time required to execute step S5 of the first embodiment is shortened.

[0162] (Example 2 of Step S5)

[0163] Reference Figure 18 to Figure 20 , Example 2 of step S5 is described. Fig.18 Yes means Fig.12 Flow chart of the subroutine flow of the second embodiment of step S5 shown.

[0164] In the second embodiment, similarly to the first embodiment, the CPU 110 performs the following operations according to the Fig.15 The conditions of the active learning process (the number of evaluation images, the first threshold value, and the second threshold value) are set in advance by inputting the screen 90 shown. However, step S5 of Example 2 includes two active learning processes. Therefore, the CPU 110 sets the conditions of each of the two active learning processes (the number of evaluation images, the first threshold value, and the second threshold value). In addition, the number of evaluation images in the two active learning processes may be the same or different. The first threshold value of the second active learning process is set to be less than the first threshold value of the first active learning process. The second threshold value of the second active learning process is set to be greater than the second threshold value of the first active learning process.

[0165] like Fig.18 As shown, the CPU 110 of the learning device 100 executes the first active learning process in step S21 and executes the second active learning process in the next step S22.

[0166] Fig.19 Yes means Fig.18 Flowchart of the subroutine flow of step S21 shown. Fig. 20 Yes Instructions to follow Fig.19 FIG. 5 is a flow chart of the active learning process shown in FIG.

[0167] like Fig.19 As shown, the CPU 110 does not use the evaluation value E1, but randomly or according to a predetermined rule, selects one or more evaluation images 4 from the image group 8 not used for learning among the plurality of images 3 (step S31). The predetermined rule is, for example, a rule that selects the image 3 at a specified number when the images are arranged according to the file name or the shooting date and time. The CPU 110 selects a predetermined number of evaluation images. Fig. 20 In the example shown, the CPU 110 selects the evaluation image group 9g' consisting of six evaluation images 40 and the evaluation image group 9ng' consisting of four evaluation images 42. The ratio of the number of evaluation images 40 to the number of evaluation images 42 is predetermined.

[0168] Next, the CPU 110 executes steps S32 to S35. The processing contents of steps S32 to S35 are respectively as follows: Fig.16 The steps S13 to S16 shown are the same. Therefore, the detailed description of steps S32 to S35 is omitted. Fig. 20 In the example shown, in step S34, the evaluation image 40b having the highest evaluation value E2 is selected as an additional image from the evaluation image group 9g'. Also, the evaluation image 42b having the highest evaluation value E2 is selected as an additional image from the evaluation image group 9ng'.

[0169] Fig.18 The processing flow of the subroutine of step S22 shown in FIG. Fig.16 The flowchart shown is the same. That is, based on the evaluation value E1, an image 3 with relatively low reliability is selected from the image group 8 not used for learning as one or more evaluation images 4. Then, using the selected one or more evaluation images 4, the relearning process is repeatedly performed until the evaluation result of the model 7 satisfies the end criterion. In addition, in step S22, the image group 8 not used for learning includes the learning images determined in step S3 and the images 3 other than the additional images selected in step S21.

[0170] As described above, since the number of learning images included in the initial training data set 6 is small, the discrimination accuracy based on the attributes of the initial model 7 obtained in step S4 is low. Therefore, when selecting one or more evaluation images 4 based on the evaluation value E1 calculated using the initial model 7, it may not be possible to select an image suitable for learning as the evaluation image 4. As a result, through the relearning of the model 7, the model 7 converges to an inappropriate state, and even if the relearning process is repeated, the discrimination accuracy of the model 7 may not meet the first criterion.

[0171] However, according to the second embodiment, in the first active learning process, the evaluation value E1 is not used, and one or more evaluation images 4 are selected from the image group 8 randomly or according to a predetermined rule. Therefore, through the first active learning process, the discrimination accuracy of the model 7 converges to a state of being high to a certain extent, and the discrimination accuracy of the model 7 easily satisfies the first criterion. Moreover, in the second active learning process, an image suitable for learning is selected as the evaluation image 4 based on the evaluation value E1 obtained by applying the model 7 whose discrimination accuracy is improved to a certain extent by the first active learning process. As a result, a model 7 with high discrimination accuracy can be generated in a shorter time.

[0172] (Example 3 of step S5)

[0173] Reference Fig.21 , Example 3 of step S5 is explained. Fig.21 Yes means Fig.12 Flow chart of the subroutine flow of the third embodiment of step S5 shown.

[0174] In the third embodiment, similarly to the first embodiment, the CPU 110 performs the following operations according to the Fig.15 The conditions of the active learning process (the number of evaluation images, the first threshold value, and the second threshold value) are set in advance by inputting the screen 90 shown in the figure. However, step S5 of the third embodiment includes M+N active learning processes. Therefore, the CPU 110 sets the conditions of each of the M+N active learning processes (the number of evaluation images, the first threshold value, and the second threshold value). In addition, M and N are integers greater than 1. Among them, at least one of M and N is greater than 2.

[0175] In addition, the number of evaluation images in the M+N active learning processes may be the same or different. The first threshold of the i-th active learning process is set to be smaller than the first threshold of the i-1-th active learning process (i is an integer greater than 2). The second threshold of the i-th active learning process is set to be greater than the second threshold of the i-1-th active learning process.

[0176] like Fig.21 As shown, the CPU 110 of the learning device 100 sequentially executes the 1st to Mth active learning processes (steps S21_1 to S21_M), and then sequentially executes the M+1th to M+Nth active learning processes (steps S22_1 to S22_N).

[0177] Fig.21 The processing flow of each subroutine of steps S21_1 to S21_M shown in FIG. Fig.19The same flowchart as shown. That is, without using the evaluation value E1, one or more evaluation images 4 are selected randomly or according to a predetermined rule from the image group 8 not used for learning among the plurality of images 3. Then, using the selected one or more evaluation images 4, the relearning process is repeatedly performed until the evaluation result of the model 7 satisfies the end criterion. In addition, in step S21_k, the image group 8 not used for learning includes the learning images determined in step S3 and the images 3 other than the additional images selected in steps S21_1 to S21_k-1.

[0178] In addition, the selection methods of the evaluation images 4 in steps S21_1 to S21_M may be different from each other. For example, in step S21_1, one or more evaluation images 4 may be randomly selected from the image group 8, and in step S21_M, one or more evaluation images 4 may be selected from the image group 8 according to a predetermined rule. In this case, the CPU 110 sets in advance the selection method of the evaluation images 4 in the first to Mth active learning processes according to user operations.

[0179] In step S21_k, one or more evaluation images 4 may be selected from the images 3 in the image group 8 other than the evaluation images 4 selected in steps S21_1 to S21_k-1.

[0180] The type of evaluation value E1 calculated in steps S22_1 to S22_N ( Fig.10 The evaluation value E1 shown, Fig.11 In this case, the CPU 110 sets in advance the types of evaluation values ​​E1 in the active learning processes from the M+1th to the M+Nth times according to user operations.

[0181] (Example 4 of step S5)

[0182] Reference Fig. 22 and Fig.23 , Example 4 of step S5 is described. Fig. 22 Yes means Fig.12 Flow chart of the subroutine flow of the fourth embodiment of step S5 shown.

[0183] In the fourth embodiment, similarly to the first embodiment, the CPU 110 performs the following operations according to the Fig.15The conditions of the active learning process (the number of evaluation images, the first threshold value, and the second threshold value) are set in advance by inputting the screen 90 shown in the figure. However, step S5 of the fourth embodiment includes three active learning processes. Therefore, the CPU 110 sets the conditions of each of the three active learning processes (the number of evaluation images, the first threshold value, and the second threshold value). In addition, as described later, in the third active learning process, all images 3 that are not used for learning are selected as evaluation images 4. Therefore, the CPU 110 does not set the number of evaluation images for the three active learning processes. Similar to the second embodiment, the number of evaluation images in the first and second active learning processes may be the same or different. Similar to the third embodiment, the first threshold value of the i-th active learning process is set to be less than the first threshold value of the i-1th active learning process. The second threshold value of the i-th active learning process is set to be greater than the second threshold value of the i-1th active learning process.

[0184] like Fig. 22 As shown, the CPU 110 of the learning device 100 executes the first active learning process in step S21, executes the second active learning process in the next step S22, and executes the third active learning process in the next step S23. Fig. 22 The processing flow of the subroutine of step S21 shown in FIG. Fig.19 The flowchart shown is the same. Fig. 22 The processing flow of the subroutine of step S22 shown in FIG. Fig.16 The flowchart shown is the same.

[0185] Fig.23 Yes means Fig. 22 The flowchart of the subroutine process of step S23 shown in FIG. Fig.23 As shown, the CPU 110 selects all images 3 not used for learning among the plurality of images 3 as evaluation images 4 (step S41). That is, the CPU 110 selects all images 3 other than the learning images determined in step S3 and the additional images selected in steps S21 and S22 as evaluation images 4.

[0186] Next, the CPU 110 executes steps S42 to S45. The processing contents of steps S42 to S45 are respectively as follows: Fig.16 The steps S13 to S16 shown are the same, and therefore, detailed description of steps S42 to S45 is omitted.

[0187] According to the fourth embodiment, in the third active learning process, all images 3 that have not been used for learning are selected as evaluation images 4. Therefore, the discrimination accuracy of the model 7 can be evaluated more accurately.

[0188] (Example 5 of Step S5)

[0189] The fifth embodiment of step S5 combines the above-mentioned third and fourth embodiments. That is, the fifth embodiment of step S5 includes Fig.21 Steps S21_1 to S21_M, S22_1 to S22_N and steps performed after step S22_N Fig. 22 Step S23 is shown.

[0190] <Results of active learning processing>

[0191] Fig.24 This is a diagram showing an example of a screen showing a result of active learning processing. Fig.24 The screen image 95 shown is generated by the CPU 110 and is displayed on the display device 600 .

[0192] like Fig.24 As shown, screen 95 includes a list of defective product images 81 and a list of qualified product images 82. In screen 95, each of the list of defective product images 81 and the list of qualified product images 82 includes: a mark 87 identifying an image 3 selected as a learning image; and a mark 88 identifying an image 3 selected as an additional image in one or more active learning processes. Thus, the user can grasp the image 3 used for learning among the plurality of images 3.

[0193] Fig.25 It is a diagram showing another example of a screen showing a result of the active learning process. Fig.25 Screen 50 shown is generated by CPU 110 and is displayed on display device 600 during execution of the active learning process.

[0194] like Fig.25 As shown, the screen 50 includes a graph 51 showing the transition of the error rate as the evaluation result of the model 7 and a display field 52 . Fig.25 The screen 50 shown is displayed during the implementation of the 12th relearning process in the second active learning process (step S22) according to the above-mentioned embodiment 2. Therefore, the graph 51 shows the transition of the error rate calculated in the first active learning process (step S21) and the error rate calculated in the first to 11th relearning processes in the second active learning process.

[0195] The display field 52 displays the ratio of the number of evaluation images 4 selected as additional images to the total number of evaluation images 4 selected in the second active learning process. Fig.25 In the example shown, the user can confirm that 12 of the 224 evaluation images labeled "conforming product" have been selected as additional images, and 6 of the 12 evaluation images labeled "defective product" have been selected as additional images.

[0196] <Modification 1>

[0197] In the above description, the plurality of images 3 include a plurality of qualified product images 30 and a plurality of unqualified product images 32, and one or more learning images are selected from the plurality of qualified product images 30 and the plurality of unqualified product images 32. However, depending on the object 2, the frequency of defects is sometimes very small. In such a case, the plurality of images 3 include a plurality of qualified product images 30 and one or more unqualified product images 32. If a small number of unqualified product images 32 are used for learning, there are not enough unqualified product images 32 for evaluation of the model 7. In such a case, machine learning can be performed using only the qualified product images 30.

[0198] Fig.26 1 is a diagram for explaining a method for obtaining a model in Variation 1. Fig.26 As shown, the CPU 110 operating as the acquisition unit 12 performs machine learning using one or more learning images labeled "qualified products" to acquire a model 7' of feature quantities on an output feature space from an input image. The CPU 110 generates a model 7' that defines a feature space such that the distance between points corresponding to feature quantities corresponding to one or more learning images is minimized.

[0199] The CPU 110 inputs one or more qualified product images 30 and one or more defective product images 32 not used for learning into the model 7', and determines a judgment boundary 60' that separates a point 65 corresponding to a feature value obtained from the qualified product image 30 and a point 66 corresponding to a feature value obtained from the defective product image 32.

[0200] The processing flow of the learning device of the modification example 1 is the same as that of the above-mentioned embodiment, and follows Fig.12 However, in step S4, the model 7' defining the feature space is acquired, but the determination boundary 60' is not determined.

[0201] The subroutine of step S5 in the modification 1 is similar to the above-mentioned embodiment and follows any one of the above-mentioned embodiments 1 to 4 (see Fig.16 , Fig.18 , Fig.19 , Figure 21 to Figure 23 ). However, in Fig.16 In step S11, the CPU 110 calculates the evaluation value E1 only for each qualified product image 30 included in the image group 8. Then, in step S12, the CPU 110 selects one or more qualified product images 30 with relatively low reliability as the evaluation image 4, and selects one or more unqualified product images 32 as the evaluation image 4. Fig.19In step S31, the CPU 110 randomly or according to a predetermined rule selects one or more qualified product images 30 included in the image group 8 as the evaluation image 4, and selects one or more unqualified product images 32 as the evaluation image 4. Fig.16 Step S13, Fig.19 Step S32 and Fig.23 In step S42 , the CPU 110 determines the determination boundary 60 ′ based on the feature amount obtained by inputting one or more evaluation images 4 into the model 7 ′, and calculates the determination accuracy and the evaluation value E2 .

[0202] Fig. 27 This is a diagram for explaining the active learning process in Modification Example 1. Fig. 27 The following is shown in Modification 1 Fig.16 The flowchart shown, Fig.18 and Fig. 22 Step S22, Fig.21 Active learning processing of steps S22_1 to S22_N.

[0203] like Fig. 27 As shown, the plurality of images 3 include a plurality of conforming product images 30 and one or more defective product images 32. By using a learning image ( Fig. 27 The initial model 7' is obtained by machine learning of the initial training data set 6 of the qualified product images 30a in the image group 8g. In addition, the image group 8g includes the qualified product images 30 that are not used for learning.

[0204] The evaluation value E1 is calculated for each qualified product image 30 of the image group 8g, and based on the evaluation value E1, a qualified product image 30 with relatively low discrimination reliability is selected as the evaluation image 4 (evaluation image 40). In addition, one or more unqualified product images 32 are also selected as the evaluation image 4. In addition, the evaluation images 40 with the label "qualified product" constitute the evaluation image group.

[0205] Then, a discrimination boundary 60' is determined based on the feature quantity obtained by inputting one or more evaluation images 4 into the model 7'. Specifically, in the feature space, a discrimination boundary 60' is determined that separates the area where points corresponding to the feature quantity calculated based on one or more evaluation images 40 exist and the area where points corresponding to the feature quantity calculated based on one or more defective product images 32 exist.

[0206] Furthermore, the evaluation value E2 is calculated for each evaluation image 40 included in the evaluation image group 9g until the evaluation result of the model 7' using one or more evaluation images 4 satisfies the end criterion, and based on the evaluation value E2, an evaluation image 40 with relatively low reliability is selected as an additional image. Then, the training data set 6 is updated in a manner including the additional image, and the model 7' is re-learned using the updated training data set 6.

[0207] Fig.28 : is a diagram showing an example of evaluation values ​​E1 and E2 in Modification Example 1. Fig.28 In FIG. 3 , the evaluation values ​​E1 and E2 calculated for the conforming product image 30 are represented by the lengths of the arrows. That is, Fig.10 The evaluation values ​​E1 and E2 shown represent the distance between the point corresponding to the feature quantity calculated based on the qualified product image 30 and the judgment boundary 60' in the feature space. In addition, when the point corresponding to the feature quantity calculated from the qualified product image 30 exists in the area 67 (the area where the feature quantity calculated based on the learning image to which the label "qualified product" is assigned exists), the evaluation values ​​E1 and E2 take positive values. In the case where the point corresponding to the feature quantity calculated based on the qualified product image 30 exists in the area 68 different from the area 67, the evaluation values ​​E1 and E2 take negative values. Fig.28 In the case of the evaluation values ​​E1 and E2 shown, the smaller the evaluation values ​​E1 and E2 are, the lower the attribute determination reliability is.

[0208] Fig.29 FIG. 2 is a diagram showing another example of evaluation values ​​E1 and E2 in Modification Example 1. Fig.29 In FIG. 3 , the evaluation values ​​E1 and E2 calculated for the conforming product image 30 are represented by the lengths of the arrows. That is, Fig.29 The evaluation values ​​E1 and E2 shown in represent the shortest distance between a point corresponding to the feature quantity calculated from the qualified product image 30 and a point corresponding to the feature quantity calculated from the learning image to which the label "qualified product" is assigned in the feature space. Fig.29 In the case of the evaluation values ​​E1 and E2 shown, the larger the evaluation values ​​E1 and E2 are, the lower the attribute determination reliability is.

[0209] According to the modification 1, in step S4, the determination boundary 60' is not determined. Therefore, in the case where the subroutine of step S5 in the modification 1 follows the above-mentioned embodiment 1, in step S11, the determination boundary 60' is calculated. Fig.29 The evaluation value E1 is shown.

[0210] When the subroutine of step S5 in the modification 1 follows any one of the above-mentioned embodiments 2 to 4, the determination boundary 60' is determined before calculating the evaluation value E1. Therefore, in steps S22, S22_1 to S22_N, the calculation Fig.28 and Fig.29 Any one of the evaluation values ​​E1 shown.

[0211] <Modification 2>

[0212] In the above embodiment, the CPU 110 performs Fig.15 The number of evaluation images 4 is set by inputting the screen 90 shown in FIG. 1 . However, the CPU 110 may set the ratio of the number of evaluation images 4 to the number of images 3 included in the image group 8 instead of the number of evaluation images 4. In this case, the CPU 110 may select the evaluation images 4 from the image group 8 so as to achieve the set ratio.

[0213] <Variation 3>

[0214] In the above embodiment, a scene of detecting whether the object 2 has defects is shown. However, the application scope of the present disclosure is not limited to such an example of appearance inspection. The present disclosure can be applied to all scenes of detecting certain features from an image of the object 2. The object 2 is not limited to a stationary object, but can also be a movable object such as a person, a vehicle (such as a car), etc.

[0215] §3 Notes

[0216] As described above, this embodiment includes the following disclosure.

[0217] (Structure 1)

[0218] A learning device (100), wherein the learning device (100) comprises:

[0219] an acquisition unit (12, 110) that acquires a model (7, 7') used for determining the attribute of an object (2) captured in an input image by performing machine learning using a training data set (6); and

[0220] an active learning unit (13, 110) that updates the model (7, 7') by executing active learning processing one or more times,

[0221] The training data set (6) includes one or more learning images selected from a plurality of images (3) to which labels representing attributes of the object (2) are assigned.

[0222] The one or more active learning processes respectively include:

[0223] selecting one or more evaluation images (4, 40, 42) from a group of images (8, 8g, 8ng) not used for learning among the plurality of images (3); and

[0224] The relearning process is repeatedly performed until the evaluation result of the model (7, 7') using the one or more evaluation images (4, 40, 42) satisfies the end criterion,

[0225] The relearning process includes:

[0226] updating the training data set (6) so as to include one or more additional images selected from the one or more evaluation images (4, 40, 42); and

[0227] relearning the model (7, 7') using the updated training data set (6),

[0228] In at least one of the one or more active learning processes, selecting the one or more evaluation images (4, 40, 42) includes:

[0229] For each image (3) included in the image group (8, 8g, 8ng), a first evaluation value indicating the reliability of determination of the attribute based on the latest model (7, 7') is calculated; and

[0230] Based on the first evaluation value, an image with relatively low reliability is selected from the image group (8, 8g, 8ng) as the one or more evaluation images (4, 40, 42).

[0231] (Structure 2)

[0232] In the learning device (100) described in Configuration 1,

[0233] The relearning process further comprises:

[0234] For each of the one or more evaluation images (4, 40, 42), a second evaluation value indicating the reliability of determination of the attribute based on the latest model (7, 7') is calculated; and

[0235] Based on the second evaluation value, an evaluation image (40a, 42a) with relatively low reliability is selected from the one or more evaluation images (4, 40, 42) as the one or more additional images.

[0236] (Structure 3)

[0237] In the learning device described in structure 1 or 2,

[0238] The model (7, 7') outputs a feature quantity representing a feature of the input image,

[0239] The feature quantity is compared with a determination boundary (60, 60') for determining the attribute,

[0240] The first evaluation value for the target image included in the image group indicates a distance between the feature amount calculated based on the target image and the determination boundary (60, 60').

[0241] (Structure 4)

[0242] In the learning device (100) described in structure 1 or 2,

[0243] The model (7, 7') outputs a feature quantity representing a feature of the input image,

[0244] The first evaluation value for the object image included in the image group (8, 8g, 8ng) represents the distance between the feature value calculated based on the object image and the feature value calculated based on the image used for learning the latest model (7, 7') and assigned the same label as the object image.

[0245] (Structure 5)

[0246] In the learning device (100) according to any one of configurations 1 to 4,

[0247] The one or more active learning processes include M active learning processes and N active learning processes that are executed after the M active learning processes and correspond to the at least one active learning process, where M and N are integers greater than 1.

[0248] In the M-time active learning process, selecting the one or more evaluation images includes: selecting the one or more evaluation images (4, 40, 42) from the image group randomly or according to a predetermined rule without using the first evaluation value.

[0249] (Structure 6)

[0250] In the learning device (100) described in Configuration 5,

[0251] The one or more active learning processes include the M+N+1th active learning process performed after the Nth active learning process.

[0252] In the M+N+1th active learning process, selecting the one or more evaluation images includes selecting all images included in the image group as the one or more evaluation images (4, 40, 42).

[0253] (Structure 7)

[0254] In the learning device (100) described in structure 5 or 6,

[0255] N is an integer greater than 2,

[0256] In the N-times active learning process, at least one of the number of evaluation images selected from the image group (8, 8g, 8ng), the type of the first evaluation value, and the end criterion is different from each other.

[0257] (Structure 8)

[0258] In the learning device (100) described in any one of the structures 5 to 7,

[0259] M is an integer greater than or equal to 2,

[0260] In the M times of active learning processing, at least one of the number of evaluation images selected from the image group (8, 8g, 8ng), the method of selecting the one or more evaluation images, and the end criterion is different from each other.

[0261] (Structure 9)

[0262] In the learning device (100) described in any one of the structures 1 to 8,

[0263] The label includes a first label indicating that the object (2) is a qualified product and a second label indicating that the object (2) is a defective product.

[0264] The plurality of images (3) include a plurality of images (30) of conforming products to which the first label is assigned and one or more images (32) of defective products to which the second label is assigned,

[0265] selecting one or more learning images from the plurality of qualified product images (30);

[0266] The one or more evaluation images include one or more images of good products (30) and one or more images of bad products (32) selected from the image group (8g).

[0267] The first evaluation value is calculated for each conforming product image included in the image group (8g).

[0268] (Structure 10)

[0269] A learning method, wherein the learning method has the following steps:

[0270] Step (S4), the computer (100) performs machine learning using a training data set (6) to obtain a model (7, 7') used to determine the attributes of an object (2) captured in an input image;

[0271] Step (S5), the computer (100) updates the model (7, 7') by performing active learning processing one or more times,

[0272] The training data set (6) includes one or more learning images selected from a plurality of images (3) to which labels representing attributes of the object (2) are assigned.

[0273] The one or more active learning processes respectively include:

[0274] selecting one or more evaluation images (4, 40, 42) from a group of images (8, 8g, 8ng) not used for learning among the plurality of images (3); and

[0275] The relearning process is repeatedly performed until the evaluation result of the model (7, 7') using the one or more evaluation images (4, 40, 42) satisfies the end criterion,

[0276] The relearning process includes:

[0277] updating the training data set (6) so as to include one or more additional images selected from the one or more evaluation images (4, 40, 42); and

[0278] relearning the model (7, 7') using the updated training data set (6),

[0279] In at least one of the one or more active learning processes, selecting the one or more evaluation images (4, 40, 42) includes:

[0280] For each image (3) included in the image group (8, 8g, 8ng), a first evaluation value indicating the reliability of determination of the attribute based on the latest model (7, 7') is calculated; and

[0281] Based on the first evaluation value, an image with relatively low reliability is selected from the image group (8, 8g, 8ng) as the one or more evaluation images (4, 40, 42).

[0282] (Structure 11)

[0283] A program, wherein the program causes a computer to execute the following steps:

[0284] By performing machine learning using a training data set (6), a model (7, 7') used to determine the attributes of an object (2) captured in an input image is obtained; and

[0285] By performing active learning processing one or more times to update the model (7, 7'),

[0286] The training data set (6) includes one or more learning images selected from a plurality of images (3) to which labels representing attributes of the object (2) are assigned.

[0287] The one or more active learning processes respectively include:

[0288] selecting one or more evaluation images (4, 40, 42) from a group of images (8, 8g, 8ng) not used for learning among the plurality of images (3); and

[0289] The relearning process is repeatedly performed until the evaluation result of the model (7, 7') using the one or more evaluation images (4, 40, 42) satisfies the end criterion,

[0290] The relearning process includes:

[0291] updating the training data set (6) so as to include one or more additional images selected from the one or more evaluation images (4, 40, 42); and

[0292] relearning the model (7, 7') using the updated training data set (6),

[0293] In at least one of the one or more active learning processes, selecting the one or more evaluation images (4, 40, 42) includes:

[0294] For each image (3) included in the image group (8, 8g, 8ng), a first evaluation value indicating the reliability of determination of the attribute based on the latest model (7, 7') is calculated; and

[0295] Based on the first evaluation value, an image with relatively low reliability is selected from the image group (8, 8g, 8ng) as the one or more evaluation images (4, 40, 42).

[0296] Although the embodiments of the present invention have been described, the embodiments disclosed this time should be considered to be illustrative in all aspects and not restrictive. The scope of the present invention is indicated by the claims, and it is intended to include all modifications within the meaning and scope equivalent to the claims.

[0297] Description of symbols

[0298] 1: system; 2: object; 3, 3a: image; 4, 4a, 40, 40a, 40b, 42, 42a, 42b: evaluation image; 6: training data set; 7, 7': model; 8, 8g, 8ng: image group; 9g, 9ng, 9g', 9ng': evaluation image group; 10: storage unit; 11: label assignment unit; 12: acquisition unit; 13: active learning unit; 30, 30a, 30b: qualified product image; 32, 32a, 32b: unqualified product image; 50, 90, 95: screen; 51: chart; 52: display bar; 60, 60': judgment boundary; 61, 62, 67, 68, 74, 83: area; 70, 80, 86: user interface screen; 71: image list; 72, 82: qualified product image list ; 73, 81: defective product image list; 75a, 75b: add button; 76: move button; 78: button; 84: frame line; 87, 88: mark; 91, 92, 93: input field; 100: learning device; 110, 210: CPU; 111, 211: RAM; 112, 212: ROM; 113, 213: communication interface; 114, 215: input interface; 115, 216: display controller; 116, 217: driver; 120, 220: memory; 122: learning program; 124: labeling program; 130: storage medium; 200: identification device; 214: external interface; 221: identification program; 300: camera; 600: display device; 700: input device; D: defect.

Claims

1. A learning device, wherein: The learning device has: an acquisition unit that acquires a model used for determining a property of an object captured in an input image by performing machine learning using a training data set; and an active learning unit that updates the model by executing active learning processing one or more times, The training data set includes one or more learning images selected from a plurality of images to which labels representing attributes of the object are assigned. The one or more active learning processes respectively include: selecting one or more evaluation images from a group of images not used for learning among the plurality of images; and The relearning process is repeatedly performed until the evaluation result of the model using the one or more evaluation images satisfies an end criterion, The relearning process includes: updating the training data set to include one or more additional images selected from the one or more evaluation images; and relearning the model using the updated training data set, In at least one of the one or more active learning processes, selecting the one or more evaluation images includes: calculating, for each image included in the image group, a first evaluation value indicating a degree of reliability of determination of the attribute based on the latest model; and Based on the first evaluation value, an image with relatively low reliability is selected from the image group as the one or more evaluation images.

2. The learning device according to claim 1, wherein: The relearning process further comprises: calculating, for each of the one or more evaluation images, a second evaluation value indicating a degree of reliability of determination of the attribute based on the latest model; and Based on the second evaluation value, an evaluation image with relatively low reliability is selected from the one or more evaluation images as the one or more additional images.

3. The learning device according to claim 1 or 2, wherein: The model outputs a feature quantity representing a feature of the input image, The feature quantity is compared with a determination boundary for determining the attribute, The first evaluation value for the target image included in the image group indicates a distance between the feature amount calculated based on the target image and the determination boundary.

4. The learning device according to claim 1 or 2, wherein: The model outputs a feature quantity representing a feature of the input image, The first evaluation value for the target image included in the image group indicates a distance between the feature amount calculated based on the target image and the feature amount calculated based on an image used for learning the latest model and assigned the same label as the target image.

5. The learning device according to any one of claims 1 to 4, wherein: The one or more active learning processes include M active learning processes and N active learning processes that are executed after the M active learning processes and correspond to the at least one active learning process, where M and N are integers greater than 1. In the M-times active learning process, selecting the one or more evaluation images includes selecting the one or more evaluation images from the image group randomly or according to a predetermined rule without using the first evaluation value.

6. The learning device according to claim 5, wherein: The one or more active learning processes include the M+N+1th active learning process performed after the Nth active learning process. In the M+N+1th active learning process, selecting the one or more evaluation images includes selecting all images included in the image group as the one or more evaluation images.

7. The learning device according to claim 5 or 6, wherein: N is an integer greater than 2, In the N-times active learning process, at least one of the number of evaluation images selected from the image group, the type of the first evaluation value, and the end criterion is different from each other.

8. The learning device according to any one of claims 5 to 7, wherein: M is an integer greater than or equal to 2, In the M-times active learning process, at least one of the number of evaluation images selected from the image group, the method of selecting the one or more evaluation images, and the end criterion is different from each other.

9. The learning device according to any one of claims 1 to 8, wherein: The labels include a first label indicating that the object is a qualified product and a second label indicating that the object is a defective product. The plurality of images include a plurality of images of conforming products to which the first label is assigned and one or more images of defective products to which the second label is assigned, selecting the one or more learning images from the plurality of conforming product images, The one or more evaluation images include one or more images of conforming products and one or more images of defective products selected from the image group. The first evaluation value is calculated for each of the conforming product images included in the image group.

10. A learning method, wherein: The learning method has the following steps: The computer performs machine learning using a training data set to obtain a model used to determine a property of an object captured in an input image; and The computer updates the model by performing active learning processing one or more times, The training data set includes one or more learning images selected from a plurality of images to which labels representing attributes of the object are assigned. The one or more active learning processes respectively include: selecting one or more evaluation images from a group of images not used for learning among the plurality of images; and The relearning process is repeatedly performed until the evaluation result of the model using the one or more evaluation images satisfies an end criterion, The relearning process includes: updating the training data set to include one or more additional images selected from the one or more evaluation images; and relearning the model using the updated training data set, In at least one of the one or more active learning processes, selecting the one or more evaluation images includes: calculating, for each image included in the image group, a first evaluation value indicating a degree of reliability of determination of the attribute based on the latest model; and Based on the first evaluation value, an image with relatively low reliability is selected from the image group as the one or more evaluation images.

11. A program, wherein: The program causes the computer to perform the following steps: Acquire a model used for determining the attribute of an object captured in an input image by performing machine learning using a training data set; and The model is updated by performing active learning processing one or more times, The training data set includes one or more learning images selected from a plurality of images to which labels representing attributes of the object are assigned. The one or more active learning processes respectively include: selecting one or more evaluation images from a group of images not used for learning among the plurality of images; and The relearning process is repeatedly performed until the evaluation result of the model using the one or more evaluation images satisfies an end criterion, The relearning process includes: updating the training data set to include one or more additional images selected from the one or more evaluation images; and relearning the model using the updated training data set, In at least one of the one or more active learning processes, selecting the one or more evaluation images includes: calculating, for each image included in the image group, a first evaluation value indicating a degree of reliability of determination of the attribute based on the latest model; and Based on the first evaluation value, an image with relatively low reliability is selected from the image group as the one or more evaluation images.

Images