IMAGE PROCESSING APPARATUS, APPEARANCE INSPECTION APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

Abstract

To provide a technique capable of easily reducing the influence of fluctuations in label selection by a worker on learning.SOLUTION: An image processing system 20 includes a labeling unit 41, a work status recording unit 43, and a self-confidence calculation unit 45. The labeling unit 41 gives a label to a tablet image 90 as teacher data based on an input by the worker which includes a selection input to select one out of multiple kinds of labels. The work status recording unit 43 records the work status of the worker based on the input by the worker. The self-confidence calculation unit 45 calculates the confidence in the input of the worker based on the work information recorded by the work status recording unit 43.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for inspecting an object. [Background technology]

[0002] Conventionally, many methods have been proposed for detecting defects using machine learning in the visual inspection process of industrial products. For example, an algorithm is known that prepares images of products without defects (good product images) and images of products with defects (defective product images) as learning data, and uses supervised machine learning to determine product defects. In general, the more correctly the image data is labeled as good or defective, the higher the accuracy of defect detection.

[0003] However, when creating training data, there may be instability in label selection due to the ability or fatigue of the training worker (the person who selects the labels to be assigned to the image data). For this reason, it is not always easy to gather the data necessary to achieve a certain level of accuracy, and it may require a large amount of cost.

[0004] In response to this, for example, Patent Document 1 proposes a method of automatically labeling without human intervention. Patent Document 2 proposes a method of correcting erroneous data labels using the reliability and certainty of the worker as class attribute information. Non-Patent Document 1 also proposes a method of improving accuracy and efficiency by increasing the number of classes, such as "Yes / Part / Container / Mixed / Missed Verification," in addition to "Yes / No" as labels when the work is done by humans. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-062544 A [Patent Document 2] JP 2018-106662 A [Non-patent literature]

[0006] [Non-Patent Document 1] Papadopoulos, Dim P., et al. "We don't need no bounding-boxes: Training object class detectors using only human verification." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016 Summary of the Invention [Problem to be solved by the invention]

[0007] However, when labeling is performed automatically as in Patent Document 1, there is a problem that the accuracy of labeling is not sufficiently guaranteed. In addition, in the case of Patent Document 2, it is necessary to collect the reliability and confidence of the operators in advance. In the case of Non-Cited Document 1, although erroneous teaching can be mitigated by increasing the number of classes, the problem of fluctuation in class definition remains.

[0008] An object of the present invention is to provide a technique for easily reducing the impact of fluctuations in label selection by an operator on learning. [Means for solving the problem]

[0009] In order to solve the above problem, a first aspect of the present invention is an image processing device, comprising: a labeling unit that assigns a label to an image serving as training data based on an input by an operator, the labeling unit including a selection input for selecting one of a plurality of types of labels; a work status recording unit that records a work status of the operator based on the input by the operator; and a confidence level calculation unit that calculates a confidence level for the input by the operator based on work information recorded by the work status recording unit. a learning unit that performs learning according to the confidence degree calculated by the confidence degree calculation unit for the image, using the image to which the label is assigned by the label assignment unit as teacher data; Equipped with The learning unit changes a learning rate for updating weights in the learning process based on the confidence level. .

[0010] A second aspect is the image processing device of the first aspect, wherein the work information includes a label selection time that is a time required for the worker to select the label.

[0011] A third aspect is the image processing device of the second aspect, wherein the work information includes an average value of each of the label selection times for a plurality of the images, and the confidence level calculation unit calculates the confidence level based on a difference between the label selection time and the average value.

[0012] A fourth aspect is the image processing device of the second or third aspect, wherein the work information includes an overall average value which is an average of the label selection time for each of the first number of images, and a partial average value which is an average of the label selection time for each of the first number of images, and the confidence level calculation unit calculates the confidence level based on a difference between the overall average value and the partial average value.

[0013] A fifth aspect is the image processing device of any one of the first to fourth aspects, wherein the input by the worker includes an enlargement input for enlarging the image, and the work information includes information relating to the enlargement input.

[0016] No. 6 The aspect is 1 Aspects from No. 5 Aspect Either one In the image processing device, when the confidence level for the image to which the label has been assigned does not satisfy a predetermined standard, the learning unit restricts the use of the image as the training data.

[0017] No. 7 The aspect is 1 From the aspect 6 In the image processing device of any one of the aspects, the label assignment unit accepts a correction of the label when the confidence level for the image to which the label has been assigned does not satisfy a predetermined standard.

[0018] No. 8 The aspect is 1From the aspect 7 The image processing device of any one of the aspects further includes an inference unit that performs inference on an image to be inspected using learned parameters generated by the learning unit.

[0019] No. 9 According to one aspect, there is provided an appearance inspection apparatus, comprising: an imaging unit that captures an image of an object; 8 and an image processing device of the present invention.

[0020] A tenth aspect is a computer-executable program, the learning unit of the image processing device of the first aspect being using the image to which the label is assigned by the label assignment unit of the image processing device as teacher data, performing learning on the image according to the degree of confidence calculated by the degree of confidence calculation unit of the image processing device, and changing a learning rate for updating weights in the learning based on the degree of confidence. The trained parameters are generated by fruit The parameters are used to cause the computer to perform inference on the image being inspected.

[0021] No. 11 An aspect of the present invention is an image processing method, comprising: The computer (b) a step of labeling images to be training data based on an operator's input including a selection input for selecting one of a plurality of labels; The computer, (c) recording the work status of the worker based on the input of the worker; The computer, calculating a degree of confidence in the input of the worker based on the work information recorded in the step (b); (d) the computer performs learning on the image based on the confidence level calculated in the step (c) using the image to which the label is assigned in the step (a) as training data; Including and the step (d) includes a step of changing a learning rate when updating weights in the learning based on the confidence level by the computer. .

[0022] A 14th aspect is the image processing method of the 13th aspect, further comprising a step of (d) using the image to which the label has been assigned by the step (a) as training data and conducting learning for the image according to the degree of confidence calculated by the step (c).

[0023] No. 12 The aspect is 11 The image processing method of the present invention includes: The computer, The method further includes a step of performing inference on the image to be inspected using the learned parameters generated in step (d).

[0024] No. 13 The embodiment is a computer readable program, and the computer reads the program to perform the following: 11 Aspects or No. 12 condition Dear An image processing method is performed. Effect of the Invention

[0025] According to the image processing device of the first aspect, the confidence level of the worker is calculated based on the work information recorded during the label selection work. Therefore, the confidence level of each labeled image can be easily calculated. Furthermore, by performing learning according to the confidence level calculated from the work information, the influence of fluctuations in the worker's label selection on learning can be easily reduced. Also, learning is done according to confidence level, which allows you to adjust the impact on learning according to confidence level. In addition, the learning rate is changed based on the confidence level, which allows the degree of influence on learning to be adjusted by adjusting the progress of learning according to the confidence level.

[0026] According to the image processing device of the second aspect, the degree of confidence can be calculated based on the time required to select a label.

[0027] According to the image processing device of the third aspect, the difference between the label selection time and the average value may depend on the hesitation of the operator. Therefore, by using the difference between the label selection time and the average value as a feature related to the hesitation, the confidence level can be appropriately calculated.

[0028] According to the image processing device of the fourth aspect, the difference between the overall average value and the partial average value may depend on the fatigue of the worker. Therefore, by using the difference between the overall average value and the partial average value as a feature amount related to fatigue, the confidence level can be appropriately calculated.

[0029] According to the image processing device of the fifth aspect, when a worker performs an enlargement operation input, the enlargement input is recorded as work information, whereby the confidence level can be calculated based on the enlargement operation input.

[0032] No. 6According to the image processing device of the aspect, it is possible to restrict images whose confidence level does not satisfy a predetermined standard from being used as training data.

[0033] No. 7 According to the image processing device of the aspect, it is possible to accept correction of the label for an image whose confidence level does not satisfy a predetermined standard. [Brief description of the drawings]

[0034] [Figure 1] 1 is a schematic diagram showing a visual inspection apparatus 100 according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing device 20. [Diagram 3] 2 is a block diagram showing an example of a functional configuration of an information processing device 20. FIG. [Figure 4] FIG. 13 is a diagram showing an example of a labeling window. [Diagram 5] FIG. 13 is a diagram conceptually illustrating how a label selection task is divided into a plurality of sections from start to finish. [Figure 6] FIG. 11 is a block diagram showing a functional configuration of an information processing device 20A according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Note that the components described in the embodiment are merely examples, and are not intended to limit the scope of the present invention. In the drawings, the dimensions and numbers of each part may be exaggerated or simplified as necessary for ease of understanding.

[0036] <1. First embodiment>

[0037] 1 is a schematic diagram showing a visual inspection apparatus 100 according to a first embodiment. The visual inspection apparatus 100 is an apparatus for inspecting the visual appearance of a tablet 9, which is an object. Note that the object is not limited to the tablet 9.

[0038] The appearance inspection apparatus 100 includes a camera 10 (imaging unit) and an information processing device 20. The camera 10 is electrically connected to the information processing device 20. The camera 10 includes an image sensor. The camera 10 outputs an image signal corresponding to the intensity of light incident on the image sensor to the information processing device 20. The camera 10 may capture an image of a tablet 9 stopped at a predetermined position, or may capture an image of a tablet 9 moving in a predetermined direction by a conveying mechanism such as a belt conveyor. The appearance inspection apparatus 100 may include one camera 10 or multiple cameras 10.

[0039] The information processing device 20 inspects the tablet 9 based on an image of the tablet 9 captured by the camera 10 (hereinafter referred to as a "tablet image"). The information processing device 20 is also capable of generating training data from the object image and performing machine learning using the generated training data. The information processing device 20 uses learned parameters (weights) generated by machine learning for inspecting the tablet 9. The information processing device 20 is an example of an image processing device.

[0040] 2 is a diagram showing an example of the hardware configuration of the information processing device 20. The information processing device 20 has a configuration as a computer. Specifically, the information processing device 20 has a processor 21, a RAM 23, a storage unit 25, a communication I / F 27, a device I / F 28, and a bus 29. The processor 21, the RAM 23, the storage unit 25, the communication I / F 27, and the device I / F 28 are electrically connected via the bus 29.

[0041] The processor 21 includes, for example, a CPU or a GPU. The RAM 23 and the storage unit 25 are storage media capable of reading and writing information. The RAM 23 includes, for example, a DRAM, an SRAM, or an SDRAM. The storage unit 25 includes, for example, a fixed drive such as a hard disk drive or a solid state drive. The storage unit 25 may include a removable medium (such as an optical disk, a magnetic disk, or a semiconductor memory) that has portability, and a media drive that reads and writes from and to the removable medium. The storage unit 25 stores a program P. The processor 21 executes the program P using the RAM 23 as a working area to realize various functions.

[0042] The communication I / F 27 is an interface for connecting the information processing device 20 to a network such as the Internet. The program P may be provided or distributed to the information processing device 20 through the network.

[0043] The device I / F 28 is an interface for electrically connecting the camera 10, the input unit 31, and the display unit 33 to the information processing device 20. The input unit 31 includes an input device such as a keyboard or a mouse. The display unit 33 displays display information such as characters and figures. The display unit 33 includes, for example, a liquid crystal display. The display unit 33 may include a touch panel. In this case, the display unit 33 can function as an input unit.

[0044] 3 is a block diagram showing an example of a functional configuration of the information processing device 20. The information processing device 20 includes a label assignment unit 41, a work status recording unit 43, a confidence level calculation unit 45, a learning unit 47, and an inference unit 49. The label assignment unit 41, the work status recording unit 43, the confidence level calculation unit 45, the learning unit 47, and the inference unit 49 are functions realized in the form of software by the processor 21 executing a program P.

[0045] The labeling unit 41 assigns a label to the tablet image serving as training data based on the input of the worker. The input of the worker includes a selection input for selecting one from a plurality of types of labels. The labeling unit 41 stores the tablet image 90 to which the label has been assigned (labeled image) in the storage unit 25. The input of the worker also includes an enlargement input for enlarging the image.

[0046] Here, the training data is data including a pair of "input data" and "label." The input data used in the information processing device 20 is a tablet image 90. The labels include "OK," "NG," "Maybe OK," or "Maybe NG." "OK" is a label indicating that the tablet image 90 has no defects, i.e., is a good product. "NG" is a label indicating that the tablet image 90 has defects, i.e., is a defective product. "Maybe OK" is a label indicating that there is a relatively high possibility that the tablet image 90 is a good product. "Maybe NG" is a label indicating that there is a relatively high possibility that the tablet image 90 is a defective product. The labels "Maybe OK" and "Maybe NG" are not required.

[0047] The work status recording unit 43 records the work status of the worker based on the input of the worker. That is, the work status recording unit 43 records information related to the selection input and information related to the enlarged input as work information. The work status recording unit 43 saves the work information in the storage unit 25.

[0048] The confidence level calculation unit 45 calculates the confidence level for the worker's input based on the work information recorded by the work status recording unit 43. The confidence level is numerical information indicating the degree of confidence of the worker in the act of selecting a label for the tablet image 90. The confidence level calculation unit 45 calculates the confidence level for each tablet image 90 to which a label has been added.

[0049] The learning unit 47 performs supervised machine learning using an inference model (inference program) with the tablet image 90 (labeled image) to which the label is added by the labeling unit 41 as training data. The learning unit 47 performs learning according to the confidence level calculated by the confidence level calculation unit 45. The learning according to the confidence level will be described later in detail. The learning unit 47 stores the learned parameters (weights) obtained by learning in the storage unit 25.

[0050] The inference unit 49 performs inference on the tablet image 90 to be inspected, using the learned parameters (weights) generated by the learning unit 47. That is, the inference unit 49 performs inference using a learned model (an inference program incorporating the learned parameters). In this example, the inference unit 49 judges whether the tablet image 90 to be inspected is "OK" or "NG".

[0051] <Label selection work> FIG. 4 is a diagram showing an example of a labeling window. The label selection window W1 shown in FIG. 4 is display information displayed on the screen of the display unit 33 by the labeling unit 41. The label selection window W1 is a GUI prepared for an operator to select a label to be applied to each tablet image 90 serving as teacher data (hereinafter referred to as a "label selection operation"). The label selection window W1 has an image display unit 51, a label selection unit 53, an enlargement operation unit 55, and an image selection unit 57.

[0052] The image display unit 51 is an area for displaying a tablet image 90 to which a label is to be applied. In the example shown in FIG. 4, the image display unit 51 can simultaneously display three tablet images 90. Of the three tablet images 90, the image displayed in the center is the processing target. That is, when the worker operates the label selection unit 53 or the enlargement operation unit 55, the label application unit 41 performs processing on the tablet image 90 to be processed according to the operation content. As shown in FIG. 4, the tablet image 90 to be processed may be displayed larger than the other tablet images 90.

[0053] The label selection unit 53 is an area that displays buttons for the worker to select a label. The label selection unit 53 includes label selection buttons 531, 532, 533, and 534. The label selection buttons 531 to 534 correspond to "OK", "NG", "Maybe OK", and "Maybe NG", respectively. That is, when the worker operates (for example, clicks) any of the label selection buttons 531 to 534, the label assignment unit 41 assigns a label corresponding to the operated selection button to the tablet image 90 to be processed. The input of the worker operating the label selection buttons 531 to 534 is an example of an input for selecting a label. The label assignment unit 41 saves information on the labels assigned to each tablet image 90 in the storage unit 25.

[0054] The enlargement operation section 55 is an area that displays a button for performing an operation to enlarge the display magnification of the tablet image 90 displayed on the image display section 51. The enlargement operation section 55 includes an enlargement button 551 and a reset button 552. When the worker operates the enlargement button 551, the labeling section 41 increases the display magnification of the tablet image 90 displayed on the image display section 51. When the worker operates the reset button 552, the labeling section 41 sets the display magnification of the tablet image 90 to a predetermined size.

[0055] With the tablet image 90 enlarged, the worker performs a predetermined operation input to change the portion to be displayed. Then, the labeling unit 41 changes the displayed portion of the tablet image 90 in response to the operation input. This allows the worker to enlarge and visually confirm any portion of the tablet image 90. The input in which the worker operates the enlarge button 551 and the reset button 552 is an example of the enlargement input.

[0056] The image selection unit 57 is an area that displays a button for selecting an image. The image selection unit 57 includes a previous image selection button 571 and a next image selection button 572. When the worker operates the previous image selection button 571, the labeling unit 41 changes the tablet image 90 to be processed that is currently displayed on the image display unit 51 to the previous tablet image 90. When the worker operates the next image selection button 572, the labeling unit 41 changes the tablet image 90 to be processed that is currently displayed on the image display unit 51 to the next tablet image 90. In this way, the image display unit 51 can display a group of tablet images 90 that are prepared in advance in order. The order in which the group of tablet images 90 are displayed can be appropriately set according to, for example, the acquisition date and time and file name of each tablet image 90. The image selection unit 57 may have an operation unit that allows the worker to arbitrarily select a specific tablet image 90 from among the multiple tablet images 90.

[0057] The work status recording unit 43 records the operation contents performed by the worker using the label selection window W1. For example, the work status recording unit 43 records the time when the worker operates each button of the label selection unit 53, the enlargement operation unit 55, or the image selection unit 57 as work information.

[0058] In order to calculate the degree of confidence, the confidence degree calculation unit 45 acquires a feature amount from the work information recorded by the work status recording unit 43. The confidence degree calculation unit 45 acquires, for example, a "label selection time" as the feature amount. The "label selection time" is the time required for the worker to select a label for one tablet image 90. For example, in the label selection window W1 shown in FIG. 4, the time from the moment one tablet image 90 is selected as a processing target via the image selection unit 57 to the moment a label for the tablet image 90 is specified via the label selection unit 53 may be set as the "label selection time."

[0059] When an operator selects a label, the operator cannot always select the label with 100% confidence every time. For example, when a defect in an image is not clear and it is difficult to tell whether or not the image contains a defect, the operator may enlarge the image or check it carefully. In this way, the operator may become hesitant and it may take a long time to select a label. In addition, if the labeling task takes a long time or the operator's physical condition, etc., the operator may become tired. Indecision and fatigue may affect the operation input in the selection task. The label selection time may be a value that reflects the operator's indecision and fatigue.

[0060] FIG. 5 is a conceptual diagram showing how the label selection task is divided into a plurality of intervals from start to finish. In FIG. 5, the horizontal axis indicates time. In the label selection task shown in FIG. 5, the total work time from the start to the end of the task is "T". In addition, in the label selection task, it is assumed that the worker has assigned labels to N tablet images 90. As shown in FIG. 5, the confidence calculation unit 45 calculates the "overall average value" and the "interval average value" as the average value of the label selection time.

[0061] The "overall average value" is the average of all label selection times in the entire label selection work performed by the worker. For example, in the case of the label selection work shown in FIG. 5, the confidence level calculation unit 45 calculates the average value of each label selection time for N tablet images 90 as the "overall average value". Note that the confidence level calculation unit 45 may calculate the value (=T / N) obtained by dividing the total work time "T" by the total number N of tablet images 90 as the overall average value.

[0062] The "interval average value" is the average (partial average value) of the label selection time in each interval when the total work time T for N tablet images 90 is divided into multiple intervals. The example shown in FIG. 5 shows each interval (first interval, second interval, third interval, ...) in which the total work time T is divided into a certain time unit (for example, 1 hour unit). The confidence degree calculation unit 45 may divide the total work time T into units of a number of tablet images 90 (for example, 100 sheets). The confidence degree calculation unit 45 calculates the average of the label selection time measured in each interval as the "interval average value" for each interval. In addition, when the number of tablet images 90 to which a worker has given a label in a certain interval is n and the work time in that interval is t, the value obtained by dividing the work time t by the number n (= t / n) may be used as the "interval average value".

[0063] The confidence level calculation unit 45 calculates a feature amount related to "hesitation" (hereinafter referred to as "hesitation feature amount") and a feature amount related to "fatigue" (hereinafter referred to as "fatigue feature amount") for each tablet image 90 based on the "label selection time", the "overall average value", and the "section average value". Specifically, the confidence level calculation unit 45 calculates, as the hesitation feature amount, the difference between the label selection time and the section average value of the section to which the label selection time belongs. In addition, the confidence level calculation unit 45 calculates, as the fatigue feature amount, the difference between the overall average value and the section average value.

[0064] When the work time is long, the worker becomes tired, and the section average value tends to become long. In other words, the section average value in the latter half of the work tends to be longer than the section average value in the first half of the work. Therefore, by calculating the difference between the section average value and the overall average value, the fatigue of the worker in each section can be quantified. In addition, by setting the hesitation feature to the difference between the label selection time and the section average value, the component of the worker's "fatigue" that depends on time can be removed, and the component that depends on "hesitation" can be extracted. Note that the confidence calculation unit 45 may calculate the difference between the label selection time and the overall average value as the hesitation feature.

[0065] The confidence level calculation unit 45 calculates the confidence level for each labeled image based on the hesitation feature amount and the fatigue feature amount. The confidence level calculation unit 45 may calculate the confidence level based on a function that uses the hesitation feature amount and the fatigue feature amount as variables. As an example, the confidence level calculation unit 45 may set the value obtained by adding the hesitation feature amount and the fatigue feature amount as the confidence level. In this case, the confidence level calculation unit 45 may weight the hesitation feature amount and the fatigue feature amount and add them together.

[0066] The confidence level calculation unit 45 may calculate the confidence level of an image labeled with "Maybe OK" or "Maybe NG" so as to be relatively lower than that of an image labeled with "OK" or "NG." This makes it possible to relatively lower the confidence level of an image labeled with "Maybe OK" or "Maybe NG" selected by the worker.

[0067] When the operator performs an operation to enlarge the tablet image 90 via the enlargement operation unit 55, the confidence degree calculation unit 45 may calculate the confidence degree according to the contents of the enlargement operation (for example, the enlargement rate). When it is difficult to visually recognize defects in the tablet image 90, it is assumed that the operator enlarges the image. Therefore, the greater the enlargement rate of the tablet image 90, the smaller the confidence degree calculated by the confidence degree calculation unit 45, so that an appropriate confidence degree can be obtained.

[0068] <Supervised Machine Learning>

[0069] The learning unit 47 learns the inference model while adjusting the degree of influence on learning, using the confidence degree calculated by the confidence degree calculation unit 45. The learning of the inference model will be described below, but the learning method is not limited to the following content.

[0070] The learning unit 47 performs learning using a predetermined inference model with the labeled image as training data. As the inference model, for example, a logistic regression model can be used. As described above, the learning unit 47 performs learning according to the confidence level. Specifically, the learning unit 47 changes the learning rate when updating the weight according to the confidence level.

[0071] Learning when using a logistic regression model as an inference model is explained using formulas. If the input feature vector is x, the model output is y, and the model weight matrix is ​​w, the logistic regression model is expressed by the following formula.

[0072]

number

[0073] The learning unit 47 normalizes the confidence calculated by the confidence calculation unit 45 so that it falls within the value range from 0 to 1. Then, the learning unit 47 multiplies the normalized confidence by the learning rate coefficient at the time of weight update. The teaching class is u, the learning rate coefficient is η, and the normalized confidence is α. Then, the weight update formula is expressed by the following formula.

[0074]

number

[0075] By optimizing based on the above formula (2), the learning unit 47 can perform learning according to the confidence level of each labeled image. For example, in the case of a labeled image with a standardized confidence level of "0.1", the weight update amount is 1 / 10 compared to the case of a labeled image with a standardized confidence level of "1". In this way, the influence of a labeled image with a low confidence level on learning is smaller than that of a labeled image with a high confidence level. By changing the learning rate according to the confidence level, the progress of learning is adjusted, and thus the influence on learning can be adjusted. The learning unit 47 stores the weights updated by learning in the storage unit 25 as learned parameters (see FIG. 3).

[0076] When the confidence level of a labeled image does not satisfy a predetermined standard, the learning unit 47 may restrict the use of the labeled image as training data. Specifically, "restricting the use as training data" is not limited to "not using as training data" and may include, for example, requesting confirmation or permission for use as training data.

[0077] The criterion for restricting the use of the labeled image as training data may be that the confidence level is less than a predetermined threshold value, or that the confidence level is an outlier. For example, the learning unit 47 may determine a value that deviates from 1σ point (a value that is outside the range of the average confidence level ±σ) as an outlier in the distribution of confidence levels. The learning unit 47 may also determine the outlier based on various tests such as a Q test or a Smirnoff test. The learning unit 47 restricts the use of the labeled image as training data according to the confidence level of the labeled image, so that a labeled image having an appropriate confidence level can be used for learning.

[0078] When the confidence level of a labeled image does not satisfy a predetermined standard, the label assignment unit 41 may accept a change of the label assigned to the labeled image. For example, the label assignment unit 41 causes the display unit 33 to display a label selection window W1 shown in FIG. 4, and causes the image display unit 51 to display a labeled image whose confidence level does not satisfy the standard. Then, the label assignment unit 41 may correct the label based on an operation of the operator on the label selection unit 53. This allows the operator to be given an opportunity to correct the label, and therefore makes it possible to generate training data to which accurate labels are assigned.

[0079] As described above, according to the information processing device 20, when the worker performs the label selection work, the work situation is simultaneously recorded as work information. Therefore, together with the labeled images serving as teacher data, the confidence level representing the confidence of the worker can be easily obtained based on the work information. Using the confidence level, a trained inference model can be created in which the degree of influence on learning of labeled images with large fluctuations in label selection is adjusted. In other words, by increasing the influence on learning of labeled images with high confidence levels and conversely decreasing the influence on learning of data with low confidence levels, the influence of fluctuations in label selection by the worker on learning can be easily reduced. This makes it possible to create a trained inference model capable of detecting defects with high accuracy.

[0080] In addition, by using the confidence level, it is possible to appropriately restrict the use of labeled images as training data and extract labeled images that require label correction.

[0081] <2. Second embodiment> Next, a second embodiment will be described. In the following description, elements having the same functions as elements already described will be given the same reference numerals or reference numerals with an additional alphabetical character, and detailed description thereof may be omitted.

[0082] In the first embodiment, the confidence level is calculated by calculation. However, the confidence level may be calculated using an inference model that estimates the confidence level (hereinafter, referred to as a "confidence level inference model"). As the confidence level inference model, for example, a logistic regression model may be used. Note that as the confidence level inference model, a model that outputs a probability value for each class, such as a support vector machine (SVM), may be used.

[0083] FIG. 6 is a block diagram showing a functional configuration of an information processing device 20A according to a second embodiment. A confidence level learning unit 61 shown in FIG. 6 uses confidence level learning data to learn a confidence level inference model, thereby generating a confidence level learned parameter. The confidence level inference model uses work information that records the work status of label selection as input data, and outputs a confidence level for the input data. For this reason, the confidence level learning data may have a pair of work information as input data and a confidence level as a correct label. The work information recorded by the work status recording unit 43 may be used as the work information of the input data.

[0084] Moreover, the confidence level learning unit 61 assigns a label of "confident" indicating a high level of confidence to the tablet image 90 to which the label assignment unit 41 has assigned "OK" or "NG". Moreover, the confidence level learning unit 61 assigns a label of "not confident" indicating a low level of confidence to the tablet image 90 to which the label assignment unit 41 has assigned "Maybe OK" or "Maybe NG". Thereby, confidence level learning data having a pair of work information and a label regarding the confidence level ("confident" or "not confident") is prepared for each tablet image 90.

[0085] The confidence level learning unit 61 may obtain a feature amount from the work information as input data. The feature amount includes, for example, a "label selection time", a "total average value", or a "section average value". The feature amount may also include a "total median", a "total standard deviation", a "section median value", or a "section standard deviation". The "total median value" is the median value of all label selection times obtained by the label selection work of the worker. The "total standard deviation" is the standard deviation of all label selection times. The "section median value" is the median value of multiple label selection times included in each section when the total work time is divided into multiple sections. The "section standard deviation" is the standard deviation of multiple label selection times included in each section. The confidence level learning unit 61 may obtain, as the feature amount, a "difference between the label selection time and the section average value" and a "difference between the section average value and the total recovery average value". The feature amount may also include a "magnification rate".

[0086] The confidence level learning unit 61 learns the confidence level inference model using confidence level learning data having pairs of task information (feature amount) and a label related to confidence level. In detail, the weights are updated so that the output when the feature amount obtained from the task information is input to the confidence level inference model matches the label ("confident" or "unconfident") indicating the confidence level corresponding to the input. The confidence level learning unit 61 stores the weights updated by learning in the storage unit 25 as confidence level learned parameters.

[0087] The confidence level calculation unit 45A calculates the confidence level by using the confidence level inference model used for learning and the confidence level learned parameters generated by the confidence level learning unit 61. The confidence level calculation unit 45A may set the output value of “confident” output by the confidence level inference model as the confidence level.

[0088] The learning unit 47 (see FIG. 3) performs learning based on the confidence level calculated by the confidence level calculation unit 45A. Note that, similarly to the first embodiment, the learning unit 47 may appropriately restrict the use of labeled images as teacher data according to the confidence level calculated by the confidence level calculation unit 45A. Furthermore, the label assignment unit 41 may extract labeled images for which label correction is to be performed based on the confidence level calculated by the confidence level calculation unit 45A.

[0089] Although the present invention has been described in detail, the above description is merely illustrative in all respects, and the present invention is not limited thereto. It is understood that countless variations not illustrated can be envisioned without departing from the scope of the present invention. The configurations described in the above embodiments and variations can be appropriately combined or omitted as long as they are not mutually contradictory. [Explanation of symbols]

[0090] 100 Visual inspection equipment 10. Camera 20 Information processing device (image processing device) 20A Information processing equipment 21 Processors 31 Input section 41 Labeling section 43 Work Status Recording Section 45,45A Confidence Calculator 47 Learning Department 49 Reasoning part 51 Image display unit 53 Label Selection Section 55 Magnification operation section 57 Image Selection Section 61 Self-Confidence Learning Department 9 tablets 90 Tablet Images P Program W1 Label selection window

Claims

1. An image processing device, a label assignment unit that assigns a label to an image serving as training data based on an input by an operator, the input including a selection input for selecting one of a plurality of types of labels; a work status recording unit that records the work status of the worker based on an input from the worker; a confidence level calculation unit that calculates a confidence level for the input of the worker based on the work information recorded by the work status recording unit; a learning unit that performs learning according to the degree of confidence calculated by the degree of confidence calculation unit for the image, using the image to which the label is assigned by the label assignment unit as training data; Equipped with The learning unit changes a learning rate used when updating weights in the learning, based on the degree of confidence.

2. 2. The image processing device according to claim 1, The work information includes a label selection time, which is a time required for the worker to select the label.

3. 3. The image processing device according to claim 2, The task information includes an average value of the label selection time for each of the plurality of images, The confidence level calculation unit calculates the confidence level based on a difference between the label selection time and the average value.

4. 4. The image processing device according to claim 2, The work information includes: an overall average value that is an average of the label selection times for the first number of images; a partial average value that is an average of the label selection times for a portion of the first number of the images; Including, The confidence level calculation unit calculates the confidence level based on a difference between the overall average value and the partial average value.

5. 5. The image processing device according to claim 1, the input by the worker includes a magnification input for magnifying the image; The image processing device, wherein the work information includes information regarding the enlarged input.

6. 6. The image processing device according to claim 1, The image processing device, wherein the learning unit restricts use of the image as the training data if the confidence level for the image to which the label has been assigned does not satisfy a predetermined standard.

7. 7. The image processing device according to claim 1, The image processing device, wherein the label assignment unit accepts a correction of the label when the degree of confidence for the image to which the label has been assigned does not satisfy a predetermined criterion.

8. 8. The image processing device according to claim 1, an inference unit that performs inference on an image to be inspected by using the trained parameters generated by the training unit; The image processing device further comprises:

9. A visual inspection apparatus comprising: An imaging unit that captures an image of an object; The image processing device according to claim 8 ; An appearance inspection device comprising:

10. A computer-executable program, 2. A program comprising: the learning unit of the image processing device according to claim 1, using the image to which the label has been assigned by the label assignment unit of the image processing device as teacher data, performing learning on the image according to the degree of confidence calculated by the confidence degree calculation unit of the image processing device, and including learned parameters generated by changing a learning rate for updating weights in the learning based on the degree of confidence, and causing the computer to perform inference on the image to be inspected using the learned parameters.

11. 1. An image processing method, comprising: (a) a step of labeling images serving as training data by a computer based on an input by an operator, the input including a selection input for selecting one of a plurality of labels; (b) the computer recording the work status of the worker based on the input of the worker; (c) calculating a degree of confidence in the input by the computer based on the work information recorded in the step (b); (d) performing learning by the computer using the image to which the label is assigned in the step (a) as training data in accordance with the degree of confidence calculated in the step (c) for the image; Including, The image processing method, wherein the step (d) includes a step of changing a learning rate used when updating weights in the learning, based on the degree of confidence, by the computer.

12. 12. The image processing method according to claim 11, (e) performing inference on an image to be inspected by the computer using the learned parameters generated by (d); The image processing method further comprises:

13. A computer-readable program, the computer being capable of reading the program and A program for causing an image processing method according to claim 11 to be executed.

Images