Information Processing Apparatus, Information Processing Method, and Program

Abstract

To make it possible to efficiently confirm whether or not learning data to be a base of a sorter is appropriate.SOLUTION: An information processing device can access a storage part which stores a plurality of feature values in association with learning images and storing the plurality of feature values in association with inspection images. The information processing device includes: a display part; a reference feature determination part for determining one or more reference features; a reference feature value range determination part for determining a reference feature value range in a space defined by a reference feature value; an image extraction part for extracting learning images and / or inspection images in which the reference feature values are included in the reference feature value range, and a display control part for displaying extraction learning images and / or extraction inspection images extracted by the image extraction part in the display part.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In manufacturing and the like, product inspection may be performed by analyzing an image obtained by imaging a product. In such an inspection, first, a classifier composed of a neural network or the like is made to perform machine learning using learning images prepared in advance by imaging, thereby optimizing the weighting for each feature amount of the learning images. Then, by inputting a desired inspection image to the machine-learned classifier, an attribute indicating the quality or the like of the inspection image is output.

[0003] The classifier generated by machine learning may make a misjudgment for various reasons. Examples of the causes of misjudgment include, for example, when the learning data is inappropriate or when the algorithm is inappropriate although the learning data is appropriate. In this regard, for example, Patent Document 1 describes an information processing system that displays a plot of feature amounts of teacher images and a plot of feature amounts of analysis images on an input / output device so that an operator can confirm the validity of teacher data and enable adjustment of the teacher database and re-learning of the classifier.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The operator may, for example, check the output results of the classifier for each inspection image to confirm whether the accuracy of the classifier is appropriate. Further, when it is found that the accuracy of the classifier is inappropriate, it is necessary to check whether the learning data is appropriate. For such a check, it is necessary to check each learning data to confirm whether the label assigned to the learning image is appropriate, etc. The burden of these confirmation operations can become greater as the number of learning images and inspection images increases.

[0006] Therefore, in one aspect, the present invention has been made in view of such circumstances, and an object thereof is to provide a technique that enables efficient confirmation of whether the learning data serving as the basis of the classifier is appropriate.

Means for Solving the Problems

[0007] In order to solve the above-described problems, the present invention employs the following configuration.

[0008] That is, an information processing apparatus according to one aspect of the present disclosure is an information processing apparatus capable of accessing a storage unit that stores a plurality of feature amounts indicating each of a plurality of features of a learning image in association with the learning image used for learning a classifier that determines an attribute of an image, and stores a plurality of feature amounts indicating each of a plurality of features of the inspection image in association with the inspection image that is an object of classification by the classifier, the information processing apparatus including a display unit, a reference feature determination unit that determines at least one reference feature that is a reference feature from among the plurality of features, a reference feature amount range determination unit that determines a reference feature amount range that is a range in a space defined by at least one reference feature amount that is a feature amount indicating at least one reference feature, an image extraction unit that extracts a learning image and / or an inspection image in which at least one reference feature amount is included in the reference feature amount range from among the learning image and / or the inspection image, and a display control unit that causes the display unit to display the extracted learning image and / or the extracted inspection image that are the learning image and / or the inspection image extracted by the image extraction unit.

[0009] In the above configuration, at least one reference feature is determined from a plurality of features of the learning images used for learning the classifier, and a reference feature amount range in a space defined by at least one reference feature amount, which is a feature amount indicating the at least one reference feature, is determined. Then, among the learning images and / or the inspection images to be classified by the classifier, the learning images and / or the inspection images whose reference feature amounts are included in the reference feature amount range are extracted, and the extracted learning images and / or inspection images are displayed on the display unit. For this reason, the operator can efficiently confirm whether the learning data serving as the basis of the classifier is appropriate.

[0010] In the information processing apparatus according to the above aspect, the reference feature determination unit may determine at least one reference feature according to a selection by the operator. According to this aspect, since the operator can select the reference feature, the convenience of the operator is improved.

[0011] In the information processing apparatus according to the above aspect, the reference feature determination unit may determine at least one reference feature by a predetermined feature selection algorithm. According to this configuration, since the reference feature is determined by the predetermined feature selection algorithm, the convenience of the operator is improved.

[0012] In the information processing apparatus according to the above aspect, the reference feature amount range determination unit may determine a reference image that is a reference learning image or inspection image from the learning images and / or the inspection images, determine a threshold value for a distance defined by at least one reference feature amount from the reference image, and determine, as the reference feature amount range, a range in which the distance from the reference image is equal to or less than the threshold value. According to this configuration, since the reference image is determined from the learning images and / or the inspection images, and the range in which the distance defined by at least one reference feature amount from the reference image is equal to or less than the threshold value is determined as the reference feature amount range, it is possible to efficiently confirm the learning images and / or the inspection images that are relatively close to the reference image.

[0013] In the information processing apparatus according to the above aspect, the reference feature amount range determination unit may receive an operation for designating a region of the reference feature amount by the operator, and determine the designated region as the reference feature amount range. According to this configuration, since the operator can determine the reference feature amount range through an operation of designating a region of the reference feature amount, the convenience of the operator is improved.

[0014] In the information processing apparatus according to the above aspect, the display control unit may further cause the display unit to display information indicating an attribute given to the extracted learning image and / or information indicating an attribute determined by a classifier for the extracted inspection image. According to this configuration, the operator can confirm whether the information indicating the attribute given to the extracted learning image is appropriate, and whether the attribute determined by the classifier for the extracted inspection image is appropriate.

[0015] In the information processing apparatus according to the above aspect, the display control unit may further cause the display unit to display at least one reference feature amount associated with the extracted learning image in the storage unit and / or at least one reference feature amount associated with the extracted inspection image in the storage unit. According to this configuration, the operator can confirm at least one reference feature amount associated with the extracted learning image and / or at least one reference feature amount associated with the extracted inspection image.

[0016] In the information processing apparatus according to the above aspect, the display control unit may further cause the display unit to display a representative point of the learning image and / or a representative point of the inspection image in a reference feature amount space that is a space defined by at least one reference feature amount. According to this configuration, the operator can grasp the relationship of the distances defined by the reference feature amounts of the learning image and / or the inspection image in the reference feature amount space.

Advantages of the Invention

[0017] According to the present invention, it is possible to provide a technique that enables efficiently confirming whether learning data serving as a basis for a classifier is appropriate.

Brief Description of the Drawings

[0018]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0019] Preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each figure, those with the same reference numerals have the same or similar configurations. In the present disclosure, "A and / or B" includes any of the cases where only A is included, only B is included, and both A and B are included.

[0020] FIG. 1 is a schematic diagram showing an example of the configuration of the inspection system 1 according to the present embodiment. The inspection system 1 according to the present embodiment is a system for inspecting the object W by determining the attributes of the inspection image generated by imaging the object W using a predetermined classifier. The inspection system 1 includes, for example, an imaging device 10 and an information processing device 20.

[0021] The imaging device 10 includes an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary MOS), and images the object W with the image sensor based on a control signal from the information processing device 20, and outputs the image obtained by imaging to the information processing device 20. Specifically, in the learning process described later, the imaging device 10 outputs a learning image obtained by imaging the object W to the information processing device 20. Also, in the inspection process described later, the imaging device 10 outputs an inspection image obtained by imaging the object W to the information processing device 20.

[0022] The object W is not particularly limited, and it does not matter whether it is a final product or a component, nor whether it is a finished product or an intermediate product. The object W may be conveyed by a conveying device 100 as shown in FIG. 1, for example. The conveying device 100 is configured as a belt conveyor, for example, and conveys the object W in the direction indicated by the arrow T in response to a control signal supplied from the information processing device 20 or another controller (not shown).

[0023]

[0024] The information processing device 20 is composed of, for example, one or more computers. The information processing device 20 includes a classifier generated by machine learning of learning data based on a learning image obtained by imaging the object W. The information processing device 20 executes the inspection of the object W by determining the attributes of the inspection image obtained by imaging the object W using the classifier.FIG. 2 is a schematic diagram showing an example of the configuration of the information processing apparatus 20 according to the present embodiment. The information processing apparatus 20 includes, for example, a display unit 21, an operation unit 22, a storage unit 23, and a processing unit 24.

[0025] The display unit 21 may be any device as long as it can display videos, images, etc. For example, it may be a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. The display unit 21 displays a video corresponding to the video data supplied from the processing unit 24, an image corresponding to the image data, and the like.

[0026] The operation unit 22 may be any device as long as it can operate the information processing apparatus 20. For example, it may be a touch panel, a key button, or the like. The operator can input characters, numbers, symbols, etc. using the operation unit 22. When the operation unit 22 is operated by the operator, it generates a signal corresponding to the operation. Then, the generated signal is supplied to the processing unit 24 as an instruction from the operator.

[0027] The storage unit 23 is, for example, an auxiliary storage device such as a hard disk drive or a solid state drive, and stores various programs executed by the processing unit 24, learning data 23a, inspection data 23b, a classifier 23c, and the like. The various programs may be installed in the storage unit 23 using a known setup program or the like from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM (a medium that accumulates information such as programs recorded on a computer or other devices and machines by an electrical, magnetic, optical, mechanical, or chemical action so that the information can be read). Further, the storage unit 23 may temporarily store temporary data related to a predetermined process.

[0028] FIG. 3 is a diagram showing an example of the data structure of the learning data 23a according to the present embodiment. The learning data includes, for example, image data of a learning image, one or more feature amounts of the learning image, and a label of the learning image.

[0029] The image ID is identification information (ID) for identifying a learning image. The image data of the learning image is, for example, image data generated by imaging an object W for generating a learning image by the imaging device 10. The image data of the learning image may be configured to include parameters such as hue, saturation, and brightness associated with each pixel constituting the learning image.

[0030] The feature amount is information that quantitatively and qualitatively indicates a predetermined feature of an image. The feature amount may be, for example, a feature amount extracted from a learning image by a feature amount extraction unit 243 described later. In the example shown in FIG. 3, as an example, four feature amounts from the first feature amount to the fourth feature amount are shown. However, the type of feature amount (feature) of the learning data is not limited to four, and may be one, two, three, or five or more. The feature amount may be information included in the image data (parameters such as hue, saturation, and brightness associated with each pixel).

[0031] The label is information indicating the attribute of the learning image. The number and content of the labels can be arbitrarily set by the operator. However, in this embodiment, as an example, it may include "good product" indicating that the object W included in the image is a good product and "defective product" indicating that the object W included in the image is a defective product. The label may be, for example, input by the operator. Note that when the machine learning for generating the classifier 23c is executed as unsupervised learning, the learning data 23a may not include labels.

[0032] FIG. 4 is a diagram showing an example of the data structure of the inspection data 23b according to the present embodiment. The inspection data includes, for example, the image data of the inspection image, one or more feature amounts of the inspection image, and the label of the inspection image.

[0033] The image ID is identification information (ID) for identifying the inspection image. The image data of the inspection image is, for example, image data generated by imaging the object W to be inspected by the imaging device 10. The image data of the inspection image may be configured to include parameters such as hue, saturation, and brightness associated with each pixel constituting the inspection image.

[0034] The feature amount is information that quantitatively and qualitatively indicates a predetermined feature of the image. The feature amount may be, for example, a feature amount extracted from the inspection image by the feature amount extraction unit 243. In the example shown in FIG. 4, as an example, four feature amounts from the first feature amount to the fourth feature amount are shown. However, the types of feature amounts (features) of the inspection data are not limited to four, and may be one, two, three, or five or more according to the learning data. The feature amount may be information included in the image data (parameters such as hue, saturation, and brightness associated with each pixel).

[0035] The label is information indicating the attribute of the inspection image. The number and content of the labels can be arbitrarily set by the operator according to the labels of the learning images. However, in this embodiment, as an example, it may include "good product" indicating that the object W included in the image is a good product, and "defective product" indicating that the object W included in the image is a defective product. The label may be, for example, output by inputting the inspection image into the classifier 23c. Note that when the machine learning for generating the classifier 23c is performed as unsupervised learning, the label in the inspection data 23b may be the result of the correct / incorrect determination.

[0036] As described above, the storage unit 23 stores, for example, a classifier 23c for determining the attributes of an image. The classifier 23c is generated, for example, by machine learning using learning data 23a, and outputs the attributes of the image in response to the input of the feature amounts of the image. For example, when the classifier 23c is configured as a neural network including an input layer, an output layer, and at least one intermediate layer, the storage unit 23 stores the weighting coefficients between these layers. The classifier 23c outputs, for example, the attributes of the inspection image related to the inspection data 23b in response to the input of the inspection data 23b. The attributes of the image output by the classifier 23c can be arbitrarily set. In this embodiment, as an example, the attributes of the inspection image include "good product" and "defective product". Here, the attribute "good product" may be an attribute indicating that the object W included in the image is a good product. Also, the attribute "defective product" may be an attribute indicating that the object W included in the image is a defective product.

[0037] The processing unit 24 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and controls each component according to information processing. The processing unit 24 controls the operations of the display unit 21, etc., so that various processes of the information processing apparatus 20 are executed in an appropriate procedure based on the programs stored in the storage unit 23 and the operations of the operation unit 22, etc. The processing unit 24 executes processing based on the programs (operating system programs, driver programs, application programs, etc.) stored in the storage unit 23. Also, the processing unit 24 can execute a plurality of programs (application programs, etc.) in parallel.

[0038] The processing unit 24 includes an imaging control unit 241, a preprocessing unit 242, a feature amount extraction unit 243, a machine learning execution unit 244, a classification execution unit 245, a reference feature determination unit 246, a reference feature amount range determination unit 247, an image extraction unit 248, and a display control unit 249. Each of these units is a functional module realized by a program executed by a processor included in the processing unit 24. Alternatively, each of these units may be implemented in the information processing apparatus 20 as an independent integrated circuit, microprocessor, or firmware.

[0039] The imaging control unit 241 transmits a control signal to the imaging device 10 to control the imaging of the imaging device 10. Further, the imaging control unit 241 acquires the image generated by imaging from the imaging device 10 and stores it in the storage unit 23. The image may be, for example, a learning image generated by imaging the object W for learning image generation, an inspection image generated by imaging the object W to be inspected, or the like.

[0040] The preprocessing unit 242 executes predetermined preprocessing in the learning process of generating the classifier 23c by machine learning using the learning data and / or the inspection process of executing determination on the inspection data using the generated classifier 23c. For example, when supervised learning is performed as the learning process, the preprocessing unit 242 may assign a label to each learning image. The assignment of the label may be executed, for example, in response to an operation via the operation unit 22 by an operator. Further, for example, when unsupervised learning is executed as the learning process, the preprocessing unit 242 may remove an image other than a desired image such as a non-defective product image as noise. Further, for example, the preprocessing unit 242 may execute a process of removing unnecessary noise from the learning data or a process of filling in missing items in the learning data in the learning process. The preprocessing unit 242 may execute a process of removing unnecessary noise from the inspection data or a process of filling in missing items in the inspection data in the inspection process.

[0041] The feature extraction unit 243 extracts predetermined features from each image in the learning process and / or the inspection process. For example, in the learning process, the feature extraction unit 243 extracts predetermined features from the learning images. The features extracted from the learning images are stored in the storage unit 23 as part of the learning data, for example. For example, in the inspection process, the feature extraction unit 243 extracts predetermined features from the inspection images. The features extracted from the inspection images are stored in the storage unit 23 as part of the inspection data, for example. The number of features extracted by the feature extraction unit 243 from each image may be one or two or more. Also, the types of features are not particularly limited. For example, in addition to parameters such as hue, saturation, and brightness associated with each pixel constituting the learning image, they may be average frequency, entropy, average luminance, etc. obtained from the parameters. Alternatively, the features can include, for example, the color of the object, the edge information of the object, the position of the object, and the like. The edge information may include the position, shape, length, etc. of the edge. The position of the object includes not only the position of the object itself but also the position of a part of the object.

[0042] The feature extraction unit 243 may extract the feature (secondary feature) generated based on at least one other feature (primary feature) in addition to the feature directly generated from the image, that is, by generating the feature (secondary feature). In other words, the feature extraction unit 243 can perform "feature compression" by generating one secondary feature from a plurality of primary features.

[0043] The machine learning execution unit 244 generates the classifier 23c by machine learning using the learning data stored in the storage unit 23. The method of machine learning is not particularly limited. For example, it may be so-called deep learning in which the initial values of the parameters of the classifier 23c are randomly determined and the parameters are adjusted while feeding back the error of image recognition output from the classifier 23c, or a method such as a well-known genetic algorithm (GA) may be adopted.

[0044] The classification execution unit 245 performs an inspection on the object W by inputting inspection data related to the inspection image into the classifier 23c to output the attributes (such as "good product" and "defective product") of the inspection image. The classification execution unit 245 may store the output attributes of the inspection image in the storage unit 23 in association with the inspection data.

[0045] The reference feature determination unit 246 determines at least one reference feature from a plurality of features associated with the learning image and / or the inspection image. For example, the reference feature determination unit 246 determines at least one reference feature from a plurality of feature amounts stored in the storage unit 23 in association with the learning image as learning data and a plurality of feature amounts stored in the storage unit 23 in association with the inspection image as inspection data. Here, the reference feature is, for example, a feature for defining a reference feature amount space and a reference feature amount range described later. That is, it can be said that the reference feature is a feature that serves as a reference when considering the accuracy of the classifier 23c. The number of reference features is not particularly limited and may be one, two, or three or more. Specifically, the reference feature determination unit 246 determines at least one feature as a reference feature from a plurality of features included in each of the learning data and / or inspection data stored in the storage unit 23. The reference feature determination unit 246 may select a feature, for example, based on the selection of a desired feature by an operator. Alternatively, the reference feature determination unit 246 may select a feature, for example, using a predetermined feature selection algorithm. The predetermined feature selection algorithm is not particularly limited as long as it is a known algorithm for selecting features suitable for the model from a plurality of features, and may include, for example, algorithms such as the filter method, the wrapper method, and the embedded method. The filter method is also called univariate feature selection and is a method of statistically verifying the relationship between each feature amount and the objective and selecting the feature amount considered to be the most dominant. The wrapper method is also called iterative feature selection and is a method of combining a plurality of feature amounts to verify the prediction accuracy and searching for the combination that gives the highest accuracy. The embedded method is also called model-based feature selection and is a method of performing machine learning and feature selection simultaneously, and decision trees and the like are typical algorithms thereof.

[0046] The reference feature amount range determination unit 247 determines a reference feature amount range that is a predetermined range in the space defined by the reference feature amount. Here, the reference feature amount range may be, for example, a range for extracting learning images and / or inspection images.

[0047] For example, the reference feature amount range determination unit 247 may determine a reference image serving as a reference from learning images and inspection images, calculate the distance defined by the reference feature amounts between each other image and the reference image, and determine the reference feature amount range based on the distance. Specifically, the reference feature amount range determination unit 247 determines a threshold value for the distance (reference feature amount distance) defined by the reference feature amount from the reference image, and may determine, as the reference feature amount range, a range in which the reference feature amount distance from the reference image is equal to or less than the threshold value. Here, the calculation method of the distance can be arbitrarily set, and may include, for example, Euclidean distance, Mahalanobis distance, and Manhattan distance. Further, the threshold value may be the reference feature amount distance for an image whose reference feature amount distance from the reference image is a predetermined rank among each learning image and each inspection image. In this case, the reference feature amount range will include images among each learning image and each inspection image whose reference feature amount distance from the reference image reaches the predetermined rank. Alternatively, the threshold value may be a predetermined value. In this case, the reference feature amount range will include images among each learning image and each inspection image whose reference feature amount distance from the reference image reaches the predetermined value. Note that when the reference feature amount distance is defined as the Euclidean distance, the reference feature amount range can be represented as a circle in the reference feature amount space. When the reference feature amount distance is defined as the Mahalanobis distance, the reference feature amount range can be represented as an ellipse in the reference feature amount space. When the reference feature amount distance is defined as the Manhattan distance, the reference feature amount range can be represented as a rectangle in the reference feature amount space.

[0048] Alternatively, in determining the reference feature amount range, it is not always necessary to determine the reference image. For example, the reference feature amount range determination unit 247 may receive an operation of designating an arbitrary region in the reference feature amount space, which is a space defined by the reference feature amount, and determine the reference feature amount range based on the region. Specifically, the reference feature amount range determination unit 247 may receive the designation of an arbitrary region in the reference feature amount space drawn by an operation such as dragging by the operator, and then determine the region as the reference feature amount range. The shape of the region is not limited to a circle, and may include a rectangle, other polygons, approximate shapes of those shapes, and any irregular shape drawn by freehand or the like.

[0049] The image extraction unit 248 extracts a desired image from the learning image and / or the inspection image. For example, the image extraction unit 248 extracts an image in which the reference feature amount is included in the reference feature amount range from among the learning image and the inspection image. Specifically, for each image to be extracted (learning image and / or inspection image), the reference feature amount stored in association with the image in the storage unit 23 or the like is specified, and it is determined whether or not the specified reference feature amount is included in the reference feature amount range. When it is determined that the specified reference feature amount is included in the reference feature amount range, the image extraction unit 248 extracts the image. When it is determined that the specified reference feature amount is not included in the reference feature amount range, the image extraction unit 248 does not extract the image.

[0050] The display control unit 249 causes the display unit 21 to display various screens and information based on various display data. For example, the display control unit 249 generates display data based on various information included in the learning data and / or inspection data stored in the storage unit 23, and causes the display unit 21 to display various information included in the learning data and / or inspection data based on the display data. Specifically, the display control unit 249 causes the learning image and / or inspection image to be displayed on the display unit 21. The learning image and / or inspection image to be displayed may be, for example, the learning image and / or inspection image extracted by the image extraction unit 248, or the learning image and / or inspection image arbitrarily selected by the operator may be displayed on the display unit 21. Further, the display control unit 249 may cause the display unit 21 to display items (features, labels, etc.) of each learning data and / or each inspection data stored in the storage unit 23 or the like. Further, the display control unit 249 may cause the display unit 21 to display a feature list that is a list of features corresponding to each of the learning image and / or inspection image in the storage unit 23 or the like. Further, the display control unit 249 may cause the display unit 21 to display representative points of the learning image and / or inspection image in a space (feature space) defined by a predetermined feature.

[0051] FIG. 5 is a diagram showing an example of an operation flow of a learning process executed by the inspection system 1 according to the present embodiment.

[0052] (S101) First, a learning image is generated by imaging the learning object W using the imaging device 10. Specifically, for example, the imaging control unit 241 controls the imaging device 10 to image the learning object W in response to the operator operating the operation unit 22. Thereby, a learning image of the object W being imaged is generated. The number of learning images to be generated is not particularly limited and may be one or two or more.

[0053] In step S201, the object W may be pre - divided into non - defective and defective products by the operator, and then the non - defective object W and the defective object W may be imaged separately to generate learning images. In this case, for example, in step S202 and the like described later, it is possible to collectively assign labels of "non - defective" or "defective" to the collectively imaged learning images.

[0054] (S102) Next, labels are assigned to each learning image. For example, the operator displays each learning image on the display unit 21 through an operation via the operation unit 22, and then performs an operation on the operation unit 22 to assign a label of "non - defective" or "defective" to each learning image. Alternatively, the operator may visually determine whether the object W is non - defective or defective, and then perform an operation on the operation unit 22 to assign a label of "non - defective" or "defective" to each learning image corresponding to the object W. In response to this operation, the pre - processing unit 242 stores the assigned label in the storage unit 23 in association with the learning image. Thereby, the assigned label is stored in the storage unit 23 as part of the learning data.

[0055] (S103) Next, the feature extraction unit 243 extracts features from each learning image, and then stores the features in the storage unit 23 in association with the learning image. Thereby, the features are stored in the storage unit 23 as part of the learning data. The number of features extracted by the feature extraction unit 243 may be one or two or more. Also, the type of features is not particularly limited. For example, in addition to parameters such as hue, saturation, and brightness associated with each pixel constituting the learning image, it may be an average frequency, entropy, average luminance, etc. obtained from these parameters. Alternatively, the features may be, for example, the color of the object, the edge information of the object, the position of the object, etc. The edge information may include the position, shape, length, etc. of the edge. The position of the object includes not only the position of the object situation but also the position of a part of the object.

[0056] (S104) Next, the machine learning execution unit 244 generates a classifier 23c by machine learning using the learning data stored in the storage unit 23. The generated classifier 23c is stored in, for example, the storage unit 23. The method of machine learning is not particularly limited, and may be, for example, deep learning, a genetic algorithm (GA), or the like. With the above, the learning process ends.

[0057] FIG. 6 is a diagram showing an example of an operation flow of an inspection process executed by the inspection system 1 according to the present embodiment.

[0058] (S201) First, an inspection image is generated by imaging an object W to be inspected using the imaging device 10. Specifically, for example, the imaging control unit 241 controls the imaging device 10 to image the object W to be inspected in response to an operator operating the operation unit 22. Thereby, an inspection image in which the object W is imaged is generated. The number of inspection images to be generated is not particularly limited, and may be one or two or more.

[0059] (S202) Next, the feature amount extraction unit 243 extracts feature amounts from each inspection image, and then stores the feature amounts in the storage unit 23 in association with the inspection images. Thereby, the feature amounts are stored in the storage unit 23 as part of the inspection data. The feature amounts extracted by the feature amount extraction unit 243 may include at least the feature amounts extracted from the learning images in the above-described learning process. That is, the number of feature amounts extracted by the feature amount extraction unit 243 may be one, or may be two or more. Also, the type of feature amount is not particularly limited, and may be, for example, parameters such as hue, saturation, and brightness associated with each pixel constituting the inspection image, as well as the average frequency, entropy, average luminance, etc. obtained from the parameters. Alternatively, the feature amount may be, for example, the color of the object, the edge information of the object, the position of the object, etc. The edge information may include the position, shape, length, etc. of the edge. The position of the object includes not only the position of the object situation but also the position of a part of the object.

[0060] (S203) The classification execution unit 245 inputs inspection data related to the inspection image into the classifier 23c to output the attributes (such as "good product" and "defective product") of the inspection image, thereby performing an inspection on the object W. The classification execution unit 245 stores the output attributes of the inspection image in the storage unit 23 in association with the inspection data. Thus, the inspection process ends.

[0061] FIG. 7 is a diagram showing an example of the operation flow of the screen display process executed by the inspection system 1 according to the present embodiment.

[0062] (S301) First, the reference feature determination unit 246 determines at least one feature among the first to nth features as the reference feature.

[0063] The reference feature determination unit 246 may, for example, accept the selection of the reference feature by the operator and determine the reference feature according to the selection. The selection of the reference feature can be performed, for example, on the screen 1000 shown in FIG. 8. The screen 1000 is displayed on the display unit 21 by the display control unit 249 based on the learning data and inspection data stored in the storage unit 23, for example. The screen 1000 may include, for example, a feature amount list 1001 that is a list of each feature amount associated with the learning image and / or the inspection image respectively. In the feature amount list 1001, for the learning image and the inspection image, the image ID, type, label, and feature amount are shown. The type of the feature amount may be one or more. In FIG. 8, as an example, the first to fourth feature amounts are shown. The order of arrangement of each image in the feature amount list is not particularly limited, but in FIG. 8, as an example, the learning image and the inspection image are arranged in the order of the image ID. The feature amount list may be sortable for each item such as the image ID, type, label, and feature amount, and the display control unit 249 may rearrange each row in the order (ascending order or descending order, etc.) of the item when sorting is selected for any item.

[0064] The operator can select a desired reference feature, for example, by selecting a cell displayed as "first feature" in the first row included in the feature list. In response to the selection, the reference feature determination unit 246 determines the feature corresponding to the selected cell as the reference feature. In the feature list, when the reference feature is determined, the display control unit 249 may sort and display the feature list by the reference feature amount, which is the feature amount corresponding to the reference feature.

[0065] In step S301, the reference feature determination unit 246 may determine features, for example, by selecting features using a predetermined feature selection algorithm such as a filter method, a wrapper method, and an embedded method. The reference feature determination unit 246 may select features using the feature selection algorithm and determine the reference feature in response to an operation by the operator instructing the determination of the reference feature by the feature selection algorithm.

[0066] (S302) Next, the display control unit 249 causes the display unit 21 to display representative points of the learning image and / or the inspection image in the reference feature amount space in response to an operation by the operator via the operation unit 22. Here, the reference feature amount space is a space defined by at least one reference feature amount determined in step S301 described above. The display control unit 249 causes the display unit 21 to display, for example, the screen 2000 shown in FIG. 9. The screen 2000 includes a graph display 2001 of the reference feature amount space.

[0067] In FIG. 9, as an example, when the reference features determined by the reference feature determination unit 246 are the p-th feature and the q-th feature, that is, when the reference feature amounts are the p-th feature amount and the q-th feature amount, the graph display 2001 of the reference feature amount space is shown. In the graph display 2001 of the reference feature amount space shown in FIG. 9, the horizontal axis represents the p-th feature amount and the vertical axis represents the q-th feature amount.

[0068] As shown in FIG. 9, the graph display 2001 of the reference feature amount space may include plots (representative points) arranged at positions corresponding to the reference feature amounts of each image (learning image and / or inspection image). In FIG. 9, white circles indicate plots of learning images (good product learning images) labeled as good products, white triangles indicate plots of learning images (defective product learning images) labeled as defective products, black circles indicate plots of inspection images (good product inspection images) labeled as good products, and black triangles indicate plots of inspection images (defective product learning images) labeled as defective products. The display control unit 249 may be arbitrarily switchable for the presence or absence and mode of display of these plots according to, for example, an operation via the operation unit 22 by an operator. For example, in the graph display 2001 of the reference feature amount space, the display control unit 249 may display only the plots of the learning images (all or part thereof), may display only the plots of the inspection images (all or part thereof), or may display both the plots of the learning images (all or part thereof) and the plots of the inspection images (all or part thereof) according to an operation on the operation unit 22 or the like. Further, the display control unit 249 may display the label associated with each image (learning image or inspection image) and each feature amount (including feature amounts other than the reference feature amount) in a mode associated with each plot.

[0069] (S303) Next, the reference feature amount range determination unit 247 determines a reference feature amount range that is a predetermined range in the space defined by the reference feature amount.

[0070] The reference feature amount range may be determined according to operations on the feature amount list 1001 shown in FIG. 8. For example, when the operator selects a desired learning image or inspection image in the feature amount list 1001, the reference feature amount range determination unit 247 determines the selected learning image or inspection image as the reference image. The operator can select the image corresponding to the row as the reference image, for example, by selecting any row included in the feature amount list. In response to the selection, the reference feature amount range determination unit 247 determines the image corresponding to the selected row as the reference image. In the feature amount list 1001, it may be possible to simultaneously select the above-described reference feature and reference image by selecting a cell at the row corresponding to a specific image and the column corresponding to a specific feature amount. That is, for example, when the cell of the first feature amount "91" for the learning image with the image ID "L-0001" shown in FIG. 8 is selected, the reference feature determination unit 246 may determine the first feature (first feature amount) as the reference feature (reference feature amount), and the image extraction unit 248 may determine the learning image with the image ID "L-0001" as the reference image.

[0071] Then, after determining a threshold value for the distance (reference feature amount distance) defined by the reference feature amount from the reference image, the reference feature amount range determination unit 247 determines, as the reference feature amount range, the range in which the reference feature amount distance from the reference image is equal to or less than the threshold value. Here, the threshold value may be the reference feature amount distance for the image whose reference feature amount distance from the reference image is in a predetermined rank among each learning image and each inspection image. In this case, the reference feature amount range will include the images whose reference feature amount distance from the reference image is up to the predetermined rank among each learning image and each inspection image. Alternatively, the threshold value may be a predetermined value. In this case, the reference feature amount range will include the images whose reference feature amount distance from the reference image is up to the predetermined value among each learning image and each inspection image.

[0072] The reference feature amount range may be determined according to operations on the graph display 2001 of the reference feature amount space shown in FIG. 9. For example, when the operator selects a desired learning image or inspection image in the graph display 2001 of the reference feature amount space, the reference feature amount range determination unit 247 determines the selected learning image or inspection image as the reference image. The operator can select any plot shown in the graph display 2001 of the reference feature amount space as the reference image. In FIG. 9, as an example, a case is shown where an image (good product inspection image A) corresponding to a plot of a black circle surrounded by a substantially pentagonal object marked with reference sign A is selected as the reference image.

[0073] Then, after determining a threshold value for the distance (reference feature amount distance) defined by the reference feature amounts from the reference image, the reference feature amount range determination unit 247 determines, as the reference feature amount range, a range in which the reference feature amount distance from the reference image is equal to or less than the threshold value. The threshold value may be a predetermined value. In other words, the reference feature amount range includes images among each learning image and each inspection image whose reference feature amount distance from the reference image is up to a predetermined value. For example, in FIG. 9, the threshold value is a predetermined value d. That is, a circle M with a radius d centered on the plot of the good product inspection image A, which is the reference image, becomes the reference feature amount range. In this way, a range in which the distance (reference feature amount distance) from the reference image in the feature amount space is within a predetermined threshold value becomes the reference feature amount range. Note that the threshold value may be the reference feature amount distance for an image whose reference feature amount distance from the reference image is a predetermined rank among each learning image and each inspection image. Note that the reference feature amount range shown in FIG. 9 is an example when the reference feature amount distance is defined as the Euclidean distance. When the reference feature amount distance is defined as the Mahalanobis distance, the reference feature amount range can be represented as an ellipse in the reference feature amount space. Also, when the reference feature amount distance is defined as the Manhattan distance, the reference feature amount range can be represented as a rectangle in the reference feature amount space.

[0074] In determining the reference feature amount range, it is not always necessary to determine a reference image in the graph display 2001 of the reference feature amount space. For example, the reference feature amount range determination unit 247 may receive an operation of designating an arbitrary region in the reference feature amount space and determine the reference feature amount range based on the region. Specifically, the reference feature amount range determination unit 247 receives the designation of an arbitrary region in the reference feature amount space drawn by an operation such as dragging by the operator. Then, the reference feature amount range determination unit 247 may determine the region as the reference feature amount range. The shape of the region is not limited to a circle and may include a rectangle, other polygons, approximate shapes of those shapes, and any irregular shape drawn by freehand or the like.

[0075] (S304) Next, the image extraction unit 248 extracts an image in which the reference feature amount is included in the reference feature amount range determined in the above-described step S303 from each of the learning images and / or each of the inspection images. Specifically, for each image to be extracted (learning image and / or inspection image), the reference feature amount stored in the storage unit 23 in association with the image is specified, and it is determined whether or not the specified reference feature amount is included in the reference feature amount range determined in the above-described step S303. When it is determined that the specified reference feature amount is included in the reference feature amount range, the image extraction unit 248 extracts the image. When it is determined that the specified reference feature amount is not included in the reference feature amount range, the image extraction unit 248 does not extract the image.

[0076] In the graph display 2001 of the reference feature amount space in FIG. 9, as an example, there are shown a plot of a white circle surrounded by a substantially rectangular object labeled with symbol B, a plot of a black circle surrounded by a substantially rectangular object labeled with symbol C, and a plot of a black triangle surrounded by a substantially rectangular object labeled with symbol D, either on or inside the circumference of circle M. This indicates that for the reference feature amount range represented by circle M, the good learning image B, the good inspection image C, and the defective inspection image D have been extracted respectively. In other words, the distance between the position coordinates represented by the p-th feature amount and the q-th feature amount of each of the good learning image B, the good inspection image C, and the defective inspection image D, and the position coordinates represented by the p-th feature amount and the q-th feature amount of the good inspection image A which is the reference image, is within d.

[0077] (S305) Next, the display control unit 249 acquires image data from the storage unit 23 for the image (learning image and / or inspection image) extracted by the image extraction unit 248, and then causes the display unit 21 to display the image extracted based on the image data. FIG. 10 shows an example of a screen 3000 on which the image extracted by the image extraction unit 248 is displayed. As shown in FIG. 10, the screen 3000 includes a graph display 3001 of the reference feature amount space. The graph display 3001 may be substantially the same as the graph display 2001 shown in FIG. 9. Also, as shown in FIG. 10, the screen 3000 includes a display area 3002 for the reference image and a display area 3003 for the extracted images other than the reference image.

[0078] The display area 3002 for the reference image includes, for example, in addition to the image of the good inspection image A which is the reference image, the type, the image ID, the label attached to the image, and the reference feature amounts (the p-th feature amount and the q-th feature amount). Note that the display area 3002 may include feature amounts other than the reference feature amounts of the reference image (good inspection image A). Note that, as described above, when the reference feature amount range is determined without determining the reference image, the screen 3000 may not include the display area 3002 for the reference image.

[0079] In the display area 3003 of the reference image, for example, in addition to the images of the extracted good learning image B, good inspection image C, and defective inspection image D, types, image IDs, labels attached to the images, or labels determined by the classifier 23c, reference feature amounts (the p-th feature amount and the q-th feature amount) are included. Note that the display area 3003 may include feature amounts other than the reference feature amounts of the reference image (good inspection image A). As described above, when the reference feature amount range is determined without determining the reference image, the display area 3002 of the reference image may not be included in the screen 3000.

[0080] The reference feature amount range determination unit 247 may update the reference image, the reference feature amount range, etc. according to operations on the graph display 3001 of the reference feature amount space. For example, the reference feature amount range determination unit 247 may update the reference image by determining, as a new reference image, the image (learning image or inspection image) corresponding to the selected plot according to the selection of the plot in the graph display 3001. Also, for example, the reference feature amount range determination unit 247 may update the reference feature amount range by determining, as a new reference feature amount range, the designated area according to the operation of designating the area of the reference feature amount range in the graph display 3001. The display control unit 249 may update the display of the graph display 3001 of the reference feature amount space included in the screen 3000, the display areas 3002 and 3003 of the images, etc. according to the update when the reference image, the reference feature amount range, etc. are updated according to operations on the graph display 3001, etc.

[0081] With the configuration of the inspection system 1 according to the present embodiment as described above, the operator can easily confirm information such as the image, feature amount, and label of a desired image (learning image or inspection image). For example, in the example shown in FIG. 10, a defective part G1 has occurred in the product included in the image of the non-defective product inspection image A which is a reference image. Therefore, the non-defective product inspection image A is an image that should originally be determined as a "defective product" by the classifier 23c. Thus, when checking the reference feature amount range represented by the circle M in the graph display 3001 of the reference feature amount space, it can be confirmed that the good learning image B and the good learning image D are extracted as learning images with a short distance defined by the reference feature amounts from the non-defective product inspection image A (originally a "defective product" inspection image). Here, when checking the images of the good learning image B and the good learning image D respectively, although no particular problem is found in the product included in the image of the good learning image B, it can be seen that a defective part G2 has occurred in the product included in the image of the good learning image D. Therefore, it is found that the good learning image D was wrongly labeled as a "good product" when it should originally have been labeled as a "defective product". Thus, the operator can update the learning data by changing the label of the good learning image D to "defective product", and can generate the classifier 23c based on the updated new learning data. In this way, in the inspection system 1 according to the present embodiment, it is possible to easily confirm whether the learning data that is the basis of the classifier is appropriate.

[0082] The embodiments described above are merely illustrative of the present invention in every respect. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. Although the data appearing in this embodiment is described in natural language, more specifically, it is specified by a quasi-language, command, parameter, machine language, etc. recognizable by a computer.

Explanation of Reference Numerals

[0083] 1…Inspection system, 10…Imaging device 10…Information processing device, 21…Display unit, 22…Operation unit, 23…Memory unit, 23a…Learning data, 23b…Inspection data, 23c…Classifier, 24…Processing unit, 241…Imaging control unit, 242…Preprocessing unit, 243…Feature extraction unit, 244…Machine learning execution unit, 245…Classification execution unit, 246…Reference feature determination unit, 247…Reference feature amount range determination unit, 248…Image extraction unit, 249…Display control unit, W…Object, 100…Conveyor

Claims

1. An information processing apparatus capable of accessing a storage unit that stores a plurality of feature amounts indicating each of a plurality of features of a learning image corresponding to the learning image used for learning a classifier for determining an attribute of an image, and stores a plurality of feature amounts indicating each of the plurality of features of the inspection image that is an object of classification by the classifier, a display unit, a reference feature determination unit that determines at least one reference feature that is a reference feature from the plurality of features, a reference feature amount range determination unit that determines a reference feature amount range that is a range in a space defined by at least one reference feature amount that is a feature amount indicating the at least one reference feature, an image extraction unit that extracts a learning image and an inspection image among the learning image and the inspection image in which the at least one reference feature amount is included in the reference feature amount range, a display control unit that causes the display unit to display the extracted learning image and the inspection image that are the extracted learning image and the extracted inspection image, An information processing apparatus comprising:

2. The information processing apparatus according to claim 1, wherein the reference feature determination unit determines the at least one reference feature according to a selection by an operator.

3. The information processing apparatus according to claim 1 or 2, wherein the reference feature determination unit determines the at least one reference feature by a predetermined feature selection algorithm.

4. The reference feature amount range determination unit, determines a reference image that is a reference learning image or inspection image from the learning image and the inspection image, determines a threshold value for a distance defined by the at least one reference feature amount from the reference image, The information processing apparatus according to any one of claims 1 to 3, wherein a range in which the distance from the reference image is equal to or less than the threshold value is determined as the reference feature amount range.

5. The information processing apparatus according to any one of claims 1 to 3, wherein the reference feature amount range determination unit receives an operation of designating a region of the reference feature amount by an operator, and determines the designated region as the reference feature amount range.

6. The information processing apparatus according to any one of claims 1 to 5, wherein the display control unit further causes the display unit to display information indicating an attribute given to the extracted learning image and / or information indicating an attribute determined by the classifier for the extracted inspection image.

7. The display control unit further causes the display unit to display at least one reference feature amount associated with the extracted learning image in the storage unit and / or at least one reference feature amount associated with the extracted inspection image in the storage unit, and the information processing apparatus according to any one of claims 1 to 6.

8. The display control unit further causes the display unit to display a representative point of the learning image and / or a representative point of the inspection image in a reference feature amount space that is a space defined by the at least one reference feature amount, and the information processing apparatus according to any one of claims 1 to 7.

9. The display control unit further causes the display unit to display a list of at least some of the plurality of feature amounts of the plurality of feature amounts associated with the learning image and / or at least some of the plurality of feature amounts of the plurality of feature amounts associated with the inspection image, and the information processing apparatus according to any one of claims 1 to 8.

10. A computer capable of accessing a storage unit that stores a plurality of feature amounts indicating each of a plurality of features of a learning image associated with the learning image used for learning a classifier that determines an attribute of an image, and stores a plurality of feature amounts indicating each of the plurality of features of the inspection image that is the object of classification by the classifier, determining at least one reference feature that is a reference feature from the plurality of features; determining a reference feature amount range that is a range in a space defined by at least one reference feature amount that is a feature amount indicating the at least one reference feature; extracting, from the learning image and the inspection image, a learning image and an inspection image in which the at least one reference feature amount is included in the reference feature amount range; causing the display unit to display the extracted learning image and inspection image, which are the extracted learning image and the extracted inspection image; An information processing method for executing.

11. In a computer capable of accessing a storage unit that stores a plurality of feature amounts indicating each of a plurality of features of a learning image associated with the learning image used for learning a classifier that determines an attribute of an image, and stores a plurality of feature amounts indicating each of the plurality of features of the inspection image that is the object of classification by the classifier, determining at least one reference feature that is a reference feature from the plurality of features; Determining a reference feature amount range, which is a range in a space defined by at least one reference feature amount that is a feature amount indicating the at least one reference feature; Extracting, from among the learning image and the inspection image, the learning image and the inspection image in which the at least one reference feature amount is included in the reference feature amount range; Causing a display unit to display the extracted learning image and inspection image, which are the extracted learning image and the extracted inspection image; A program for causing the above to be executed.

Images