Method for collecting teacher data

Abstract

The present application provides a kind of teacher data collection method, the quality and quantity of the unqualified product data collected as teacher data can be quantitatively and properly evaluated, sufficient quality and quantity of teacher data can be used, classification model is learned with high accuracy, and the collection working hours of unqualified product data is minimized.The present application is a teacher data collection method for collecting unqualified product data as teacher data for learning classification model for classifying test objects G as normal product / abnormal product, collecting many unqualified product data (step 1 of figure 3), extracting multiple characteristic quantities from each of the many unqualified product data (step 2), calculating the state of each multiple characteristic quantity extracted for the many unqualified product data and Zk (step 3), calculating the logarithmic sum of multiple state and Zk as index value HEG (step 4), when index value HEG becomes given target value HEGTGT or more, the collection of unqualified product data is ended (step 5).

Description

Technical Field

[0001] This invention relates to a method for collecting teacher data used in learning, such as in a testing device that utilizes machine learning functions employing neural networks. Background Technology

[0002] In recent years, the development of automated inspection technologies has been advancing, utilizing inspection devices with neural network-based machine learning capabilities to determine whether various industrial products and components are normal (qualified) or abnormal (unqualified). In such inspection devices, machine learning is performed by reading image data of the appearance of numerous inspected objects classified as qualified and unqualified as teacher data, allowing the classification model to learn. Using this learned classification model, the inspection device can then classify new inspected objects captured by a camera as qualified or unqualified.

[0003] As mentioned above, in machine learning classification models, image data of both qualified and unqualified products are used as teacher data. Therefore, to improve inspection accuracy (classification accuracy), a large number of image data points are needed for either qualified or unqualified products. However, in manufacturing sites for industrial products, production is generally carried out in a way that minimizes the generation of unqualified products. Therefore, while the number of qualified products is large, the number of unqualified products is very small. Consequently, the collection of image data for qualified products (hereinafter referred to as "qualified product data") is relatively easy, while the collection of image data for unqualified products (hereinafter referred to as "unqualified product data") tends to be more difficult.

[0004] Furthermore, in such an inspection device, if the image data used as the inspection object contains data patterns that the classification model has not learned, it is difficult to classify it properly. Therefore, the teacher data used for learning the classification model requires high-quality data that encompasses as many data patterns as possible. In other words, if the collection of teacher data is insufficient, or if poor-quality teacher data with uncaptured data patterns is provided, it is impossible to build a high-accuracy classification model. As mentioned above, to ensure the accuracy of machine learning-based classification models, especially for substandard teacher data, in addition to sufficient quantity, it is also necessary to have high-quality data that encompasses many data patterns.

[0005] Conventionally, as a system for collecting teacher data used in machine learning as described above, for example, the system disclosed in Patent Document 1 is known. In this system, a user environment is set up to perform an inspection job that uses a classification model to classify the processing state of the inspected object into normal / abnormal states, and a learning device is provided to enable the classification model to learn and provide it to the user environment. Labeled data obtained during the inspection job is sent from the user environment to the learning device. The learning device uses an existing classification model to determine whether the label of the data is correct. Furthermore, if the determination is appropriate, the labeled data is used as supplementary teacher data to reconstruct the classification model; conversely, if the determination is inappropriate, the labeled data is excluded from the teacher data.

[0006] Prior art literature

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2020-194355 Summary of the Invention

[0009] The problems that the invention needs to solve

[0010] The existing teacher data collection system described above can prevent data with inappropriate labels from being mixed in with the teacher data. However, this system does not evaluate the quality and quantity of the collected teacher data itself. Therefore, the results of learning a classification model when the quality or quantity of collected substandard data is insufficient may not guarantee high classification accuracy. Furthermore, for the same reason, it is impossible to accurately determine when to stop collecting substandard data, and it is possible that even if sufficient quality and quantity of teacher data have been obtained, it is unnecessary to continue collecting teacher data.

[0011] This invention was made to solve the problems mentioned above, and its purpose is to provide a method for collecting teacher data that can quantitatively and appropriately evaluate the quality and quantity of non-conforming data collected as teacher data. As a result, a classification model can be learned with high accuracy using sufficient quality and quantity of teacher data, and the collection of non-conforming data can be completed at the optimal time to minimize the collection time.

[0012] Methods for solving problems

[0013] To achieve this objective, the invention involved in technical solution 1 is a method for collecting image data, i.e., non-conforming product data, obtained by photographing the appearance of an inspection object G that is determined to be abnormal by an inspector, as teacher data for learning a classification model that categorizes the inspection object G as normal / abnormal. The method is characterized by comprising: a non-conforming product data collection step, which collects a plurality of non-conforming product data (…). Figure 3 Step 1); Feature extraction process, extracting multiple features from each of the many nonconforming product data collected (Step 2); State and calculation process, calculating the state and Zk of each of the extracted multiple features for the many nonconforming product data using the following formula (1) (Step 3); Index value calculation process, calculating the logarithmic sum of the calculated multiple states and Zk as the index value HEG using the following formula (2) (Step 4); and Nonconforming product data collection termination process, ending the collection of nonconforming product data when the calculated index value HEG becomes above the given target value HEGTGT (Step 5).

[0014] Formula 1

[0015]

[0016] Zk: The state of each feature in all nonconforming item data and

[0017] N: Number of non-conforming items

[0018] xi: The value of the characteristic quantity in each nonconforming item data

[0019] mu N: The average value of each characteristic in all nonconforming product data.

[0020] σN: Standard deviation

[0021] a: Standardization constant

[0022] Formula 2

[0023]

[0024] HEG: Indicator Value

[0025] F: Dimension of the feature quantity.

[0026] In the teacher data collection method of the present invention, image data obtained by photographing the appearance of inspection objects that are determined to be abnormal by inspectors, i.e., non-conforming product data, is collected as teacher data for learning a classification model that categorizes inspection objects into normal / abnormal products. According to the present invention, multiple feature quantities are extracted from each of the many non-conforming product data collected, and the state and Zk of each of the multiple feature quantities are calculated for the many non-conforming product data using Equation (1).

[0027] The technical significance of state and Zk is as follows. Specifically, when non-experts (newcomers, operators with few years of experience) perform inspection work on the objects being inspected, the selected non-conforming product data is generally limited to data patterns that are relatively easy to generate. Therefore, when extracting characteristic quantities from many non-conforming product data selected by such non-experts, the characteristic quantity becomes a distribution state biased towards the overall average of the non-conforming product data, and the standard deviation becomes smaller. Furthermore, according to equation (1), state and Zk have the characteristic that the larger the standard deviation σN, the larger they become, and the smaller the standard deviation σN, the closer they are to the value of 1. Thus, state and Zk well reflect the quality of each characteristic quantity as teacher data, and the larger their value, the better the state in which the characteristic quantity is dispersed from the average. Additionally, the standardization constant a is a coefficient used to adjust so that the logarithm of state and Zk becomes 0.

[0028] Next, using equation (2), the logarithmic sum of multiple states and Zk for each characteristic quantity is calculated as the index value HEG. Based on the characteristics of the states and Zk described above, the index value HEG effectively reflects the overall quality and quantity of the non-conforming data collected as teacher data; a larger value indicates higher quality and quantity of teacher data. Therefore, according to the present invention, when the calculated index value HEG is above a given target value, it is considered that teacher data with sufficient quality and quantity has been obtained, and the collection of non-conforming data is terminated.

[0029] As described above, the quality and quantity of nonconforming data collected as teacher data can be quantitatively and appropriately evaluated using the HEG index. This allows for the high-precision learning of the classification model using sufficient high-quality and high-quantity teacher data, and enables the collection of nonconforming data to be completed at the optimal time, minimizing collection time. Furthermore, by appropriately setting the standardization constant 'a', the logarithm of the states and Zk, which are less affected by the features, becomes close to 0. Therefore, terms related to such features can be automatically eliminated, and the HEG index can be calculated appropriately and easily. Attached Figure Description

[0030] Figure 1It is a diagram that roughly represents an inspection system that uses a classification model learned based on collected teacher data to inspect the objects being inspected.

[0031] Figure 2 This is a diagram representing the device for collecting non-conforming product data and the learning device for the classification model.

[0032] Figure 3 This is a flowchart illustrating the collection and processing of teacher data.

[0033] Figure 4 It means through Figure 3 A diagram of an example of the feature quantity obtained by the transformation process.

[0034] Figure 5 It means Figure 4 A graph showing the relationship between the features, the state, and Zk.

[0035] Figure 6 It means Figure 5 A graph showing the relationship between the state of Zk and the index value HEG.

[0036] Figure 7 This is a graph showing the relationship between the HEG index value and the accuracy of the classification model. Detailed Implementation

[0037] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Figure 1 An inspection system is shown that has learned a classification model using data of numerous images of non-conforming products (hereinafter referred to as "non-conforming product data") collected by a collection method described later, and data of numerous images of conforming products (hereinafter referred to as "conforming product data"). This inspection system 1 is installed, for example, in a vehicle parts manufacturing plant, and automatically determines whether a manufactured vehicle part (e.g., a cylinder block) is a normal product (conforming product) or an abnormal product (non-conforming product) by inspecting the appearance of the vehicle part using the classification model. Hereinafter, the vehicle part to be inspected is referred to as the "inspection object".

[0038] like Figure 1 As shown, the inspection system 1 includes: a conveyor 2 that transports an object G to be inspected at a given speed in one direction; and an inspection device 3 that determines whether the object G is good or bad when it reaches a given inspection position. Objects G that are determined to be defective by the inspection device 3 are removed from the conveyor 2 and transported to a dedicated storage area for defective products.

[0039] The inspection device 3 is composed of an information processing device such as a computer, and includes a control unit 4, an image acquisition unit 5, a storage unit 6, a learning unit 7, an input unit 8, an output unit 9, and a camera 10.

[0040] The control unit 4 has a CPU and controls the other parts 5-9 of the inspection device 3, as well as the camera 10. The image acquisition unit 5 acquires image data of the appearance of the object to be inspected G captured by the camera 10 as digital data. The storage unit 6 has ROM and RAM; various programs for controlling the inspection device 3 are stored in the ROM, and various data acquired for control are stored in the RAM. The learning unit 7 has a classification model for determining whether the object to be inspected G is good or bad. The input unit 8 has a keyboard operated by the operator and can input data and signals from the outside. The output unit 9 has a display for showing the determination result of whether the object to be inspected G is good or bad.

[0041] like Figure 2 As shown, the learning system 11 that enables the aforementioned classification model to learn includes a teacher data collection device 12 and a classification model learning device 13. The collection device 12 collects data on defective products actually determined as defective by the inspection device 3 as teacher data with the required quality and quantity through a collection process described later. Similar to the inspection device 3, the collection device 12 is composed of an information processing device such as a computer, and includes a control unit with a CPU, a storage unit with ROM and RAM, an input unit, and an output unit (all not shown).

[0042] The learning device 13 uses a neural network to learn a classification model by collecting a large number of qualified product teacher data from a large number of qualified product data actually determined to be qualified products by the inspection device 3, and a large number of unqualified product teacher data collected by the collection device 12.

[0043] Figure 3 This indicates the collection and processing of teacher data performed in the collection device 12. In this process, firstly in step 1 (illustrated as "S1"; the same applies below), a number of defective product data (number of data = N) obtained by the inspection device 3 are acquired.

[0044] Next, in step 2, each of the N defective product data points is transformed into a given set of features (feature dimension = F) and stored in the storage unit. This transformation from defective product data to features can be performed, for example, using SIFT (Scale-Invariant Feature Transform) or CNN (Convolutional Neural Network). Thus, as... Figure 4 As shown, for each of the N defective product data, the feature quantity of dimension F is obtained. Figure 4 This is an example of a case where the number of nonconforming data N = 956 and the dimension F = 50.

[0045] Next, in step 3, the state and Zk are calculated using the following formula (1).

[0046] Formula 1

[0047]

[0048] Zk: The state of each feature in all nonconforming item data and

[0049] N: Number of non-conforming items

[0050] xi: The value of the characteristic quantity in each nonconforming item data

[0051] muN: The average value of each characteristic in all nonconforming product data.

[0052] σN: Standard deviation

[0053] a: Standardization constant

[0054] Therefore, as Figure 5 As shown, the state and Zk (Z1~Z50) of each characteristic quantity across all nonconforming data were calculated. As mentioned earlier, the state and Zk well reflect the quality of each characteristic quantity as teacher data; the larger the value, the better the state of the characteristic quantity, indicating that it is dispersed from the mean. Furthermore, the state and Zk exhibit the characteristic that the larger the standard deviation σN, the larger it becomes, and the smaller the standard deviation σN, the closer it is to the value of 1. Additionally, the standardization constant a is a coefficient used to adjust the logarithm of the state and Zk to a value of 0, while observing... Figure 4 The result of the characteristic quantity shown is that the logarithm of the state and Zk is manually set to a value of 0.

[0055] Next, in step 4, the logarithmic sum of the calculated states and Zk is used as the index value HEG by formula (2). Thus, as... Figure 6 As shown, a single index value (HEG) is calculated that covers all nonconforming product data and all feature quantities. As mentioned earlier, the HEG index value effectively reflects the overall quality and quantity of nonconforming product data collected as teacher data; a higher value indicates higher quality and quantity of teacher data.

[0056] Formula 2

[0057]

[0058] HEG: Indicator Value

[0059] F: Dimension of the feature quantity

[0060] Next, in step 5, it is determined whether the calculated indicator value HEG is above the given target value HEGTGT. If the determination result is "No" and the indicator value HEG does not reach the target value HEGTGT, it is considered that the quality and / or quantity of non-conforming data collected as teacher data is insufficient, and the process returns to step 1 above to obtain more non-conforming data. On the other hand, if the determination result in step 5 is "Yes" and the indicator value HEG reaches the target value HEGTGT, it is considered that the quality and quantity of non-conforming data collected as teacher data have been sufficiently ensured, and the collection of non-conforming data ends, thus concluding this process.

[0061] As described above, according to this embodiment, the quality and quantity of non-conforming product data collected as teacher data can be quantitatively and appropriately evaluated using the index value HEG. As a result, the classification model can be trained with high accuracy using sufficient quality and quantity of teacher data, and the collection of non-conforming product data can be completed at the optimal time, minimizing the collection time.

[0062] Furthermore, by appropriately setting the standardization constant a, the logarithm of the state and Zk, which are related to features with little influence, becomes a value close to 0. Therefore, terms related to such features can be automatically eliminated, and the index value HEG can be calculated appropriately and easily.

[0063] Figure 7 This figure illustrates the relationship between the calculated indicator value HEG, as described above, and the classification accuracy of test data (both acceptable and unacceptable data) using a classification model learned from nonconforming data collected before reaching that HEG value. As shown, a higher HEG value corresponds to higher classification accuracy based on the classification model, confirming the effectiveness of HEG as a parameter representing both the quality and quantity of teacher data. Furthermore, based on this relationship between HEG and classification accuracy, the target value HEGTGT, which serves as the benchmark for determining the end of teacher data collection, is set to be equivalent to the target value ACCTGT for classification accuracy based on the classification model.

[0064] Furthermore, the present invention is not limited to the described embodiments and can be implemented in various ways. For example, in one embodiment, by applying a standardization constant a to the state and Zk, terms related to features with little influence are automatically eliminated from the index value HEG, but the standardization constant a can also be omitted.

[0065] Furthermore, the dimension F of the feature quantities and the number of defective product data N shown in the embodiment are examples, but of course, other suitable values ​​can also be used. Moreover, the embodiment shows an example of checking whether a vehicle part is a qualified or unqualified product by using a classification model learned from collected teacher data, but the invention is not limited to this; any classification model used for checking qualified / unqualified products can be widely applied to the collection of teacher data for this purpose. Furthermore, within the scope of the invention, the structure of minor parts can be appropriately modified.

[0066] Symbol Explanation

[0067] 1. Inspection System

[0068] 3. Inspection device

[0069] 12. Device for collecting non-conforming product data

[0070] 13. Learning apparatus for classification models

[0071] G. Object to be inspected

[0072] Zk states and

[0073] HEG index value

[0074] The target value for the HEGTGT indicator.

Claims

1. A method for collecting teacher data, comprising collecting image data (i.e., non-conforming product data) obtained by photographing the appearance of inspection objects determined by inspectors to be abnormal, and using this data as teacher data for learning a classification model that categorizes the inspection objects as normal / abnormal products, wherein, The method for collecting teacher data includes: The non-conforming product data collection process collects data on many of the aforementioned non-conforming products; The feature extraction process extracts multiple features from each of the many non-conforming product data collected. The state and calculation process, through the following formula (1), calculates the state and Zk of each of the extracted multiple feature quantities for the multiple nonconforming product data; The index value calculation process involves using the following formula (2) to calculate the logarithmic sum of the calculated states and Zk as the index value HEG. as well as The non-conforming product data collection process ends when the calculated HEG index value reaches or exceeds a given target value. Formula 1 Zk: The state of each feature in all nonconforming item data and N: Number of non-conforming items xi: The value of the characteristic quantity in each nonconforming item data muN: The average value of each characteristic in all nonconforming product data. σN: Standard deviation a: Standardization constant Formula 2 HEG: Indicator Value F: Dimension of the feature quantity.

Images