MÉTODO DE INSPEÇÃO, MÉTODO DE CLASSIFICAÇÃO, MÉTODO DE GERENCIAMENTO, MÉTODO DE PRODUÇÃO DE MATERIAL DE AÇO, MÉTODO DE GERAÇÃO DE MODELO DE TREINAMENTO, MODELO DETREINAMENTO, DISPOSITIVO DE INSPEÇÃO E INSTALAÇÃO DE PRODUÇÃO DE MATERIAL DE AÇO

A two-stage defect detection method combining conventional machine learning and convolutional neural networks addresses computational complexity and non-detection/overdetection issues, improving accuracy and efficiency in surface inspection.

BR112023027343B1Active Publication Date: 2026-11-10JFE STEEL CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
BR112023027343
Authority / Receiving Office
BR · BR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-07-08
Filing Date
2022-06-20
Publication Date
2026-11-10
Estimated Expiration
2042-06-20

AI Technical Summary

Technical Problem

Existing surface inspection methods using convolutional neural networks face challenges of high computational complexity and non-detection/overdetection, leading to inefficient production processes and defective product leakage.

Method used

A two-stage defect detection method using a first defect determination with conventional machine learning and a second determination with a convolutional neural network, reducing the candidate defect parts through initial screening to manage processing capacity and improve accuracy.

Benefits of technology

The method enhances detection accuracy and reduces processing time by leveraging convolutional neural networks effectively on a larger number of defect candidates while maintaining high discrimination capacity.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

Inspection method, classification method, management method, steel material production method, learning model generation method, learning model, inspection device, and steel material production equipment. An inspection method, a classification method, a management method, a steel material production method, a training model generation method, a training model, an inspection device, and a steel material production facility are provided that allow for both improved detection accuracy and reduced processing time. This inspection method is for detecting a surface defect on an inspection target.The inspection method includes: an image generation step (s1) to acquire an image obtained by capturing a surface of the inspection target; an extraction step (s3) to extract candidate defect parts from the image; a screening step (s4) to sort the extracted candidate defect parts by a first defect determination; and an inspection step (s5) to define, as a target, a candidate defect part sorted by the first defect determination and detect whether the surface defect is harmful or harmless, by a second defect determination using a convolutional neural network.
Need to check novelty before this filing date? Find Prior Art

Description

1 / 25 “INSPECTION METHOD, CLASSIFICATION METHOD, MANAGEMENT METHOD, STEEL MATERIAL PRODUCTION METHOD, TRAINING MODEL GENERATION METHOD, TRAINING MODEL, INSPECTION DEVICE AND STEEL MATERIAL PRODUCTION INSTALLATION” TECHNICAL FIELD

[001] This disclosure relates to an inspection method, a classification method, a management method, a steel material production method, a learning model generation method, a learning model, an inspection device and a steel material production equipment. BACKGROUND OF THE INVENTION

[002] In the manufacturing industry, surface defects in products can lead to defective products in terms of appearance and strength. Surface inspection to ensure defect-free products is significantly important to prevent the flow of defective products. For example, in the steel industry, an optical surface inspection device using a line sensor and a line light source, such as that described in Patent Literature (PTL) 1, was introduced to implement automatic inspection, including automatic defect extraction, determining them as harmful or harmless, and classifying the types or similarities of defects. PTL 1 describes a surface inspection device that optically detects surface defects using feature values. Examples of feature values ​​include those based on defect sizes, those based on defect types, and those based on defect grades. LIST OF QUOTES

[003] Patent Literature Petition 870230113723, dated 12 / 22 / 2023, page 12 / 71 2 / 25 PTL 1: JP H09-257723A SUMMARY (Technical problem)

[004] Here, convolutional neural networks, which have been increasingly used for image inspection in recent years, have greater discrimination capabilities than general machine learning methods. Examples of general machine learning methods include machine learning methods using feature values, such as regression models, decision tree models, random forests, Bayesian estimation models, Gaussian mixture models, and other methods in which these machine learning models are enhanced. Examples of regression models include linear regression, logistic regression, multiple regression, support vector machines, and those with nonlinear kernels.

[005] The machine learning method described in PTL 1 does not possess discrimination capabilities comparable to those of a convolutional neural network, and more frequently causes non-detection, in which harmful candidate parts are erroneously determined as harmless, and overdetection, in which harmless candidate parts are erroneously determined as harmful. This non-detection and overdetection can lead to the leakage of defective products or a decrease in the efficiency of the entire process, because redetermination is necessary in a post-inspection process as a countermeasure. It is therefore necessary to reduce this non-detection and overdetection.

[006] Convolutional neural networks can reduce the occurrence of non-detection and overdetection due to their high discrimination capacity as described above. However, convolutional neural networks face challenges of high computational complexity and Petition 870230113723, dated 12 / 22 / 2023, page 13 / 71 3 / 25 cannot process a large number of potentially defective parts at the same time. For example, when used for inspection or classification of steel materials in continuous motion along a production line, it is necessary to parallelize expensive processing devices to perform continuous determination using convolutional neural networks with high computational complexity. Alternatively, it is necessary to compromise production efficiency in favor of inspection, for example.

[007] It would be useful to provide an inspection method, a classification method, a management method, a steel material production method, a learning model generation method, a learning model, an inspection device and a steel material production equipment that can improve both detection accuracy and reduce processing time. (Solution to the problem)

[008] An inspection method according to an embodiment of this disclosure is an inspection method for detecting surface defects on an inspection target, the inspection method including: an image generation step to acquire an image of a target inspection surface; an extraction step to identify potential defects from the image; a screening stage to sort out candidate defect parts extracted by an initial defect determination; and an inspection stage to detect harmful or harmless surface defects by a second defect determination using a convolutional neural network, the second defect determination being Petition 870230113723, dated 12 / 22 / 2023, page 14 / 71 4 / 25 directed to parts that are candidates for defects after screening for the initial defect determination.

[009] A classification method according to an embodiment of the present disclosure is a classification method for classifying surface defects on an inspection target, the classification method including: an image generation step to acquire an image of a target inspection surface; an extraction step to identify potential defects from the image; A screening stage to sort out candidate defect parts extracted through an initial defect determination; and a classification stage to classify types and / or degrees of surface defects by a second defect determination using a convolutional neural network, the second defect determination being directed at candidate defect parts after screening by the first defect determination.

[0010] A management method according to an embodiment of this disclosure includes a management step to classify the inspection target, based on the types and / or degrees to which surface defects were classified by the classification method above.

[0011] A method for producing steel material according to an embodiment of the present disclosure includes: A production step to produce a steel material; and the inspection step included in the inspection method above, where in the inspection step, the steel material produced in the production step is the target of inspection. Petition 870230113723, dated 12 / 22 / 2023, page 15 / 71 5 / 25

[0012] A method for producing steel material according to an embodiment of the present disclosure includes: A production step to produce a steel material; and the management step included in the management method above, where in the management step, the steel material produced in the production step is the target of inspection.

[0013] A learning model generation method according to an embodiment of the present disclosure is a learning model generation method for generating a learning model to be used in an inspection method for detecting surface defects on an inspection target, the learning model generation method including a learning model generation step by a convolutional neural network using teaching data, which includes candidate defect parts after screening by a first defect determination on an image of an inspection target that was acquired beforehand as input record data, and results indicating whether the input record data is harmful or harmless as output result data.

[0014] A learning model according to an embodiment of the present disclosure is generated by the learning model generation method above.

[0015] An inspection device according to an embodiment of the present disclosure is an inspection device configured to detect or classify surface defects on an inspection target, the inspection device including: an imaging unit configured to acquire a Petition 870230113723, dated 12 / 22 / 2023, page 16 / 71 6 / 25 image of a surface of the inspection target; and an arithmetic unit configured to extract candidate defect parts from the image, sort the extracted candidate defect parts by a first defect determination, and perform surface defect determination by a second defect determination using a convolutional neural network, the second defect determination being directed at the candidate defect parts after sorting by the first defect determination, wherein the surface defect determination performed by the second defect determination includes the detection of harmful or harmless surface defects or the classification of types and / or degrees of surface defects.

[0016] Steel production equipment according to an embodiment of this disclosure includes: Production equipment configured to produce a steel material; and the inspection device above, in which the steel material produced by the production equipment is the target of inspection. (Beneficial Effect)

[0017] According to this disclosure, the inspection method, the classification method, the management method, the steel material production method, the learning model generation method, the learning model, the inspection device and the steel material production equipment that can improve both detection accuracy and reduce processing time are provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the attached drawings: Petition 870230113723, dated 12 / 22 / 2023, page 17 / 71 7 / 25 FIG. 1 illustrates an example of a surface inspection processing flow; FIG. 2 illustrates an example of a determinant generation flow; and FIG. 3 illustrates an example of a configuration for an inspection device according to one of the modes. DETAILED DESCRIPTION

[0019] Next, an inspection method or similar method according to an embodiment of this disclosure will be described with reference to the drawings. (Inspection Method)

[0020] The inspection method according to the present embodiment detects surface defects on an inspection target. FIG. 1 illustrates an example of a surface inspection processing flow performed by the inspection method according to the present embodiment. First, an image of the inspection target surface is acquired (Step S1: image generation step). Although the inspection targets in the Examples described later are steel plates, the inspection target is not limited to a steel plate.

[0021] After the image generation stage, preprocessing is performed on the image (S2 Stage). Preprocessing is performed to remove disturbances from the obtained image and facilitate the detection of defect signals (signals that indicate surface defects). Preprocessing may include, for example, inspection area extraction, luminance irregularity correction, frequency filtering, differential filtering, or similar.

[0022] After preprocessing, the candidate part extraction processing for candidate part extraction from Petition 870230113723, dated 12 / 22 / 2023, page 18 / 71 8 / 25 of the image is processed (Step S3, extraction step). The defect candidate part extraction process is the process for extracting and labeling parts (defect candidate parts) where the luminance differs from neighboring parts compared to neighboring parts using thresholding. In the present embodiment here, thresholding is used to extract defect candidate parts, but other types of processing, by which defect candidate parts can be identified, can also be used. Furthermore, the entire image can be extracted as a defect candidate part.

[0023] After the extraction step, the first defect determination process is executed to sort out the candidate defect parts extracted by the first defect determination (Step S4: screening step). Unlike the second defect determination described later, the first defect determination does not use a convolutional neural network. By the first defect determination, the number of candidate defect parts is reduced to the extent that they can be processed by the second defect determination described later. In the first defect determination according to the present embodiment, the screening is performed in such a way as to exclude candidate defect parts that are certainly determined to be harmless.

[0024] In the first defect determination, for example, feature values ​​can be extracted from the image and the feature values ​​can be used to sort out candidate defect parts. In the first defect determination, a machine learning model using feature values ​​extracted from a past image of a target inspection surface can be used to sort out candidate defect parts, as in the Examples described later. Details of the learning model used in the first defect determination Petition 870230113723, dated 12 / 22 / 2023, page 19 / 71 9 / 25 will be described later.

[0025] After the screening stage, the second defect determination process to detect harmful or harmless surface defects by second defect determination using a convolutional neural network is executed, the second defect determination process being directed at candidate defect parts after screening by the first defect determination (S5 Stage: Inspection Stage). Here, harmless surface defects refer to surface defects that do not render the inspection target defective. For example, harmless surface defects include those that have physical characteristics detected as signs different from most perfect parts and are extracted as defect candidates, but which are not actually defects because they do not negatively affect quality. Harmless surface defects also include, for example, those that are defects but are small enough to meet quality standards.Harmful surface defects refer to surface defects that are not harmless surface defects. Details of the convolutional neural network, i.e., a learning model to be used in determining the second defect, will be described later.

[0026] Now, in another embodiment, the second defect determination process can be performed as a process to classify surface defects on the inspection target. That is, after the screening step, instead of the aforementioned inspection step, the second defect determination process, being directed at the candidate parts for defects after screening by the first defect determination, can be performed (Step S5: classification step). The second defect determination process classifies the types and / or degrees of surface defects by the second defect determination using a Petition 870230113723, dated 12 / 22 / 2023, page 20 / 71 10 / 25 Convolutional neural network. At this point, a classification method is implemented to classify surface defects on the inspection target in more detail than the inspection method (detection of harmful or harmless defects). Such a classification method can be applied to a management method that includes the management step to classify the inspection target based on the types and / or degrees of surface defects detected on the inspection target. Alternatively, in a case where the inspection target is a steel material, such as a steel plate, as in the Examples described later, a steel material production method can be implemented. The steel material production method includes the production step for producing a steel material and the inspection step or management step described above, where the inspection step or management step manages the steel material produced in the production step as the inspection target.Here, the steel material production stage can be designed to include known processes. The production stage can be designed to include a roughing rolling process, a finishing rolling process, a cooling process, or similar, for example, in a case where the inspection target is a steel sheet.

[0027] A result of the second defect determination process is issued as defect information (Step S6). Defect determination may involve the detection of harmful or harmless surface defects or the classification of the types and / or degrees of surface defects. The issued defect information may be used for quality control of the inspection target in the management method and the steel material production method as described above. (Inspection Device)

[0028] The inspection method described above is performed, for example, by Petition 870230113723, dated 12 / 22 / 2023, page 21 / 71 11 / 25 an inspection device. FIG. 3 illustrates an example of an inspection device configuration according to one embodiment. The inspection device is controlled by a host system (e.g., a process computer that manages the production processes of the inspection target). The inspection device may be supplied as one of the devices that constitute production equipment for the inspection target.

[0029] The inspection device can be configured to include, for example, an arithmetic unit and a camera. The inspection device can also include a machine learning unit, as needed. The camera is an example of an image generation unit and captures an image of the surface of the inspection target (steel sheet in the example in Figure 3) illuminated by a light source. The arithmetic unit performs preprocessing, defect candidate part extraction processing, first defect determination processing, and second defect determination processing on the image. The arithmetic unit can also output data related to the above processing to the host system. The machine learning unit acquires teaching data from the arithmetic unit and generates a learning model (second determiner) to be used in the second defect determination.In the case where a learning model is used in the first defect determination, the machine learning unit also acquires teaching data from the arithmetic unit and generates the learning model (first determinant) to be used in the first defect determination. Here, in the case where a learning model is not used in the first defect determination, the first determinant can be designed manually. The first determinant and the second determinant that were generated can be stored in memory. Petition 870230113723, dated 12 / 22 / 2023, page 22 / 71 12 / 25 accessible by the arithmetic unit, and the arithmetic unit can read them from memory when the first fault determination and the second fault determination are performed. (Method for Generating Determinants)

[0030] FIG. 2 illustrates an example of a determinant generation flow. In FIG. 2, the same surface inspection processing flow from FIG. 1 is represented by dashed lines. The processing flow in the dashed lines indicates that the processing is not executed simultaneously with the determinant generation flow, and is illustrated together to indicate the relationship with Steps S11, S12, and S21 of the determinant generation flow. After the determinants are generated according to the determinant generation flow, the surface inspection processing flow is executed using the generated determinants.

[0031] The first determinant is generated before the second determinant as follows. First, an image is acquired, preprocessing is performed, and candidate defect parts are extracted by thresholding. The above processing corresponds to Steps S1 to S3 above and can be performed by the methods described above. Here, the acquired image is an image (passed image) of an inspection target on which actual defects have already been measured.

[0032] Then, the extracted candidate parts are linked to the actual defects (Step S11). Based on the candidate parts that have been linked to the actual defects, the first determinant is generated by machine learning or manual design (Step S12). In a case where the first determinant is generated by machine learning, the first determinant (learning model) is a model generated in advance by machine learning using feature values. Petition 870230113723, dated 12 / 22 / 2023, page 23 / 71 13 / 25 extracted from a previously obtained image of a surface of any inspection target. Feature values ​​are feature values ​​that are linked to real defects and are used as teaching data.

[0033] In the first defect determination using the first determinant, a determination method is used whereby all candidate defect parts after screening must be determined in time by the second defect determination, even when the method has low discrimination capability. That is, the first defect determination requires a determination method that does not take much computation time. For example, feature values ​​such as size, shape, orientation, and hue are extracted from the image, and defect determination is performed using these feature values. In other words, a determination method other than a convolutional neural network is preferable. Conditional branches can be created manually for determination using the feature values ​​(manual design in Step S12).General machine learning methods using feature values, such as regression models, decision tree models, random forests, Bayesian estimation models, Gaussian mixture models, or other methods that enhance these machine learning models, can also be used for determination using feature values ​​(machine learning in Step S12). Here, examples of regression models include linear regression, logistic regression, multiple regression, support vector machines, and those with nonlinear kernels. Furthermore, it is not essential that the first defect determination be made using feature values, as long as sufficiently high-speed processing is possible. The first determinant is designed to identify candidate parts for defects, such as harmless surface defects that are not needed, and remove them. Petition 870230113723, dated 12 / 22 / 2023, page 24 / 71 14 / 25 the parts that are candidates for defect are unnecessary from those to be determined by the second defect determination.

[0034] Using the first generated determinant, candidate parts for defects are selected. This processing corresponds to Step S4 above. Based on the selected candidate parts for defects, the second determinant is generated using a convolutional neural network (Step S21). In Step S21, a learning model is generated using teaching data, which includes candidate parts for defects after screening by the first defect determination in the previously acquired image as input log data and results indicating whether the input log data are harmful or harmless as output result data. The second determinant is generated so that it can determine defects (e.g., harmful defects) that require further determination using a convolutional neural network after the first defect determination, and so that the results obtained are final determination results.

[0035] As described above, it is preferable that the number of candidate parts to be subjected to the second defect determination be adjusted based on a threshold or parameter defined for the first determinant, so that it does not reach the upper limit of the second determinant's defect determination processing capacity. Here, by increasing the number of candidate parts selected by the first defect determination as long as the second determinant's defect determination processing capacity is not exceeded, it is possible to increase the number of targets to be determined by the convolutional neural network with high discrimination capacity, thus improving the detection accuracy. Furthermore, since the number of candidate parts is reduced by the first defect determination, it does not reach the upper limit of the processing capacity. Petition 870230113723, dated 12 / 22 / 2023, page 25 / 71 With a 15 / 25 defect determination from the second detector, the processing time is not delayed in the second defect determination, and the total processing time can be reduced.

[0036] Here, in convolutional neural networks that use deep learning, the time required for determination and discrimination capability varies depending on the layer structures. It is preferable to determine a layer structure considering the time allowed for determination and the required discrimination capability. (Example 1)

[0037] An Example (Example 1) of an inspection method implemented using the inspection device of FIG. 3 will be described below. In this Example, the defect determination processing was performed using determinators built in advance using machine learning, and this allowed for accurate discrimination between harmful and harmless defects.

[0038] First, the determinant was generated. As illustrated in FIG. 3, a camera and a light source were installed on a steel sheet production line, and a whole image of a steel sheet surface was captured by the camera. A monochrome line camera was used as the camera and a line light source was used as the light source. The arithmetic unit acquired the above image and performed the necessary processing. Specifically, the arithmetic unit performed luminance irregularity removal processing. The arithmetic unit also normalized the image by frequency filtering and subsequently extracted candidate defect parts by thresholding, labeling, or similar. From the extracted candidate defect parts, 100 or more feature values ​​related to shape, orientation, size, hue, or similar were calculated by processing of Petition 870230113723, dated 12 / 22 / 2023, page 26 / 71 16 / 25 image.

[0039] The machine learning unit linked the feature values ​​extracted from the candidate parts to the actual defects, labeling them as harmful or harmless, for example. Thus, teaching data (“primary teaching data” for generating the first determinant) was obtained. The machine learning unit automatically constructed the first determinant with a binary decision tree using machine learning and the primary teaching data. By using the first determinant generated by the machine learning unit, the arithmetic unit became capable of performing the first defect determination in real time.

[0040] Here, the primary teaching data set included 50 data pieces (harmful data) linked as harmful and 400 data pieces (innocuous data) considered innocuous. In constructing the binary decision tree, some branches of the tree were optimally pruned by cross-validation to avoid overfitting. Specifically, the teaching data was randomly divided into 10 parts, with nine parts used as training data to construct the determinant with the binary decision tree, and one part used as evaluation data to assess the determination of the constructed determinant. Training and evaluation were repeated 10 times, and the evaluation was then performed by cross-validation where all evaluation data were separated from the training data.At this point, an overfitting prevention penalty was applied to the number of branches to avoid overfitting, and the ideal pruning was performed by setting the overfitting prevention penalty so that the best cross-validation evaluation results could be achieved. Furthermore, the first determinant was designed to detect... Petition 870230113723, dated 12 / 22 / 2023, page 27 / 71 17 / 25 reliably identified harmful data, leaving, to some extent, harmless data. In other words, the first determinant was designed to exclude only data that was certainly harmless. For this reason, the first determinant was tuned so that it would not identify harmful data as harmless in the first defect identification. As a specific example of the tuning, the first error penalty for incorrectly identifying harmful data as harmless data during training was set to 10 times greater than the second error penalty for incorrectly identifying harmless data as harmful data. This setting allowed the first determinant to correctly identify all harmful data.

[0041] The candidate parts determined to be harmful in the first defect determination include those that are linked as harmless in terms of actual defects. It is preferable that this harmless data be correctly determined as harmless in the second defect determination. In order to achieve greater discrimination capability than the first determinant, the second determinant was generated using deep learning (specifically, a convolutional neural network). Different data from the primary teaching data were prepared. In the prepared data, the data on the candidate parts that were determined to be harmful by the first defect determination were linked to actual defects and used as teaching data (“secondary teaching data” to generate the second determinant).

[0042] Here, the number of pieces of secondary teaching data included 58 pieces of harmful data and 390 pieces of harmless data. As in training in determining the first defect, the data Petition 870230113723, dated 12 / 22 / 2023, page 28 / 71 18 / 25 secondary school students were randomly divided into training and validation data, and all data were evaluated. Since convolutional neural networks require all images to be normalized to the same size, the secondary school data were normalized to a size of 256 x 256 through resizing. For the secondary school data, images that were flipped up and down and left and right, and up and down and left and right after resizing were added, and additional images that were rotated -10°, -5°, +5°, and +10° were added for each image, in order to increase the number of training data. After adding the rotated images, the final secondary school data were obtained by cropping the 224 x 224 size from the center of each image to make all images the same size.The above inversion and rotation processing allowed the number of teaching data seconds to be increased by a factor of 20.

[0043] ResNet152 was used as the network framework for deep learning. Global average clustering (GAP) was applied to ResNet152 in order to train it to discriminate data into two classes, i.e., harmful data and harmless data. After that, a fully connected layer configured for classification into 1000 classes was removed and, in its place, two fully connected layers with 128 output nodes and a fully connected layer configured for classification into two classes were added, thus building a network framework. TensorFlow (version 2.0.0) with the ImageNet dataset was used as a deep learning framework.

[0044] For training, the ResNet152 weight was first obtained and the network parameters were optimized by transfer learning using stochastic gradient descent (SGD). We defined a moment Petition 870230113723, dated 12 / 22 / 2023, page 29 / 71 19 / 25 of SGD of 0.9, a drop in the learning rate of 5e4, an initial learning rate of 0.01, and a maximum number of learning epochs of 10. As a result, an ideal epoch was found to be 5, with a cross-validation correct response rate of 98.9%.

[0045] The arithmetic unit became capable of performing the second defect determination in real time using the second determinant with a convolutional neural network that was generated by the machine learning unit.

[0046] Here, in this example, the machine learning unit was a device (another computer) with a processor separate from the arithmetic unit. The machine learning unit was able to communicate with the arithmetic unit in order to transmit and receive the necessary data.

[0047] Information about defects that were ultimately determined to be detrimental in the two-step defect determination above is sent to the host system via communication and is used to determine if the steel sheet requires maintenance and if it can be shipped. Additionally, the types and degrees of detriment (values ​​indicating the degrees of detriment) of the detrimental surface defects can be determined. The host system can take optimal action depending on the types and degrees of detriment of the detrimental surface defects. Examples of surface defects include, but are not limited to, flaws.

[0048] After the first and second determinations were generated, the surface of the steel sheet was inspected according to the processing flow illustrated in FIG. 1. In the present Example, 98% of the candidate defect parts that were determined to be harmless in terms of actual defects in the extraction processing of Petition 870230113723, dated 12 / 22 / 2023, page 30 / 71 20 / 25 candidate defective parts (Step S3: extraction step) were determined to be harmless in the first defect determination. In the second defect determination, 99.97% of the candidate defective parts were determined to be harmless in combination with the first defect determination.

[0049] As described above, with the above processes and configuration, the inspection method and inspection device according to the present embodiment can improve detection accuracy and reduce processing time. That is, according to the inspection method and inspection device of the present embodiment, the number of defect candidates is limited by the first defect determination, in which the detected defect candidates are determined by a conventional machine learning method or similar with high processing speed that uses feature values ​​or similar. After that, a convolutional neural network is applied as a second defect determination. This makes it possible to apply the convolutional neural network with high discrimination capacity even to a large number of defect candidates.Currently, the number of parts considered defective due to the limitation imposed by the first defect determination can be adjusted to be less than or equal to the number that can be processed by the second defect determination, thus reducing the total processing time. Furthermore, according to the inspection method and inspection device of this embodiment, limits, parameters, and layer structures can be individually adjusted for the respective defect detectors in the two-stage defect determination. This has the advantageous effect of providing greater adjustment freedom compared to a case where a single detector is provided. (Example 2: First Determinant Adjustment Method) Petition 870230113723, dated 12 / 22 / 2023, page 31 / 71 21 / 25

[0050] An example (Example 2) relating to a method of adjusting the first determinant will be described below.

[0051] The goal of the first defect determination is screening. It is therefore preferable to make adjustments so that candidate parts determined to be harmful in terms of actual defects are correctly and reliably determined to be harmful, while allowing some candidate parts determined to be harmless in terms of actual defects to be erroneously determined to be harmful, to some extent. In a case where the first defect determination is performed by machine learning, there is a trade-off between erroneously determining harmful defects as harmless and erroneously determining harmless defects as harmful defects. Implementing machine learning without considering this relationship may lead to the possibility that some candidate parts determined to be harmful in terms of actual defects cannot be reliably determined to be harmful.

[0052] For example, consider a case where a determinant is created by machine learning using N data points on candidate parts determined to be harmful in terms of actual defects and M data points on candidate parts determined to be harmless in terms of actual defects as training data. Training is assumed to be simply performed so that, among (N + M) data points, a percentage of data correctly determined by the determinant is maximized. In this case, even when determined as harmful or harmless in terms of actual defects, data that are significantly different from the population tend to be determined erroneously. Therefore, it is difficult to reliably determine which candidate parts are determined to be harmful in terms of Petition 870230113723, dated 12 / 22 / 2023, page 32 / 71 22 / 25 actual defects as being harmful.

[0053] To solve this, an evaluation function was used and machine learning was performed so that the evaluation function was maximized or minimized. In this example, machine learning was performed to maximize the evaluation function E represented by the equation below including the weight α. This allowed the above trade-off to be adjusted. This example therefore made it possible to reliably determine which candidate parts are identified as harmful in terms of actual harmful defects, preferably correctly identifying them as harmful.

[0054] E = α X Roí + (1 - α )

[0055] Here, Roí represents the number of correct responses for candidate defects determined to be harmful in terms of actual defects, and R02 represents the number of correct responses for candidate defects determined to be harmless in terms of actual defects.

[0056] A determinant is generated by machine learning such that the higher the value of α, the higher the rate of correct answers for candidate defects determined as harmful in terms of actual defects, and the lower the value of α, the higher the rate of correct answers for candidate defects determined as harmless in terms of actual defects. Thus, by adjusting α, it is possible to generate a determinant that reliably and correctly determines which candidate parts are determined as harmful in terms of actual defects. Here, instead of using the weight α, the same effect can be achieved by increasing the number of training data chunks, replicating the N data chunks on the candidate parts determined as harmful in terms of actual defects during machine learning.

[0057] Here, α can be adjusted by performing machine learning. Petition 870230113723, dated 12 / 22 / 2023, page 33 / 71 23 / 25 while varying α, and seeking conditions that satisfy the requirements of the first defect determination while actually classifying the performance of the determinant obtained using test data and evaluating the results. The test data can be identical to the training data, but are preferably different from the training data to avoid overfitting. In a case where a correct response rate for candidate parts determined to be harmful in terms of actual target defects can be used, it is possible to solve it as an optimization problem where the correct response rate serves as a constraint condition, such that maximizing the correct response rate for candidate parts determined to be harmless in terms of actual defects is achieved.

[0058] Here, although the above modality describes classification into two classes, that is, harmful and harmless classes, the classes can be further divided into even smaller classes, and screening conditions can be defined for each class. For example, harmful defects can be divided into three categories, namely, severe, moderate, and slight, depending on the degree of impact they cause. For example, conditions are established so that a correct response rate for candidate parts determined to be severely harmful in terms of actual defects is 100%. For example, conditions are established so that a correct response rate for candidate parts determined to be moderately harmful is 75%. For example, conditions are established so that a correct response rate for candidate parts determined to be slightly harmful is 50%.

[0059] When solving the optimization problem using the evaluation function E represented by the equation below including the weights α, β, and γ, Petition 870230113723, dated 12 / 22 / 2023, p. 34 / 71 24 / 25 The correct response rate for candidate parts determined to be harmless in terms of actual defects can be maximized.

[0060] E = α X R11 + βΧ R12 + γΧ R13 + (1 - α - β - γ ) x R14

[0061] Here, R11 represents the number of correct answers for candidate defects determined as severely detrimental in terms of actual defects. R12 represents the number of correct answers for candidate defects determined as moderately detrimental in terms of actual defects. R13 represents the number of correct answers for candidate defects determined as slightly detrimental in terms of actual defects. R14 represents the number of correct answers for candidate defects determined as harmless in terms of actual defects.

[0062] Although an embodiment of the present disclosure has been described with reference to the drawings and examples, it should be noted that various modifications and alterations may be made by those skilled in the art based on the present disclosure. Consequently, such modifications or alterations are included within the scope of this disclosure. For example, functions or the like included in each component, each step or the like may be rearranged without logical inconsistency, and a plurality of components, steps or the like may be combined into one or divided. An embodiment according to this disclosure may be implemented as a program that is executed by a processor included in the device or as a storage medium in which the program is recorded. It should be understood that these are included within the scope of this disclosure.

[0063] Although a line camera is used in the above method, an area camera can be used. In addition, a color camera can Petition 870230113723, dated 12 / 22 / 2023, page 35 / 71 A 25 / 25 camera can be used instead of a monochrome camera. The camera can capture images under multiple conditions, and the information in the images can be combined for inspection. For example, the initial defect determination can be performed using feature values ​​calculated separately or in combination from images of candidate defects obtained under different conditions. For example, two images obtained with two different optical systems under regular reflection and diffuse reflection conditions can be used, or a color image from each channel can be used. Furthermore, fusion or calculation results (e.g., addition and subtraction) of candidate defect images obtained under different conditions can be used in the initial defect determination.

[0064] Although in this modality only images were used to determine the second defect, feature values ​​can be used in addition to images. At this time, feature values ​​can be used in addition to images in the construction of the second determinant using deep learning.

[0065] Although the present method is intended for the surface inspection of steel plates, it goes without saying that it can be applied to the surface inspection not only of steel plates, but also of various steel materials and various industrial materials and products. Petition 870230113723, dated 12 / 22 / 2023, pp. 36 / 71

Claims

1 / 5 CLAIMS 1. Inspection method for detecting surface defects on an inspection target, the inspection method CHARACTERIZED in that it includes: an image generation step to acquire an image of a surface of the inspection target; an extraction step to extract candidate defect parts from the image; a screening step to sort the extracted candidate defect parts by a first defect determination using any one of a regression model, decision tree model, random forest, Bayesian estimation model, Gaussian mixture model or other method in which these models are enhanced as a first determinant;and an inspection step to detect harmful or harmless surface defects by a second defect determination using a convolutional neural network, as a second determinant, the second defect determination being directed at candidate defect parts after screening by the first defect determination, where the first determinant is designed in such a way that the first error determination penalty for erroneously determining harmful data as harmless data during training is set greater than the second error determination penalty for erroneously determining harmless data as harmful data.

2. Classification method for classifying surface defects on an inspection target, the classification method CHARACTERIZED by the fact that it includes: an image generation step to acquire an image of a Petition 870260068717, dated 10 / 07 / 2026, page 12 / 21 2 / 5 surface of the inspection target; an extraction step to extract candidate defect parts from the image; a screening step to sort the extracted candidate defect parts by means of a first defect determination using any one of a regression model, a decision tree model, a random forest, a Bayesian estimation model, a Gaussian mixture model or another method in which these models are enhanced as a first determinant;and a classification step to classify types and / or degrees of surface defects by a second defect determination using a convolutional neural network, as a second determinant, the second defect determination being directed at candidate parts for defects after screening by the first defect determination, where the first determinant is designed in such a way that the first error determination penalty for erroneously determining harmful data as harmless data during training is set greater than the second error determination penalty for erroneously determining harmless data as harmful data.

3. Management method, CHARACTERIZED in that it comprises a management step to classify the inspection target, based on the types and / or degrees to which the surface defects were classified by the classification method, as defined in claim 2.

4. Method of producing steel material, CHARACTERIZED in that it comprises: a production step to produce a steel material; and the inspection step included in the inspection method, as defined in claim 1, in which, in the inspection step, the steel material produced in the production step is the target of inspection.

5. A method for producing steel material, CHARACTERIZED in that it comprises: a production step to produce a steel material; and the management step included in the management method, as defined in claim 3, wherein in the management step, the steel material produced in the production step is the target of inspection.

6. Learning model generation method for generating a learning model to be used as a second determinant in the inspection method, as defined in claim 1, the learning model generation method CHARACTERIZED in that it comprises a learning model generation step by a convolutional neural network using teaching data, which includes candidate defect parts after screening by a first determinant used for a first defect determination in an image of an inspection target that was acquired beforehand as input record data, and results indicating whether the input record data are harmful or harmless as output result data.

7. Learning model to be used as a second determinant in the inspection method as defined in claim 1, the learning model, CHARACTERIZED in that it is generated by the learning model generation method as defined in claim 6.

8. Inspection device configured to detect or classify surface defects on an inspection target, the inspection device CHARACTERIZED in that it comprises: an image generation unit configured to acquire an image of a surface of the inspection target; and an arithmetic unit configured to extract candidate defect parts from the image, sort the extracted candidate defect parts by a first determinant using any one of a regression model, a decision tree model, a random forest, a Bayesian estimation model, a Gaussian mixture model or another method in which these models are improved, and perform surface defect determination by a second determinant using a convolutional neural network, the second defect determination being directed to the candidate defect parts after sorting by the first defect determination,wherein the surface defect determination performed by the second determinant includes the detection of harmful or harmless surface defects or the classification of types and / or degrees of surface defects, wherein the first determinant is designed in such a way that the first error penalty for erroneously determining harmful data as harmless data during training is set greater than the second error penalty for erroneously determining harmless data as harmful data.

9. Steel material production equipment, CHARACTERIZED in that it comprises: production equipment configured to produce a steel material; and an inspection device, as defined in claim 8, wherein, in the inspection device, the steel material produced by the production equipment is the target of inspection.