Steel quality inspection system based on xai and large scale deep learning hpc system
The system addresses long inspection times and inaccurate 2D-based detection by using a deep learning algorithm with 3D point clouds and XAI for optimized steel defect detection, enhancing accuracy and error identification.
Patent Information
- Application Number
- KR1020200187002
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-30
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2040-12-30
AI Technical Summary
Conventional steel inspection methods, including human visual inspection and machine vision, face challenges such as long inspection times, variable accuracy, and inability to improve detection beyond certain limits, especially with 2D-based algorithms failing to accurately detect three-dimensional defects in steel.
A steel quality inspection system utilizing a deep learning algorithm that processes 2D images and 3D point clouds, combined with a modular XAI algorithm, to optimize defect detection and provide explainable results, enabling accurate localization, sizing, and shaping of defects.
The system enhances defect detection accuracy by distinguishing between various types of defects using 2D and 3D algorithms, and the XAI algorithm allows for error identification, improving performance and consistency in defect visualization.
Smart Images

Figure 112020143103446-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a steel quality inspection system based on explainable artificial intelligence and a large-scale deep learning HPC system. Background Technology
[0003] According to conventional technology, to detect cracks and defects contained in steel, experts with advanced expertise in the field and accumulated long-term quality control experience visually inspected the surface of the steel to identify defective areas. However, in the case of steel plates used in large heavy equipment, the inspection time is very long due to the large surface area, and different results are obtained depending on the inspector's condition and fatigue. Furthermore, there are problems in that detection accuracy decreases when inspecting a large volume of materials within a proposed time.
[0004] In addition, quality inspection utilizing machine vision technology, which has recently begun to be introduced in earnest centered on smart factories, shows limitations such as being highly vulnerable to changes in the working environment like lighting and being unable to improve accuracy beyond a certain level, despite improved detection performance compared to visual inspection.
[0005] Furthermore, there are limitations to improving the algorithm's performance because, when an error occurs in the algorithm's defect detection, the exact cause of the error cannot be identified due to the black box problem where the process used to infer the result is unknown. The problem to be solved
[0007] The present invention is proposed to solve the aforementioned problems, and aims to provide a system capable of detecting defects under optimal conditions and expressing the accurate location, size, and shape of defects by applying an optimized algorithm based on the type of cracks and defects in steel materials, using a deep learning algorithm that takes 2D images as input, a Point CNN that takes 3D point clouds as input, and a modular XAI algorithm with explainability, thereby solving the problem of reduced detection accuracy of existing algorithms that are uniform regardless of the type of defect. means of solving the problem
[0009] The steel quality inspection system based on explainable artificial intelligence and large-scale deep learning HPC system according to the present invention includes a memory in which a steel quality inspection program based on explainable artificial intelligence and large-scale deep learning HPC system is stored and a processor that executes said program, wherein the processor acquires information on cracks and defects of steel through images and detects and quantifies them based on artificial intelligence.
[0010] The above processor acquires images based on mobile stereo imaging.
[0011] The above processor detects the cracks and defects based on explainable artificial intelligence.
[0012] The above processor visualizes and diagrams the crack and defect information.
[0013] The above processor performs training of artificial intelligence algorithms for detection based on a large-scale deep learning HPC system.
[0014] The above processor improves Heatmap and Captioning performance for objects using a benchmark dataset and then transfers the weight matrix through transfer learning.
[0015] The above processor performs detection of cracks and defects contained in contaminated temporary construction materials using an artificial intelligence filter specialized for temporary construction materials. Effects of the invention
[0017] According to the present invention, by applying an algorithm optimized according to the type of cracks and defects in steel based on a deep learning algorithm that receives a 2D image as input, a Point CNN that receives a 3D point cloud as input, and a modular XAI algorithm with explainability, it is possible to detect defects under optimal conditions and represent the accurate location, size, and shape of the defects.
[0018] It is possible to improve detection performance by applying an optimal algorithm that distinguishes between impurities, chemical by-products, scratches, and rust that can be detected by a 2D image-based algorithm regardless of the size of the defect, indentations and pitting that can be detected by a 2D or 3D-based algorithm depending on the size of the defect, and deformation, breakage, and wear of steel that can be detected by a 3D-based algorithm.
[0019] In addition, by utilizing the explanatory function of the XAI algorithm to visually output defect detection results as heatmaps and captions, the consistency between the algorithm's inference process and classification results can be verified, and users can identify errors in the inference process, thereby enabling performance improvement.
[0020] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing
[0022] FIG. 1 illustrates a configuration diagram of an explainable artificial intelligence algorithm for detecting defects occurring on the surface of a steel material according to an embodiment of the present invention. FIG. 2 illustrates the types of defects that occur on the surface of steel according to an embodiment of the present invention. FIG. 3 illustrates the types of three-dimensional defects that occur in steel according to an embodiment of the present invention. FIG. 4 illustrates a flowchart of an artificial intelligence algorithm for detecting and quantifying defects occurring on the surface of a steel material according to an embodiment of the present invention. Figure 5 illustrates a Captioning output result explaining the characteristics of a defect (chemical byproduct) that occurred on the surface of a steel material according to an embodiment of the present invention. Figure 6 illustrates a Captioning output result explaining the characteristics of a defect (scratch) that occurred on the surface of a steel material according to an embodiment of the present invention. FIG. 7 illustrates a heatmap output result expressing the characteristics of defects occurring on the steel surface according to an embodiment of the present invention by weight. FIG. 8 illustrates benchmark dataset training to improve heatmap performance according to an embodiment of the present invention. FIG. 9 illustrates benchmark dataset training to improve Captioning performance according to an embodiment of the present invention. FIG. 10 illustrates a process for stereoscopically representing defects based on stereoimaging technology according to an embodiment of the present invention. FIG. 11 illustrates a large-scale deep learning HPC system according to an embodiment of the present invention. Specific details for implementing the invention
[0023] The aforementioned objectives of the present invention, as well as other objectives, advantages, and features, and the methods for achieving them, will become clear from the embodiments described in detail below together with the accompanying drawings.
[0024] However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms, and the following embodiments are provided merely to easily inform those skilled in the art of the purpose, structure, and effects of the invention, and the scope of the rights of the present invention is defined by the description in the claims.
[0025] Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used in this specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.
[0027] According to an embodiment of the present invention, which is proposed to solve the problems of existing deep learning algorithms, by performing crack and defect detection of steel based on XAI, point cloud, and HPC systems, it is possible to solve the problems of limitations in performance improvement, inability to explain the reason for inference, inability to visualize defects in three dimensions, and long computation time, which are disadvantages of 2D-based deep neural networks.
[0028] FIG. 1 illustrates a configuration diagram of an explainable artificial intelligence algorithm for detecting defects occurring on a steel surface according to an embodiment of the present invention; FIG. 2 illustrates the types of defects occurring on a steel surface according to an embodiment of the present invention; FIG. 3 illustrates the types of three-dimensional defects occurring on steel according to an embodiment of the present invention; FIG. 4 illustrates a flowchart of an artificial intelligence algorithm for detecting and quantifying defects occurring on a steel surface according to an embodiment of the present invention; FIG. 5 illustrates a Captioning output result explaining the characteristics of defects (chemical by-products) occurring on a steel surface according to an embodiment of the present invention; FIG. 6 illustrates a Captioning output result explaining the characteristics of defects (scratches) occurring on a steel surface according to an embodiment of the present invention; FIG. 7 illustrates a heatmap output result expressing the characteristics of defects occurring on a steel surface by weight according to an embodiment of the present invention; FIG. 8 illustrates benchmark dataset training to improve heatmap performance according to an embodiment of the present invention; FIG. 9 illustrates benchmark dataset training to improve Captioning performance according to an embodiment of the present invention; and FIG. 10 illustrates three-dimensionalizing defects based on stereoimaging technology according to an embodiment of the present invention. Figure 11 illustrates a large-scale deep learning HPC system according to an embodiment of the present invention.
[0029] Unlike surface defects such as cracks, indentations, and chemical byproducts, three-dimensional defects occurring in steel do not have a color difference between the defect area and the background and lack distinct boundaries, making it very difficult to detect them using image and 2D-based artificial intelligence algorithms.
[0030] According to an embodiment of the present invention, for defects that can be distinguished from and expressed three-dimensionally as surface defects of steel, such as deformation, breakage, and wear, two images are superimposed to extract features of the overlapping area, and after matching the two images based on these features, a stereo imaging technique is applied to create a point cloud based on the SfM (Structure from Motion) algorithm.
[0031] In cases where it is difficult to create a three-dimensional model due to the small degree of defects such as minute deformation, it is possible to improve accuracy by combining it with the detection results of a 2D-based artificial intelligence algorithm.
[0032] A steel quality inspection system based on explainable artificial intelligence and a large-scale deep learning HPC system according to an embodiment of the present invention includes a memory in which a steel quality inspection program based on explainable artificial intelligence and a large-scale deep learning HPC system is stored, and a processor that executes said program, wherein the processor acquires information on cracks and defects of steel through images, and detects and quantifies them based on artificial intelligence.
[0033] The above processor acquires images based on mobile stereo imaging.
[0034] The above processor detects the cracks and defects based on explainable artificial intelligence.
[0035] The above processor visualizes and diagrams the crack and defect information.
[0036] The above processor performs training of artificial intelligence algorithms for detection based on a large-scale deep learning HPC system.
[0037] The above processor improves Heatmap and Captioning performance for objects using a benchmark dataset and then transfers the weight matrix through transfer learning.
[0038] The above processor performs detection of cracks and defects contained in contaminated temporary construction materials using an artificial intelligence filter specialized for temporary construction materials.
Claims
Claim 1 A steel quality inspection system based on explainable artificial intelligence and a large-scale deep learning HPC system, comprising: a memory storing a steel quality inspection program based on explainable artificial intelligence and a large-scale deep learning HPC system; and a processor for executing said program, wherein the processor acquires information on cracks and defects of steel through images based on portable stereo imaging, detects and quantifies said cracks and defects based on explainable artificial intelligence, visualizes and draws said cracks and defects information, and wherein the processor extracts a bounding box for an input image and extracts a segmented mask of a shape and region different from said bounding box when analyzing said images using a convolutional neural network and explainable artificial intelligence, extracts a mask from which noise has been removed from said segmented mask using a multi-noise filter, and outputs information on area, width, and length as a result of detecting said cracks and defects of steel. Claim 2 A steel quality inspection system based on an explainable artificial intelligence and large-scale deep learning HPC system according to claim 1, wherein the processor generates a bounding box for a defect candidate region using a Region Proposal Network (RPN) based on a convolutional neural network for an input image, generates a pixel-unit segmented mask different from the shape of the bounding box using a Segmentation Neural Network (SegNet) for the region inside the bounding box, divides the segmented mask into patch-wise units, applies a Classification Neural Network to each patch to distinguish between defect patches and non-defect patches, generates a denoised mask with false detection regions removed by applying a Multiple Noise Filter considering multiple noise characteristics, and quantitatively calculates the area, width, and length of cracks and defects of the steel through a Quantification Neural Network (M-Net) using the denoised mask as input. Claim 3 In claim 1, the processor receives a 2D color image as input, detects candidate regions for cracks and defects in steel using a Convolutional Neural Network (CNN) including a Region Proposal Network (RPN) on the input image, generates a Heat Map and Captioning information serving as the basis for defect detection through an Explainable Artificial Intelligence (XAI) module linked to the CNN-based defect detection process, generates a segmented mask different from the bounding box shape by detecting defect boundaries pixel by pixel using a Segmentation Neural Network (SegNet) on the candidate defect regions, classifies the type of defect using a classification algorithm including a Fully Connected Layer (FCL) based on the segmented mask, removes noise caused by changes in the shooting environment, surface texture, and lighting by applying a Multi-layer Noise Filter to the classified defect mask, and a Feed Forward Neural Network that takes the noise-removed defect mask as input, A steel quality inspection system based on an explainable artificial intelligence and large-scale deep learning HPC system, characterized by measuring the width, length, and area of cracks and defects in steel using FFNN, and quantifying, visualizing, and drawing the measured defect data. Claim 4 In claim 1, the processor acquires color images of the same steel at multiple viewpoints using a portable stereo imaging device, extracts features including edges, boundaries, and points from the color images at multiple viewpoints using a feature extraction algorithm (SURF algorithm, SIFT algorithm), resizes the images to correct errors due to differences in resolution and viewpoint, performs matching between features, generates a 3D point cloud of the steel surface using a Structure from Motion (SfM) algorithm using the matching results, detects a three-dimensional defect region for cracks and defects in the steel based on the point cloud, refines the defect location and shape by mutually correcting the three-dimensional defect region detection result with the detection result of a Convolutional Neural Network (CNN) of the RPN family, which is a 2D-based defect detection algorithm, and calculates the width, length, and area of the cracks and defects in the steel in 3D using a Feed Forward Neural Network (FFNN) with the corrected defect region as input. A steel quality inspection system based on explainable artificial intelligence and large-scale deep learning HPC systems, characterized by measuring based on standards, quantifying measured defect data, and visualizing and drawing it. Claim 5 In claim 1, the steel quality inspection system based on explainable artificial intelligence and a large-scale deep learning HPC system, wherein the processor performs artificial intelligence algorithm training for detection based on a large-scale deep learning HPC system. Claim 6 A steel quality inspection system based on an explainable artificial intelligence and large-scale deep learning HPC system, wherein the processor improves Heatmap and Captioning performance for objects using a benchmark dataset and then transfers a weight matrix through transfer learning. Claim 7 A steel quality inspection system based on explainable artificial intelligence and large-scale deep learning HPC systems, wherein the processor performs detection of cracks and defects contained in contaminated temporary construction materials using an artificial intelligence filter specialized for temporary construction materials.
Citation Information
Patent Citations
Method for determining surface quality of strip
KR1020180123775A
Deep learning based classification system using image data augmentation, and cotrol method thereof
KR1020200084444A
Apparatus And Method for Detecting A Surface Defect Using A Deep Neural Network And Noise Reduction
KR1020190042384A
Systems and methods for visualizing and analyzing a rail surface
US20200164904A1
