Steel surface defect identification method based on FPGA and MnasNet

Abstract

The invention relates to the technical field of steel surface defect detection, and provides a steel surface defect identification method based on FPGA and MnasNet, and the method comprises the following steps: 1, capturing an image of a steel surface in real time by using an industrial camera; step 2, converting the color image into a grayscale image so as to reduce the calculation amount and the storage demand; step 3, removing noise in the image by using median filtering; 4, identifying brightness change in the image through edge detection in image processing, and calculating a gradient approximate value of image brightness; and 5, in image processing, further processing or analysis is carried out by selecting a specific area in the image by a user. By combining the deep learning technology with the FPGA hardware platform, efficient, accurate and real-time processing of steel surface defect identification is realized, a new thought and method are provided for quality control in the steel production process, and the processing speed and efficiency are also remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of steel surface defect detection, and in particular to a steel surface defect recognition method based on FPGA and MnasNet. Background Art

[0002] With the rapid development of modern industrial technology, steel, as an important material for infrastructure construction and industrial manufacturing, has a vital impact on the overall performance and reliability of the product. Surface defects of steel, such as cracks, scars, scratches, rust, etc., not only affect the appearance of the product, but may also significantly reduce its strength, corrosion resistance and fatigue life, thus posing a potential threat to safety in use. Therefore, how to efficiently and accurately identify defects on the surface of steel has become the focus of attention of steel companies and scientific research institutions.

[0003] Traditional methods for detecting surface defects of steel mainly include manual visual inspection and non-destructive testing technology based on machine vision. Although manual visual inspection is flexible, it has disadvantages such as low efficiency, easy fatigue, and strong subjectivity, and it is difficult to adapt to the fast pace and high-precision requirements of modern industrial production. Traditional machine vision technology, such as laser scanning and CCD detection methods, has improved detection efficiency and accuracy to a certain extent, but its complex equipment, high cost, and susceptibility to environmental interference have limited its widespread application.

[0004] In recent years, with the rapid development of deep learning technology, especially the successful application of MnasNet in the field of image recognition, a new solution has been provided for the recognition of steel surface defects. With its powerful feature extraction and pattern recognition capabilities, MnasNet can accurately identify tiny defects on the surface of steel under complex backgrounds, and has the advantages of fast detection speed, high accuracy, and high degree of automation. However, when processing large-scale data, CNN models based on CPU or GPU often face problems such as high computing resource consumption and poor real-time performance.

[0005] In response to the above challenges, a steel surface defect recognition method based on FPGA and MnasNet is proposed, combining the advantages of field programmable gate array (FPGA) technology. As a high-performance programmable logic device, FPGA has the advantages of strong parallel processing capability, low power consumption, and good real-time performance, which can significantly accelerate the calculation process of the MnasNet model. By deploying the MnasNet model on FPGA, real-time, efficient, and accurate recognition of steel surface defects can be achieved.

[0006] The steel surface defect recognition method based on FPGA and MnasNet first uses the MnasNet model to train a large number of steel surface defect images and extract the feature representation of the defects. Then, the trained MnasNet model is converted into hardware description language (HDL) code executable by FPGA, and the parallel computing of the model is realized through FPGA. In practical applications, images of the steel surface are collected in real time through a camera or image acquisition card, and input into the MnasNet model in the FPGA for processing. FPGA uses its parallel computing capability to quickly complete tasks such as image preprocessing, feature extraction and defect recognition, and feeds back the recognition results to the control system or operator in real time. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a steel surface defect recognition method based on FPGA and MnasNet, aiming to improve the accuracy and efficiency of industrial detection. The method combines the parallel computing capability of FPGA and the deep learning feature extraction capability of MnasNet to achieve real-time automatic recognition and classification of steel surface defects. Through FPGA hardware acceleration, large-scale steel surface image data streams can be processed to quickly and accurately analyze various defect types, including cracks, pits, bubbles, etc. This method that integrates deep learning and hardware acceleration not only improves the speed and accuracy of detection, but also enhances the reliability and application scope of the system, and is suitable for real-time quality control and defect analysis tasks on industrial production lines.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: A steel surface defect recognition method based on FPGA and MnasNet, comprising the following steps:

[0009] Step 1: Use an industrial camera to capture images of the steel surface in real time;

[0010] Step 2: Convert the color image to a grayscale image to reduce the amount of computation and storage requirements;

[0011] Step 3: Use median filtering to remove noise from the image;

[0012] Step 4: In image processing, edge detection is used to identify brightness changes in the image and calculate the gradient approximation of the image brightness;

[0013] Step 5: In image processing, a specific area in the image is selected by the user for further processing or analysis;

[0014] Step 6: Normalize the image pixel values ​​to a specific range to meet the input requirements of the MnasNet model;

[0015] Step 7: Use MnasNet to detect and recognize the processed image and accelerate it through FPGA.

[0016] Preferably, the grayscale formula in step 2 is:

[0017] l gray =0.299·R+0.587·G+0.114·B

[0018] Among them, R, G, and B represent the red, green, and blue components of each pixel in the image, respectively. gray is the grayscale value after conversion.

[0019] Preferably, the median filtering principle formula in step three is:

[0020] y[n]=median(x[nk],...,x[n+k])

[0021] Among them, x[n] is the original signal, y[n] is the filtered signal, n is the current position, k is half of the window size, and the median(.) function represents the median operation.

[0022] Preferably, the specific method of edge detection in step 4 is:

[0023] Horizontal direction:

[0024] Vertical direction:

[0025] Preferably, in step 6, the image pixel value is normalized using the formula:

[0026]

[0027] Among them, l nom is the normalized pixel value, I is the original pixel value, l min and l max are the minimum and maximum pixel values ​​in the image, respectively.

[0028] Preferably, in step seven, MnasNet detects and identifies the processed image:

[0029] During the forward propagation process, the preprocessed image data first enters the input layer of MnasNet, and then passes through the convolution layer using the depth-separable convolution technique to reduce the number of parameters and the amount of computation, while maintaining the ability to extract image features, extracting the high-level semantic features of the image, and mapping it to the predetermined category space through the fully connected layer (often converted to a global average pooling layer in classification tasks to reduce parameters), and outputting the confidence score for each category. After the convolution layer, a pooling layer is usually followed to reduce the spatial dimension of the feature map, further reduce the amount of computation and introduce a certain degree of translation invariance. Between the convolution and pooling layers, a nonlinear activation function may also be inserted to enhance the nonlinear expression ability of the network.

[0030] Preferably, the convolution layer is a core component in CNN, and features are extracted from input data through convolution operation. The purpose of convolution operation is to extract local features of input data. The basic formula of convolution operation can be expressed as:

[0031]

[0032] Among them, Output(i,j) is the value of position (i,j) on the output feature map, Input(i+m,j+n) is the value of the corresponding position on the input feature map, and Kernel(m,n) is the weight of position (m,n) on the convolution kernel.

[0033] Preferably, the pooling layer is mainly used to reduce the dimension of the feature map while retaining important feature information. The window is set to slide on the feature map and the data in the window is aggregated to reduce the size of the feature map. For each sub-region in the pooling window, the maximum value in the region is taken as the output.

[0034] Preferably, the fully connected layer connects all neurons of the previous layer to each neuron of the current layer to achieve feature integration and classification. The formula is:

[0035] y i =f(∑ j w ij x j +b i )

[0036] Among them, y i is the output of the i-th neuron in the current layer, x j is the output of the jth neuron in the previous layer, w ij is the weight between the i-th and j-th neurons in the two layers, b i is the bias term and f is the activation function.

[0037] The present invention provides a steel surface defect recognition method based on FPGA and MnasNet. It has the following beneficial effects:

[0038] 1. This invention creatively combines the efficient deep learning model MnasNet with the FPGA hardware platform to achieve high precision and real-time steel surface defect recognition. This combination not only takes advantage of MnasNet's advantages in complex feature extraction, but also makes full use of the high parallel processing capability and low latency characteristics of FPGA, providing a new solution for real-time quality control.

[0039] 2. By converting the trained MnasNet model into HDL code executable by FPGA, the present invention realizes the seamless conversion of deep learning models from software to hardware. This process not only retains the high accuracy of the model, but also significantly improves the processing speed and efficiency, allowing the model to achieve instant response in the actual production environment.

[0040] 3. The present invention can capture high-definition images of the steel surface in real time by integrating hardware devices such as cameras or image acquisition cards, and directly perform image preprocessing, feature extraction and defect identification in FPGA. This end-to-end real-time processing flow greatly shortens the response time of defect identification, making it possible for rapid response and timely adjustment of the production line.

[0041] 4. The application of the FPGA-based MnasNet model in steel surface defect recognition significantly improves the automation level and quality control level of the production line. By instantly feeding back defect information to the control system or operator, corresponding measures can be taken quickly to reduce the generation of defective products and improve product quality and production efficiency.

[0042] 5. The reconfigurability of the FPGA of the present invention makes the system of the present invention have good flexibility and scalability. With the continuous changes in the requirements for steel surface defect recognition and the continuous optimization of deep learning models, the FPGA can be reconfigured to adapt to new models and algorithms without replacing hardware equipment, reducing upgrade costs and maintenance difficulties.

[0043] 6. The present invention combines deep learning technology with the FPGA hardware platform to achieve efficient, accurate and real-time processing of steel surface defect identification, providing new ideas and methods for quality control in the steel production process. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The steel surface defect image of the present invention;

[0045] Figure 2 The MnasNet network structure of the present invention;

[0046] Figure 3 is the search space of MNasNet of the present invention;

[0047] Figure 4 are the model weights and key parameters of the present invention;

[0048] Figure 5 It is the overall module architecture of the present invention;

[0049] Figure 6 The crack of the present invention;

[0050] Figure 7 The interlayer of the present invention

[0051] Figure 8 It is the cross-shaped segregation of the present invention;

[0052] Fig. 9 The medical image of the present invention is rotated 60° counterclockwise;

[0053] Fig.10 The central crack of the present invention

[0054] Fig.11 The strip-shaped surface interlayer of the present invention;

[0055] Fig.12 The PL acceleration module simulation 1 of the present invention;

[0056] Fig.13 It is the PL acceleration module simulation 2 of the present invention;

[0057] Fig.14 is a power consumption diagram of the neural network accelerator of the present invention;

[0058] Fig.15 It is the PS end processing time of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The embodiment of the present invention provides a steel surface defect recognition method based on FPGA and MnasNet, comprising the following steps:

[0061] Step 1: Use an industrial camera to capture images of the steel surface in real time;

[0062] Step 2: Convert the color image to a grayscale image to reduce the amount of calculation and storage requirements. The grayscale formula is:

[0063] lgray =0.299·R+0.587·G+0.114·B

[0064] Among them, R, G, and B represent the red, green, and blue components of each pixel in the image, respectively. gray is the grayscale value after conversion;

[0065] Step 3: Use median filtering to remove noise from the image. The median filtering principle formula is:

[0066] y[n]=median(x[nk|,...,x[n+k])

[0067] Where x[n] is the original signal, y[n] is the filtered signal, n is the current position, k is half the window size, and the median(.) function represents the median operation;

[0068] Step 4: In image processing, edge detection is used to identify brightness changes in the image and calculate the gradient approximation of the image brightness. The specific method of edge detection is:

[0069] Horizontal direction:

[0070] Vertical direction:

[0071] Step 5: In image processing, a specific area in the image is selected by the user for further processing or analysis;

[0072] Step 6: Normalize the image pixel values ​​to a specific range to meet the input requirements of the MnasNet model. The image pixel value normalization formula is:

[0073]

[0074] Among them, l nom is the normalized pixel value, I is the original pixel value, l min and l max are the minimum and maximum pixel values ​​in the image, respectively;

[0075] Step 7: Used in the forward propagation process, the preprocessed image data first enters the input layer of MnasNet, and then uses the depth-separable convolution technology through the convolution layer to reduce the number of parameters and the amount of calculation, while maintaining the ability to extract image features, extracting the high-level semantic features of the image, and mapping it to the predetermined category space through the fully connected layer (often converted to a global average pooling layer in classification tasks to reduce parameters), and outputting the confidence score of each category. After the convolution layer, it is usually followed by a pooling layer to reduce the spatial dimension of the feature map, further reduce the amount of calculation and introduce a certain degree of translation invariance. Between the convolution and pooling layers, nonlinear activation functions may also be inserted to enhance the nonlinear expression ability of the network, and accelerate it through FPGA.

[0076] The convolution layer is the core component of CNN, which extracts features from input data through convolution operation. The purpose of convolution operation is to extract local features of input data. The basic formula of convolution operation can be expressed as:

[0077]

[0078] Among them, Output(i, j) is the value of position (i, j) on the output feature map, Input(i+m, j+n) is the value of the corresponding position on the input feature map, and Kernel(m, n) is the weight of position (m, n) on the convolution kernel.

[0079] The pooling layer is mainly used to reduce the dimension of the feature map while retaining important feature information. By setting the window to slide on the feature map, the data in the window is aggregated to reduce the size of the feature map. For each sub-region in the pooling window, the maximum value in the region is taken as the output.

[0080] The fully connected layer connects all neurons in the previous layer to each neuron in the current layer to achieve feature integration and classification. The formula is:

[0081] y i =(∑ j w ij x j +b i )

[0082] Among them, y i is the output of the i-th neuron in the current layer, x j is the output of the jth neuron in the previous layer, w ij is the weight between the i-th and j-th neurons in the two layers, b i is the bias term, and f is the activation function

[0083] Example:

[0084] The technical details of the present invention are analyzed and described in detail, especially the process of model training and optimization in Python environment using MobileNetV3 derivative architecture - MNasNet in the efficient neural network architecture search (NAS) technology for 512×512 resolution steel surface defect detection task. This process not only demonstrates the outstanding application potential of deep learning in the field of industrial automation, but also integrates cutting-edge search space construction strategies to achieve high-precision (over 90%) defect recognition performance.

[0085] This experiment uses a dataset containing steel surface defect images with a resolution of 512×512 pixels as the object, such as Figure 1 As shown in the figure, Python and its powerful data processing libraries (such as NumPy and Pandas) are used for efficient data loading and preprocessing. The preprocessing steps may include image normalization, enhancement (such as rotation, scaling, noise addition), etc. to enhance the generalization ability of the model.

[0086] As a variant of MobileNetV3, MNasNet introduces the Factorized Hierarchical Search Space mechanism. The network structure is as follows Figure 2 As shown in the figure, efficient network design is achieved under resource constraints (such as edge devices). This architecture is designed for mobile and embedded vision tasks, and achieves excellent performance in steel defect detection tasks by balancing accuracy and computational complexity.

[0087] like Figure 3 As shown in Figure 1, the search space construction process of MNasNet is highly dependent on the ingenious combination of automated machine learning and manual design. By dividing the network into multiple modules (blocks), each module contains multiple repeated layers, and independently performing structural searches for each layer, such as convolution kernel size, expansion ratio, etc., the optimal architecture configuration is explored. In particular, downsampling between modules is achieved by adjusting the step size of the first layer to 2, keeping the step size of subsequent layers to 1, and retaining the identity mapping between layers to enhance information flow and gradient transfer.

[0088] MNasNet is trained using a deep learning framework (such as TensorFlow or PyTorch) on the Python platform. During the training process, the cross entropy loss function is used to evaluate the difference between the model prediction and the true label, and the network weights are adjusted with the help of optimization algorithms (such as Adam or SGD) to minimize the loss function value. Through multiple iterations of training, the model gradually learns the complex feature representation of steel surface defects and ultimately achieves high-precision defect recognition.

[0089] After training is completed, the model weights and key parameters (such as Figure 4 The model can be saved to the PC for subsequent model deployment on low-power hardware such as FPGA.

[0090] In the modular design framework on the FPGA side, we built a highly integrated and efficient system with the overall module architecture as follows: Figure 5 As shown in the figure, it is carefully divided into multiple key sub-modules to realize the complete processing flow from image capture to final display. These modules not only reflect the advantages of FPGA in parallel processing and real-time performance, but also integrate the latest progress in image processing and deep learning technology.

[0091] Image acquisition module: This module integrates high-speed interfaces and advanced signal processing technology, and is responsible for capturing image data from industrial-grade high-resolution cameras in real time. Through precise timing control and data synchronization mechanisms, it ensures the integrity and accuracy of image data, laying a solid foundation for subsequent processing.

[0092] Image preprocessing module: This module integrates a variety of advanced image processing algorithms, including but not limited to median filtering (used to remove image noise), Canny edge detection (accurately extract edge information in images), and morphological operations (such as dilation algorithms, used to enhance edge features and fill small holes). When these algorithms are implemented on FPGA, they achieve a low-latency, high-efficiency image preprocessing process by optimizing hardware resource allocation and parallel processing strategies.

[0093] MnasNet neural network module: As the core of the system, the MnasNet neural network module is highly customized and deployed on FPGA. By utilizing the lightweight and efficient characteristics of the MnasNet architecture and combining the parallel computing advantages of FPGA, this module can achieve rapid feature extraction and defect recognition of pre-processed images. Through the trained model weights and parameters, the module can accurately identify various defects on the steel surface in real time.

[0094] LCD display module: Finally, the LCD display module is responsible for displaying the processed image and recognition results to the user intuitively. The module is connected to the FPGA through a high-speed data interface, receives the processed image data, and displays it in high definition on the LCD screen. At the same time, the module also supports multiple display modes and interactive functions to meet the needs of use in different scenarios.

[0095] After the above hardware and software overall implementation plan Figure 6 As shown in the figure, the pictures collected by the industrial camera will go through advanced image processing steps such as median filtering, Canny edge detection, morphological dilation algorithm and possible image cropping, and finally generate the following Figures 7 to 11 Clear, accurate images shown.

[0096] After the module construction was completed, we immediately entered the detailed simulation verification phase. This key step aims to accurately evaluate the performance and reliability of each module through advanced simulation tools. Figure 12 to Figure 13 As shown, we used a high-performance simulation environment to conduct comprehensive simulation tests on the PL (programmable logic) acceleration module to ensure the feasibility of the experimental design and the consistency of the expected results.

[0097] During the simulation process, we paid special attention to the performance of the PL acceleration module in processing complex data streams and computationally intensive tasks, and verified its acceleration effect in key links such as image preprocessing, feature extraction, and neural network reasoning by simulating real-world operation scenarios. This simulation not only tests the design optimization of the module, but also conducts a rigorous evaluation of its stability and efficiency in actual applications.

[0098] Fig.14 It intuitively shows the power consumption performance of the neural network accelerator during the simulation process. Through accurate power consumption measurement and analysis, we can deeply understand the energy consumption characteristics of the accelerator under different workloads. This analysis is not only crucial for optimizing accelerator design and improving energy efficiency, but also provides valuable data support for subsequent system-level power management strategies.

[0099] We compared the performance of the neural network model accelerated by FPGA and the network model based on the PS (processor subsystem) when performing the same task, such as Fig.15 This comparative experiment not only reveals the great potential of FPGA in accelerating deep learning reasoning, but also intuitively demonstrates the significant performance improvement brought by FPGA acceleration strategy through quantitative data.

[0100] Specifically, experimental results show that the neural network model accelerated by FPGA achieves more than three times the speed of the traditional PS-side implementation. This extraordinary acceleration ratio is due to the parallel processing architecture and customized design of FPGA, which allows large amounts of data to be processed simultaneously and effectively reduces data movement and storage delays, thereby greatly improving computing efficiency.

[0101] Further analysis shows that FPGA-accelerated neural networks have shown outstanding advantages in real-time, throughput, and energy efficiency. Its highly parallel computing units can quickly complete complex matrix operations and convolution operations, which is particularly important for processing large-scale image data or performing high-frame rate video analysis tasks. At the same time, FPGA's power management strategy also helps to reduce the overall energy consumption of the system and extend the battery life of the device, and has broad application prospects in edge computing and IoT applications.

[0102] In summary, through this experimental comparison, we not only verified the excellent ability of FPGA in accelerating neural network reasoning, but also provided strong data support for its application in high-performance computing and real-time processing. This achievement not only promotes the further development of deep learning technology, but also provides valuable reference and inspiration for the design and implementation of future intelligent systems.

[0103] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A steel surface defect recognition method based on FPGA and MnasNet, characterized in that: The following steps are involved: Step 1: Use an industrial camera to capture images of the steel surface in real time; Step 2: Convert the color image to a grayscale image to reduce the amount of computation and storage requirements; Step 3: Use median filtering to remove noise from the image; Step 4: In image processing, edge detection is used to identify brightness changes in the image and calculate the gradient approximation of the image brightness; Step 5: In image processing, a specific area in the image is selected by the user for further processing or analysis; Step 6: Normalize the image pixel values ​​to a specific range to meet the input requirements of the MnasNet model; Step 7: Use MnasNet to detect and recognize the processed image and accelerate it through FPGA.

2. The steel surface defect recognition method based on FPGA and MnasNet according to claim 1 is characterized in that the grayscale formula in step 2 is: l gray =0.299·R+0.587·G+0.114·B in, R, G, and B represent the red, green, and blue components of each pixel in the image, respectively. gray is the grayscale value after conversion.

3. The steel surface defect recognition method based on FPGA and MnasNet according to claim 1 is characterized in that: The median filtering principle formula in step 3 is: y[n]=median(x[nk],...,x[n+k]) Among them, x[n] is the original signal, y[n] is the filtered signal, n is the current position, k is half of the window size, and the median(.) function represents the median operation.

4. The steel surface defect recognition method based on FPGA and MnasNet according to claim 1 is characterized in that: The specific method of edge detection in step 4 is: Horizontal direction: Vertical direction:

5. The method for identifying steel surface defects based on FPGA and MnasNet according to claim 1, characterized in that: In step 6, the image pixel value is normalized using the formula: Among them, l nom is the normalized pixel value, I is the original pixel value, l min and l max are the minimum and maximum pixel values ​​in the image, respectively.

6. The method for identifying steel surface defects based on FPGA and MnasNet according to claim 1, characterized in that: In step 7, MnasNet detects and identifies the processed image: During the forward propagation process, the preprocessed image data first enters the input layer of MnasNet, and then passes through the convolution layer using the depth-separable convolution technique to reduce the number of parameters and the amount of computation, while maintaining the ability to extract image features, extracting the high-level semantic features of the image, and mapping it to the predetermined category space through the fully connected layer (often converted to a global average pooling layer in classification tasks to reduce parameters), and outputting the confidence score for each category. After the convolution layer, a pooling layer is usually followed to reduce the spatial dimension of the feature map, further reduce the amount of computation and introduce a certain degree of translation invariance. Between the convolution and pooling layers, a nonlinear activation function may also be inserted to enhance the nonlinear expression ability of the network.

7. The method for identifying steel surface defects based on FPGA and MnasNet according to claim 6, characterized in that: The convolution layer is the core component of CNN, which extracts features from input data through convolution operation. The purpose of convolution operation is to extract local features of input data. The basic formula of convolution operation can be expressed as: Among them, Output(i,j) is the value of position (i,j) on the output feature map, Input(i+m,j+n) is the value of the corresponding position on the input feature map, and Kernel(m,n) is the weight of position (m,n) on the convolution kernel.

8. The method for identifying steel surface defects based on FPGA and MnasNet according to claim 6, characterized in that: The pooling layer is mainly used to reduce the dimension of the feature map while retaining important feature information. The window is set to slide on the feature map and the data in the window is aggregated to reduce the size of the feature map. For each sub-region in the pooling window, the maximum value in the region is taken as the output.

9. The method for identifying steel surface defects based on FPGA and MnasNet according to claim 6, characterized in that: The fully connected layer connects all neurons in the previous layer to each neuron in the current layer to achieve feature integration and classification. The formula is: yi=f(∑j wijxj+bi) Among them, y i is the output of the i-th neuron in the current layer, x j is the output of the jth neuron in the previous layer, w ij is the weight between the i-th and j-th neurons in the two layers, b i is the bias term and f is the activation function.

Images