A method and application for detecting surface defects in steel based on the vision of an intelligent vehicle

Abstract

This invention discloses a method and application for steel surface defect detection based on intelligent vehicle vision, belonging to the field of machine vision inspection. Through path planning, the intelligent vehicle automatically reaches a designated detection area, collects defect images from multiple angles and under different lighting conditions, constructs and enhances a dataset, trains it with an improved YolovV5 model, generates a pt format model, converts it to ONNX format, and transmits it to an embedded mobile device. Real-time defect detection is achieved using ONNXRuntime and OpenCV. Detection results are uploaded to a host computer for remote monitoring and data storage, ensuring efficient and automated detection and management of steel defects. By optimizing the YolovV5 model to a lightweight MobileNetV3 network and introducing the SE attention mechanism, the computational complexity and number of parameters of the model are significantly reduced, and the inference speed is improved. This enables embedded mobile devices to efficiently complete real-time detection of steel surface defects, solving the problems of low frame rate and inability to detect in real time caused by the large computational load of traditional methods.

Description

Technical Field

[0001] This invention relates to the field of machine vision inspection, specifically to a method and application for detecting surface defects in steel based on the vision of an intelligent vehicle. Background Technology

[0002] The steel industry is an important part of the manufacturing industry. Improving product quality is very important for steel companies. However, in the steel quality inspection process, traditional manual inspection is difficult to detect by human eyes. Furthermore, it is slow, inaccurate, and costly. Therefore, using machines to replace manual inspection has become a trend.

[0003] For the detection of surface defects in steel, existing technologies mainly include the following methods: "Zhu Hongping proposed a surface defect detection algorithm for strip steel based on convolutional neural networks in "A Method for Detecting Steel Surface Defects Based on Convolutional Neural Networks" [J]. Industrial Control Computer, 2020, 033(8): 83-84. This algorithm enables the judgment of surface defects in strip steel. "Shi Weijie et al. compared the detection methods of steel image defects based on two U-shaped networks" [J]. Journal of Beijing Information Science and Technology University (Natural Science Edition), 2021, 036(1): 63-68. They used U-Net and Res-UNet to detect defects in steel images. The results showed that Res-UNet was more accurate in identifying the location and category of defects. However, the above methods failed to combine defect classification and localization, and could not obtain the defect category and location at the same time, which limited their application in industrial production. To solve this problem, many researchers have proposed improved schemes based on the YOLO algorithm to achieve dual detection of defect classification and localization.

[0004] To this end, many domestic researchers have proposed improved schemes based on the YOLO algorithm, aiming to simultaneously classify and detect surface defects in steel. Fang Yexiang et al., in “Research on Improved YOLOv3 Algorithm for Metal Surface Defect Detection” [J]. Mechanical Science and Technology 39.9(2020): 5., adopted an improved YOLOv3 algorithm, mainly through histogram equalization for image preprocessing, data augmentation to simulate the real working environment, and optimization of the loss function to improve the accuracy of surface defect classification; Zhang Shuguang et al., in “Improved Tiny-YOLOv3 Algorithm for Industrial Steel Defect Detection” [J]. Mechanical Design and Manufacturing, 2024(5): 97-10. To improve the speed of steel defect detection, the R-Tiny-YOLOv3 algorithm based on Tiny-YOLOv3 was proposed in 1.”. This algorithm incorporates a residual network and a spatial pyramid pooling (SPP) module, and finally selects CIOU as the loss function. In addition, patent CN115294089A discloses a steel surface defect detection method based on an improved YolovV5 algorithm. This method improves the original YolovV5 algorithm for different types of defects that may appear on the steel surface, thereby improving detection accuracy and efficiency.

[0005] However, for steel surface defect detection, the models trained using the above-mentioned related technical solutions are often large in size and cannot be mounted on embedded mobile devices with limited resources and weak computing power, thus failing to meet the requirements of real-time detection. Therefore, this invention provides a steel surface defect detection method based on intelligent vehicle vision, which ensures high defect recognition accuracy. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and application for detecting steel surface defects based on the vision of an intelligent vehicle, which solves the technical problem of low frame rate and inability to achieve real-time detection when a neural network model is mounted on a resource-constrained embedded mobile device.

[0007] To achieve the above objectives, the following technical solutions can be used:

[0008] This invention provides a method for detecting surface defects in steel based on the vision of an intelligent vehicle, comprising the following steps:

[0009] S1. Intelligent vehicle path planning and dynamic navigation: Real-time environmental information is collected by the lidar sensor on the vehicle. Combined with the improved dynamic window path planning algorithm, the vehicle's travel path is planned, and multi-sensor data fusion is used to achieve accurate positioning and obstacle avoidance.

[0010] S2. Defect Image Acquisition and Multimodal Dataset Construction: During the vehicle's movement, the embedded mobile device acquires images of steel surface defects from different angles and under different lighting conditions via a camera. At the same time, it acquires surface temperature and texture information to construct a multimodal dataset of surface defect images. The images are then labeled, classified, and divided into training and testing sets.

[0011] S3, Data Augmentation and Lightweight Model Training: Data augmentation is performed on images in multimodal data, and simulations of lighting and pollution in an industrial environment are added. The images are then input into the improved YolovV5 model for lightweight training to generate a model suitable for embedded mobile devices.

[0012] S4. Model Conversion and Real-time Detection: The lightweight model is loaded through the ONNXRuntime framework to perform inference on the real-time images captured by the camera, automatically identify the type and location of defects on the steel surface, and upload them to the host computer.

[0013] S5. Host computer remote monitoring and data storage: The host computer monitors the test results in real time and automatically generates maintenance suggestions based on the defect type. At the same time, the test data is stored in a local database for subsequent analysis.

[0014] As a further aspect of the present invention, the specific method steps of step S1 are as follows:

[0015] S11. Activate the lidar sensor on the car to collect planar point cloud data of the surrounding environment in real time, and combine it with the visual information collected by the camera to perform multimodal environment modeling.

[0016] S12. A path planning algorithm based on the dynamic window method is adopted, combined with multi-sensor fusion technology including LiDAR, inertial navigation and visual sensors, to generate a dynamic path that adapts to changes in obstacles.

[0017] S13. Use SLAM technology and Gmapping algorithm to generate an environment map, and visualize the point cloud data using the rviz tool to monitor the vehicle's positioning and mapping effect, set the start and end points, and achieve automatic navigation.

[0018] As a further aspect of the present invention, the specific method steps of step S2 are as follows:

[0019] S21. Use an embedded mobile device to connect a USB camera to collect multi-angle, multi-light images of steel surface defects, and combine this with an infrared temperature sensor to obtain surface temperature information to generate a multimodal dataset.

[0020] S22. Use LabelImg software to manually annotate defects in image data. Defect types include six categories: cracks, inclusions, pits, patches, scratches, and rolled scales. At the same time, record the temperature and texture information corresponding to the image.

[0021] S23. Based on the collection environment and task requirements, the multimodal data is divided into training and testing sets in a 4:1 ratio, and the industrial environment is simulated based on changes in illumination or surface contamination conditions to expand the complexity of the dataset.

[0022] As a further aspect of the present invention, in step S3, the improved method of the YolovV5 model is to replace the original backbone network with a lightweight MobileNetV3 network and add an SE attention mechanism during the training phase. The specific steps are as follows:

[0023] S31. Perform image enhancement processing on the images in the pre-training dataset, including scaling, rotation, and HSV color transformation operations. Simultaneously, incorporate lighting and dirt simulations under industrial conditions. The processed images are randomly cropped and stitched together to generate more diverse training samples.

[0024] S32. The enhanced image is processed by combining depthwise separable convolution, improved Hard-Swish activation function and attention module of MobileNetV3 network to extract features;

[0025] S33. Enhance the extracted feature map through the SE module to improve the detection capability of important features. The specific steps are as follows: First, perform global average pooling or global max pooling on each channel of the feature map to compress the spatial information into a single value representation and capture global context information; then, generate importance weights for each channel through fully connected layers and non-linear activation functions, thereby adjusting the weights of each channel in the feature map so that the model focuses more on important features.

[0026] S34. During the model training phase, specific industrial scene noise simulations are added, and then the model size is further optimized through global average pooling to adapt to embedded mobile devices. The results are then input into the classifier for prediction, and the final results are output.

[0027] As a further aspect of the present invention, in step S32, the improved Hard-Swish activation function is obtained by modifying the ReLU6 activation function. It not only retains the simplicity of ReLU6 but also significantly reduces the detection of minute defects during the input range by providing a smooth transition. The modification principle is as follows:

[0028] RELU6(x) = min(max(0,x),6);

[0029]

[0030] As a further aspect of the present invention, the specific method steps of step S4 are as follows:

[0031] S41. Use the ONNXRuntime framework to convert the pt format model file to ONNX format and upload the ONNX file to the embedded mobile device;

[0032] S42. Preprocess the real-time images captured by the camera using OpenCV, including erosion, dilation, and edge detection;

[0033] S43. Use ONNXRuntime to load the ONNX model file uploaded to the embedded system, and input the preprocessed image into the model for inference, outputting the classification results and confidence scores of steel surface defects.

[0034] S44. Perform post-processing of defect detection based on the classification results output by the model, mark the defects in the image and display the classification information, generate repair suggestions, and upload the detection results to the host computer or cloud in real time.

[0035] This invention also provides an application of a steel surface defect detection method based on intelligent vehicle vision. The above-described steel surface defect detection method is used in the steel production and processing process to automate the detection of steel surface defects. The specific steps are as follows:

[0036] T1. Train the improved YolovV5 model and generate a pt format file. Use ONNXRuntime to convert the model to ONNX format for inference on the Raspberry Pi. Use MobaXterm software to upload the ONNX file to the Raspberry Pi.

[0037] T2. Use OpenCV for image preprocessing, including erosion, dilation, and thresholding. Then, load the ONNX model through ONNXRuntime for real-time inference, label the defect location and category, and upload the results to the host computer in real time.

[0038] T3. Log in to the Raspberry Pi Ubuntu system, set the account and password for remote connection to the VNC service, remotely connect the local host to the Raspberry Pi Ubuntu system, and enter the command line terminal;

[0039] T4. Run the LiDAR Python script in the Raspberry Pi terminal to start the LiDAR to scan the surrounding environment and generate a map. After the car moves to the designated position, turn on the camera to detect the surface defects of the steel in real time and save the detected defect types and locations to the Raspberry Pi's SD card.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] 1. This invention optimizes the YolovV5 model into a lightweight MobileNetV3 network and combines it with the SE attention mechanism, significantly reducing computational complexity and parameter count while improving inference speed. It enables efficient real-time detection of steel surface defects in embedded devices, overcoming the problems of low frame rates and inability to perform real-time detection due to high computational demands in traditional methods.

[0042] 2. This invention, through an improved YolovV5 model, integrates the MobileNetV3 network and the SE module, enhancing the model's ability to identify key features and significantly improving the accuracy of defect classification and location. It can still accurately identify steel surface defects under different lighting and angle conditions, meeting the high quality requirements of industrial production.

[0043] 3. This invention optimizes computational efficiency by converting the trained deep learning model into ONNX format and using the ONNXRuntime framework for inference, enabling the model to run smoothly in embedded mobile devices. It also combines OpenCV for image preprocessing to meet the industrial production requirements for low power consumption and real-time performance.

[0044] 4. Through data augmentation technology and model improvement, this invention enables the detection model to have higher robustness and adaptability, and can operate stably under various complex production conditions (such as changes in light intensity, surface contamination, etc.), ensuring the consistency and reliability of detection results.

[0045] 5. This invention integrates remote monitoring functionality, allowing operators to view test results in real time via a host computer, enabling effective monitoring and management of the defect detection process. Test data can also be stored locally or in the cloud for convenient subsequent data retrieval, analysis, and production optimization. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the method for implementing the present invention.

[0047] Figure 2 These are images of a dataset of steel surface defects collected in this invention.

[0048] Figure 3 This invention involves labeling and classifying images in a dataset.

[0049] Figure 4 This is a diagram showing the effect of steel surface defect detection according to the present invention.

[0050] Figure 5 This is a comparison chart of the recognition accuracy before and after the improvement of the YolovV5 model in this invention.

[0051] Figure 6This is a comparison chart of various performance indicators of the improved YolovV5 model of this invention.

[0052] Figure 7 This is a frame rate comparison chart of the present invention running on a Raspberry Pi.

[0053] Figure 8 This is a diagram of the remote login terminal for the Raspberry Pi according to the present invention. Detailed Implementation

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Example 1:

[0056] like Figure 1-6 As shown in the figure, an embodiment of the present invention provides a method for detecting surface defects in steel based on the vision of an intelligent vehicle, comprising the following steps:

[0057] S1. Intelligent vehicle path planning and positioning: Real-time environmental information is collected by the lidar sensor on the vehicle. Combined with the improved dynamic window path planning algorithm, the vehicle's travel path is planned, and multi-sensor data fusion is used to achieve accurate positioning and obstacle avoidance.

[0058] S2. Defect Image Acquisition and Dataset Construction: During the movement of the vehicle, the embedded device acquires images of steel surface defects under different angles and lighting conditions through a camera. At the same time, it acquires surface temperature and texture information to construct a multimodal dataset of surface defect images. After the images are labeled and classified, they are divided into training and test sets.

[0059] S3, Data Augmentation and Model Training: Data augmentation is performed on images in multimodal data, adding simulations of lighting and pollution in an industrial environment, and then inputting the improved YolovV5 model for lightweight training to generate a model suitable for embedded devices.

[0060] S4. Model Conversion and Real-time Detection: The lightweight model is loaded through the ONNXRuntime framework to perform inference on the real-time images captured by the camera, automatically identify the type and location of defects on the steel surface, and upload them to the host computer.

[0061] S5. Host computer remote monitoring and data storage: The host computer monitors the test results in real time and automatically generates maintenance suggestions based on the defect type. At the same time, the test data is stored in a local database for subsequent analysis.

[0062] In this embodiment, the specific method steps of step S1 are as follows:

[0063] S11. Activate the lidar sensor on the vehicle to collect planar point cloud data of the surrounding environment in real time, and combine it with visual information collected by the camera to perform multimodal environment modeling. By activating the lidar to collect point cloud data and combining it with camera visual information for multimodal environment modeling, the perception capability of the surrounding environment is significantly improved. Multimodal data fusion compensates for the lack of detailed information from a single sensor, making the modeling more comprehensive and accurate, while also enhancing the system's adaptability to complex environments.

[0064] S12. A path planning algorithm based on the dynamic window method is adopted, combined with multi-sensor fusion technology including LiDAR, inertial navigation, and visual sensors, to generate a dynamic path that adapts to obstacle changes. This method ensures the flexibility and reliability of path planning, stably responding to obstacle changes even in dynamic environments, and providing efficient navigation support.

[0065] S13. SLAM (Simultaneous Localization and Mapping) technology and the Gmapping algorithm are used to generate an environmental map. The point cloud data is then visualized using the rviz tool to monitor the vehicle's localization and mapping performance. Starting and ending points are set to enable automatic navigation. This allows the vehicle to automatically complete navigation tasks, while operators can make real-time adjustments to ensure accurate mapping and efficient task execution, laying the foundation for long-term operation in complex environments.

[0066] In this embodiment, as Figure 2-4 As shown, the specific steps of step S2 are as follows:

[0067] S21. Connect a USB camera to a Raspberry Pi, start the camera, and acquire images of defects on the steel surface in real time. During the acquisition process, ensure that images are taken under different types, angles, and lighting conditions to guarantee the diversity and comprehensiveness of the dataset. Simultaneously, combine this with an infrared temperature sensor to obtain surface temperature information and generate a multimodal dataset.

[0068] S22. LabelImg software was used to manually annotate different types of steel surface defects in the dataset. During the annotation process, the types of steel surface defects were subdivided into six categories: cracking, inclusion, patches, pitted, scratches, and rolled scales. Temperature and texture information corresponding to the images were also recorded. Through meticulous annotation and data recording, the quality of the dataset and the model's ability to identify diverse defects were improved.

[0069] S23. Based on the data collection environment and task requirements, the labeled dataset is divided into training and testing sets in a 4:1 ratio. 80% of the data is used to train the model, and 20% is used to test the model's accuracy and performance. Furthermore, the dataset's complexity is increased by simulating industrial environmental factors such as changes in lighting and surface contamination. This step improves the robustness and adaptability of the dataset, laying the foundation for building a more accurate and generalized defect detection model.

[0070] In this embodiment, as Figure 5-6 As shown, in step S3, the improvement method for the YolovV5 model is to replace the original backbone network with a lightweight MobileNetV3 network and add the SE attention mechanism during the training phase. The specific steps are as follows:

[0071] S31. Perform image enhancement processing on the images in the pre-training dataset. The operations include scaling, rotation, and HSV color transformation. Simulation of lighting and dirt in an industrial environment is also added. The processed images are randomly cropped and stitched together to generate more diverse training samples.

[0072] S32. The processed image is processed by combining depthwise separable convolution, improved Hard-Swish activation function and attention module to extract features.

[0073] The improved Hard-Swish activation function is derived from the ReLU6 activation function. It not only retains the simplicity of ReLU6 but also significantly reduces the detection of minute defects during the process by providing a smooth transition within the input range. The principle behind this modification is as follows:

[0074] The definition of RELU6 is: RELU6(x) = min(max(0,x),6), which restricts the input to the range of 0 to 6. The RELU6 function sets all values ​​less than 0 to 0 and restricts all values ​​greater than 6 to 6, with a range of (0,6).

[0075] Hard-Swish modifies RELU6, and its formula is as follows:

[0076]

[0077] This formula can be divided into two parts:

[0078] 1. Offset: Add 3 to the input value, i.e., x+3. This shifts the input range so that the output of RELU6(x+3) is between (0,6).

[0079] 2. Scaling and weighting: Divide the result of RELU6(x+3) by 6 to make the range between (0,1), and then multiply it by the original input value x.

[0080] In this way, Hard-Swish creates a smooth activation region within the input range, gradually activating the input within the non-negative region, thus enhancing the model's non-linear expressive power. Compared to the ordinary Swish function, Hard-Swish is simpler and has lower computational cost, making it very suitable for lightweight networks such as MobileNetV3. Hard-Swish retains the efficiency of ReLU6 while improving feature extraction performance through smooth non-linear activation.

[0081] S33. Enhance the extracted feature map using the SE (Squeeze-and-Excitation) module to improve the ability to perceive important features. The specific steps are as follows: First, perform global average pooling or global max pooling on each channel of the feature map to compress the spatial information into a single value representation and capture global contextual information; then, generate importance weights for each channel through fully connected layers and non-linear activation functions, thereby adjusting the weights of each channel in the feature map so that the model focuses more on important features.

[0082] S34. During the model training phase, specific industrial scene noise simulations are added, and the model size is further optimized by global average pooling to adapt to embedded mobile devices (generally reduced to within 5M). The results are then input into the classifier for prediction, and the final result is output.

[0083] It should be noted that, for example Figure 6 As shown, the optimized YolovV5 model has a network layer count, number of parameters, and GFLOPs that fall between YolovV5s and YolovV5-Lite. Compared to YolovV5s, the number of network layers increased by 74.5%, the number of parameters decreased by nearly half, and GFLOPs decreased by about 58%. Compared to YolovV5-Lite, the number of network layers decreased by about 11%, the number of parameters increased by 57.7%, and GFLOPs increased by 75.6%.

[0084] like Figure 5 As shown, the accuracy rate of steel surface defect detection is... Figure 5The left side shows the unmodified YolovV5 model, and the right side shows the model combined with MobileNetV3. The data below shows that the overall recognition accuracy before and after the improvement is 0.824 and 0.810, respectively. The improved model, with a 74% increase in network layers and a reduction of nearly half the number of parameters, only decreased the recognition accuracy by about 2%. Both models have high recognition accuracy for patches, pitted, and scrapes, but low recognition accuracy for crazing, inclusion, and rolled defects. The improved model has a higher recognition accuracy for crazing than the unmodified model, but a lower accuracy for inclusion.

[0085] In summary, the optimized YolovV5 model improves computational efficiency for steel surface defect detection while maintaining good recognition accuracy for most defect types. Although it performs slightly worse on certain defect types (such as "inclusion"), its overall performance still meets the needs of most practical applications, making it particularly suitable for resource-constrained industrial scenarios. Furthermore, the improved model offers higher inference speed and lower computational cost, providing a balanced solution of high efficiency and high accuracy for industrial automated inspection systems.

[0086] In this embodiment, the specific method steps of step S4 are as follows:

[0087] S41. Use ONNXRuntime to convert the trained pt format model from step S3 into the ONNX file format, which can run on embedded mobile devices. Transfer the converted ONNX file to the storage of embedded mobile devices such as Raspberry Pi, ensuring that the device has the corresponding operating environment and dependent libraries. ONNX (Open Neural Network Exchange) is an open format that allows models to be migrated between different platforms and frameworks.

[0088] S42. Using the OpenCV framework on an embedded mobile device, process real-time images captured by a camera, including erosion, dilation, and edge detection. A commonly used embedded mobile device is the Raspberry Pi. On the Raspberry Pi, images of the steel surface are captured in real-time via a camera, and preprocessed using the OpenCV framework, including:

[0089] Erosion and dilation: Remove image noise and highlight defective areas;

[0090] Thresholding segmentation: Converts the image into a binary image, which facilitates the separation of defect areas;

[0091] Other processing methods include grayscale conversion, smoothing, and edge detection.

[0092] S43. Use ONNXRuntime to load the ONNX model file uploaded to the embedded system, and input the preprocessed image into the model for inference, outputting the classification results and confidence scores of steel surface defects.

[0093] S44. Perform post-processing of defect detection based on the classification results output by the model, mark the defects in the image and display the classification information, generate repair suggestions, and upload the detection results to the host computer or cloud in real time.

[0094] The entire methodology ensures the efficient operation of the defect detection system on lightweight embedded mobile devices, providing real-time monitoring and data feedback. Key challenges include the inference speed and accuracy of the model on embedded mobile devices, as well as the limitations of device resources, requiring a balance between inference speed and accuracy.

[0095] Example 2:

[0096] like Figure 7-8 As shown in the figure, this embodiment of the invention provides an application of a steel surface defect detection method based on intelligent vehicle vision. This method is used to automate the detection of steel surface defects during steel production and processing. The specific steps are as follows:

[0097] T1. Train the improved YolovV5 model and generate a pt format file. Use ONNXRuntime to convert the model to ONNX format for inference on the Raspberry Pi. Use MobaXterm software to upload the ONNX file to the Raspberry Pi. Note that in addition to MobaXterm software, you can also use SCP (Secure Copy), SFTP (SSH File Transfer Protocol), rsync, upload via USB drive, or web interface to upload to the Raspberry Pi.

[0098] T2. Use OpenCV for image preprocessing, including erosion, dilation, and thresholding. Then, load the ONNX model through ONNXRuntime for real-time inference, label the defect location and category, and upload the results to the host computer in real time.

[0099] T3. Log in to the Raspberry Pi Ubuntu system, set the account and password for remote connection to the VNC service, remotely connect the local host to the Raspberry Pi Ubuntu system, and enter the command line terminal;

[0100] T4. Run the LiDAR Python script in the Raspberry Pi terminal to start the LiDAR to scan the surrounding environment and generate a map. After the car moves to the designated position, turn on the camera to detect the surface defects of the steel in real time and save the types and locations of the detected defects to the Raspberry Pi's SD card, and monitor the frame rate in real time.

[0101] It is worth noting that a Raspberry Pi was chosen as the test device in this embodiment after the trained model was deployed to an embedded mobile device. Besides the Raspberry Pi, other embedded mobile devices can also be selected, such as OpenMV, Jetson Nano, or K210. The actual frame rate will vary depending on the processing power of the different devices. See also... Figure 8 Through testing, this embodiment found that the original YolovV5s model had an average frame rate of 2.5 to 3.2 frames per second on Raspberry Pi, while the improved model increased to 5.8 to 7.2 frames per second. This indicates that the improved model significantly improves processing speed when running on embedded mobile devices such as Raspberry Pi.

[0102] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting surface defects in steel based on the vision of an intelligent vehicle, characterized in that, Includes the following steps: S1. Intelligent Vehicle Path Planning and Dynamic Navigation: The system uses the lidar sensor on the vehicle to collect environmental information in real time. Combined with a dynamic window path planning algorithm based on multi-sensor fusion, it plans the vehicle's path and utilizes multi-sensor data fusion to achieve precise positioning and obstacle avoidance. The specific process includes: S11. Activate the lidar sensor on the car to collect planar point cloud data of the surrounding environment in real time, and combine it with the visual information collected by the camera to perform multimodal environment modeling. S12. A path planning algorithm based on the dynamic window method is adopted, combined with multi-sensor fusion technology including LiDAR, inertial navigation and visual sensors, to generate a dynamic path that adapts to changes in obstacles. S13. Use SLAM technology and Gmapping algorithm to generate an environmental map, and visualize the point cloud data using the rviz tool to monitor the vehicle's positioning and mapping effect, set the starting point and ending point, and achieve automatic navigation; S2. Defect Image Acquisition and Multimodal Dataset Construction: During the vehicle's movement, the embedded mobile device acquires images of steel surface defects from different angles and under different lighting conditions via a camera. At the same time, it acquires surface temperature and texture information to construct a multimodal dataset of surface defect images. The images are then labeled, classified, and divided into training and testing sets. S3. Data Augmentation and Lightweight Model Training: Data augmentation is performed on the images in the multimodal data, incorporating simulations of lighting and pollution in an industrial environment. This data is then input into an improved YolovV5 model for lightweight training, generating a model suitable for embedded mobile devices. The improvement method for the YolovV5 model involves replacing the original backbone network with a lightweight MobileNetV3 network and adding an SE attention mechanism during the training phase. The specific steps are as follows: S31. Perform image enhancement processing on the images in the pre-training dataset, including scaling, rotation, and HSV color transformation operations. Simultaneously, incorporate lighting and dirt simulations under industrial conditions. The processed images are randomly cropped and stitched together to generate more diverse training samples. S32. The enhanced image is processed using a combination of depthwise separable convolution, a Hard-Swish activation function, and an attention module from the MobileNetV3 network to extract features. The Hard-Swish activation function is a modified version of the ReLU6 activation function. RELU6(x) = min(max(0,x),6), x is the input value; S33. Enhance the extracted feature map through the SE module to improve the detection capability of important features. The specific steps are as follows: First, perform global average pooling or global max pooling on each channel of the feature map to compress the spatial information into a single value representation and capture global context information; then, generate importance weights for each channel through fully connected layers and non-linear activation functions, thereby adjusting the weights of each channel in the feature map so that the model focuses more on important features. S34. During the model training phase, specific industrial scene noise simulation is added, and then the model size is further optimized through global average pooling to adapt to embedded mobile devices. The results are input into the classifier for prediction, and the final results are output. S4. Model Conversion and Real-time Detection: The lightweight model is loaded through the ONNXRuntime framework to perform inference on the real-time images captured by the camera, automatically identify the type and location of defects on the steel surface, and upload them to the host computer. S5. Host computer remote monitoring and data storage: The host computer monitors the test results in real time and automatically generates maintenance suggestions based on the defect type. At the same time, the test data is stored in a local database for subsequent analysis.

2. The method for detecting steel surface defects based on intelligent vehicle vision according to claim 1, characterized in that, The specific steps of step S2 are as follows: S21. Use an embedded mobile device to connect a USB camera to collect multi-angle, multi-light images of steel surface defects, and combine this with an infrared temperature sensor to obtain surface temperature information to generate a multimodal dataset. S22. Use LabelImg software to manually annotate defects in image data. Defect types include six categories: cracks, inclusions, pits, patches, scratches, and rolled scales. At the same time, record the temperature and texture information corresponding to the image. S23. Based on the collection environment and task requirements, the multimodal data is divided into training and testing sets in a 4:1 ratio, and the industrial environment is simulated based on changes in illumination or surface contamination conditions to expand the complexity of the dataset.

3. The method for detecting steel surface defects based on intelligent vehicle vision according to claim 1, characterized in that, The specific steps of step S4 are as follows: S41. Use the ONNXRuntime framework to convert the pt format model file to ONNX format, and upload the ONNX file to the embedded mobile device; S42. Preprocess the real-time images captured by the camera using OpenCV, including erosion, dilation, and edge detection; S43. Use ONNXRuntime to load the ONNX model file uploaded to the embedded system, and input the preprocessed image into the model for inference, outputting the classification results and confidence scores of steel surface defects. S44. Perform post-processing of defect detection based on the classification results output by the model, mark the defects in the image and display the classification information, generate repair suggestions, and upload the detection results to the host computer or cloud in real time.

4. The application of the steel surface defect detection method based on intelligent vehicle vision according to any one of claims 1-3, characterized in that, The detection method is used in the steel production and processing process to automatically detect surface defects in steel. The specific steps are as follows: T1. Train the improved YolovV5 model and generate a pt format file. Use ONNXRuntime to convert the model to ONNX format for inference on the Raspberry Pi. Use MobaXterm software to upload the ONNX file to the Raspberry Pi. T2. Use OpenCV for image preprocessing, including erosion, dilation, and thresholding. Then, load the ONNX model through ONNXRuntime for real-time inference, label the defect location and category, and upload the results to the host computer in real time. T3. Log in to the Raspberry Pi Ubuntu system, set the account and password for remote connection to the VNC service, remotely connect the local host to the Raspberry Pi Ubuntu system, and enter the command line terminal; T4. Run the LiDAR Python script in the Raspberry Pi terminal to start the LiDAR to scan the surrounding environment and generate a map. After the car moves to the designated position, turn on the camera to detect the surface defects of the steel in real time and save the detected defect types and locations to the Raspberry Pi's SD card.

Images