Steel surface defect detection method based on intelligent trolley vision and application

Abstract

The invention discloses a steel surface defect detection method based on intelligent trolley vision and application, and relates to the field of machine vision detection.Through path planning, an intelligent trolley automatically arrives at a designated detection area, defect images at multiple angles and under illumination are collected, a data set is constructed and enhanced, an improved YolovV5 model is used for training, and the defect detection accuracy is improved. And after generating a pt format model, converting the pt format model into an onx format model, and transmitting the pt format model into the embedded mobile equipment. Real-time defect detection is realized through ONNXRuntime and OpenCV, and a detection result is uploaded to an upper computer for remote monitoring and data storage, so that efficient and automatic detection and management of steel defects are ensured; a YovV5 model is optimized into a lightweight MobileNetV3 network, and an SE attention mechanism is introduced, so that the calculation complexity and parameter quantity of the model are remarkably reduced, the reasoning speed is increased, embedded mobile equipment can also efficiently complete real-time detection of steel surface defects, and the detection efficiency is improved. The problems of low frame rate and incapability of real-time detection caused by large calculation amount of a traditional method are solved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision detection, and in particular to a steel surface defect detection method and application based on intelligent vehicle vision. Background Art

[0002] The steel industry is one of the important components of the manufacturing industry. Improving product quality is very important for steel companies. However, in the process of steel quality inspection, traditional manual inspection is unable to detect small defects by the human eye, and the recognition speed is slow, the accuracy is low, and the cost is high. It has become a trend to use machines instead of manual inspection.

[0003] For steel surface defect detection, the existing technology mainly includes the following methods: "Zhu Hongping proposed a strip surface defect detection algorithm based on convolutional neural network in "Steel Surface Defect Detection Method Based on Convolutional Neural Network" [J]. Industrial Control Computer, 2020, 033(8): 83-84.", which realized the judgment of strip surface defects; "Shi Weijie et al. used U-Net and Res-UNet to detect and compare steel image defects in "Comparison of Steel Image Defect Detection Methods Based on Two U-Networks" [J]. Journal of Beijing Information Science and Technology University (Natural Science Edition), 2021, 036(1): 63-68." The results showed that Res-UNet was more accurate in identifying defect locations and categories. However, the above methods failed to combine defect classification and positioning, and could not simultaneously obtain defect categories and locations, which limited their application in industrial production. To solve this problem, many researchers have proposed improved solutions based on the YOLO algorithm to achieve dual detection of defect classification and positioning.

[0004] To this end, many domestic researchers have proposed improved solutions based on the YOLO algorithm, aiming to simultaneously achieve the classification and detection of steel surface defects. Fang Yexiang et al. adopted an improved YOLOv3 algorithm in "Research on Improved YOLOv3 Algorithm for Metal Surface Defect Detection" [J]. Mechanical Science and Technology 39.9 (2020): 5. The improved YOLOv3 algorithm was mainly used for image preprocessing and data enhancement through histogram equalization to simulate the real working environment, and the loss function was optimized to improve the accuracy of surface defect classification; Zhang Shuguang et al. in "Improved Tiny-YOLOv3 Industrial Steel Defect Detection Algorithm" [J]. Mechanical Design and Manufacturing, 2024 (5): 97-10 1." In order to improve the speed of steel defect detection, the R-Tiny-YOLOv3 algorithm based on Tiny-YOLOv3 is proposed. The algorithm adds a residual network and a spatial pyramid pooling SPP module and finally selects CIOU as the loss function; in addition, patent CN115294089A discloses a steel surface defect detection method based on improved YolovV5. According to different types of defects that may appear on the steel surface, the original YolovV5 algorithm is improved, thereby improving the detection accuracy and efficiency.

[0005] However, for steel surface defect detection, the trained model using the above-mentioned related technical solutions is often large in size and cannot meet the requirements of real-time detection when installed on embedded mobile devices with limited resources and weak computing power. Therefore, the present invention provides a steel surface defect detection method based on intelligent vehicle vision while ensuring high defect recognition accuracy. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a steel surface defect detection method and application based on intelligent vehicle vision, which solves the technical problems of low running frame rate and inability to achieve real-time detection when a neural network model is installed on a resource-constrained embedded mobile device.

[0007] To achieve the above purpose, the following technical solutions can be used:

[0008] The present invention provides a steel surface defect detection method based on intelligent vehicle vision, comprising the following steps:

[0009] S1. Intelligent car path planning and dynamic navigation: The car collects environmental information in real time through the lidar sensor on the car, combines the improved dynamic window path planning algorithm to plan the car's travel path, and uses multi-sensor data fusion to achieve precise positioning and obstacle avoidance;

[0010] S2. Defect image acquisition and multimodal dataset construction: When the vehicle is moving, the embedded mobile device uses a camera to collect steel surface defect images at different angles and lighting conditions, and also collects surface temperature and texture information to construct a multimodal dataset of surface defect images. The images are annotated and classified and then divided into training and test sets.

[0011] S3, data enhancement and lightweight model training: perform data enhancement on images in multimodal data, add illumination and contamination simulation in industrial environments, and then input the improved YolovV5 model for lightweight training to generate a model suitable for embedded mobile devices;

[0012] S4, model conversion and real-time detection: load the lightweight model through the ONNXRuntime framework, perform inference on the real-time image collected by the camera, automatically identify the type and location of steel surface defects, and upload them to the host computer;

[0013] S5. Host computer remote monitoring and data storage: The host computer monitors the test results in real time and automatically generates maintenance suggestions based on the defect type. At the same time, the test data is stored in the local database for subsequent analysis.

[0014] As a further solution of the present invention, the specific method steps of step S1 are as follows:

[0015] S11, activate the laser radar sensor on the car, collect the plane point cloud data of the surrounding environment in real time, and perform multimodal environment modeling in combination with the visual information collected by the camera;

[0016] S12, using a path planning algorithm based on the dynamic window method, combined with multi-sensor fusion technology including lidar, inertial navigation and visual sensors, to generate a dynamic path that adapts to obstacle changes;

[0017] S13. Use SLAM technology and Gmapping algorithm to generate environmental maps, and visualize point cloud data through rviz tools to monitor the positioning and mapping effects of the car, set the starting point and end point, and realize automatic navigation.

[0018] As a further solution of the present invention, the specific method steps of step S2 are as follows:

[0019] S21. Use an embedded mobile device connected to a USB camera to collect multi-angle and multi-lighting images of steel surface defects, and use an infrared temperature sensor to obtain surface temperature information to generate a multimodal data set.

[0020] S22. Use LabelImg software to manually annotate defects in the image data. The defect types include six categories: cracks, inclusions, pits, patches, scratches, and rolled scales. Meanwhile, the temperature and texture information corresponding to the image are recorded.

[0021] S23. According to the collection environment and task requirements, the multimodal data is divided into training set and test set in a ratio of 4:1, and the industrial environment is simulated based on lighting changes or surface contamination conditions to expand the complexity of the data set.

[0022] As a further solution of the present invention, in step S3, the improvement method of the YolovV5 model is to replace the original backbone network with a lightweight MobileNetV3 network and add the SE attention mechanism in the training stage. The specific method steps are as follows:

[0023] S31, perform image enhancement processing on the images in the pre-training dataset, including scaling, rotation, and HSV color transformation operations, and add illumination and contamination simulation in industrial environments. The processed images are randomly cropped and spliced ​​to generate training samples with higher diversity;

[0024] S32, extract features from the enhanced image by combining the depth-separable convolution, improved Hard-Swish activation function and attention module of the MobileNetV3 network;

[0025] S33, enhance the extracted feature map through the SE module to improve the detection ability of important features. The specific steps are as follows: first, perform global average pooling or global maximum pooling on each channel of the feature map to compress the spatial information into a single value representation and capture the global context information; then, generate the importance weight of each channel through the fully connected layer and nonlinear activation function, so as to adjust the weight of each channel in the feature map so that the model can focus more on important features;

[0026] S34. Add specific industrial scene noise simulation during the model training phase, and then further optimize the model size through global average pooling to adapt to embedded mobile devices. Input the results into the classifier for prediction and output the final results.

[0027] As a further solution of the present invention, in step S32, the improved Hard-Swish activation function is modified on the basis of the RELU6 activation function, which not only retains the simplicity of ReLU6, but also significantly reduces the detection of minor defects by smooth transition within the input range. The modification principle is:

[0028] RELU6(x)=min(max(0,x),6);

[0029]

[0030] As a further solution of the present invention, the specific method steps of step S4 are as follows:

[0031] S41. Use the ONNXRuntime framework to convert the model file in pt format into ONNX format, and upload the ONNX file to the embedded mobile device;

[0032] S42, preprocessing the real-time image collected by the camera based on OpenCV, including corrosion, expansion and edge detection;

[0033] S43, using ONNXRuntime to load the ONNX model file uploaded to the embedded system, and input the preprocessed image into the model for reasoning, and output the classification results and confidence of the steel surface defects;

[0034] S44. Perform post-processing of defect detection based on the classification results output by the model, mark the defects in the image and display the classification information, generate repair suggestions, and upload the detection results to the host computer or cloud in real time.

[0035] The present invention also provides an application of a steel surface defect detection method based on intelligent vehicle vision. The steel surface defect detection method described above is used in the steel production and processing process to automatically detect steel surface defects. The specific method steps are as follows:

[0036] T1. Use the improved YolovV5 model to train and generate pt format files, use ONNXRuntime to convert the model to ONNX format for inference on the Raspberry Pi, and use MobaXterm software to upload the onnx file to the Raspberry Pi;

[0037] T2. Use OpenCV to preprocess the image, including corrosion, expansion, and threshold segmentation. Then load the ONNX model through ONNXRuntime for real-time reasoning, mark the defect location and category, and upload the results to the host computer in real time.

[0038] T3. Log in to the Raspberry Pi Ubuntu system, set the account and password for remote connection to the VNC service, remotely connect the local host to the Raspberry Pi Ubuntu system, and enter the command line terminal;

[0039] T4. Run the Python script of the laser radar in the Raspberry Pi terminal, start the laser radar to scan the surrounding environment and generate a map. After the car runs to the specified position, turn on the camera to detect the surface defect image of the steel in real time, and save the detected defect type and location in the SD card of the Raspberry Pi.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This paper optimizes the YolovV5 model into a lightweight MobileNetV3 network and combines it with the SE attention mechanism, which significantly reduces the computational complexity and parameter amount and improves the reasoning speed. It realizes efficient real-time detection of steel surface defects in embedded devices, overcoming the low frame rate and inability to detect in real time caused by the large amount of calculation in traditional methods.

[0042] 2. The present invention improves the model's ability to identify key features by integrating the MobileNetV3 network and SE module through the improved YolovV5 model, significantly improving the accuracy of defect classification and positioning accuracy. Under different lighting and angle conditions, it can still accurately identify steel surface defects, meeting the high requirements of industrial production for detection quality.

[0043] 3. The present invention optimizes computing efficiency by converting the trained deep learning model into ONNX format and using the ONNX Runtime framework for reasoning, so that the model can run smoothly in embedded mobile devices, and combines OpenCV for image preprocessing to meet the requirements of industrial production for low power consumption and real-time performance.

[0044] 4. Through data enhancement technology and model improvement, the present invention makes the detection model more robust and adaptable, and can operate stably under a variety of complex production conditions (such as lighting changes, surface contamination, etc.), ensuring the consistency and reliability of the detection effect.

[0045] 5. The present invention integrates a remote monitoring function. Operators can view the test results in real time through the host computer to achieve effective monitoring and management of the defect detection process. The test data can also be stored locally or in the cloud to facilitate subsequent data query, analysis and production optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flow chart of the method implemented by the present invention

[0047] Figure 2 This is the steel surface defect dataset image collected by the present invention.

[0048] Figure 3 The dataset images are labeled and classified for the present invention.

[0049] Figure 4 This is a diagram showing the effect of steel surface defect detection according to the present invention.

[0050] Figure 5 This is a comparison chart of the recognition accuracy before and after the YolovV5 model is improved by the present invention.

[0051] Figure 6This is a comparison chart of various performance indicators of the improved YolovV5 model of the present invention.

[0052] Figure 7 This is a frame rate comparison chart of the present invention running on a Raspberry Pi.

[0053] Figure 8 This is a diagram of the remote login to the Raspberry Pi terminal of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Embodiment 1:

[0056] like Figure 1-6 As shown, a steel surface defect detection method based on intelligent vehicle vision provided by an embodiment of the present invention comprises the following steps:

[0057] S1. Path planning and positioning of intelligent vehicles: The vehicle's path is planned by collecting environmental information in real time through the lidar sensor on the vehicle, combined with an improved dynamic window path planning algorithm, and multi-sensor data fusion is used to achieve precise positioning and obstacle avoidance;

[0058] S2. Defect image acquisition and dataset construction: When the vehicle is moving, the embedded device uses a camera to collect images of steel surface defects at different angles and lighting conditions, and also collects surface temperature and texture information to construct a multimodal dataset of surface defect images. The images are annotated and classified and then divided into training and test sets.

[0059] S3, data enhancement and model training: perform data enhancement on images in multimodal data, add illumination and contamination simulation in industrial environments, and then input the improved YolovV5 model for lightweight training to generate a model suitable for embedded devices;

[0060] S4, model conversion and real-time detection: load the lightweight model through the ONNXRuntime framework, perform inference on the real-time image collected by the camera, automatically identify the type and location of steel surface defects, and upload them to the host computer;

[0061] S5. Host computer remote monitoring and data storage: The host computer monitors the test results in real time and automatically generates maintenance suggestions based on the defect type. At the same time, the test data is stored in the local database for subsequent analysis.

[0062] In this embodiment, the specific method steps of step S1 are as follows:

[0063] S11. Activate the LiDAR sensor on the car to collect the plane point cloud data of the surrounding environment in real time, and combine it with the visual information collected by the camera to perform multimodal environment modeling. By activating the LiDAR to collect point cloud data and combining it with the camera's visual information to perform multimodal environment modeling, the ability to perceive the surrounding environment is significantly improved. Multimodal data fusion makes up for the lack of detailed information of a single sensor, making the modeling more comprehensive and accurate, and enhancing the system's adaptability to complex environments.

[0064] S12. A path planning algorithm based on the dynamic window method is used, combined with multi-sensor fusion technology including LiDAR, inertial navigation and visual sensors, to generate a dynamic path that adapts to obstacle changes. A path planning algorithm based on the dynamic window method is used, combined with LiDAR, inertial navigation and visual sensors for multi-sensor fusion, to achieve real-time dynamic path adjustment and efficient obstacle avoidance. This method ensures path planning flexibility and reliability, and can stably respond to obstacle changes even in dynamic environments, providing efficient navigation support.

[0065] S13, use SLAM (Simultaneous Localization and Mapping) technology and Gmapping algorithm to generate environmental maps, and use rviz tools to visualize point cloud data to monitor the positioning and mapping effects of the car, set the starting point and end point, and achieve automatic navigation. This allows the car to automatically complete navigation tasks, while the operator can make real-time adjustments to ensure accurate mapping and efficient task execution, laying the foundation for long-term operation in complex environments.

[0066] In this embodiment, if Figure 2-4 As shown, the specific method steps of step S2 are as follows:

[0067] S21. Use the Raspberry Pi to connect a USB camera, start the camera and collect images of defects on the steel surface in real time. During the collection process, it is necessary to ensure that the images are taken under different types, angles and lighting conditions to ensure the diversity and comprehensiveness of the data set. At the same time, the infrared temperature sensor is used to obtain surface temperature information to generate a multimodal data set.

[0068] S22. Use LabelImg software to manually label different types of steel surface defects in the dataset. During the labeling process, the types of steel surface defects are subdivided into six categories: cracking, inclusion, pitting, scratches, and rolled scales. At the same time, the temperature and texture information corresponding to the image are recorded. Through meticulous labeling and data recording, the quality of the dataset and the model's ability to identify diverse defects are improved.

[0069] S23. Based on the collection environment and task requirements, the labeled data set is divided into a training set and a test set in a ratio of 4:1, that is, 80% of the data is used to train the model, and 20% of the data is used to test the accuracy and performance of the model. The complexity of the data set is expanded by simulating industrial environmental factors such as lighting changes and surface contamination. This step improves the robustness and adaptability of the data set, laying the foundation for building a more accurate and generalized defect detection model.

[0070] In this embodiment, if Figure 5-6 As shown, in step S3, the improvement method of the YolovV5 model is to replace the original backbone network with a lightweight MobileNetV3 network and add the SE attention mechanism in the training stage. The specific steps are as follows:

[0071] S31, performing image enhancement processing on the images in the pre-training data set, including scaling, rotation, HSV color transformation, adding illumination and contamination simulation in industrial environments, and randomly cropping and splicing the processed images to generate training samples with higher diversity;

[0072] S32, extract features from the processed image through the depth-separable convolution of MobileNetV3, the improved Hard-Swish activation function and the attention module combination;

[0073] The improved Hard-Swish activation function is modified based on the RELU6 activation function. It not only retains the simplicity of ReLU6, but also significantly reduces the detection of minor defects by smooth transition within the input range. The modification principle is as follows:

[0074] The definition of RELU6 is: RELU6(x)=min(max(0,x),6), which means limiting the input to between 0 and 6. The RELU6 function sets all values ​​less than 0 to 0 and limits all values ​​greater than 6 to 6, ranging from (0,6).

[0075] Hard-Swish is modified based on RELU6, and its formula is:

[0076]

[0077] The formula can be broken down into two parts:

[0078] 1. Offset: Add 3 to the input value, that is, x+3, which shifts the input range so that the output of RELU6(x+3) is between (0,6).

[0079] 2. Scaling and weighting: Divide the result of RELU6(x+3) by 6 so that the range becomes between (0,1), and then multiply it by the original input value x.

[0080] In this way, Hard-Swish creates a smooth activation region within the input range, gradually activates the input within the non-negative region, and enhances the nonlinear expression ability of the model. Compared with the ordinary Swish function, Hard-Swish is more concise and has a lower computational overhead, making it very suitable for applications in lightweight networks such as MobileNetV3. Hard-Swish not only retains the high efficiency of ReLU6, but also improves the feature extraction effect through smooth nonlinear activation.

[0081] S33. Enhance the extracted feature map through the SE (Squeeze-and-Excitation) module to improve the perception of important features. The specific steps are as follows: first, perform global average pooling or global maximum pooling on each channel of the feature map to compress the spatial information into a single value representation and capture global context information; then, generate the importance weight of each channel through a fully connected layer and a nonlinear activation function, thereby adjusting the weights of each channel in the feature map to make the model more focused on important features;

[0082] S34. Add specific industrial scene noise simulation during the model training phase, and further optimize the model size through global average pooling to adapt to embedded mobile devices (generally reduced to less than 5M in size). Input the results into the classifier for prediction and output the final results.

[0083] It should be noted that if Figure 6 As shown in the figure, the number of network layers, number of parameters, and GFLOPs of the optimized YolovV5 model are between YolovV5s and YolovV5-Lite. Compared with YolovV5s, the number of network layers increased by 74.5%, the number of parameters decreased by nearly half, and GFLOPs decreased by about 58%. Compared with YolovV5-Lite, the number of network layers decreased by about 11%, the number of parameters increased by 57.7%, and GFLOPs increased by 75.6%.

[0084] like Figure 5 As shown in the figure, the recognition accuracy of steel surface defect detection is Figure 5The left side is the unimproved YolovV5 model, and the right side is the model combined with MobileNetV3. From the data below, it can be concluded that the total recognition accuracy before and after the improvement is 0.824 and 0.810 respectively. The improved model increases the number of network layers by 74% and reduces the number of parameters by nearly half, while the recognition accuracy only decreases by about 2%. The two models have higher recognition accuracy for patches, pitted, and scratches, and lower recognition accuracy for crazing, inclusion, and rolled defects. The improved model has a higher recognition accuracy for the crazing category than the model before the improvement, but a lower recognition accuracy for the inclusion category than the model before the improvement.

[0085] In summary, the optimized YolovV5 model can maintain good recognition accuracy for most defect types while improving the computational efficiency of steel surface defect detection tasks. Although it performs slightly worse on some types of defects (such as "Inclusion"), the overall performance still meets most practical application requirements, and is especially suitable for resource-constrained industrial scenarios. In addition, the improved model can provide higher reasoning speed and lower computing consumption, providing a solution that balances efficiency and accuracy for industrial automation detection systems.

[0086] In this embodiment, the specific method steps of step S4 are as follows:

[0087] S41. Use ONNXRuntime to convert the trained model in pt format in step S3 into an ONNX file format that can be run on an embedded mobile device, and transfer the converted ONNX file to the storage of an embedded mobile device such as Raspberry Pi, ensuring that the device has the corresponding operating environment and dependent libraries. ONNX (Open Neural Network Exchange) format is an open format that allows models to be migrated between different platforms and frameworks.

[0088] S42. Based on the OpenCV framework, the real-time image captured by the camera is processed on the embedded mobile device, including corrosion, expansion and edge detection; the commonly used embedded mobile device is Raspberry Pi, and the steel surface image is collected in real time through the camera on the Raspberry Pi, and the OpenCV framework is used for preprocessing, including:

[0089] Erosion and dilation: remove image noise and highlight defect areas;

[0090] Threshold segmentation: convert the image into a binary image to facilitate the separation of defect areas;

[0091] Other processing: such as grayscale, smoothing, edge detection, etc.

[0092] S43, using ONNXRuntime to load the ONNX model file uploaded to the embedded system, and input the preprocessed image into the model for reasoning, and output the classification results and confidence of the steel surface defects;

[0093] S44. Perform post-processing of defect detection based on the classification results output by the model, mark the defects in the image and display the classification information, generate repair suggestions, and upload the detection results to the host computer or cloud in real time.

[0094] The entire method step process ensures that the entire defect detection system runs efficiently on lightweight embedded mobile devices and provides real-time monitoring and data feedback. The key challenges include the inference speed, accuracy and device resource limitations of the model on embedded mobile devices, and it is necessary to find a balance between inference speed and accuracy.

[0095] Embodiment 2:

[0096] like Figure 7-8 As shown, the embodiment of the present invention provides an application of a steel surface defect detection method based on intelligent vehicle vision, which is used to apply the steel surface defect detection method described in the first embodiment to the steel production and processing process to automatically detect steel surface defects. The specific method steps are as follows:

[0097] T1. Use the improved YolovV5 model to train and generate pt format files, use ONNXRuntime to convert the model to ONNX format for inference on the Raspberry Pi, and use MobaXterm software to upload the onnx file to the Raspberry Pi. It should be noted that in addition to using MobaXterm software, you can also use SCP (Secure Copy), SFTP (SSHFile Transfer Protocol), rsync, upload through USB drive, web interface, etc. to upload to the Raspberry Pi;

[0098] T2. Use OpenCV to preprocess the image, including corrosion, expansion, and threshold segmentation. Then load the ONNX model through ONNXRuntime for real-time reasoning, mark the defect location and category, and upload the results to the host computer in real time.

[0099] T3. Log in to the Raspberry Pi Ubuntu system, set the account and password for remote connection to the VNC service, remotely connect the local host to the Raspberry Pi Ubuntu system, and enter the command line terminal;

[0100] T4. Run the Python script of the laser radar in the Raspberry Pi terminal, start the laser radar to scan the surrounding environment and generate a map. After the car runs to the specified position, turn on the camera to detect the surface defect image of the steel in real time, and save the detected defect type and location in the SD card of the Raspberry Pi, with real-time detection frame rate.

[0101] It should be noted that after the trained model is deployed to the embedded mobile device, this embodiment selects the Raspberry Pi as the test device. In addition to the Raspberry Pi, other embedded mobile devices can also be selected, such as OpenMV, Jetson Nano or K210, etc. The actual running frame rate will vary depending on the processing power of different devices. Figure 8 Through testing, this embodiment found that the average frame rate of the original YolovV5s model on the Raspberry Pi was between 2.5 and 3.2 frames, while the improved model was increased to 5.8 to 7.2 frames, indicating that the processing speed of the improved model was significantly improved when running on embedded mobile devices such as the Raspberry Pi.

[0102] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A steel surface defect detection method based on intelligent vehicle vision, characterized in that: The following steps are involved: S1. Intelligent car path planning and dynamic navigation: The car collects environmental information in real time through the lidar sensor on the car, combines the improved dynamic window path planning algorithm to plan the car's travel path, and uses multi-sensor data fusion to achieve precise positioning and obstacle avoidance; S2. Defect image acquisition and multimodal dataset construction: When the vehicle is moving, the embedded mobile device uses a camera to collect steel surface defect images at different angles and lighting conditions, and also collects surface temperature and texture information to construct a multimodal dataset of surface defect images. The images are annotated and classified and then divided into training and test sets. S3, data enhancement and lightweight model training: perform data enhancement on images in multimodal data, add illumination and contamination simulation in industrial environments, and then input the improved YolovV5 model for lightweight training to generate a model suitable for embedded mobile devices; S4, model conversion and real-time detection: load the lightweight model through the ONNXRuntime framework, perform inference on the real-time image collected by the camera, automatically identify the type and location of steel surface defects, and upload them to the host computer; S5. Host computer remote monitoring and data storage: The host computer monitors the test results in real time and automatically generates maintenance suggestions based on the defect type. At the same time, the test data is stored in the local database for subsequent analysis.

2. The method for detecting steel surface defects based on intelligent vehicle vision according to claim 1 is characterized in that: The specific method steps of step S1 are as follows: S11, activate the laser radar sensor on the car, collect the plane point cloud data of the surrounding environment in real time, and perform multimodal environment modeling in combination with the visual information collected by the camera; S12, using a path planning algorithm based on the dynamic window method, combined with multi-sensor fusion technology including lidar, inertial navigation and visual sensors, to generate a dynamic path that adapts to obstacle changes; S13. Use SLAM technology and Gmapping algorithm to generate environmental maps, and visualize point cloud data through rviz tools to monitor the positioning and mapping effects of the car, set the starting point and end point, and realize automatic navigation.

3. The method for detecting steel surface defects based on intelligent vehicle vision according to claim 1 is characterized in that: The specific method steps of step S2 are as follows: S21. Use an embedded mobile device connected to a USB camera to collect multi-angle and multi-lighting images of steel surface defects, and use an infrared temperature sensor to obtain surface temperature information to generate a multimodal data set. S22. Use LabelImg software to manually annotate defects in the image data. The defect types include six categories: cracks, inclusions, pits, patches, scratches, and rolled scales. Meanwhile, the temperature and texture information corresponding to the image are recorded. S23. According to the collection environment and task requirements, the multimodal data is divided into training set and test set in a ratio of 4:1, and the industrial environment is simulated based on lighting changes or surface contamination conditions to expand the complexity of the data set.

4. The method for detecting steel surface defects based on intelligent vehicle vision according to claim 1 is characterized in that: In step S3, the improvement method of the YolovV5 model is to replace the original backbone network with a lightweight MobileNetV3 network and add the SE attention mechanism in the training stage. The specific steps are as follows: S31, perform image enhancement processing on the images in the pre-training dataset, including scaling, rotation, and HSV color transformation operations, and add illumination and contamination simulation in industrial environments. The processed images are randomly cropped and spliced ​​to generate training samples with higher diversity; S32, extract features from the enhanced image by combining the depth-separable convolution, improved Hard-Swish activation function and attention module of the MobileNetV3 network; S33, enhancing the extracted feature map through the SE module to improve the detection ability of important features, the specific steps are as follows: first, performing global average pooling or global maximum pooling on each channel of the feature map to compress the spatial information into a single value representation and capture the global context information; Then, through the fully connected layer and nonlinear activation function, the importance weight of each channel is generated, so as to adjust the weight of each channel in the feature map, so that the model can focus more on important features; S34. Add specific industrial scene noise simulation during the model training phase, and then further optimize the model size through global average pooling to adapt to embedded mobile devices. Input the results into the classifier for prediction and output the final results.

5. The method for detecting steel surface defects based on intelligent vehicle vision according to claim 4 is characterized in that: In step S32, the improved Hard-Swish activation function is modified based on the RELU6 activation function, which not only retains the simplicity of ReLU6, but also significantly reduces the detection of minor defects by smooth transition within the input range. The modification principle is: RELU6(x)=min(max(0,x),6); 6. The method for detecting steel surface defects based on intelligent vehicle vision according to claim 1 is characterized in that: The specific method steps of step S4 are as follows: S41. Use the ONNXRuntime framework to convert the model file in pt format into ONNX format, and upload the ONNX file to the embedded mobile device; S42, preprocessing the real-time image collected by the camera based on OpenCV, including corrosion, expansion and edge detection; S43, using ONNXRuntime to load the ONNX model file uploaded to the embedded system, and input the preprocessed image into the model for reasoning, and output the classification results and confidence of the steel surface defects; S44. Perform post-processing of defect detection based on the classification results output by the model, mark the defects in the image and display the classification information, generate repair suggestions, and upload the detection results to the host computer or cloud in real time.

7. Application of a steel surface defect detection method based on intelligent vehicle vision according to any one of claims 1 to 6, characterized in that: The detection method is used in the process of steel production and processing to automatically detect steel surface defects. The specific method steps are as follows: T1. Use the improved YolovV5 model to train and generate pt format files, use ONNXRuntime to convert the model to ONNX format for inference on the Raspberry Pi, and use MobaXterm software to upload the onnx file to the Raspberry Pi; T2. Use OpenCV to preprocess the image, including corrosion, expansion, and threshold segmentation. Then load the ONNX model through ONNXRuntime for real-time reasoning, mark the defect location and category, and upload the results to the host computer in real time. T3. Log in to the Raspberry Pi Ubuntu system, set the account and password for remote connection to the VNC service, remotely connect the local host to the Raspberry Pi Ubuntu system, and enter the command line terminal; T4. Run the Python script of the laser radar in the Raspberry Pi terminal, start the laser radar to scan the surrounding environment and generate a map. After the car runs to the specified position, turn on the camera to detect the surface defect image of the steel in real time, and save the detected defect type and location in the SD card of the Raspberry Pi.

Images