Steel plate surface defect detection method and system based on multi-source data fusion

Abstract

The invention provides a steel plate surface defect detection method and system based on multi-source data fusion, and relates to the technical field of steel production automatic detection.The method comprises the steps that water stain pretreatment is conducted on the surface of a steel plate, the residual humidity is reduced to be below a preset low humidity threshold value, and a dry surface is obtained; based on the dry surface, collecting multi-source data through a 3D laser line scanning camera and a 2D industrial line scanning camera which are synchronously triggered, obtaining line scanning laser point cloud data, a line scanning reflectivity gray level image, a line scanning depth gray level image and an area array reflectivity color image, and establishing a space corresponding relation among the multi-source data; and forming a multi-source data set based on the spatial correspondence among the multi-source data. According to the method, a high-precision and high-robustness steel plate surface defect detection system covering the whole detection process is constructed by eliminating water stain interference, unifying a multi-source data reference, optimizing data quality, accurately identifying defects and realizing automatic feedback and data closed loop.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic detection of steel production, in particular to a steel plate surface defect detection method and system based on multi-source data fusion. BACKGROUND

[0002] In the steel industry production, the steel plate surface is prone to cracks, holes, pits, and scale indentation during rolling, cooling, and transportation. The surface quality directly affects the subsequent processing performance and safety of the product. The current mainstream detection method mainly relies on manual visual inspection or machine vision technology based on single 2D images.

[0003] With the advancement of technology, 3D point cloud technology provides a new way for surface three-dimensional feature extraction. However, the existing technology has the following shortcomings in data processing: first, the water stain interference problem is prominent. The residual cooling water in the production process can seriously affect data acquisition: abnormal noise points are generated in 3D point cloud data, and reflection artifacts are formed in 2D images. The existing technology lacks effective water stain preprocessing mechanism, which may lead to high false detection rate and missed detection rate in a humid environment. Second, the three-dimensional defect recognition capability is insufficient. Traditional 2D image detection methods cannot accurately obtain surface depth information, and may have principle limitations in detecting pits, bumps, and other three-dimensional defects, resulting in high missed detection rate. Third, the multi-source data collaborative utilization is insufficient. The existing method fails to effectively combine the complementary advantages of 3D point cloud and 2D image: 3D point cloud is good at geometric shape representation but weak in texture recognition, while 2D image is good at texture capture but lacks depth information. The lack of effective cross-modal feature fusion mechanism may lead to insufficient detection stability of composite defects such as cracks with water stains. In addition, the data preprocessing self-adaptive ability is lacking. The existing system mostly uses fixed parameter configuration, which is difficult to adapt to the detection needs of steel plates of different specifications and different surface states, especially in high-speed production line environment, and cannot effectively balance the processing efficiency and detection accuracy. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a steel plate surface defect detection method and system based on multi-source data fusion, which eliminates water stain interference, unifies multi-source data reference, optimizes data quality, accurately identifies defects, and realizes automatic feedback and data closed loop, thereby constructing a high-precision and high-robustness steel plate surface defect detection system covering the whole detection process.

[0005] To solve the above technical problems, the technical solution of the present application is as follows: In a first aspect, a steel plate surface defect detection method based on multi-source data fusion is provided, which comprises: water stain preprocessing of the steel plate surface to reduce the residual humidity to below a predetermined low humidity threshold, obtaining a dry surface; Based on the dry surface, multi-source data is collected by a 3D laser line scanning camera and a 2D industrial line scanning camera triggered synchronously, line scanning laser point cloud data, line scanning reflectivity grayscale image, line scanning depth grayscale image and area array reflectivity color image are obtained, a spatial correspondence relationship among the multi-source data is established, and a multi-source data set is formed based on the spatial correspondence relationship among the multi-source data; Based on the multi-source data set, a multi-source data calibration area is established on the surface of the steel plate and is divided into a partition grid to obtain a grid framework; the multi-source data corresponding to the grid framework is analyzed to obtain a preprocessing parameter adjustment value for image enhancement and point cloud filtering; the preprocessing parameter adjustment value is used to perform adaptive bilateral filtering and grayscale stretching processing on the line scanning reflectivity grayscale image, the line scanning depth grayscale image and the area array reflectivity color image, and to perform outlier removal on the line scanning laser point cloud data, to obtain a high-quality multi-source data set after preprocessing; The high-quality multi-source data set after preprocessing is input into a pre-trained multi-data fusion neural network, texture and geometric features are extracted, and after fusion through a cross-modal attention mechanism, a detection result is obtained; Based on the detection result, a structured defect report is obtained, which is transmitted to a PLC system through an OPC UA protocol to trigger automatic marking and quality grading operations, and detection data is stored in a blockchain database for quality traceability and model iteration.

[0006] In a second aspect, a steel plate surface defect detection system based on multi-source data fusion includes: A surface preprocessing module is configured to perform water stain preprocessing on the surface of the steel plate to reduce residual humidity to below a predetermined low humidity threshold to obtain a dry surface. An acquisition module is configured to collect multi-source data based on the dry surface by a 3D laser line scanning camera and a 2D industrial line scanning camera triggered synchronously, to obtain line scanning laser point cloud data, line scanning reflectivity grayscale image, line scanning depth grayscale image and area array reflectivity color image, to establish a spatial correspondence relationship among the multi-source data, and to form a multi-source data set based on the spatial correspondence relationship among the multi-source data. A processing module is configured to establish a multi-source data calibration area on the surface of the steel plate and divide it into a partition grid based on the multi-source data set to obtain a grid framework; to analyze the multi-source data corresponding to the grid framework to obtain a preprocessing parameter adjustment value for image enhancement and point cloud filtering; and to use the preprocessing parameter adjustment value to perform adaptive bilateral filtering and grayscale stretching processing on the line scanning reflectivity grayscale image, the line scanning depth grayscale image and the area array reflectivity color image, and to perform outlier removal on the line scanning laser point cloud data to obtain a high-quality multi-source data set after preprocessing. A defect analysis module is configured to input the preprocessed high-quality multi-source data set into a pre-trained multi-data fusion neural network, extract texture and geometric features, and obtain a detection result after fusion through a cross-modal attention mechanism; A result feedback module is configured to obtain a structured defect report based on the detection result, transmit the structured defect report to a PLC system through an OPC UA protocol to trigger automatic marking and quality grading operations, and store detection data in a blockchain database for quality traceability and model iteration.

[0007] In a third aspect, a computing device includes: one or more processors; a storage device storing one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method.

[0008] In a fourth aspect, a computer-readable storage medium stores a program, which, when executed by a processor, implements the method.

[0009] The above-mentioned scheme of the present application has at least the following beneficial effects: The residual humidity is controlled to be below a predetermined low humidity threshold through water stain preprocessing, interference factors are eliminated from the data acquisition source, point cloud abnormal noise and image reflection artifacts caused by water stains are avoided, a clean and stable surface basis is provided for subsequent multi-source data acquisition, and the authenticity and purity of the original data are ensured; the acquisition of multiple types of data by two types of cameras is triggered synchronously, ensuring the consistency of data timing, and taking into account geometric morphology and texture detail information; a correspondence relationship between multi-source data spaces is established and a data set is formed, realizing the unification of spatial references of different modal data, providing structured and clearly related data support for cross-modal fusion; the data is disassembled into fine local units through partition grid division, focusing on subtle data differences to avoid feature masking in global analysis; adaptive preprocessing parameters are generated, so that filtering and enhancement operations adapt to different regional data characteristics; the quality of images and point clouds is optimized synchronously, retaining defect details while suppressing noise, and obtaining high-quality data that takes into account completeness and clarity; texture and geometric features are extracted in parallel through double-branch extraction, fully exploiting the core value of multi-source data; cross-modal attention mechanisms strengthen inter-modal related features and weaken irrelevant interference, so that the fusion features fully represent defect attributes; complete detection information including categories, positions and sizes is output, providing data basis for subsequent operations; structured defect reports adapt to the transmission and analysis needs of industrial systems, ensuring smooth data flow; automatic marking and grading are triggered, improving the degree of automation of the production process; blockchain storage establishes an unalterable quality traceability record, integrates data to provide samples that fit actual scenarios for model iteration, and forms a data closed loop to improve comprehensive utilization value. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of a steel plate surface defect detection method based on multi-source data fusion provided by an embodiment of the present application.

[0011] Figure 2 is a schematic diagram of a steel plate surface defect detection system based on multi-source data fusion provided by an embodiment of the present application. DETAILED DESCRIPTION

[0012] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0013] As shown in Figure 1 An embodiment of the present application proposes a steel plate surface defect detection method based on multi-source data fusion, which comprises the following steps: Step 100, water stain pretreatment is performed on the surface of the steel plate to reduce the residual humidity to below a predetermined low humidity threshold, thereby obtaining a dry surface; Step 200, based on the dry surface, multi-source data is collected by synchronously triggered 3D laser line scanning camera and 2D industrial line scanning camera, line scanning laser point cloud data, line scanning reflectivity grayscale image, line scanning depth grayscale image and area array reflectivity color image are obtained, spatial correspondence between multi-source data is established, and multi-source data set is formed based on the spatial correspondence between multi-source data; Step 300, based on the multi-source data set, a multi-source data calibration area is established on the surface of the steel plate and is divided into a grid, thereby obtaining a grid framework; the multi-source data corresponding to the grid framework is analyzed to obtain a pretreatment parameter adjustment value for image enhancement and point cloud filtering; the pretreatment parameter adjustment value is used to perform adaptive bilateral filtering and grayscale stretching processing on the line scanning reflectivity grayscale image, line scanning depth grayscale image and area array reflectivity color image, and at the same time, outlier removal is performed on the line scanning laser point cloud data, thereby obtaining a pretreated high-quality multi-source data set; Step 400, the pretreated high-quality multi-source data set is input into a pre-trained multi-data fusion neural network, texture and geometric features are extracted, and after cross-modal attention mechanism fusion, a detection result is obtained; Step 500, based on the detection result, a structured defect report is obtained, which is transmitted to the PLC system through the OPC UA protocol to trigger automatic marking and quality grading operations, and at the same time, detection data is stored in a blockchain database for quality traceability and model iteration.

[0014] In the embodiment of the present application, the water stains on the surface of the steel plate are removed, the residual humidity is reduced, the interference of the water stains on subsequent data acquisition is reduced, and the foundation for high-quality data acquisition is laid; multiple types of data are synchronously acquired, the spatial correspondence between the data is established, the three-dimensional geometric information and two-dimensional texture information are integrated, the data dimension is enriched, complete and closely related data support is provided for multi-source data fusion; the preprocessing parameters are optimized by partitioning the grid, the image detail definition is adaptively improved, the point cloud outliers are effectively filtered, the purity and consistency of the multi-source data are improved, and the recognizability of the internal features of the data is strengthened; the texture and geometric features are fully extracted, the feature depth fusion is realized through the cross-modal attention mechanism, the correlation of different types of defect features is strengthened, and the comprehensive extraction effect of defect information is improved; structured defect information is generated, efficient linkage between the detection results and the production system is realized, the security and traceability of data storage are ensured, data support is provided for continuous optimization of the model, and the practicability and sustainability of the detection system are improved.

[0015] In a preferred embodiment of the present application, the above step 100, the water stain pretreatment is performed on the surface of the steel plate, the residual humidity is reduced to below a predetermined low humidity threshold, and a dry surface is obtained, comprising: Step 101, the humidity of the steel plate surface entering the detection area is monitored by an infrared humidity sensor to obtain a surface humidity state signal, specifically including: first, the monitoring range of the infrared humidity sensor is calibrated to match the width of the channel through which the steel plate enters the detection area, to ensure that the sensor can fully cover the to-be-detected area of the steel plate surface, then the real-time data acquisition frequency of the sensor is set to adapt to the transmission speed of the steel plate production line, to realize continuous capture of humidity data at different positions on the surface of the steel plate, and the sensor will convert the sensed physical quantity of the humidity of the steel plate surface into a digital quantity signal recognizable by the subsequent processing unit through an internal signal conversion unit, and then form a complete surface humidity state signal to provide basic data support for subsequent liquid water interference judgment.

[0016] Step 102: Process the surface humidity status signal. When liquid water interference is detected, a water removal device activation signal is obtained. Specifically, this includes: first, setting a threshold for liquid water interference humidity detection. This threshold needs to be determined in conjunction with the anti-interference requirements of subsequent 3D point cloud acquisition and 2D image acquisition. Basic data is obtained through multiple sets of comparative experiments. Specifically, 3D point cloud data and 2D image data of the steel plate surface are collected under different humidity conditions. The noise rate of the 3D point cloud and the proportion of reflection artifacts in the 2D image corresponding to different humidity levels are statistically analyzed. When the humidity drops to a certain value, the noise rate of the 3D point cloud can be stably less than or equal to 3%, and... The initial critical value is set at 10% or less for 2D image reflection artifacts. Considering the differences in physical properties of steel plate surfaces under different production scenarios such as cold rolling and hot rolling, the above-mentioned comparative experiments need to be conducted for different steel plate specifications to obtain critical value ranges suitable for different scenarios. For different steel plate specifications, such as thickness of 6 to 20 mm and width of 500 to 2000 mm, the lower limit of this range is set as the general threshold for judging humidity interference from liquid water. If a specific production line is to be selected, the critical value of the corresponding scenario of the production line is selected as the judgment threshold to ensure that the threshold setting has both universality and scenario adaptability.

[0017] Subsequently, the digital data in the surface humidity status signal obtained in step 101 is analyzed in real time. First, according to the monitoring area division rules of the steel plate surface, the monitoring range of the infrared humidity sensor is divided into several monitoring zones matching the width of the steel plate. For example, when the steel plate width is 1800mm, it is divided into 6 monitoring zones of 300mm each. 3 to 5 evenly distributed monitoring points are set in each monitoring zone. During the analysis process, the humidity digital data of each monitoring point needs to be extracted one by one according to the preset analysis frequency. The preset analysis frequency is adapted to the transmission speed of the steel plate production line. For example, when the production line speed is 0.3m / s, the analysis frequency is set to be greater than or equal to 5Hz. At the same time, the validity of the extracted humidity data is verified, and abnormal data caused by instantaneous fluctuations of the sensor is eliminated. Specifically, when the difference between two consecutive humidity data collected at a certain point exceeds ±10% of the previous data, the data is determined to be an abnormal value and the average of the two valid data is used to replace it, ensuring that the analyzed data can truly reflect the humidity of the steel plate surface.

[0018] Next, the humidity values ​​at each point after validity verification are compared with the preset liquid water interference judgment threshold one by one in real time. The comparison process follows the rule of combining zone judgment and overall judgment: for each monitoring zone, if the humidity values ​​of two or more consecutive monitoring points in the zone are higher than the judgment threshold, the zone is judged to have liquid water interference; when two or more monitoring zones on the entire steel plate surface are judged to have liquid water interference, or when more than half of the monitoring points in a single zone have humidity values ​​higher than the judgment threshold, the entire steel plate surface is judged to have liquid water interference.

[0019] When the determination result is that there is liquid water interference, the operation of the signal generation module is triggered, which generates a water removal device start signal within 100 ms after receiving the determination result, and simultaneously embeds the position information of the super-threshold monitoring point (including the monitoring partition to which it belongs and the specific coordinates in the partition) into the start signal, so that the subsequent water removal device can adjust the operation parameters according to the position information, and provide instructions with position guidance for the accurate triggering and efficient execution of subsequent water removal operations.

[0020] Step 103, according to the water removal device start signal, controlling the high-pressure air blowing unit and the mechanical scraping unit to perform cooperative water removal operation to obtain the steel plate surface after preliminary water removal, specifically including: after receiving the water removal device start signal generated in step 102, first obtaining the actual width specification and surface humidity distribution data of the steel plate to be processed, and determining the vertical distance from the nozzle outlet of the high-pressure air blowing unit to the steel plate surface and taking this distance as the height of the cone; then based on the cone side area algorithm, determining the lateral range that the air flow needs to cover on the steel plate surface based on the width of the steel plate, corresponding the lateral coverage range to the arc length parameter of the cone base, calculating the effective coverage area of the high-pressure air flow by calculating the cone side area to ensure that the effective coverage area can completely cover the water removal area on the steel plate surface, avoiding partial water stains due to incomplete air flow coverage; on this basis, combining the position and humidity value of the high-humidity area in the surface humidity distribution data, setting the output air pressure value of the high-pressure air blowing unit according to the matching relationship between the effective coverage area and the air flow density, so that the high-humidity area can obtain sufficient air flow intensity within the effective coverage area to quickly remove water stains, and the low-humidity area can avoid excessive air blowing to cause resource waste through the adaptive air flow intensity; at the same time, according to the air flow coverage length parameter calculated by the cone side area and the transmission speed of the steel plate production line, the blowing time of the high-pressure air blowing unit is set through the ratio relationship between the two, to ensure that each position on the steel plate surface can stay in the air flow coverage area for enough time during the transmission process, realizing the full effect on water stains at different positions; at the same time, the moving speed of the water scraping plate of the mechanical scraping unit is set according to the transmission speed of the steel plate production line, and the moving path of the water scraping plate is kept synchronous with the effective coverage area of the high-pressure air flow, that is, the coverage area corresponding to the cone side area, to ensure that the water scraping plate can maintain an adaptive adhesion force with the steel plate surface, ensuring the scraping effect and avoiding damage to the steel plate surface; then send operation instructions to the high-pressure air blowing unit and the mechanical scraping unit respectively, control the high-pressure air blowing unit to output high-pressure air flow according to the set air pressure value and time, and the mechanical scraping unit to perform scraping operation along the steel plate surface according to the set moving speed; the two units cooperatively complete the preset operation period, realize efficient removal of liquid water on the steel plate surface, and finally obtain the steel plate surface after preliminary water removal.

[0021] Step 104, real-time humidity monitoring is performed on the surface of the preliminary water-removed steel plate. When the monitoring data indicates that the residual humidity reaches a predetermined low humidity threshold, an operation termination signal is obtained and a dry surface is confirmed. Specifically, it includes: first, the determination of the predetermined low humidity threshold. The setting of the threshold is based on the core standard of ensuring that the subsequent 3D point cloud and 2D image acquisition are not disturbed. It is completed through a plurality of controlled experiments and scene adaptation verification. First, select steel plate samples that match the actual application scene, covering different surface treatment types such as cold rolling, hot rolling, and galvanizing, covering common specifications with a thickness of 6 to 20 mm and a width of 500 to 2000 mm. Simulate the detection conditions of the production line in the laboratory environment. Control the humidity of the steel plate surface at different gradient values such as 1%, 3%, 5%, 7%, and 9%. Synchronously collect 3D point cloud data and 2D image data for each humidity gradient. Statistics and analysis of the noise rate of 3D point cloud and the proportion of reflection artifacts of 2D image under each gradient. When the humidity gradient is reduced to 5%, the 3D point cloud noise rate is stable and less than or equal to 3%, the 2D image reflection artifact proportion is less than or equal to 10%, and the false detection rate of subsequent defect detection under this state can be controlled within 5%. This 5% is taken as the basic low humidity threshold in the general scene. At the same time, supplementary experiments are carried out for different production line speeds and steel plate surface states. Different production line speeds, such as 0.3 m / s, 1 m / s, and 2 m / s, and steel plate surface states such as with scale and coated layer. If the production line speed is greater than or equal to 2 m / s or the steel plate surface is galvanized, the humidity needs to be further controlled to less than or equal to 4.5% to ensure that the data acquisition is not disturbed. At this time, the basic threshold is dynamically fine-tuned for this specific scene, and the predetermined low humidity threshold that adapts to the current detection scene is finally determined.

[0022] Subsequently, the infrared humidity sensor is started to continuously collect humidity data from the surface of the preliminary water-removed steel plate. The collection frequency is adapted to the steel plate production line transmission speed, such as setting the collection frequency to be greater than or equal to 8 Hz when the production line speed is 0.3 m / s, to ensure that the residual humidity changes at different positions on the steel plate surface can be captured in real time. The collected residual humidity data is processed by the signal conversion module inside the sensor, converting the analog humidity sensing signal into a 16-bit precision digital signal to form a digital signal that can be recognized by the data processing unit. The digital signal is analyzed in real time according to the preset analysis rule, that is, abnormal data caused by sensor instantaneous jitter is first removed. When the humidity data of a certain collection point and the average value of the adjacent 5 collection points exceed ±0.5%, it is determined as an abnormal value and replaced by the adjacent average value. Then, the average humidity value of each monitoring partition is calculated, which is consistent with the monitoring partition divided in step 102. The actual residual humidity value of this partition is taken as the actual residual humidity value of this partition.

[0023] The actual residual humidity values of each monitoring partition are continuously compared with the aforementioned determined predetermined low humidity threshold value, and the comparison process needs to meet the dual requirements of time continuity and spatial full coverage. In terms of time, it needs to be continuous within 10 collection periods, where each collection period is the inverse of the collection frequency, and the actual residual humidity values of all monitoring partitions are less than or equal to the predetermined low humidity threshold value. In terms of space, it needs to ensure that all monitoring partitions meet the threshold value requirement, and no partition's actual residual humidity value exceeds the threshold value. When the above two requirements are met at the same time, it is determined that the actual residual humidity value has stabilized and meets the standard.

[0024] At this time, the signal generation module is triggered to run. The module generates a water removal device operation termination signal within 50 ms after receiving the stable standard judgment result. At the same time, the final residual humidity values of each monitoring partition, the length of time when the standard is met, and other data are transmitted to the system control unit through the data feedback channel, and the control unit confirms that the steel plate surface has met the drying requirements, providing a stable and interference-free surface environment for the accurate collection of multi-source data of 3D laser line scanning cameras and 2D industrial line scanning cameras.

[0025] In a preferred embodiment of the present application, step 200, based on the dry surface, obtains line scanning laser point cloud data, line scanning reflectivity grayscale images, line scanning depth grayscale images, and face array reflectivity color images by synchronously triggering 3D laser line scanning cameras and 2D industrial line scanning cameras to collect multi-source data, establishes spatial correspondence between multi-source data, and forms a multi-source data set based on the spatial correspondence between multi-source data, including: Step 201, based on the dry surface, the 3D laser line scanning camera and the 2D industrial line scanning camera are controlled by a synchronous trigger signal to perform synchronous acquisition, and original 3D sensing data and 2D image data are obtained, specifically including: first, the core configuration of the synchronous trigger signal needs to be determined in combination with the real-time running parameters of the steel plate production line. The real-time transmission speed of the steel plate is obtained through the production line PLC system first. The speed range is suitable for the high-speed production line demand mentioned in the background technology, usually 0.3 m / s to 2 m / s. The synchronous trigger frequency is calculated according to the transmission speed. For example, when the production line speed is 0.3 m / s, the trigger frequency is set to be greater than or equal to 5 Hz, ensuring that the 3D laser line scanning camera and the 2D industrial line scanning camera can complete full-width non-missing data acquisition during the process of each steel plate passing through the detection area. Secondly, the acquisition parameters of the two types of cameras are calibrated. The 3D laser line scanning camera uses a 532 nm laser emitter, and the scanning frequency is set to 300 Hz to match the trigger signal. At the same time, the laser power is adjusted to 60% to avoid excessive laser reflection on the dry surface, which may cause data distortion. The 2D industrial line scanning camera uses a 12K resolution CCD sensor, and the scanning frequency is set to be greater than or equal to 20 kHz. The exposure time is adjusted to 80 μs to ensure that the acquisition timing of the 3D camera is consistent. Then, the synchronous trigger module is started. The module generates a synchronous pulse signal based on the preset trigger frequency, and the signal transmission delay is controlled to be less than or equal to 1 μs. The signal is sent to the control unit of the 3D laser line scanning camera and the 2D industrial line scanning camera respectively. After receiving the signal, the two types of cameras start the acquisition action synchronously, and the three-dimensional geometric information and two-dimensional texture information of the dry surface of the steel plate are captured in real time. Finally, the initial original 3D sensing data and 2D image data are obtained. The original 3D sensing data includes laser distance signals and reflection intensity signals, and the 2D image data includes RGB color signals.

[0026] Step 202, point cloud reconstruction and reflectivity analysis are performed on the original 3D sensing data to obtain line scanning laser point cloud data, line scanning reflectivity grayscale image and line scanning depth grayscale image, specifically including: first, the laser distance signal in the original 3D sensing data is processed by point cloud reconstruction. The preset intrinsic parameters (including focal length, pixel size and distortion coefficient) and extrinsic parameters (including 45° angle parameters between camera optical axis and steel plate surface and installation height parameters) of the 3D laser line scanning camera are called first. The distance signal of each laser scanning point is converted into three-dimensional space coordinate value, that is, X axis corresponds to production line transmission direction, Y axis corresponds to steel plate width direction, and Z axis corresponds to steel plate surface vertical direction. On this basis, the coordinate data is optimized in point density. The sparse area is supplemented by neighborhood interpolation algorithm to ensure that the finally generated line scanning laser point cloud data has a point density greater than or equal to 200 points / cm 2, meet the accuracy requirements of subsequent geometric feature extraction; secondly, the reflection intensity signal in the original 3D sensing data is analyzed, and the reflection intensity value, which is usually a digital quantity of 0 to 4095, is converted to a gray value of 0 to 255 through a gray mapping rule. In the mapping process, the intensity abnormal value is truncated, wherein the intensity abnormal value is, for example, a value lower than 100 or higher than 3800. Then, the small fluctuations are eliminated through mean filtering to generate a line-scan reflectivity gray image with a resolution of 1000x3200. Finally, the Z-axis coordinate value of the line-scan laser point cloud data is analyzed in depth, and the effective range of the Z-axis coordinate is also mapped to a gray value of 0 to 255. The effective range of the Z-axis coordinate is usually -5mm to 5mm, with the steel plate reference surface as the zero point. The smaller the Z-axis coordinate (concave), the lower the gray value, and the larger the Z-axis coordinate (convex), the higher the gray value. After edge smoothing, a line-scan depth gray image with a resolution of 1000x3200 is generated.

[0027] In step 203, color space conversion and pixel calibration are performed on the 2D image data to obtain a face array reflectivity color image, specifically including: first, the original 2D image data is converted in color space. The original data is usually stored in RGB color space. In order to eliminate the influence of uneven lighting on the dry surface on texture recognition, the color signal in RGB space is converted to HSV color space. In the conversion process, the hue (H) channel value is kept unchanged, the saturation (S) channel value is increased by 10% as a whole to enhance the texture contrast, and the brightness (V) channel value is adaptively adjusted according to the lighting intensity of different regions of the steel plate surface, for example, the V value of the region with strong lighting is reduced by 5% to 8%, and the V value of the region with weak lighting is increased by 8% to 12%, to ensure uniform overall color brightness. Secondly, pixel calibration processing is performed. A high-precision chessboard calibration board, i.e., the size of the calibration board grid is 10mmx10mm, is used to pre-acquire the distortion parameters of the 2D industrial line-scan camera, including the radial distortion coefficient and the tangential distortion coefficient. Based on the distortion parameters, each pixel coordinate of the 2D image data is corrected, and the position of the distorted pixel is adjusted to its true corresponding position on the steel plate surface. The correction error is controlled to be less than or equal to 1 pixel. Then, the calibrated image is subjected to reflectivity feature enhancement. The relative reflectivity of each pixel is obtained by calculating the ratio of the HSV value of each pixel to the HSV value of a standard white pixel. Then, the relative reflectivity is mapped back to the RGB color space to generate a face array reflectivity color image with a resolution greater than or equal to 12000x8000, ensuring that the texture features such as oxide scale and scratches on the steel plate surface in the image are clear and distinguishable.

[0028] At step 204, based on the line-scan laser point cloud data and the area-array reflectivity color image, a spatial correspondence between the line-scan laser point cloud data and the area-array reflectivity color image is established through a pre-calibrated sensor pose transformation matrix, which specifically includes: first, the pre-calibration of the sensor pose transformation matrix needs to be completed, which needs to rely on standard equipment and standardized operation, and the specific steps are as follows: first, a standard calibration block with three-dimensional coordinate markers is selected, which needs to meet the requirement that the number of feature points is greater than or equal to 20, and the three-dimensional coordinates of each feature point are known quantities, and the coordinate accuracy needs to be controlled within ±0.01 mm, and each feature point is provided with a high-contrast mark on the surface, such as a black circular mark with a diameter of 2 mm, to ensure that the subsequent image acquisition can be accurately positioned.

[0029] After the placement of the calibration block is completed, the 3D line-scan laser camera is started to collect line-scan laser point cloud data of the standard calibration block, and the parameters of the 3D camera are kept consistent with the actual detection, such as laser power 60%, scanning frequency 300 Hz; after the collection is completed, based on the geometric features of the point cloud data, such as the spherical convex structure of the feature points and the corner shape, through the point cloud segmentation and feature extraction algorithm, the three-dimensional coordinates of each known feature point in the 3D point cloud coordinate system are recognized and extracted, and the coordinates of each feature point are measured three times during the extraction process, and the average value is taken as the final three-dimensional coordinate value of the feature point to reduce the measurement error.

[0030] At the same time, the 2D industrial line-scan camera is started to collect the area-array reflectivity color image of the standard calibration block, and the collection parameters are also kept consistent with the actual detection, such as exposure time 80μs, scanning frequency greater than or equal to 20kHz; after the collection is completed, through the image feature matching algorithm, such as the SIFT-based feature point matching algorithm, the high-contrast mark of each feature point in the image is recognized, and then the two-dimensional pixel coordinates of each feature point in the 2D image pixel coordinate system are determined, and the two-dimensional coordinates of each feature point are also identified and extracted three times, and the average value is taken as the final two-dimensional pixel coordinate value.

[0031] Based on the extracted feature point three-dimensional coordinates and two-dimensional pixel coordinates, the spatial coordinate conversion principle (such as Perspective-n-Point algorithm) is used to calculate the pose transformation matrix, which needs to include translation parameters and rotation parameters, where the translation parameters are specifically the offset of the 3D point cloud coordinate system relative to the 2D image pixel coordinate system in X, Y, Z three directions, and the rotation parameters are specifically the rotation angle of the 3D point cloud coordinate system relative to the 2D image pixel coordinate system around the X, Y, Z axes; In order to further improve the accuracy of the matrix, more than or equal to 5 times of repeated collection and calculation are required during calibration, and the average value of the pose transformation matrix parameters obtained by each calculation is taken, and the final obtained matrix needs to meet the requirement that the conversion error is less than or equal to 0.1mm, that is, the deviation of the coordinates of any feature point after conversion by the matrix and the actual coordinates needs to be less than or equal to 0.1mm, to ensure that the matrix can accurately realize the conversion of the two coordinate systems.

[0032] After completing the pre-calibration of the sensor pose transformation matrix, the matrix can be used to establish the spatial correspondence between the line-scan laser point cloud data and the area-array reflectivity color image: for each three-dimensional coordinate point in the line-scan laser point cloud data, its coordinate value is substituted into the pre-calibrated pose transformation matrix for calculation, and through the translation and rotation parameters in the matrix, the two-dimensional pixel coordinates corresponding to the three-dimensional coordinate point in the area-array reflectivity color image are obtained; At the same time, for each pixel point in the area-array reflectivity color image, its pixel coordinates are substituted into the pose transformation matrix for inverse operation, and the three-dimensional space coordinates corresponding to the pixel point in the 3D point cloud coordinate system are calculated, that is, the three-dimensional coordinate interval corresponding to the steel plate surface area covered by the pixel point.

[0033] In order to verify the accuracy of the spatial correspondence, more than or equal to 100 feature points are randomly selected for verification, that is, 100 feature points not participating in the matrix calculation are selected, or 100 feature points are randomly selected from the selected feature points, and their corresponding coordinates are obtained through matrix conversion and compared with the actual coordinates, to ensure that the corresponding accuracy of three-dimensional coordinates and two-dimensional pixel coordinates is greater than or equal to 99.5%, so as to determine the stable spatial mapping relationship between the line-scan laser point cloud data and the area-array reflectivity color image.

[0034] At step 205, based on the spatial correspondence and by using the inherent index association between the line-scan reflectivity grayscale image, the line-scan depth grayscale image and the line-scan laser point cloud data, all multi-source data are unified into the same spatial coordinate system to form a multi-source data set with unified spatial correspondence, specifically including: first, a unified spatial coordinate system is determined, taking the center point of the steel plate inlet end of the detection station as the coordinate origin, the transmission direction of the production line as the positive direction of the X axis, the width direction of the steel plate as the positive direction of the Y axis, and the vertical direction of the steel plate surface upward as the positive direction of the Z axis, and the length unit of the coordinate system is set as mm, and the precision is retained to one decimal place; second, based on the spatial correspondence established at step 204, the two-dimensional pixel coordinates of the area array reflectivity color image are converted into the three-dimensional coordinate range in the unified coordinate system through the pose transformation matrix, so that each pixel in the image corresponds to a specific spatial region in the unified coordinate system; meanwhile, by using the inherent index association between the line-scan reflectivity grayscale image, the line-scan depth grayscale image and the line-scan laser point cloud data, that is, the three types of data are synchronously collected by the same 3D laser line-scan camera, there is a one-to-one correspondence between the frame index and the pixel index, each pixel of the line-scan reflectivity grayscale image and the line-scan depth grayscale image is matched and associated to the index of the corresponding three-dimensional coordinate point in the line-scan laser point cloud data, and then is mapped to the unified coordinate system. Subsequently, the spatial alignment verification of the multi-source data is performed, the natural feature points on the surface of the steel plate, such as the edge inflection points and the surface small pits, are selected, the coordinate or pixel information of the feature points is extracted in the four types of data, that is, the line-scan laser point cloud data, the line-scan reflectivity grayscale image, the line-scan depth grayscale image and the area array reflectivity color image, the consistency in the unified coordinate system is verified, and if the deviation exceeds 0.1 mm, the index association parameters are adjusted for correction. Finally, the four types of data are integrated and sorted according to the X axis and Y axis coordinates of the unified coordinate system, each spatial coordinate point corresponds to the storage of the three-dimensional coordinate value of the line-scan laser point cloud, the line-scan reflectivity grayscale value, the line-scan depth grayscale value and the area array reflectivity color value, and a structured multi-source data set is formed, thereby providing a unified data benchmark for the cross-modal feature extraction of the subsequent deep learning model.

[0035] In a preferred embodiment of the present application, the step 300 described above, based on the multi-source data set, establishes a multi-source data calibration region on the surface of the steel plate and performs partition grid division to obtain a grid framework; analyzes the multi-source data corresponding to the grid framework to obtain a pre-processing parameter adjustment value for image enhancement and point cloud filtering; and uses the pre-processing parameter adjustment value to perform adaptive bilateral filtering and grayscale stretching processing on the line-scan reflectivity grayscale image, the line-scan depth grayscale image and the area array reflectivity color image, and simultaneously performs outlier removal on the line-scan laser point cloud data to obtain a pre-processed high-quality multi-source data set, including: In step 301, based on the unified spatial coordinate system of the multi-source data set, the boundary coordinates of the effective detection area of the steel plate surface are extracted; the geometric center position and the normal vector direction of the detection area are calculated according to the boundary coordinates of the effective detection area of the steel plate surface, specifically including: first, relying on the unified spatial coordinate system established by the multi-source data set, the boundary coordinates of the effective detection area of the steel plate surface are extracted; specifically, first, the points with Z-axis coordinates in the reasonable range of the thickness of the steel plate are selected from the line-scan laser point cloud data in the multi-source data set, such as the steel plate thickness of 6 to 20 mm, the Z-axis coordinate is limited to the range of ±10 mm of the reference surface, and the region with pixel gray value in the surface texture feature interval of the area array reflectivity color image is combined, the background black area and the highlight interference area are excluded, and the regions after screening of the two types of data are spatially superimposed, and the extreme values of the superimposed region in the X-axis (production line transmission direction) and Y-axis (steel plate width direction) are taken as the boundary coordinates of the effective detection area, that is, the X-axis boundary is the minimum and maximum values of the X coordinates of all points in the superimposed region, and the Y-axis boundary is the minimum and maximum values of the Y coordinates of all points in the superimposed region; after obtaining the boundary coordinates, the geometric center position of the detection area is calculated, specifically the arithmetic average of the minimum and maximum values of the X-axis boundary as the center X coordinate, the arithmetic average of the minimum and maximum values of the Y-axis boundary as the center Y coordinate, and the Z-axis coordinate taking the Z value of the steel plate reference surface, which together constitute the geometric center position; at the same time, the normal vector direction is calculated, by selecting more than 20 evenly distributed feature points in the effective detection area, such as edge corner points and surface flat area sampling points, a spatial coordinate matrix of these points is constructed, a covariance matrix of the matrix is solved and its eigenvector is calculated, wherein the eigenvector consistent with the perpendicular direction of the steel plate surface is the normal vector direction of the detection area.

[0036] Step 302, based on the geometric center position and the normal vector direction, a local coordinate system of the steel plate surface is established; according to the horizontal and vertical scanning field angles of the 3D laser line scanning camera, the boundary coordinates of the scanning coverage range are calculated in the local coordinate system, specifically including: on the basis of completing the calculation of the geometric center position and the normal vector direction, the local coordinate system of the steel plate surface is established; the geometric center position is taken as the origin of the local coordinate system, the normal vector direction is set as the positive direction of the Z-axis of the local coordinate system, the production line transmission direction is consistent with the X-axis of the unified space coordinate system and is set as the positive direction of the X-axis of the local coordinate system, and the positive direction of the Y-axis of the local coordinate system is determined to be consistent with the width direction of the steel plate through the right-hand rule, so as to ensure that the local coordinate system is consistent with the shape of the steel plate surface; then, the boundary coordinates of the scanning coverage range are calculated according to the horizontal and vertical scanning field angles of the 3D laser line scanning camera; first, the hardware parameters of the camera are obtained, wherein the horizontal scanning field angle a generally takes a value range of 30° to 60°, the vertical scanning field angle β generally takes a value range of 5° to 15°, and the vertical distance d from the camera lens center to the geometric center of the steel plate surface is determined according to the installation parameters, which is usually 500 to 800 mm; in the local coordinate system, the scanning coverage half-width in the horizontal direction (Y-axis direction) is d x tan(a / 2), and the scanning coverage half-length in the vertical direction (X-axis direction) is d x tan(β / 2); taking the origin of the local coordinate system as the center, the horizontal coverage half-width is extended to the positive and negative directions of the Y-axis, and the vertical coverage half-length is extended to the positive and negative directions of the X-axis, so as to obtain the X-axis, Y-axis boundary coordinates of the scanning coverage range in the local coordinate system, and the Z-axis boundary coordinates are consistent with the steel plate reference surface.

[0037] Step 303, based on the boundary coordinates of the scanning coverage range, the spatial range of the multi-source data calibration region on the steel plate surface is determined, and the multi-source data calibration region is established, specifically including: based on the calculation results of the scanning coverage range boundary coordinates, the spatial range of the multi-source data calibration region on the steel plate surface is determined; specifically, the intersection region of the effective detection region boundary obtained in step 301 and the scanning coverage range boundary obtained in step 302 is taken, the overlapping interval of the effective detection region X-axis boundary and the scanning coverage range X-axis boundary is taken in the X-axis direction, that is, the X-axis calibration boundary is the larger value of the two X-axis minimum values and the smaller value of the two X-axis maximum values; in the Y-axis direction, the larger value of the two Y-axis minimum values and the smaller value of the two Y-axis maximum values are taken; in the Z-axis direction, the range of the steel plate reference surface ± 5 mm is still taken, so as to ensure that the calibration region not only covers the range of the effective data that can be actually collected by the camera, but also focuses on the core detection region of the steel plate surface, so as to avoid the interference of invalid background data or unscanned regions on the subsequent processing, and finally form the complete spatial range parameters of the multi-source data calibration region.

[0038] Step 304, based on the spatial range of the multi-source data calibration area, calculate the reference grid cell size that meets the point cloud data feature analysis requirements, specifically including: based on the spatial range of the multi-source data calibration area, calculate the reference grid cell size that meets the point cloud data feature analysis requirements; first determine the basic requirements of point cloud data feature analysis, that is, a single grid cell needs to contain a sufficient number of point cloud data (usually greater than or equal to 10 points) to support density distribution feature calculation, while avoiding that the unit is too large to cover the local subtle features; first, count the average point density of the line-scan laser point cloud data in the calibration area in the multi-source data set, such as the known point density is greater than or equal to 200 points / cm 2 , according to the average point density, calculate the minimum area of a single grid cell, that is, the minimum area = the required number of points / the average point density, if the required number of points is 12 and the average point density is 200 points / cm 2 , then the minimum area = 12 / 200 = 0.06 cm 2 ; combined with the X-axis and Y-axis lengths of the calibration area, convert the minimum area into the side length of an approximate square, if the minimum area is 0.06 cm 2 , then the side length ≈ 0.245 cm, converted to 2.45 mm, for the convenience of subsequent calculation and division, round to 2.5 mm, finally determine the reference grid cell size as 2.5 mm × 2.5 mm, ensure that the number of point clouds in a single cell can meet the feature analysis requirements and reflect the differences in local area point cloud distribution.

[0039] Step 305, according to the reference grid cell size, calculate the number of grid divisions along the longitudinal and transverse directions of the steel plate surface, specifically including: according to the determined reference grid cell size, calculate the number of grid divisions along the longitudinal direction (X-axis direction) and transverse direction (Y-axis direction) of the steel plate surface; first, obtain the X-axis length L x and Y-axis length L y of the multi-source data calibration area, which are calculated by the boundary coordinates determined in step 303, that is, the maximum X-axis value minus the minimum X-axis value is L x , and the maximum Y-axis value minus the minimum Y-axis value is L y ; the calculation method of the longitudinal division number N x is L x divided by the reference grid cell side length, if the calculation result has a decimal, round up, and slightly adjust the cell side length (the adjustment amplitude is less than or equal to 0.1 mm) to ensure that N x is an integer, for example, L x = 2000 mm, the cell side length is 2.5 mm, N x = 2000 / 2.5 = 800, no adjustment is needed; the calculation method of the transverse division number N y is consistent with the longitudinal direction, that is, L yDivide by the reference grid cell edge length, also round up and fine-tune the cell edge length, ensure N y is an integer, for example, L y =1800mm, cell edge length 2.5mm, N y =1800 / 2.5=720, finally get the longitudinal N x and the grid division number of the transverse N y , ensure that the grid after division can completely cover the calibration area.

[0040] Step 306, based on the grid division number, establish regular geometric units in the multi-source data calibration area, get a grid framework that completely matches the spatial range of the multi-source data calibration area, specifically including: based on the longitudinal N x and the grid division number of the transverse N y , establish regular geometric units in the multi-source data calibration area; first from the minimum value of X axis (X min ) and Y axis (Y min ) of the calibration area, along the positive direction of X axis, divide Nx units in turn, the X axis range of each unit is X min +(i-1)×cell edge length to X min +i×cell edge length, where i is an integer from 1 to N x ; along the positive direction of Y axis, divide N y units in turn, the Y axis range of each unit is Y min +(j-1)×cell edge length to Y min +j×cell edge length, where j is an integer from 1 to N y ; the spatial range of each regular geometric unit is determined by the corresponding X axis interval and Y axis interval, and the Z axis interval is still the reference surface of the steel plate ±5mm, forming N x ×N y uniformly distributed rectangular grid cells; after division, verify whether the maximum value of X axis of all grid cells is consistent with the maximum value of X axis of the calibration area, and whether the maximum value of Y axis is consistent with the maximum value of Y axis of the calibration area, to ensure that the grid framework completely matches the spatial range of the calibration area, without area omission or exceeding.

[0041] Step 307, based on the grid framework, respectively calculate the density distribution characteristics of the point cloud data and the gray gradient characteristics of the image data in each grid cell, specifically including: based on the established grid framework, respectively calculate the density distribution characteristics of the point cloud data and the gray gradient characteristics of the image data in each grid cell; for the point cloud data density distribution characteristics, traverse each grid cell, count the total number of line-scan laser point cloud data falling within the unit space, divide the total number of points by the area of the unit, that is, the product of the unit side length, to obtain the point cloud density value of the unit, and the point cloud density value of all units as the density distribution characteristics of the point cloud data; for the gray gradient characteristics of the image data, for the line-scan reflectivity gray image, line-scan depth gray image, and face array reflectivity color image (converted to gray image), respectively, take the image area corresponding to a single grid cell as the object, calculate the gray difference value of each pixel in the region and its horizontal and vertical adjacent pixels, square the sum of the horizontal and vertical gray difference values of each pixel to obtain the gradient value of the pixel, traverse all pixels in the unit and count the distribution of the gradient value, such as the maximum, minimum, and median of the gradient value, as the gray gradient characteristics of the image data in the unit.

[0042] Step 308, normalize the density distribution characteristics of the point cloud data to obtain the normalized density coefficient, and statistically analyze the amplitude of the gray gradient characteristics of the image data to obtain the average gradient intensity, specifically including: normalizing the point cloud density values of all grid cells to obtain the normalized density coefficient; first find the maximum value ρ max and the minimum value ρ min of all unit point cloud density values; for the density value ρ i of each unit, calculate it in the form of (ρ i -ρ min ) / (ρ max -ρ min ) to obtain the normalized density coefficient of the unit, ensure that the coefficient value is between 0 and 1, eliminate the influence of the absolute value difference of the density between different units, and make the density characteristics have a unified contrast standard; at the same time, statistically analyze the amplitude of the gray gradient characteristics of the image data of each unit to obtain the average gradient intensity; for the image gradient value set of each unit, calculate the arithmetic mean of all gradient values, if there are abnormal pixels with gradient value of 0, such as background area, first remove such abnormal values and then calculate the average value to obtain the average gradient intensity of the unit.

[0043] Step 309, the normalized density coefficient and the average gradient intensity are weighted and fused according to a predetermined weight ratio to obtain a weighted fusion result, specifically including: the normalized density coefficient and the average gradient intensity of each grid cell are weighted and fused according to a predetermined weight ratio, and the determination of the predetermined weight ratio needs to be carried out around the influence degree of the characteristics of multi-source data on the subsequent preprocessing effect, and needs to be calibrated through multiple sets of control experiments to ensure balance; Specifically, first, select steel plate samples covering different defect types such as pits, cracks, and scale indentation, and different surface humidity states, design multiple weight combinations for control experiments, and the weight proportions of the normalized density coefficient and the average gradient intensity in each weight combination are 0.3 to 0.7, 0.4 to 0.6, and 0.5 to 0.5; For each weight combination, perform subsequent image enhancement and point cloud filtering preprocessing on the multi-source data of the sample, and then detect the point cloud denoising effect and image enhancement effect of the preprocessed data, wherein the point cloud denoising effect such as outlier removal rate and effective defect point retention rate, and the image enhancement effect such as texture detail clarity and noise suppression degree, and the comprehensive processing effect corresponding to each weight is counted; After multiple rounds of experimental verification, it is found that when the weight of the normalized density coefficient is 0.4 and the weight of the average gradient intensity is 0.6, the abnormal noise points can be accurately removed in the point cloud filtering process without losing effective geometric features, and the defect texture details can be clearly retained in the image enhancement process without excessive amplification of noise, and the preprocessing effect of the two types of data reaches the optimal balanced state, so the weight ratio is determined as the predetermined weight ratio.

[0044] After determining the predetermined weight ratio, for each grid cell, the weighted fusion value of the cell is calculated by multiplying the normalized density coefficient by 0.4 and adding the average gradient intensity by 0.6; After the fusion values of all cells are calculated, the values need to be range checked to determine whether each fusion value is within the reasonable range of 0 to 1; If there is a fusion value less than 0, it is adjusted to 0 through truncation processing, and if there is a fusion value greater than 1, it is adjusted to 1 through truncation processing; Through this checking and adjusting process, the effectiveness and consistency of all cell fusion results are guaranteed, providing balanced and reliable feature basis for subsequent synchronous generation of image enhancement and point cloud processing preprocessing parameter adjustment values.

[0045] At step 310, based on the weighted fusion result, the pre-processing parameter adjustment value for image enhancement processing and the pre-processing parameter adjustment value for point cloud processing are generated synchronously, specifically including: based on the weighted fusion result of each grid unit, the pre-processing parameter adjustment value for image enhancement processing and the pre-processing parameter adjustment value for point cloud processing are generated synchronously; first, a mapping relationship between the fusion result and the parameter adjustment value is established, which is determined by the pre-experimental calibration. When the fusion result tends to 1, it indicates that the point cloud density is high, the image gradient is large, the details are rich, the parameter adjustment value for image enhancement needs to tend to 0.8, which corresponds to a smaller filtering strength to retain details, and the parameter adjustment value for point cloud processing needs to tend to 0.9, which corresponds to a smaller outlier threshold to accurately denoise. When the fusion result tends to 0, it indicates that the point cloud density is low, the image gradient is small, the details are blurred, the parameter adjustment value for image enhancement needs to tend to 0.3, which corresponds to a larger filtering strength to suppress noise, and the parameter adjustment value for point cloud processing needs to tend to 0.2, which corresponds to a larger outlier threshold to avoid deleting effective points. For the fusion result of each unit, the image enhancement parameter adjustment value and the point cloud processing parameter adjustment value corresponding to the unit are calculated according to the above mapping relationship by linear interpolation, the synchronous generation of the two types of parameters is realized, the accurate matching of parameter adjustment and multi-source data characteristics in the unit is ensured, and the insufficient adaptability caused by independent generation of parameters is avoided.

[0046] Step 311, based on the pre-processing parameter adjustment value of the image enhancement processing, respectively calculate the spatial domain sigma value and the gray domain sigma value corresponding to the line-scan reflectivity gray image, the line-scan depth gray image and the area-array reflectivity color image; use the spatial domain sigma value and the gray domain sigma value to dynamically configure the bilateral filter parameter of each image; execute the filter processing on each image through the configured bilateral filter parameter, and obtain the image data after noise suppression, specifically including: based on the image enhancement processing parameter adjustment value of each grid unit, respectively calculate the spatial domain sigma value and the gray domain sigma value corresponding to the three types of images; first, establish the association rule of the parameter adjustment value and the sigma value, the greater the parameter adjustment value, the smaller the spatial domain sigma value, for example, the adjustment value 0.8 corresponds to the spatial sigma=0.6, the adjustment value 0.3 corresponds to the spatial sigma=1.5, and the gray domain sigma value decreases with the increase of the adjustment value, for example, the adjustment value 0.8 corresponds to the gray sigma=0.5, and the adjustment value 0.3 corresponds to the gray sigma=1.2; according to the rule, the image enhancement parameter adjustment value of each unit is converted to obtain the spatial domain sigma value and the gray domain sigma value of the line-scan reflectivity gray image, the line-scan depth gray image and the area-array reflectivity color image in the unit region, and the sigma values of the three types of images are calculated according to the same rule to ensure consistency of processing; then, use these sigma values to dynamically configure the bilateral filter parameter of each image, put the spatial domain sigma value into the spatial weight calculation module of the filter, and put the gray domain sigma value into the gray weight calculation module; execute the filter processing on each image according to the grid unit one by one, that is, for each pixel in the image, according to the filter parameter of the unit where the pixel is located, calculate the spatial weight and the gray weight of all pixels in the neighborhood of the pixel, wherein, for example, the neighborhood of the pixel is 3*3 neighborhood, the spatial weight is based on the distance between pixels and the spatial sigma value, and the gray weight is based on the gray difference between pixels and the gray sigma value, multiply the two weights to obtain the comprehensive weight, then take the comprehensive weight to average the gray values of the neighborhood pixels to obtain the filtered gray value of the pixel, and after traversing all pixels, the image data after noise suppression is obtained.

[0047] Step 312, based on the point cloud processing pre-processing parameter adjustment value, the dynamic distance threshold value of the line-scan laser point cloud data is calculated; the parameters of the statistical filter are configured using the dynamic distance threshold value of the line-scan laser point cloud data; based on the parameters of the statistical filter, the k-nearest neighbor-based statistical outlier filtering processing is performed on the line-scan laser point cloud data to obtain the denoised three-dimensional point cloud data, specifically including: based on the point cloud processing pre-processing parameter adjustment value of each grid cell, the dynamic distance threshold value of the line-scan laser point cloud data is calculated; first, the threshold calculation reference is determined, and the average distance deviation of the point cloud of the flat area of the steel plate (usually 0.05 mm) is taken as the basis to establish the conversion relationship between the parameter adjustment value and the distance threshold value, the larger the parameter adjustment value, the smaller the dynamic distance threshold value, for example, the adjustment value 0.9 corresponds to the threshold value 0.08 mm, and the adjustment value 0.2 corresponds to the threshold value 0.3 mm; according to the relationship, the point cloud processing parameter adjustment value of each cell is converted into the dynamic distance threshold value of the cell. Then the parameters of the statistical filter are configured using the dynamic distance threshold value, and the number of k-nearest neighbors of the statistical filter is set to 6, that is, the optimal value verified by the previous experiment, that is, each point needs to calculate the average distance of the 6 nearest neighbors around it; based on the configured filter parameters, the outlier filtering processing is performed on the line-scan laser point cloud data according to the grid cell, and the point cloud data in each cell is traversed. The average distance of each point and the 6 nearest neighbors is calculated, and if the average distance is greater than the dynamic distance threshold value of the cell, the point is determined to be an outlier and is removed, and if it is less than or equal to the threshold value, it is retained as an effective point. After traversing all cells, the denoised three-dimensional point cloud data is obtained, which ensures that the abnormal noise points caused by water stains are removed while the real defect points are retained.

[0048] Step 313, the contrast-enhanced gray scale stretching processing is performed on the noise-suppressed image data to obtain enhanced image data; the enhanced image data and the denoised three-dimensional point cloud data are combined to form a pre-processed high-quality multi-source data set, specifically including: first, the contrast-enhanced gray scale stretching processing is performed on the noise-suppressed line-scan reflectivity gray scale image, the line-scan depth gray scale image, and the face array reflectivity color image (converted into a gray scale image); for a single image, first, the gray scale values of all pixels are counted, and the minimum gray scale value G min and the maximum gray scale value G max are found out; if the difference between G min and G max is too small, such as less than 50, G min' =0 and G max' =255 are manually set to expand the gray scale range; for the gray scale value G of each pixel in the image, (G-G min ) / (G max -G minThe stretched gray value G' is calculated in a manner of G' = (G - 255) * 255, if G' is less than 0, 0 is taken, and if G' is greater than 255, 255 is taken, so as to ensure that the gray value of the stretched image is distributed in the interval of 0 to 255, the image contrast and the texture detail definition are improved, and the enhanced image data is obtained. After the image enhancement is completed, the enhanced three types of image data and the three-dimensional point cloud data after the denoising are associated according to a unified spatial coordinate system, that is, the spatial corresponding relationship established through step 204, so as to ensure that each pixel of the image data and the corresponding three-dimensional coordinate of the point cloud data are accurately matched, and a preprocessed high-quality multi-source data set is formed, so as to provide high-quality data input for subsequent cross-modal feature extraction of a deep learning model.

[0049] In a preferred embodiment of the present application, the step 400 inputs the preprocessed high-quality multi-source data set into the pre-trained multi-data fusion neural network, extracts texture and geometric features, and obtains a detection result after the cross-modal attention mechanism fusion, including: Step 401, input the enhanced image data in the preprocessed high-quality multi-source data set into the 2D feature extraction branch of the pre-trained multi-data fusion neural network, extract the two-dimensional feature vector containing texture details through the improved ResNet-101 network, and input the denoised three-dimensional point cloud data into the 3D feature extraction branch, extract the three-dimensional feature vector containing geometric shape through the point cloud convolution network, specifically including: before input data processing, the construction and training of the pre-trained multi-data fusion neural network need to be completed first, the construction of the network takes adapting multi-source data feature extraction and cross-modal fusion as the core goal, the specific construction process is as follows: adopt hierarchical design idea to build the overall architecture of the network, which contains 2D feature extraction branch, 3D feature extraction branch, subsequent cross-modal attention fusion module and output branch, wherein the 2D feature extraction branch is specially used for processing the texture features of image data, the 3D feature extraction branch is specially used for processing the geometric features of point cloud data, and the two are designed in parallel to ensure the synchronous extraction of multi-source data features; for the 2D feature extraction branch, ResNet-101 network is selected as the basic architecture and improved, at the output end of each bottleneck block of the network 3 convolution stage (containing 23 bottleneck blocks) and the network 4 convolution stage (containing 36 bottleneck blocks), a spatial attention mechanism (SAM) module is connected in series, the SAM module is composed of global average pooling layer, global maximum pooling layer, 1*1 convolution layer and Sigmoid activation function, wherein the channel number of the 1*1 convolution layer is set to 1 / 4 of the output channel number of the corresponding bottleneck block, so as to realize the channel dimension compression and attention weight accurate generation; for the 3D feature extraction branch, the PointNet network is improved as the basis, the neighborhood grouping module and the multi-scale convolution layer are added, the neighborhood grouping module is realized by using the ball query algorithm, the multi-scale convolution layer is set to 3 layers and the channel dimension is 64, 128 and 512 respectively, and the batch normalization layer and the ReLU activation function are connected in series after each convolution, so as to strengthen the geometric feature extraction ability.

[0050] The training process of the network needs to be carried out in two stages of pre-training and transfer learning fine-tuning. The specific training process is as follows: first, prepare the training data set, adopt the mixed data set mode of public defect data set and self-acquired data set, wherein the public defect data set selects NEU-DET data set, which contains 6 kinds of steel plate defects such as cracks and scratches, a total of 3000 samples, and the self-acquired data set is obtained by collecting the current production line, which contains 10000 steel plate defect samples of different specifications and different surface states, wherein the 10000 different specifications are 500 to 2000 mm in width and 6 to 20 mm in thickness, and the different surface states are oxide skin and plating layer. Both data sets need to be labeled with the texture area (for image) and geometric coordinates (for point cloud) of the defect; then the data set is processed by data enhancement, for image data, ±10° rotation, 0.7 to 1.3 times scaling, brightness ±15% adjustment, for point cloud data, ±5mm translation, ±10° local rotation, random down sampling, and 70% to 90% point cloud is reserved to improve the generalization ability of the network; in the pre-training stage, the AdamW optimization algorithm is adopted, the learning rate is set to 5e-5, the weight decay is set to 0.01, the 2D feature extraction branch optimizes the texture feature extraction accuracy with the cross entropy loss function, the 3D feature extraction branch optimizes the geometric feature extraction accuracy with the mean square error loss function, and the whole iterative training is carried out for 100 rounds. After each round of training, the verification set, which accounts for 20% of the total data set, is used to evaluate the feature extraction effect. When the 2D feature texture recognition accuracy on the verification set is greater than or equal to 94% and the 3D feature geometric representation error is less than or equal to 0.15mm, the pre-training stage is completed; after pre-training, the transfer learning fine-tuning stage is entered, 500 steel plate samples (containing various defects) collected on the current production line are selected as the fine-tuning data set, the learning rate is adjusted to 1e-5, and the iterative training is carried out for 20 rounds. Through the fine-tuning of the weight parameters of each layer of the network, the network is adapted to the steel plate features of the current production line. When the feature extraction error on the fine-tuning data set is stable and less than 10% of the error in the pre-training stage, the network training is completed, and the pre-trained multi-data fusion neural network is obtained.

[0051] After the network construction and training are completed, input data processing and feature extraction can be carried out. First, the processing specification of the input data is determined. In the preprocessed high-quality multi-source data set obtained in step 313, the enhanced image data, including the line-scan reflectivity gray image, the line-scan depth gray image, and the area-array reflectivity color image, are converted into a unified format. The area-array reflectivity color image needs to be converted into a 3-channel gray image first. The conversion is realized by weighted average RGB channel values. The R channel weight is set to 0.299, the G channel weight is set to 0.587, and the B channel weight is set to 0.114, so as to ensure that the converted gray image can retain the texture information of the original color image. The line-scan reflectivity gray image and the line-scan depth gray image are single-channel format, and the single-channel features are directly retained. Finally, the three types of images are adapted to the input requirements of the 2D feature extraction branch in single-channel or 3-channel gray format. Then, the standardized image data is input into the 2D feature extraction branch in batches. The batch size is set to 8. This size is determined based on the parallel computing capability of the subsequent GPU (such as NVIDIA A100), which can ensure efficient use of GPU resources and avoid calculation delay caused by too large batch size.

[0052] The feature extraction process of the 2D feature extraction branch is as follows: the standardized image data is first input into the first convolution stage of the improved ResNet-101 network. After being processed by a 7×7 convolution kernel (step size 2), a batch normalization layer, and a ReLU activation function, an initial feature map is obtained, which has a size of 500×1600×64. Then, it is sequentially input into the second convolution stage (3 bottleneck blocks, output feature map size 250×800×256), the third convolution stage (23 bottleneck blocks). After the output of each bottleneck block in the third convolution stage, a SAM module is connected: first, the feature map output by the bottleneck block, which has a size of 250×800×256, is subjected to global average pooling and global maximum pooling in the channel dimension to obtain two 1×1×256 feature vectors. After the two vectors are concatenated into a 1×1×512 vector and input into a 1×1 convolution layer (output channel 64), a 1×1×256 spatial attention weight map is generated by a Sigmoid activation function. The weight map is multiplied pixel by pixel with the original bottleneck block output feature map to strengthen the feature response of key detail areas such as crack texture and scale indentation texture. Then, the feature map enters the fourth convolution stage (36 bottleneck blocks). A SAM module is also connected after the output of each bottleneck block in this stage to further optimize the texture feature extraction effect. Finally, the feature map is processed by a global average pooling layer and then input into a fully connected layer with an output dimension of 1024 to obtain a two-dimensional feature vector containing texture details.

[0053] At the same time, the denoised three-dimensional point cloud data (point density greater than or equal to 200 points / cm 2The 3D feature extraction branch is input, and the feature extraction process of the branch is specifically as follows: first, the input point cloud is randomly down-sampled to fix the number of points in each frame to 1024 points, which is determined through experiments, which can avoid low computational efficiency caused by too many point clouds, and can avoid loss of geometric features caused by too few point clouds; then, the neighborhood grouping of each point after down-sampling is performed through a ball query algorithm, the grouping radius is set to 0.5 mm, and each group contains 32 neighborhood points, so that the local geometric information of each point can be fully captured; the grouped point cloud features are input into three convolutional layers stacked in turn, wherein the grouped point cloud features contain 3D coordinates and 3D normal vectors for each point, a total of 6 dimensions, the first convolutional layer maps the 6-dimensional features to 64-dimensional features, the second convolutional layer enhances the 64-dimensional features to 128-dimensional features, and the third convolutional layer further enhances the 128-dimensional features to 512-dimensional features; after each convolutional layer, the feature distribution offset is eliminated through a batch normalization layer, and then the non-linear expression ability is enhanced through a ReLU activation function; finally, the 512-dimensional local features output by the third convolutional layer are input into a global max-pooling layer to aggregate the local features of all points, and a 3D feature vector with a dimension of 512 containing geometric shape information such as pit depth distribution and convex contour is obtained.

[0054] At step 402, based on the two-dimensional feature vector and the three-dimensional feature vector, a fusion feature is obtained by channel dimension splicing; the fusion feature is input into a cross-modal attention mechanism to calculate the correlation weight between different modal features, and the fusion feature is weighted and reconstructed based on the correlation weight to obtain a cross-modal fusion feature vector, specifically including: first, the two-dimensional feature vector (1024 dimensions) and the three-dimensional feature vector (512 dimensions) output in step 401 are spliced in the channel dimension, specifically, the channel dimension of the three-dimensional feature vector and the channel dimension of the two-dimensional feature vector are directly stacked along the network feature channel axis (i.e. dimension axis) to form a fusion feature with a total channel dimension of 1024+512=1536, which ensures that the original information of the two types of modal features is included in the fusion link without loss, avoiding information loss caused by feature truncation.

[0055] The 1536-dimensional fusion feature is then input into a cross-modal attention mechanism module. The module first splits the fusion feature in the channel dimension to separate a 1024-dimensional channel part corresponding to the original two-dimensional feature and a 512-dimensional channel part corresponding to the original three-dimensional feature. Then, the correlation similarity between the two parts of features is calculated. Specifically, the inner product of the transpose matrix of the original two-dimensional feature channel part and the original three-dimensional feature channel part is calculated by matrix multiplication operation to obtain a similarity matrix with a dimension of 1024x512. Each element in the matrix represents the correlation degree between a channel of the two-dimensional feature and a channel of the three-dimensional feature. The elements of each row of the similarity matrix are normalized by softmax to convert the element values into correlation weights in the range of 0 to 1, and the sum of the weights of each row is 1. The weights quantitatively represent the importance of each channel of the three-dimensional feature to the corresponding channel of the two-dimensional feature.

[0056] The fusion feature is weighted and reconstructed based on the above correlation weights. Specifically, each channel feature value of the original two-dimensional feature channel part is multiplied by the correlation weight of its corresponding row, and each channel feature value of the original three-dimensional feature channel part is multiplied by the correlation weight of its corresponding column. The correlation weights are obtained by softmax normalization in the row direction of the similarity matrix. Then, the two parts of weighted features are recombined along the channel axis to obtain a cross-modal fusion feature vector with a dimension of 1536. The vector strengthens the key correlation features for defect recognition, such as the correlation between crack texture and crack depth geometric features, by weight distribution, weakens irrelevant feature interference, and improves the feature representation ability for composite defects.

[0057] Step 403, input the cross-modal fusion feature vector into the defect classification branch and the regression branch, output the defect category probability distribution through the softmax activation function of the defect classification branch, and output the defect position coordinates and three-dimensional size parameters through the linear activation function of the regression branch, finally generate the detection result containing the defect category, position information and three-dimensional size, specifically including: first, input the 1536-dimensional cross-modal fusion feature vector obtained in step 402 into the defect classification branch and the regression branch of the network, realize parallel calculation of the two types of outputs; wherein the defect classification branch includes two fully connected layers, the first fully connected layer maps the 1536-dimensional feature to a 256-dimensional feature through a weight matrix, and the second fully connected layer further maps the 256-dimensional feature to a 6-dimensional feature, corresponding to the 6 types of target defects of cracks, holes, pits, scale indentation, protrusions and scratches; after the output of the second fully connected layer, a softmax activation function is connected, the core function of the softmax activation function is to convert the 6-dimensional feature values output by the second fully connected layer, which have no explicit numerical range, into non-negative probability values and the sum of all probability values is 1, so as to convert the abstract feature output into a defect category probability expression conforming to the probability statistical logic, and at the same time, by strengthening the weight of the category corresponding to the high feature value and weakening the weight of the category corresponding to the low feature value, the discrimination degree between different defect categories is improved, and the precise classification requirement of the 6 types of target defects is adapted.

[0058] The specific conversion process is to convert each feature value into a non-negative value through a natural exponential function, and then divide each non-negative value by the sum of the exponential conversion of all 6 feature values to obtain the probability value of each defect category; these generated probability values have two core uses, on the one hand, as a direct basis for preliminary defect category determination, selecting the category with the highest probability value as the preliminary defect category of the current region, ensuring that the category determination has clear quantitative support; on the other hand, as a core index for subsequent effectiveness verification, providing a reference for judging whether there is an effective defect in the current region, avoiding the one-sidedness caused by relying on feature output for category determination. Through the above processing, a defect category probability distribution is formed, and the category with the highest probability value is the preliminary defect category determination result of the current region.

[0059] Meanwhile, the regression branch also contains two fully connected layers, the first fully connected layer maps the 1536-dimensional fusion features to 128-dimensional features, and the second fully connected layer maps the 128-dimensional features to 5-dimensional features, which correspond to the position coordinates and three-dimensional size parameters of the defect, i.e., the X-axis starting coordinate, the Y-axis starting coordinate, the defect length, the defect width, and the defect depth, wherein the X-axis starting coordinate corresponds to the steel plate transmission direction, the Y-axis starting coordinate corresponds to the steel plate width direction, the defect length is the X-axis direction span, the defect width is the Y-axis direction span, and the defect depth is the Z-axis direction deviation; a linear activation function is connected after the output of the second fully connected layer, and the 5-dimensional feature values of the output are directly retained as the final parameter results without nonlinear conversion to ensure the consistency of the parameter values and the actual physical size, and the parameter unit is mm, which is consistent with the unit of the unified spatial coordinate system.

[0060] Finally, the effectiveness of the classification and regression results is verified, if the probability value of a certain defect is less than 0.5, it is determined that there is no effective defect in the region, and the corresponding regression parameters are excluded; if the regression parameters exceed the actual specification range of the steel plate, such as the defect length being greater than the steel plate width and the depth absolute value being greater than the steel plate thickness, the parameter mean value of the adjacent effective defects of the same batch of steel plates is called to correct; after the verification is completed, the information is integrated in a structured format according to the defect category, position coordinates (X, Y), and three-dimensional size (length, width, and depth) to generate a detection result containing complete defect information, which can be directly used for subsequent OPC UA protocol transmission and production system interaction.

[0061] In a preferred embodiment of the present application, the step 500 obtains a structured defect report based on the detection result, transmits it to the PLC system through the OPC UA protocol to trigger automatic marking and quality grading operations, and stores the detection data to the blockchain database for quality traceability and model iteration, including: Step 501, based on the defect category, location information and three-dimensional size parameters in the detection result, a structured defect report containing defect feature description, spatial coordinates and geometric size is obtained, specifically including: first, determine the core field system of the structured defect report, which needs to cover the three core dimensions of defect features, spatial position and geometric size, and the field format needs to adapt to the subsequent OPC UA protocol transmission and PLC system analysis requirements; Specific fields include defect feature description, X-axis starting coordinate, Y-axis starting coordinate, defect length, defect width, defect depth, detection timestamp, a total of 7 fields, wherein the defect feature description field needs to combine the defect category in the detection result, integrate the key texture features and geometric features corresponding to the defect to construct the field content, for example, the crack is 2.5mm long, 0.3mm wide, and extends along the X-axis direction, ensuring that the description has both category identification and detailed features; The spatial coordinate field needs to extract the X and Y axis coordinate values of the defect position information in the detection result, and the coordinate values need to be consistent with the unified spatial coordinate system determined in step 205, and are kept to one decimal place, with a unit of mm, to avoid subsequent positioning errors due to coordinate system deviation; The geometric size field needs to extract the three-dimensional size parameters in the detection result, also kept to one decimal place, with a unit of mm, to ensure that the length, width and depth parameters correspond to the actual shape of the defect.

[0062] After extracting the above parameters, data format standardization processing is required, converting the defect feature description to a string format, with a character length of less than or equal to 50 characters, the spatial coordinates and geometric size to a numerical format, and the detection timestamp to a year-month-day hour-minute-second.millisecond format, accurate to 1 millisecond; Further check the data integrity, if some field parameters are missing, such as the defect depth is not detected, mark it as not detected and supplement the remarks, such as the planar defect has no depth information, to avoid empty values causing report parsing failure; Finally, integrate the fields in the fixed order of defect feature description, spatial coordinates (X, Y), geometric size (length, width, depth), and detection timestamp to form a structured defect report, ensuring that the report does not require additional format conversion in the subsequent transmission and analysis process, improving data flow efficiency.

[0063] At step 502, the structured defect report is transmitted to the production line PLC control system through the OPC UA communication protocol, so that the PLC system obtains the defect detection information. Specifically, first, the parameter configuration of the OPC UA communication protocol is completed, which needs to match the communication interface parameters of the production line PLC control system, specifically including setting the communication port number, the sampling period and the timeout time. The communication port number is usually set to 4840, which conforms to the OPC UA standard port specification. The sampling period is set according to the production line speed, for example, when the production line speed is 0.3 m / s, the sampling period is set to 100 ms to ensure real-time performance. The timeout time is set to 500 ms to avoid transmission interruption caused by network delay. Then, the mapping relationship between the structured defect report field and the PLC system variable node is established, for example, the X-axis starting coordinate is mapped to the DefectX variable node of the PLC, and the defect category is mapped to the DefectType variable node, so that each report field has a unique corresponding PLC variable node, and data mapping confusion is avoided.

[0064] After the mapping relationship is established, the structured defect report is encapsulated by the OPC UA protocol, and the field data of the report is converted into a communication data packet in the binary coding format specified by the protocol. A data check bit, i.e., a CRC32 check algorithm, is added in the encapsulation process to ensure that the data is not tampered with and lost during transmission by calculating the check value of the data packet. After encapsulation, the protocol communication link is started, a connection request is first sent to the PLC system, and after receiving the connection confirmation signal of the PLC system, the data packet is transmitted frame by frame according to the preset sampling period. The communication state is monitored in real time during transmission. If transmission timeout or check failure occurs, the retransmission mechanism is triggered immediately, and the retransmission number is less than or equal to 3. If the retransmission still fails, an alarm signal is sent to the system control unit, so that the PLC system can accurately and timely obtain the defect detection information and provide data support for subsequent production operation triggering.

[0065] At step 503, based on the defect detection information, the automatic marking device is triggered to perform an inkjet marking operation at the defect position, and the quality grading module is activated to determine the quality grade of the steel plate, obtaining production data containing marking and grading results. Specifically, first, the trigger logic of the automatic marking device is processed. After receiving the defect detection information, the PLC system needs to determine the extension direction of the defect by using the tangent equation algorithm of the parabola to optimize the adaptability of the marking geometric parameters. Specifically, the characteristic point coordinates of the defect along the surface extension direction are extracted from the detection results, and three key characteristic points, i.e., the starting point, the midpoint, and the endpoint of the defect, are selected. The X and Y axis spatial coordinates of each point are extracted and kept consistent with the unified spatial coordinate system, with a precision of 0.1 mm. For example, the starting point (1498.5, 800.2), the midpoint (1500.0, 800.3), and the endpoint (1501.5, 800.4) are extracted. The coordinates of the three characteristic points are substituted into the parabola fitting model, and the coefficients of the quadratic term, the linear term, and the constant term of the parabola are adjusted by the least squares method to make the deviation of the fitting curve from the coordinates of the three characteristic points less than or equal to 0.1 mm. Finally, the parabola equation representing the extension trend of the defect is obtained.

[0066] Based on the parabola equation, the tangent direction at the midpoint of the defect (i.e., the center point of the subsequent marking) is further calculated. The X-axis coordinate of the midpoint is substituted into the tangent equation calculation formula of the parabola to determine the tangent direction by solving the slope value of the tangent. For example, the slope is calculated to be 0.067, and the angle between the tangent and the positive direction of the X-axis is about 3.8°. This angle is the extension direction of the defect. Through this process, the deviation between the tangent direction and the actual trend of the defect is ensured to be less than or equal to 1°, providing a basis for the directional design of the subsequent marking geometric parameters.

[0067] After determining the extension direction of the defect, the X and Y axis spatial coordinates of the defect are converted into mechanical coordinates of the automatic marking device. The conversion process needs to consider the relative position parameters of the marking device and the detection area, including the X-axis offset and Y-axis offset of the marking device. These parameters are obtained through pre-calibration with a precision of ±0.1 mm. For example, the X-axis spatial coordinate of the midpoint of the defect is 1500.0 mm, and the X-axis offset of the marking device is 50 mm. Therefore, the mechanical X coordinate is 1500.0 + 50 = 1550.0 mm. The Y-axis coordinate is converted in the same way to ensure that the converted mechanical coordinates accurately correspond to the midpoint of the defect.

[0068] Subsequently, the geometric parameters of the inkjet mark are calculated according to the defect length and the tangent direction, and an elliptical mark is used to adapt to the extension form of the defect instead of a fixed circular mark; if the defect length is less than or equal to 5 mm, the length of the major axis of the ellipse is set to 3 mm, the length of the minor axis is set to 2 mm, and the direction of the major axis is consistent with the tangent direction; if the defect length is greater than 5 mm, the length of the major axis of the ellipse is set to 4 mm, the length of the minor axis is set to 2 mm, and the direction of the major axis is also along the tangent direction; through this parameter design, it is ensured that the elliptical mark can completely cover the defect area, while avoiding excessive extension in the direction of the minor axis to affect the normal surface of the surrounding steel plate; after the parameter calculation is completed, the PLC system sends a trigger signal to the automatic marking device, the mechanical arm of the control device adjusts the nozzle angle along the tangent direction, moves to the target mechanical coordinates, and then performs the inkjet marking operation according to the set elliptical parameters, and after the marking is completed, the device feeds back a marking success signal to the PLC system, and simultaneously records parameters such as the angle, the major axis or the minor axis size of the marking.

[0069] At the same time, activate the quality grading module to determine the quality grade, which needs to be combined with the steel plate specifications (width, thickness) and the defect influence degree to formulate, specifically, set the basic score according to the defect type, such as negative 30 points for crack basic score and negative 10 points for oxide skin indentation basic score; set the additional score according to the defect size, such as negative 2 points for each 1 mm increase in length and negative 5 points for each 0.1 mm increase in depth; set the steel plate basic score as 100 points, and the total score = basic score + basic score + additional score; according to the total score, the quality grade is divided, and the total score is greater than or equal to 90 points for excellent product, the total score is less than 70 points for qualified product, and the total score is less than 70 points for unqualified product; the specification parameters of the current steel plate need to be called during grading, which are obtained from the production line MES system, such as width 1800 mm and thickness 10 mm, to ensure that the grading standard adapts to different specifications of steel plates, for example, the depth of the indentation is more sensitive to the steel plate with a thickness less than or equal to 8 mm, and the additional score for the depth is adjusted to negative 8 points for each 0.1 mm increase; after grading, the automatic marking result and the quality grade determination result are integrated, wherein the automatic marking result includes: marking position, elliptical parameter, angle, and the quality grade determination result includes: grade, total score, to form production data containing timestamp, steel plate number, marking parameter, and grade parameter, which is fed back to the PLC system for temporary storage.

[0070] Step 504, integrate the production data, detection results and multi-source data to form a complete detection data package, store it in the blockchain database to establish a quality traceability record with a timestamp, and provide iterative updated training samples for deep learning models. Specifically, first determine the structure framework of the complete detection data package, which needs to include production data, detection results and multi-source data. The production data is the data generated in step 503, which includes labeling and grading results, and all fields are retained. The detection results are the original detection data generated in step 403, which include defect categories, locations and sizes, and key parameters are retained while redundant intermediate values are removed. The multi-source data selects pre-processed enhanced image key frames (each steel plate retains image frames of the defect area and the surrounding 20mm range, with a resolution compressed to 1000x800 to balance storage and clarity) and denoised point cloud key frames (point cloud data of the defect area is retained, with a point density reduced to 100 points / cm 2 to reduce storage occupancy), ensuring that the data package contains full process information and avoids excessive data redundancy.

[0071] In the data integration process, a unique data association identifier is assigned to each steel plate, which is composed of the steel plate production batch number and the detection timestamp, such as Batch20240501-14:30:25.123. The production data, detection results and multi-source data are associated and bound through this identifier to form a complete detection data package. Further, the timestamp and data hash value required for blockchain storage are generated, where the timestamp is accurate to 1 millisecond and consistent with the detection timestamp, and the data hash value is calculated by SHA-256 algorithm to ensure data uniqueness and non-tamperability. Then, the complete detection data package, timestamp and data hash value are packaged according to the writing format requirements of the blockchain database, and a writing request is submitted through the API interface of the blockchain node. After receiving the request, the database verifies the validity of the data hash value. If the verification is passed, the data is written to the block and synchronized to all nodes, establishing a quality traceability record with a timestamp, ensuring that the data is traceable and verifiable in the subsequent traceability process.

[0072] Meanwhile, training samples suitable for iteration of the deep learning model are screened from the complete detection data package, and the screening criteria are: clear defect category, complete three-dimensional size parameters, and clear key frames of multi-source data, wherein the clear defect category means that the probability value is greater than or equal to 0.8, the complete three-dimensional size parameters mean that there is no undetected field, and the clear key frames of multi-source data mean that there is no blur or noise; for the data package meeting the criteria, an image block (size 200x200 pixels) and a point cloud segment (200 points) of the defect area are extracted, a pixel-level mask of the defect is supplemented, that is, generated based on the position and size in the original detection result, and a training sample with complete labeling is formed; more than or equal to 200 such new samples are summarized every month, and are divided into a training set and a verification set at a ratio of 7:3, and are stored in a model iteration database to provide sample data fitting the actual production scene for online learning and updating of the deep learning model, and help the model continuously adapt to the detection requirements of the production line.

[0073] As shown in Figure 2 Embodiments of the present application also provide a steel plate surface defect detection system based on multi-source data fusion, comprising: a surface pretreatment module for pretreating the surface of the steel plate with water stains to reduce residual humidity to below a predetermined low humidity threshold, and obtaining a dry surface; a collection module for collecting multi-source data based on the dry surface by synchronously triggered 3D laser line scanning cameras and 2D industrial line scanning cameras, obtaining line scanning laser point cloud data, line scanning reflectivity grayscale images, line scanning depth grayscale images, and face array reflectivity color images, establishing spatial correspondence between the multi-source data, and forming a multi-source data set based on the spatial correspondence between the multi-source data; a processing module for establishing a multi-source data calibration area on the surface of the steel plate based on the multi-source data set and performing partition grid division to obtain a grid framework; analyzing the multi-source data corresponding to the grid framework to obtain pretreatment parameter adjustment values for image enhancement and point cloud filtering; using the pretreatment parameter adjustment values, performing adaptive bilateral filtering and grayscale stretching processing on the line scanning reflectivity grayscale images, the line scanning depth grayscale images, and the face array reflectivity color images, and simultaneously performing outlier removal on the line scanning laser point cloud data to obtain a pretreated high-quality multi-source data set; a defect analysis module for inputting the pretreated high-quality multi-source data set into a pre-trained multi-data fusion neural network, extracting texture and geometric features, and obtaining a detection result after cross-modal attention mechanism fusion; a result feedback module for obtaining a structured defect report based on the detection result, transmitting the structured defect report to a PLC system through an OPC UA protocol to trigger automatic marking and quality grading operations, and storing detection data to a blockchain database for quality traceability and model iteration.

[0074] It should be noted that the system is a system corresponding to the above method, all the implementation manners in the above method embodiment are applicable to this embodiment, and the same technical effects can be achieved. The above is the preferred embodiment of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A steel plate surface defect detection method based on multi-source data fusion, characterized in that, The method comprises: Step 100, water spot pretreatment is performed on the surface of the steel plate to reduce the residual humidity to below a predetermined low humidity threshold, and a dry surface is obtained; Step 200, based on the dry surface, multi-source data is collected by a 3D laser line scanning camera and a 2D industrial line scanning camera triggered synchronously, line scanning laser point cloud data, line scanning reflectivity grayscale images, line scanning depth grayscale images, and area array reflectivity color images are obtained, a spatial correspondence between the multi-source data is established, and a multi-source data set is formed based on the spatial correspondence between the multi-source data; Step 300, based on the multi-source data set, a multi-source data calibration region is established on the surface of the steel plate and is divided into a partition grid to obtain a grid framework; the multi-source data corresponding to the grid framework is analyzed to obtain a pretreatment parameter adjustment value for image enhancement and point cloud filtering; the pretreatment parameter adjustment value is used to perform adaptive bilateral filtering and grayscale stretching processing on the line scanning reflectivity grayscale images, the line scanning depth grayscale images, and the area array reflectivity color images, and to perform outlier removal on the line scanning laser point cloud data to obtain a high-quality multi-source data set after pretreatment; Step 400, the high-quality multi-source data set after pretreatment is input into a pre-trained multi-data fusion neural network, texture and geometric features are extracted, and a detection result is obtained after fusion through a cross-modal attention mechanism; Step 500, based on the detection result, a structured defect report is obtained, which is transmitted to a PLC system through an OPC UA protocol to trigger automatic marking and quality grading operations, and detection data is stored in a blockchain database for quality traceability and model iteration.

2. The steel sheet surface defect detection method based on multi-source data fusion according to claim 1, characterized by, The step 100 comprises: A humidity monitoring of the surface of the steel plate entering a detection area is performed through an infrared humidity sensor to obtain a surface humidity state signal; The surface humidity state signal is processed, and a water removal device starting signal is obtained when it is determined that there is liquid water interference; According to the water removal device starting signal, a high-pressure air blowing unit and a mechanical scraping unit are controlled to perform cooperative water removal work to obtain a steel plate surface after preliminary water removal; Real-time humidity monitoring is performed on the steel plate surface after preliminary water removal, and when the monitoring data indicates that the residual humidity reaches a predetermined low humidity threshold, a work termination signal is obtained and a dry surface is confirmed. 3.The steel plate surface defect detection method based on multi-source data fusion according to claim 2, characterized in that, The step 200 comprises: Based on the dry surface, the 3D laser line scanning camera and the 2D industrial line scanning camera are controlled to perform synchronous collection through a synchronous triggering signal to obtain original 3D sensing data and 2D image data; Point cloud reconstruction and reflectivity analysis are performed on the original 3D sensing data to obtain line scanning laser point cloud data, line scanning reflectivity grayscale images, and line scanning depth grayscale images; Color space conversion and pixel calibration are performed on the 2D image data to obtain an area array reflectivity color image; Based on the line scanning laser point cloud data and the area array reflectivity color image, a spatial correspondence between the line scanning laser point cloud data and the area array reflectivity color image is established through a pre-calibrated sensor pose transformation matrix; Based on the spatial correspondence relationship, and by using the inherent index correlation between the line-scan reflectivity gray image, the line-scan depth gray image and the line-scan laser point cloud data, all the multi-source data are unified to the same spatial coordinate system to form a multi-source data set with unified spatial correspondence relationship. 4.The steel plate surface defect detection method based on multi-source data fusion according to claim 3, characterized in that, The step 300 comprises: Based on the unified spatial coordinate system of the multi-source data set, the boundary coordinates of the effective detection region on the steel plate surface are extracted; and the geometric center position and the normal vector direction of the detection region are calculated according to the boundary coordinates of the effective detection region on the steel plate surface; Based on the geometric center position and the normal vector direction, a local coordinate system of the steel plate surface is established; and the boundary coordinates of the scanning coverage range are calculated in the local coordinate system according to the horizontal and vertical scanning field angles of the 3D laser line-scan camera; Based on the boundary coordinates of the scanning coverage range, the spatial range of the multi-source data calibration region on the steel plate surface is determined, and the multi-source data calibration region is established; Based on the spatial range of the multi-source data calibration region, the reference grid cell size meeting the requirements of point cloud data feature analysis is calculated; According to the reference grid cell size, the grid division quantity is calculated along the longitudinal and transverse directions of the steel plate surface, respectively; Based on the grid division quantity, regular geometric cells are established in the multi-source data calibration region to obtain a grid framework completely matching the spatial range of the multi-source data calibration region. 5.The steel plate surface defect detection method based on multi-source data fusion according to claim 4, characterized in that, The step 300 further comprises: Based on the grid framework, the density distribution feature of the point cloud data and the gray gradient feature of the image data in each grid cell are calculated, respectively; The density distribution feature of the point cloud data is normalized to obtain a normalized density coefficient, and the amplitude statistics of the gray gradient feature of the image data is performed to obtain an average gradient intensity; The normalized density coefficient and the average gradient intensity are weighted and fused according to a predetermined weight ratio to obtain a weighted fusion result; Based on the weighted fusion result, the preprocessing parameter adjustment value for image enhancement processing and the preprocessing parameter adjustment value for point cloud processing are generated synchronously. 6.The steel plate surface defect detection method based on multi-source data fusion according to claim 5, characterized in that, The step 300 further comprises: Based on the preprocessing parameter adjustment value for image enhancement processing, the spatial domain sigma value and the gray domain sigma value corresponding to the line-scan reflectivity gray image, the line-scan depth gray image and the area array reflectivity color image are calculated, respectively; the spatial domain sigma value and the gray domain sigma value are used to dynamically configure the bilateral filter parameters of each image; and the images are filtered by using the configured bilateral filter parameters to obtain image data after noise suppression; Based on the preprocessing parameter adjustment value for point cloud processing, the dynamic distance threshold value of the line-scan laser point cloud data is calculated; the parameters of the statistical filter are configured by using the dynamic distance threshold value of the line-scan laser point cloud data; and the statistical outlier filtering processing based on k-nearest neighbors is performed on the line-scan laser point cloud data based on the parameters of the statistical filter to obtain three-dimensional point cloud data after denoising. Performing contrast enhancement gray scale stretching processing on the noise-reduced image data to obtain enhanced image data; combining the enhanced image data with the denoised three-dimensional point cloud data to form a preprocessed high-quality multi-source data set.

7. The steel sheet surface defect detection method based on multi-source data fusion according to claim 6, characterized by, The step 400 comprises: Inputting the enhanced image data in the preprocessed high-quality multi-source data set into a 2D feature extraction branch of a pre-trained multi-data fusion neural network, extracting a two-dimensional feature vector containing texture details through an improved ResNet-101 network, and simultaneously inputting the denoised three-dimensional point cloud data into a 3D feature extraction branch, extracting a three-dimensional feature vector containing geometric shapes through a point cloud convolution network; Based on the two-dimensional feature vector and the three-dimensional feature vector, a channel dimension splicing is performed to obtain a fusion feature; the fusion feature is input into a cross-modal attention mechanism to calculate the correlation weight between different modal features, and the fusion feature is weighted and reconstructed based on the correlation weight to obtain a cross-modal fusion feature vector; The cross-modal fusion feature vector is input into a defect classification branch and a regression branch, and a defect category probability distribution is output through a softmax activation function of the defect classification branch, while a defect position coordinate and a three-dimensional size parameter are output through a linear activation function of the regression branch, and finally a detection result containing a defect category, position information and three-dimensional size is generated. 8.The steel plate surface defect detection method based on multi-source data fusion according to claim 7, characterized in that, The step 500 comprises: Based on the defect category, position information and three-dimensional size parameter in the detection result, a structured defect report containing defect feature description, spatial coordinates and geometric size is obtained; The structured defect report is transmitted to a production line PLC control system through an OPC UA communication protocol, so that the PLC system obtains defect detection information; Based on the defect detection information, an automatic marking device is triggered to perform an inkjet marking operation at the defect position, and a quality grading module is activated to determine the quality grade of the steel plate, and production data containing marking and grading results are obtained; The production data, the detection result and the multi-source data are integrated to form a complete detection data package, which is stored in a blockchain database to establish a time-stamped quality traceability record, and provides an iteratively updated training sample for a deep learning model.

9. A steel plate surface defect detection system based on multi-source data fusion, the system implements the method of any one of claims 1 to 8, characterized in that, Comprise: A surface pretreatment module for water stain pretreatment of the surface of the steel plate, reducing residual humidity to below a predetermined low humidity threshold, and obtaining a dry surface; A collection module for collecting multi-source data based on the dry surface through a synchronously triggered 3D laser line scan camera and a 2D industrial line scan camera, obtaining line scan laser point cloud data, line scan reflectivity grayscale image, line scan depth grayscale image and area array reflectivity color image, establishing a spatial correspondence relationship between the multi-source data, and forming a multi-source data set based on the spatial correspondence relationship between the multi-source data; A processing module for establishing a multi-source data calibration area on the surface of the steel plate based on the multi-source data set and performing partition grid division to obtain a grid framework; analyzing the multi-source data corresponding to the grid framework to obtain a preprocessing parameter adjustment value for image enhancement and point cloud filtering; The preprocessing parameter adjustment value is used to perform adaptive bilateral filtering and gray stretching processing on the line-scan reflectivity gray image, the line-scan depth gray image and the area-array reflectivity color image, and to perform outlier removal on the line-scan laser point cloud data, to obtain a preprocessed high-quality multi-source data set; The defect analysis module is configured to input the preprocessed high-quality multi-source data set into a pre-trained multi-data fusion neural network, extract texture and geometric features, and obtain a detection result after cross-modal attention mechanism fusion; The result feedback module is configured to obtain a structured defect report based on the detection result, transmit the structured defect report to a PLC system through an OPC UA protocol to trigger automatic marking and quality grading operations, and store detection data in a blockchain database for quality traceability and model iteration.

10. A computing device, comprising: The method comprises the following steps: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

Images