A multi-view stereo vision steel plate flatness detection method and system based on laser feature points

Abstract

The application discloses a kind of based on laser feature point's multi-view stereo vision steel plate flatness detection method and system, belong to the field of steel plate flatness detection, comprising: by projecting parallel laser stripe to steel plate surface, at least two groups of binocular vision module with overlapping field of view are used to synchronous acquisition image;Image processing is extracted laser stripe subpixel center coordinates, by stereo matching and three-dimensional reconstruction obtains local three-dimensional point cloud;Further, each point cloud is spliced and fused into complete three-dimensional point cloud covering the whole surface of steel plate;Finally, through plane fitting and gridding distance calculation, the quantitative evaluation of steel plate overall and local flatness is realized.The system can realize non-contact, high efficiency, high precision online automatic detection, effectively overcome the problems of low efficiency, incomplete data, poor environmental adaptability of traditional contact measurement, suitable for steel plate quality automatic detection and digital management in continuous production line.

Description

Technical Field

[0001] This invention belongs to the field of steel plate flatness detection, and particularly relates to a multi-view stereo vision method and system for steel plate flatness detection based on laser feature points. Background Technology

[0002] Steel plate flatness is a key quality indicator that measures the degree of deviation of the steel plate surface from an ideal plane, directly affecting the quality and efficiency of subsequent processes such as cutting, welding, and forming. Currently, the industrial field mainly relies on contact or quasi-contact measurement methods to detect steel plate flatness. Typical technical solutions include two categories: one is based on a mechanical slide rail driving a contact sensor (such as a capacitive probe) to perform single-point or single-line scanning, and calculates the flatness by measuring the change in distance between the probe and the steel plate surface; the other is to use a pressure plate with a pressure-sensing layer to press down on the steel plate surface, and determine the uniformity of flatness by analyzing the pressure feedback spectrum. Although these methods can achieve a basic judgment of whether flatness exists, they are generally static or low-speed intermittent measurements in terms of detection mode.

[0003] However, the aforementioned existing technologies have significant drawbacks in practical industrial applications. First, their detection efficiency is low: mechanical scanning methods are limited by the speed of the moving mechanism, while pressure sensing methods require the steel plate to stop transport and undergo multiple "move-stop-press" operations, neither of which can meet the cycle time requirements of real-time, online detection in continuous production lines. Second, their data dimensions are insufficient and their completeness is poor: existing methods can usually only obtain discrete point, line, or two-dimensional pressure distribution information, making it difficult to reconstruct the full-field three-dimensional morphology of the steel plate surface. This easily leads to the omission of complex defects such as localized minor warping and wave deformation, and the data provided cannot be directly used to guide automated leveling or welding path planning. In addition, contact measurement also has applicability issues such as damaging the steel plate surface and being susceptible to interference from on-site vibration and dust. Therefore, there is currently a lack of an automated flatness detection solution that can achieve full-field, non-contact, high-precision, and environmentally robust flatness detection during the continuous transport of large-size steel plates. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-view stereo vision method for steel plate flatness detection based on laser feature points, comprising: Multiple parallel and spaced laser stripes are projected onto the surface of the steel plate to be tested, and images of the steel plate surface with laser stripes are simultaneously acquired by at least two binocular vision modules with overlapping fields of view. The acquired images are processed to extract the sub-pixel coordinates of the center line of the laser stripes in each image; Based on the internal and external parameters of each binocular vision module, stereo matching and three-dimensional reconstruction are performed according to the extracted laser stripe center line to obtain the three-dimensional point cloud of the steel plate surface in the field of view of each module. The 3D point clouds obtained by each binocular vision module are uniformly transformed to the same global coordinate system and then stitched and fused to generate complete 3D point cloud data covering the entire surface of the steel plate. Based on the complete 3D point cloud data, the flatness of the steel plate in its overall and local areas is quantitatively evaluated through plane fitting and distance calculation.

[0005] Optionally, the step of extracting the sub-pixel coordinates of the laser stripe center line includes: The region of interest containing laser stripes is located and segmented from the acquired images; A center extraction algorithm based on the Hessian matrix is ​​adopted. By calculating the Hessian matrix of each pixel in the image and obtaining the feature vector corresponding to its maximum eigenvalue, the pixel-level normal direction of the laser stripes in the region of interest is determined. Along the normal direction, the gray-level distribution function of the laser stripe cross section is expanded by a second-order Taylor series. By solving for the points where the first derivative of the expansion is zero, the sub-pixel coordinates of the laser stripe center line are obtained.

[0006] Optionally, after obtaining the sub-pixel coordinates of the laser stripe centerline, the method further includes: A smoothing filter based on local polynomial least squares fitting is used to perform sliding window filtering on the sub-pixel coordinate sequence to filter out high-frequency jitter caused by sensor noise or surface micro-texture, while retaining the true waveform characteristics of the steel plate surface.

[0007] Optionally, the step of performing 3D reconstruction based on the intrinsic and extrinsic parameters of each binocular vision module includes: Based on the fundamental matrix obtained from the calibration, calculate the epipolar equation of the laser center point extracted from the left image in the right image; A one-dimensional search is performed near the epipolar line, using the vertical distance from the pixel to the epipolar line as the matching metric to filter out pairs of feature points with the same name that satisfy the geometric constraints. Using the pixel coordinates of the corresponding point pair and the left and right camera projection matrices obtained through calibration, the three-dimensional coordinates of the point in space are solved by the least squares method.

[0008] Optionally, the step of uniformly transforming each 3D point cloud to the same global coordinate system and then stitching and fusing them includes: Based on the rotation matrix and translation vector between the reference camera of each binocular vision module and the global reference camera of the system obtained through global calibration, rigid body transformation is performed on each local 3D point cloud to transform it into the global coordinate system. For spatially adjacent points formed by different point clouds within the overlapping field of view, fusion weights are assigned based on the position of each point in its source camera's field of view or reconstruction confidence, and their weighted average coordinates are calculated to achieve smooth fusion of point cloud data.

[0009] Optionally, the step of quantitatively evaluating the flatness of the steel plate includes: The least squares method is used to perform plane fitting on the complete three-dimensional point cloud data, and the equation coefficients of the reference plane are obtained by solving the equation. The surface of the steel plate is divided into multiple grid regions according to a preset size, and the vertical distance from all three-dimensional points in each grid region to the reference plane is calculated. The set of vertical distances within each grid area is statistically analyzed, and at least one of its maximum, average, or root mean square value is calculated as a quantitative indicator of the flatness of that area.

[0010] Optionally, a system calibration step is also included before acquiring the steel plate image: Using the calibration board, single-target calibration is performed on the left and right cameras in each binocular vision module to obtain the internal parameter matrix, lens distortion coefficient, and external pose parameters of each camera relative to the calibration board. By placing at least one common target within the overlapping field of view of multiple binocular vision modules, or by using multiple targets with known geometric relationships, a pose transformation chain between cameras of different modules is established, and the unified rotation and translation relationship of all cameras relative to a selected reference camera is solved, thus completing the global calibration of the multi-view system.

[0011] To address the aforementioned technical problems, this invention provides a multi-view stereo vision steel plate flatness detection system based on laser feature points, comprising: The structured light projection module is used to project multiple parallel and spaced laser stripes onto the surface of a moving steel plate. The multi-view image acquisition module contains at least two binocular vision modules with overlapping fields of view. Each module consists of two industrial cameras for synchronously acquiring images of steel plates with laser stripes. The core processing module is used to perform image processing, subpixel coordinate extraction, 3D reconstruction, point cloud stitching, and flatness calculation. A synchronization control module is used to receive production line signals and precisely control the timing of the coordinated operation of the structured light projection module and the multi-view image acquisition module; The human-computer interaction module is used for system parameter setting, process monitoring, and visualization of test results.

[0012] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0013] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0014] Compared with the prior art, the present invention has the following advantages and technical effects: This invention proposes a multi-view stereo vision method and system for steel plate flatness detection based on laser feature points, effectively solving the industry problems of low efficiency, incomplete data, and difficulty in online application of traditional contact detection technology. The system projects parallel laser stripes onto the steel plate surface and simultaneously acquires images using two sets of binocular vision modules with overlapping fields of view. Combined with a sub-pixel-level light stripe center extraction algorithm and 3D reconstruction technology, it efficiently reconstructs a high-precision 3D point cloud covering the entire plate surface. Then, through point cloud stitching, plane fitting, and meshed partitioning calculation, it achieves accurate quantitative evaluation of the overall and local flatness of the steel plate.

[0015] This method boasts outstanding advantages such as non-contact operation, full automation, and high environmental adaptability: the inspection process requires no interruption of the steel plate, and a single measurement can be completed within 15 seconds, meeting the requirements of continuous production cycles; it employs PLC synchronous control and a dedicated protective design to effectively resist dust and vibration interference in industrial environments; the measurement accuracy can reach ±1.5mm, and it can provide complete two-dimensional morphological data, providing a direct basis for subsequent automated correction. The system is also equipped with an intuitive human-machine interface, enabling process monitoring, result visualization, and digital report generation, significantly improving the automation and intelligence level of steel plate quality inspection, and contributing to the digital transformation and quality control upgrade of the heavy manufacturing industry. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the overall steel plate flatness detection system according to an embodiment of the present invention; Figure 2 This is a deployment diagram of the steel plate flatness detection system according to an embodiment of the present invention; Figure 3 This is a calibration diagram of a multi-view camera according to an embodiment of the present invention; Figure 4 This is a laser centerline extraction diagram according to an embodiment of the present invention; Figure 5This is a diagram of a binocular multi-line structured light measurement model according to an embodiment of the present invention; Figure 6 This is a schematic diagram of reprojection matching according to an embodiment of the present invention; Figure 7 This is a 3D point cloud mosaic image according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the flatness according to an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0019] Example 1 This embodiment provides a multi-view stereo vision method for steel plate flatness detection based on laser feature points, including: Multiple parallel and spaced laser stripes are projected onto the surface of the steel plate to be tested, and images of the steel plate surface with laser stripes are simultaneously acquired by at least two binocular vision modules with overlapping fields of view. The acquired images are processed to extract the sub-pixel coordinates of the center line of the laser stripes in each image; Based on the internal and external parameters of each binocular vision module, stereo matching and three-dimensional reconstruction are performed according to the extracted laser stripe center line to obtain the three-dimensional point cloud of the steel plate surface in the field of view of each module. The 3D point clouds obtained by each binocular vision module are uniformly transformed to the same global coordinate system and then stitched and fused to generate complete 3D point cloud data covering the entire surface of the steel plate. Based on the complete 3D point cloud data, the flatness of the steel plate in its overall and local areas is quantitatively evaluated through plane fitting and distance calculation.

[0020] To eliminate camera lens distortion and unify the spatial coordinate systems of each camera, a step-by-step calibration is required. First, single-camera calibration is performed within the field of view of each camera using a calibration board, obtaining internal parameters including focal length and principal point coordinates, as well as distortion coefficients. Then, global calibration of the multi-view system is performed, establishing the relative pose relationships (rotation and translation matrices) between the four cameras. Through calibration calculations, the coordinate systems of the four independent fields of view are transformed to the reference camera coordinate system, laying the foundation for subsequently stitching together the scattered local point clouds to form a complete 3D topography of the steel plate surface.

[0021] Before the steel plate enters the inspection area, the system scans the reference plane to obtain reference data. Once the steel plate enters the field of view, the PLC triggers a linear laser source to project purple laser stripes onto the steel plate surface at 20cm intervals, simultaneously triggering the camera to acquire images. To address the complex background characteristics of industrial environments, a deep learning-based laser region extraction module was designed. A neural network model is trained using a pre-collected and labeled laser stripe dataset. The system then uses this model to infer the acquired images, accurately locating and extracting the regions of interest (ROIs) containing the laser stripes, effectively filtering out environmental background noise interference.

[0022] Before the steel plate enters the inspection area, the system first scans the reference plane to obtain reference data. Once the steel plate enters the field of view, the PLC triggers a linear laser source to project 450nm purple laser stripes onto the steel plate surface at 20cm intervals, simultaneously triggering four industrial cameras to acquire images. Addressing the complex backgrounds and ambient light interference in industrial environments, this invention employs a region of interest (ROI) extraction strategy based on traditional image processing.

[0023] The system first uses an adaptive threshold segmentation algorithm to separate the high-brightness laser stripes from the dark background. Then, it combines morphological opening and closing operations to filter out speckle noise and discontinuous reflection stray light in the image, thereby accurately locating and extracting the effective area containing the laser stripes. While ensuring the real-time performance of the algorithm, it effectively filters out the interference of environmental background noise.

[0024] For the extracted ROI region, the Steger algorithm based on the Hessian matrix is ​​first used to extract the sub-pixel centerline of the light stripe. This algorithm obtains the normal direction of the light stripe by calculating the Hessian matrix of the image, and fits the gray-level distribution function using a second-order Taylor polynomial expansion, thereby accurately calculating the sub-pixel position of the light stripe center.

[0025] To further eliminate high-frequency jitter in the extracted coordinates caused by camera sensor noise or microscopic textures on the steel plate surface, the system then employs the Savitzky-Golay (SG) filtering algorithm to smooth and denoise the extracted light stripe center coordinate sequence. The SG filter effectively removes high-frequency noise while preserving the true waveform characteristics of the steel plate surface (such as minor edge warping) to the greatest extent possible, avoiding signal distortion. Finally, in the 3D reconstruction stage, the system performs calculations based on a distributed binocular vision model.

[0026] The two binocular modules on the left (cameras 1 and 2) and the right (cameras 3 and 4) respectively use epipolar constraints and stereo matching principles to transform the feature points of the left and right views after SG filtering into three-dimensional spatial point clouds in the local coordinate system. Then, according to the global calibration parameters, the point cloud data of the right module is uniformly transformed into the reference coordinate system of the left for spatial fusion to generate complete three-dimensional point cloud data covering the entire surface of the steel plate.

[0027] Based on the restored 3D point cloud of the steel plate surface, the least squares method is used to fit the current plane equation of the steel plate surface. To refine the flatness assessment, the 5m×2m steel plate is divided into 12 small regions. The vertical distance from the spatial coordinate point on the center line of the surface stripe in each region to the fitted plane is calculated, and key indicators such as the maximum deviation value and surface undulation of each region are statistically analyzed. The system automatically determines the flatness level of the entire plate and each local region by comparing it with the preset flatness standard.

[0028] Specifically: First, the intrinsic and extrinsic parameters of the monocular camera are calibrated using the camera calibration module, such as... Figure 3 As shown, by taking multiple images of the calibration plate at different positions with the camera and then processing the images, the camera's intrinsic parameters can be obtained, thus completing single-target calibration.

[0029] To achieve the measurement objective, the coordinate systems of cameras 1 to 4 are set as follows: , , and ; It is a world coordinate system with the center point of the calibration plate as the origin. Let the coordinates of a feature point on the calibration board be the three-dimensional coordinates. We have: ; in, As a scale factor, and From world coordinate system to camera The rotation matrix and translation vector in the coordinate system are the camera's external parameters. The camera's intrinsic parameter matrix can be written as: ; in, The coordinates of the principal point in the image. and These are the scale factors for the u-axis and v-axis in the image pixel coordinate system, respectively.

[0030] Considering the radial and tangential distortion of the camera lens, set points... Ideal imaging point The coordinates in the image coordinate system are ,point actual imaging point The coordinates in the image coordinate system are Image distortion is corrected based on the following distortion model: ; in, It is the radial distortion coefficient of the lens. This represents the tangential distortion coefficient of the lens. .

[0031] Based on the aforementioned imaging and distortion models, the intrinsic parameter matrix of a single camera can be obtained by processing calibration board images in different poses. and distortion coefficient .

[0032] After performing monocular camera calibration and solving the initial camera intrinsic parameter matrix, the 3D world coordinates of the target center point captured by the camera can be obtained. The camera parameters are then reprojected back onto the image plane based on the initially solved camera parameter matrix to obtain the reprojected pixel coordinates. Therefore, the second norm of the reprojection error for all feature points is calculated, and the following equation is used as the objective function to be optimized: ; In the formula, j is the number of target center points identified by the camera; n represents the number of frames of the image captured by the camera; Let be the coordinates of the reprojected point, representing the j-th corner point calibrated by the camera in the i-th frame; Let be the actual coordinates of the feature points on the image; where h is a 3D-to-2D projection operator, representing the mapping from homogeneous 3D coordinates to image coordinates given internal and extrinsic parameters. The spatial coordinates at their minimum value are calculated using the Levenberg-Marquardt algorithm, which represents the optimal solution for the measured values. Ultimately, the optimal solutions for the intrinsic, distortion coefficients, and extrinsic parameters of a single camera are obtained, achieving high-precision calibration of a large field-of-view monocular camera and providing high-precision data for subsequent calibration of large-size steel plate flatness detection systems.

[0033] The multi-view camera calibration process includes: The system employs a customized concentric circle array calibration target. This target integrates feature points of different scales and specific encoding patterns. This multi-scale encoding design significantly improves the extraction accuracy of feature circle centers, thereby enhancing the robustness and stability of the system calibration. The camera's intrinsic parameters and relative pose parameters are calculated based on Zhang Zhengyou's planar calibration method.

[0034] like Figure 3As shown, the measurement module on the left consists of camera C1 and camera C4. The calibration target is placed within the common field of view of both cameras. Using a stereo calibration algorithm, the pose transformation matrix of the subordinate camera C4 relative to the reference camera C1 is calculated. : ; Similarly, the measurement module on the right consists of camera C2 and camera C3. After acquiring the corresponding calibration images, the transformation matrix of camera C3 relative to its local reference camera C2 is calculated. : ; To achieve spatial fusion of measurement data from the left and right sides, a connection between local reference points needs to be established. Utilizing the overlapping field of view between cameras C1 and C2, the rigid transformation matrix of the right-side reference camera C2 relative to the global master camera C1 is directly calculated by photographing a global calibration reference object within this region. .

[0035] Since there is no common field of view between the edge camera C3 and the global master camera C1, direct calibration cannot be implemented. Their spatial relationship needs to be indirectly derived through the intermediate node camera C2.

[0036] Based on the transitivity of rigid body motion, the coordinate transformation path from camera C3 to global master camera C1 is established as follows: ; Based on the matrix multiplication rule of Lie group spaces, the two transformation processes mentioned above are cascaded. The equivalent transformation matrix of the edge camera C3 relative to the global main camera C1 is... The calculation is as follows: ; Substituting each block matrix into the formula and expanding it, we obtain the final global parameter expression: ; Right now: ; Therefore, the coordinate systems of all cameras within the system are uniformly mapped to the global principal coordinate system. This achieves precise unification of measurement standards across the entire field.

[0037] The process of 3D scene reconstruction includes: Laser centerline extraction: such as Figure 4As shown, the Steger algorithm is used to extract the center of laser stripes on the steel plate surface. Based on the Hessian matrix, the normal to the light stripe is obtained, and then the center point is obtained by Taylor series expansion, thus realizing the extraction of sub-pixel coordinates of the center of the light stripe. From the above light intensity distribution map, it can be seen that the center of the line should be located at the zero point of the first derivative and the minimum value of the second derivative. These two conditions can be used to extract the center point of the line edge. The basic idea of ​​this method is to obtain the normal direction of the center line of the light stripe by solving the eigenvalues ​​and eigenvectors of the Hessian matrix of the image, and then perform Taylor series expansion along this normal direction to obtain the sub-pixel coordinates of the center point of the light stripe. The basic steps are as follows: ① By performing a convolution operation with a Gaussian kernel function in the form of its partial derivative, the image pixels are obtained. The Hessian matrix; ② Solve for the eigenvalues ​​and eigenvectors of the Hessian matrix. The normal direction of the center line of the light stripe can be obtained by using the eigenvector corresponding to the eigenvalue with the largest absolute value. ③ Expand the gray-scale distribution function of the light stripe along the normal direction using a second-order Taylor series. By taking the extreme value condition of the quadratic polynomial, i.e., the first derivative passes through the zero point, the sub-pixel coordinates of the center point of the light stripe can be obtained.

[0038] The above Hessian matrix can be expressed as: ; In the formula, A two-dimensional Gaussian function: ; In the formula, The mean square error of Gaussian can affect the smoothing effect of light stripe images. The larger the value, the more significant the smoothing effect on the light stripe image, but when... When the value is too large, it will cause the light stripe image to appear blurry and fuzzy, and The value is related to the width of the laser stripe.

[0039] Let the unit vector of the normal direction obtained by solving the Hessian matrix above be... With point Using the base point as an example, and then performing a second-order Taylor series expansion on the gray-level distribution function of the light fringe cross section, the points on the light fringe cross section are then... The grayscale value is as follows: ; In the formula, , From light stripe image pixels The results are obtained by convolving each kernel with a Gaussian kernel, namely: ; make According to the above formula, we can obtain: ; The center point of the light stripe is Subpixel coordinates are .

[0040] The process of structured light 3D reconstruction based on distributed binocular components includes: This invention employs a distributed binocular vision architecture, dividing four industrial cameras into two binocular measurement modules (camera 1 and camera 2 form the left measurement module, and camera 3 and camera 4 form the right measurement module). The binocular multi-line structured light measurement technology combines the advantages of active structured light projection and passive binocular vision. By projecting high-contrast laser stripes onto the surface of the target object, it provides high-precision matching features for steel plate surfaces lacking texture features, thereby significantly improving the robustness and reconstruction accuracy of stereo matching.

[0041] During implementation, the system first performs independent 3D reconstruction on each binocular module. Taking the left module as an example, the 3D spatial coordinates of the laser line stripe center in the camera 1 coordinate system are accurately calculated using the binocular line structured light model.

[0042] Figure 5 This is a binocular multi-line structured light measurement model. It captures images of a target object with laser lines from different angles using a binocular camera and accurately extracts the center point of the structured light in the image, thereby solving the three-dimensional coordinates of the real-world points corresponding to the left and right feature points.

[0043] The pixel coordinates of the intersection point between the midline structured light and one side of the calibration board in the obtained image. Assume that the three-dimensional coordinates of the measured point in the reference world coordinate system are: but: ; In the formula, the symbol This represents the scaling factor, which is closely related to the distance between the optical center of the camera and the point being measured in space.

[0044] This represents the projection matrix that incorporates both intrinsic and extrinsic camera parameters. The expression is: ; For the left and right cameras, respectively: ; In the formula, based on the projection matrix of the left camera And the projection matrix of the camera on the right. The three-dimensional coordinates of the corresponding point in the reference world coordinate system are obtained using the least squares method. .

[0045] In binocular stereo vision geometry, the projection of any point P in space onto the left camera image plane is: Then its corresponding point on the right camera image plane It is necessarily constrained by epipolar geometry. Specifically, the corresponding points It must lie on a specific straight line in the right image plane; this line is called the epipolar line. Utilizing this property, a two-dimensional full-image search can be reduced to a one-dimensional epipolar search, greatly narrowing the search range for feature points, such as... Figure 6 As shown.

[0046] This system uses the camera intrinsic parameters obtained from calibration and the stereo relative pose matrix to calculate the fundamental matrix. For the laser center point extracted from the left image Its corresponding polar line in the right image Satisfy the equation: ; in, The coordinates of the candidate matching points on the right image are given. The system performs reprojection matching verification by calculating the vertical distance from the laser stripe pixel to the corresponding epipolar line. A distance threshold is set. Only feature point pairs whose distance is within the threshold range are retained as corresponding points, thereby effectively filtering out noise points and false positive matches that do not meet geometric constraints.

[0047] Furthermore, considering the potential for minute texture interference and laser speckle noise on the surface of steel plates in industrial settings, the directly calculated 3D point cloud data often contains high-frequency noise, leading to pseudo-ripples in flatness detection. To preserve the true undulation features of the steel plate surface (such as edge warping due to stress) while smoothing and denoising, this invention employs a Savitzky-Golay (SG) filter to smooth the extracted laser center point sequence or the generated 3D contour line.

[0048] SG filtering is a filtering method based on local polynomial least squares fitting. Its principle is to use a sliding window of length 2m+1 to fit the 2m+1 data points within the window to a single value using the least squares method. A polynomial of order 1 is used to calculate the smoothed value of the center point. Compared with traditional moving average filtering, the SG filter can better maintain the width and height of the waveform signal while filtering out high-frequency noise, avoiding the "smoothing" effect on the subtle defects on the steel plate surface, thus ensuring high fidelity in subsequent flatness calculations.

[0049] The spatial stitching and fusion process of multi-module point clouds includes: After the aforementioned binocular reconstruction process, the system obtained two independent sets of 3D point cloud data: the left region point cloud generated by cameras 1 and 2 (based on the camera 1 coordinate system) and the right region point cloud generated by cameras 3 and 4 (based on the camera 3 coordinate system). In order to obtain the full-field 3D topography of the steel plate surface, these two sets of data must be unified into the system's global coordinate system.

[0050] In this embodiment, the optical center coordinate system of camera 1 is selected as the system's global coordinate system. Through global calibration, the pose transformation relationship between the right-side module reference camera (camera 3) and the global reference camera (camera 1), i.e., the rotation matrix, is obtained in advance. With translation vector As shown below: ; Based on this pose transformation relationship, the local 3D point cloud measured by the right-side binocular module can be transformed. (i.e., the point set in the camera's 3-coordinate system) is uniformly transformed to the global coordinate system. Let... For any point in the point cloud on the right, its transformed coordinates The calculation formula is: ; By performing rigid body transformation on all measurement points of the right module, it can be compared with the direct measurement point cloud of the left module. Spatial registration is complete.

[0051] ; For the overlapping region of the two binocular modules, the system uses a weighted average fusion algorithm. For close spatial points within the overlapping region, weighted fusion is performed based on their confidence levels in their respective camera fields of view to eliminate the "step effect" caused by stitching errors.

[0052] In summary, through distributed binocular reconstruction and global stitching technology, the system scans both with and without a steel plate, ultimately obtaining a standard reference plane point cloud without a steel plate and a full-field 3D point cloud of the measured surface with a steel plate. This provides complete high-precision data support for subsequent flatness meshing calculations. The full-field 3D point cloud of the measured steel plate surface is shown below. Figure 7 As shown.

[0053] The flatness inspection process includes: This system first uses the least squares method to fit the spatial coordinates of points along the center line of the laser stripes to derive the standard plane equation for the area without steel plate coverage. Then, it calculates the distance from the spatial coordinates of the points along the center line of the laser stripes on the steel plate surface to the corresponding standard plane equation, which serves as the standard for judging flatness. Figure 8 As shown.

[0054] set up , , These are three points in a 3D point cloud that are not on the same straight line, and their object point spatial coordinates are respectively... These three points form a plane W. For any point on this plane... All satisfy the following formula: ; After expansion, the plane equation of W can be obtained as follows: ; in, , , , Since the standard plane does not pass through the origin, the plane can be simplified again: ; in, , , To accurately determine the coefficients of the standard plane equation, it is necessary to obtain the data of the remaining three-dimensional coordinate points along the center line of the laser stripe. Substitute the equations into the plane equations and list the error equations: ; Let the first one be substituted into the equation , , The obtained equation coefficients are the initial values. , , Therefore, the error equation used in actual calculations is the sum of the initial value and the adjusted value, as shown in the following formula: ; in , , The values ​​within the right parentheses are correction values ​​for the unknown plane parameters, and are treated as constants. Then the error equation formed by the coordinates of all three-dimensional points on the median line can be rewritten in the following matrix-vector form: ; In the formula, , , , .

[0055] By minimizing the error using the least squares method, the solution for the corrected vector X of the plane parameters can be obtained as follows: ; Therefore, the coefficients of the standard plane equation after correction can be obtained.

[0056] The steel plate is divided into 12 smaller planes. For each smaller plane, the distance from the center point of all laser stripes to the standard plane is calculated, as shown in the following formula: ; The distance between the center point of each laser stripe on each small plane The flatness of each small surface is constituted The formula is as follows: ; For the flatness obtained from each small surface With a preset balance threshold By making comparisons, the flatness of the steel plate surface in a small area can be determined.

[0057] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0058] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0059] Example 2 This embodiment provides a multi-view stereo vision steel plate flatness detection system based on laser feature points, including: like Figure 1As shown, the system mainly consists of four high-resolution industrial cameras, four sets of linear laser emitters, a programmable logic controller (PLC), an industrial control computer (ICC), and a display screen. The four industrial cameras and four sets of laser emitters are mounted on the columns surrounding the flattening machine using customized protective components, arranged in an overlapping field of view to cover the large steel plate area. The ICC, as the core processing unit of the system, is connected to the four industrial cameras via a gigabit Ethernet port (GigE) and is responsible for high-speed acquisition and processing of image data. Simultaneously, the ICC establishes a connection with the PLC via the TCP / IP communication protocol to achieve the interaction of control commands. The PLC is responsible for monitoring the production line status and controlling the start and stop of the laser emitters and the triggering sequence of the cameras. The steel plate to be tested is conveyed to the detection area via a roller conveyor. After the PLC detects the arrival signal, it triggers the 450nm wavelength purple linear laser to turn on through the IO port, and simultaneously triggers the industrial cameras to acquire images. The acquired image data with laser stripes is transmitted to the ICC, where it is processed using deep learning and 3D reconstruction algorithms to calculate the 3D point cloud data and flatness index of the steel plate surface. Finally, the measurement results, panoramic images, and 3D point clouds are displayed in real time on a screen, enabling online automated detection of the flatness of the steel plate.

[0060] The steel plate flatness detection system designed in this invention is mainly composed of hardware modules and software modules working together, such as... Figure 2 As shown, the hardware module monitors production line signals in real time through a PLC logic control module, precisely and synchronously controlling the start and stop of the 450nm laser emitter and the hard-triggered acquisition of the industrial camera. Combined with dedicated protective components, this ensures that the core sensor stably acquires high-resolution images of the steel plate surface in dusty and vibrating industrial environments and transmits them at high speed to the industrial control computer. The software module first obtains the intrinsic parameters of each camera and the relative pose relationships between the binocular modules through a multi-view vision calibration module to construct a globally unified coordinate system. Then, in the image preprocessing stage, it sets the region of interest (ROI) and uses threshold segmentation and morphological filtering to remove background noise. Finally, it applies Steg through a sub-pixel center extraction module. The ER algorithm accurately calculates the normal direction of the light stripe and the sub-pixel coordinates of the laser center point. Based on this, the Savitzky-Golay (SG) filtering algorithm is used to smooth and denoise the extracted light stripe center coordinate sequence. Then, the 3D reconstruction and stitching module uses binocular stereo matching and epipolar constraints to convert the 2D data into a 3D point cloud and fuse the data from the left and right binocular modules into the global space. Finally, the flatness calculation module uses the least squares method to fit the reference plane and performs grid partitioning to calculate the flatness error of each region. The 3D reconstruction effect, panoramic image and quantitative indicators are displayed in real time through the human-computer interaction module, realizing the automated detection and monitoring of the whole process.

[0061] The hardware modules include: Industrial Cameras and Optical Imaging Module: This invention employs four ultra-high-resolution industrial cameras (5120×5120 resolution) to construct a multi-view vision acquisition array. The cameras are connected to an industrial control computer via a Gigabit Ethernet interface (GigE) to ensure high-speed and stable transmission of massive amounts of image data. At the software level, a C++ control program is developed based on the camera's native SDK, which can dynamically adjust key imaging parameters such as exposure time, gain, and white balance according to ambient lighting conditions. A 450nm wavelength violet line laser emitter is used in conjunction with the cameras. Compared to traditional red lasers, 450nm violet light has better diffuse reflection characteristics on metal surfaces and stronger resistance to ambient light interference, generating high-contrast characteristic fringes. To ensure a stable relative positional relationship between the multi-view cameras and the measured plane, and to cope with harsh environments, each of the four industrial cameras is mounted in a dedicated camera protective housing. The protective housing is securely locked to the crossbeam support using high-strength connecting components, providing not only necessary mechanical support but also dustproof, shockproof, and impact-resistant functions. This design aims to reduce the impact of severe vibrations generated during the operation of the flattening machine on the camera's optical axis, thereby ensuring that the camera's intrinsic and extrinsic parameters remain consistent during long-term operation, avoiding measurement errors caused by sensor displacement, and greatly improving the robustness and reliability of the system.

[0062] The control and communication module employs a master-slave architecture of PLC + industrial computer as the system's control center. The PLC, acting as the slave computer, is responsible for real-time acquisition of photoelectric switch and encoder signals from the production line, directly controlling the laser's power supply via hardwiring to ensure microsecond-level response speed. The industrial control computer, acting as the master computer, communicates with the PLC via TCP / IP protocol. When the PLC determines that the steel plate has reached the predetermined detection position, it notifies the industrial computer to enter the "ready to acquire" state via TCP message and simultaneously sends a trigger signal, thus achieving precise synchronization between production line actions and visual acquisition.

[0063] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-view stereo vision method for steel plate flatness detection based on laser feature points, characterized in that, include: Multiple parallel and spaced laser stripes are projected onto the surface of the steel plate to be tested, and images of the steel plate surface with laser stripes are simultaneously acquired by at least two binocular vision modules with overlapping fields of view. The acquired images are processed to extract the sub-pixel coordinates of the center line of the laser stripes in each image; Based on the internal and external parameters of each binocular vision module, stereo matching and three-dimensional reconstruction are performed according to the extracted laser stripe center line to obtain the three-dimensional point cloud of the steel plate surface in the field of view of each module. The 3D point clouds obtained by each binocular vision module are uniformly transformed to the same global coordinate system and then stitched and fused to generate complete 3D point cloud data covering the entire surface of the steel plate. Based on the complete 3D point cloud data, the flatness of the steel plate in its overall and local areas is quantitatively evaluated through plane fitting and distance calculation.

2. The method according to claim 1, characterized in that, The step of extracting the sub-pixel coordinates of the laser stripe center line includes: The region of interest containing laser stripes is located and segmented from the acquired images; A center extraction algorithm based on the Hessian matrix is ​​adopted. By calculating the Hessian matrix of each pixel in the image and obtaining the feature vector corresponding to its maximum eigenvalue, the pixel-level normal direction of the laser stripes in the region of interest is determined. Along the normal direction, the gray-level distribution function of the laser stripe cross section is expanded by a second-order Taylor series. By solving for the points where the first derivative of the expansion is zero, the sub-pixel coordinates of the laser stripe center line are obtained.

3. The method according to claim 2, characterized in that, After obtaining the sub-pixel coordinates of the laser stripe center line, the following is also included: A smoothing filter based on local polynomial least squares fitting is used to perform sliding window filtering on the sub-pixel coordinate sequence to filter out high-frequency jitter caused by sensor noise or surface micro-texture, while retaining the true waveform characteristics of the steel plate surface.

4. The method according to claim 1, characterized in that, The steps for 3D reconstruction based on the intrinsic and extrinsic parameters of each binocular vision module include: Based on the fundamental matrix obtained from the calibration, calculate the epipolar equation of the laser center point extracted from the left image in the right image; A one-dimensional search is performed near the epipolar line, using the vertical distance from the pixel to the epipolar line as the matching metric to filter out pairs of feature points with the same name that satisfy the geometric constraints. Using the pixel coordinates of the corresponding point pair and the left and right camera projection matrices obtained through calibration, the three-dimensional coordinates of the point in space are solved by the least squares method.

5. The method according to claim 1, characterized in that, The steps of uniformly transforming each 3D point cloud to the same global coordinate system and then stitching and fusing them include: Based on the rotation matrix and translation vector between the reference camera of each binocular vision module and the global reference camera of the system obtained through global calibration, rigid body transformation is performed on each local 3D point cloud to transform it into the global coordinate system. For spatially adjacent points formed by different point clouds within the overlapping field of view, fusion weights are assigned based on the position of each point in its source camera's field of view or reconstruction confidence, and their weighted average coordinates are calculated to achieve smooth fusion of point cloud data.

6. The method according to claim 1, characterized in that, The steps for quantitatively evaluating the flatness of the steel plate include: The least squares method is used to perform plane fitting on the complete three-dimensional point cloud data, and the equation coefficients of the reference plane are obtained by solving the equation. The surface of the steel plate is divided into multiple grid regions according to a preset size, and the vertical distance from all three-dimensional points in each grid region to the reference plane is calculated. The set of vertical distances within each grid area is statistically analyzed, and at least one of its maximum, average, or root mean square value is calculated as a quantitative indicator of the flatness of that area.

7. The method according to claim 1, characterized in that, Before acquiring images of the steel plate, a system calibration step is also included: Using the calibration board, single-target calibration is performed on the left and right cameras in each binocular vision module to obtain the internal parameter matrix, lens distortion coefficient, and external pose parameters of each camera relative to the calibration board. By placing at least one common target within the overlapping field of view of multiple binocular vision modules, or by using multiple targets with known geometric relationships, a pose transformation chain between cameras of different modules is established, and the unified rotation and translation relationship of all cameras relative to a selected reference camera is solved, thus completing the global calibration of the multi-view system.

8. A multi-view stereo vision steel plate flatness detection system based on laser feature points, characterized in that, include: The structured light projection module is used to project multiple parallel and spaced laser stripes onto the surface of a moving steel plate. The multi-view image acquisition module contains at least two binocular vision modules with overlapping fields of view. Each module consists of two industrial cameras for synchronously acquiring images of steel plates with laser stripes. The core processing module is used to perform image processing, subpixel coordinate extraction, 3D reconstruction, point cloud stitching, and flatness calculation. A synchronization control module is used to receive production line signals and precisely control the timing of the coordinated operation of the structured light projection module and the multi-view image acquisition module; The human-computer interaction module is used for system parameter setting, process monitoring, and visualization of test results.

9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-7.

Images