A method for intelligent detection of a bending line of a sheet metal structural member

Abstract

The present application belongs to the technical field of sheet metal processing intelligent detection, and provides a kind of sheet metal structural member bending line intelligent detection method, comprising: the structure image of sheet metal structural member is collected, it is judged whether structure image occurs motion blur, structure image is marked as clear frame or fuzzy frame, and it is judged whether to trigger fuzzy frame clear processing, if trigger, combined with optical flow method, construct and adopt deep learning to blur network to restore fuzzy frame, output corresponding clear frame, generate clear image sequence, combined with the digital twin model of the sheet metal structural member calculated real-time six degrees of freedom pose, determine the dynamic region of interest of bending line and extract bending line position information, at the end of collection period, introduce Kalman filtering algorithm to the time series filtering fusion optimization of bending line position information, generate optimized bending line data, calculate the actual deviation of bending line key geometric parameter, and make qualified or unqualified detection decision for sheet metal structural member bending line.

Description

Technical Field

[0001] This invention belongs to the field of intelligent inspection technology for sheet metal processing, specifically a method for intelligent inspection of bending lines of sheet metal structural parts. Background Technology

[0002] Sheet metal structural parts are widely used in automobiles, home appliances, construction machinery and other fields. The position and angle accuracy of their bending lines directly determine the fit of the parts and the performance of the products. Therefore, bending line inspection is the core link of production quality control. As the manufacturing industry upgrades to high speed and high precision, some existing automated vision inspection solutions can achieve dimensional measurement in static or offline states, but when applied to continuously operating production lines, they expose many technical bottlenecks.

[0003] First, the motion blur of workpieces caused by the dynamic operation of the production line is the primary technical obstacle. Existing methods often ignore this problem or simply increase the shutter speed at the expense of the image signal-to-noise ratio, resulting in unstable quality of the acquired image sequence, which directly restricts the accuracy and reliability of subsequent analysis. Second, traditional visual inspection schemes are usually based on absolute measurements using a fixed coordinate system, which cannot effectively distinguish between the overall positional deviation of the workpiece on the production line and the processing error of the bending line itself. This leads to the system misjudging a workpiece as unqualified even if the bending line processing is completely correct, generating a large number of "false positive" alarms due to slight changes in the workpiece's position on the conveyor belt, which seriously affects the practicality and confidence of the inspection system.

[0004] Furthermore, existing methods lack robustness in edge extraction under complex lighting conditions and lack a temporal filtering mechanism for interference factors such as instantaneous vibration and measurement noise, resulting in fluctuations in detection results and making it difficult to meet the stringent requirements of industrial-grade detection for stability and accuracy.

[0005] To address the above problems, this invention proposes an intelligent detection method for bending lines of sheet metal structural parts. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0007] The technical solution adopted by this invention to solve the technical problem is: an intelligent detection method for bending lines of sheet metal structural parts, comprising: During the acquisition period, structural component images of sheet metal structural components are acquired and original image sequences are generated. It is determined whether motion blur occurs in the structural component images, and the structural component images are marked as clear frames or blurry frames. It is also determined whether blurry frame clearing processing is triggered. If triggered, the motion vector field is estimated using optical flow for the blurred frame, a deep learning deblurring network is constructed and used, and the blurred frame is restored by combining the motion vector field, and the corresponding clear frame is output. Organize clear frames to generate clear image sequences. During the acquisition period, combine the constructed digital twin model of the sheet metal structure to calculate the real-time six-degree-of-freedom pose, determine the dynamic region of interest of the bending line in real time, and extract the bending line position information. At the end of the acquisition period, a Kalman filter algorithm is introduced to perform temporal filtering and fusion optimization on the bent line position information extracted in the clear image sequence to generate optimized bent line data. Based on the real-time projection results of optimized bending line data and digital twin model, the actual deviation of key geometric parameters of bending line is calculated, and the bending line of sheet metal structural parts is judged as qualified or unqualified. The method for determining whether a structural component image has motion blur is as follows: The structural component image is preprocessed by grayscale conversion, and convolution operation is performed by the Laplacian operator to obtain the second-order differential feature map of the structural component image. The sharpness evaluation score of the structural component image is obtained by calculating the variance of all pixel values ​​of the second-order differential feature map. If the sharpness evaluation score of the structural component image is higher than the minimum sharpness threshold, it is determined that the structural component image does not have motion blur; otherwise, it is determined that motion blur exists. The method for determining whether to trigger blur frame sharpening is as follows: During the acquisition period, if the structural component image acquired at the current moment is a blurry frame, and the first structural component image in the original image sequence is a clear frame, and the number of blurry frames in the original image sequence at the current moment is less than the preset blur standard, it is determined that blurry frame clearing processing is triggered. The method for obtaining the motion vector field is as follows: The blurred frame acquired at the current moment and the structural component image of the adjacent previous frame in the original image sequence are used as input. After preprocessing, two preprocessed images are obtained. The Farneback dense optical flow algorithm is used to approximate the local signal structure of the preprocessed image in the neighborhood of each pixel through a polynomial transformation model. Thus, an approximate relationship based on polynomial expansion is constructed between the two preprocessed images. By calculating the displacement of the polynomial expansion coefficients between the two preprocessed images, the global dense optical flow field is solved, and a two-dimensional motion vector is calculated for each pixel in the preprocessed image. The two-dimensional motion vectors of all pixels are integrated according to their corresponding pixel coordinates to form a motion vector field; The specific method for restoring blurred frames is as follows: A deep learning deblurring network based on an encoder-decoder architecture is constructed. The deep learning deblurring network performs end-to-end image transformation in a feedforward manner. The encoder part is responsible for extracting feature maps of deep features from the input blurred frame and using motion vector fields to perform motion-aware modulation and alignment of the feature maps. The decoder part is responsible for reconstructing the potential sharp image from the modulated and aligned feature map and finally outputting the sharp frame corresponding to the blurred frame. The deep learning deblurring network is trained using a pairing database, which contains paired data consisting of blurred frames and corresponding sharp frames actually collected from the production line, as well as paired data consisting of blurred frames and corresponding sharp frames generated by simulating different motion states. The calculation method for real-time six-degree-of-freedom pose is as follows: During the acquisition period, the last clear frame in the clear image sequence is the current frame. Based on the precise geometric information and three-dimensional geometric information of the bending line of the sheet metal structural parts obtained from the product design end, a digital twin model of the sheet metal structural parts is constructed. The current frame is matched with the digital twin model in real time. The ORB feature extraction algorithm is used to extract and match key point pairs from the current frame and the digital twin model. The key point pairs include the two-dimensional image points of the current frame and the corresponding three-dimensional model points of the digital twin model. A set of correspondences between two-dimensional image points and three-dimensional model points is established. Based on the correspondence set, the perspective n-point localization problem is solved. A cost function is constructed to transform the perspective n-point localization problem into an optimization problem that minimizes the reprojection error. The cost function obtains the real-time six-degree-of-freedom pose of the optimal solution by minimizing the reprojection error of all key point pairs. The method for determining the region of interest dynamically is as follows: Based on the calculated real-time six-degree-of-freedom pose, the three-dimensional geometric information of the bending line in the digital twin model is called. Through coordinate system transformation and perspective projection, the position of the bending line in the digital twin model is accurately projected onto the two-dimensional plane of the current frame. A dynamic region of interest that completely surrounds the projected bending line is dynamically predicted and generated according to the projection position. The dynamic region of interest is updated in real time with the movement of the sheet metal structure. The method for extracting the bending line position information is as follows: A phase-consistent edge detection algorithm is used within the dynamic region of interest to extract candidate edge points of the bend line. The phase-consistent edge detection algorithm defines a phase consistency function in the frequency domain to measure the edge intensity. When the phase of all frequency components reaches consistency at the pixel position, the phase consistency value approaches the maximum value. For each pixel position, if the phase consistency value of the pixel position is greater than the preset PC threshold, the pixel position is determined to be an edge candidate point. For the extracted edge candidate points, a phase consistency value distribution curve is fitted with a cubic polynomial along the edge normal direction. The extreme points of the curve are taken as sub-pixel edge coordinates. Isolated noise points are removed from all sub-pixel edge coordinates. The discrete sub-pixel edge coordinates are connected into a continuous bending line using Hough transform. The sub-pixel edge coordinate sequence is obtained by integrating according to the connection order and marked as bending line position information. The optimized method for obtaining bend line data is as follows: At the end of the acquisition period, the position information of the bending line corresponding to all clear frames in the clear image sequence is obtained. The bending line data sequence is obtained by organizing according to the time sequence. The Kalman filter algorithm is used to perform time-series filtering on the bending line data sequence. A state vector containing the key geometric parameters of the bending line is defined. The state transition model and process noise covariance are determined. The measurement vector is defined as the key geometric parameters of the bending line extracted from the clear image sequence. The observation matrix and measurement noise covariance are determined. Starting from the first clear frame in the clear image sequence, the Kalman filter algorithm is applied to the bend line position information of each clear frame in turn, including a prediction step and an update step. In the prediction step, the state vector at the current time is estimated based on the state vector at the previous time step. In the update step, the measured vector at the current time step is compared with the estimated predicted state vector. The optimized and updated state estimation vector is obtained by weighted fusion by calculating the Kalman gain. The clear image sequence is traversed, and the finally updated state estimation vector is marked as the optimized bend line data. The calculation method for the actual deviation of the key geometric parameters of the bending line is as follows: Optimizing the bending line data includes the key geometric parameters of the bending line, obtaining the real-time six-degree-of-freedom pose calculated from the last clear frame in the clear image sequence, and projecting the real-time six-degree-of-freedom pose based on the digital twin model to obtain the theoretical values ​​of the key geometric parameters of the bending line under the current pose, which are then marked as theoretical bending line data. The key geometric parameters of the bending line are divided into positional parameters and angle parameters. The actual deviations of the key geometric parameters of the bending line are calculated, including positional deviation and angle deviation. The positional deviation is obtained by calculating the Euclidean distance between the positional parameters in the optimized bending line data and the theoretical bending line data. The angle deviation is obtained by calculating the absolute difference between the angle parameters in the optimized bending line data and the theoretical bending line data.

[0008] The beneficial effects of this invention are as follows: 1. This invention fundamentally solves the core challenges in dynamic inspection by constructing a dynamic image quality assurance system and a precise positioning model based on digital twins. The solution employs hardware synchronization and online sharpness assessment, combined with optical flow-guided deep learning deblurring technology, ensuring the acquisition of high-quality, unblurred image sequences even under high-speed production line operation. This provides a reliable data foundation for subsequent analysis. The innovative introduction of a digital twin model for real-time pose estimation and virtual online calibration transforms the detection benchmark from fixed image coordinates to model coordinates that move with the workpiece. This intelligently eliminates the impact of overall sheet metal part positional deviations on the detection results, achieving precise discrimination of bending line processing errors and significantly improving the accuracy and robustness of the inspection.

[0009] 2. This invention achieves a significant leap in detection efficiency and automation level through deep integration of multiple technologies and a closed-loop design throughout the entire process. The solution organically combines algorithms such as phase consistency edge detection and Kalman filter timing fusion, which improves sub-pixel level detection accuracy while effectively suppressing interference such as instantaneous vibration and illumination changes, ensuring the stability and consistency of the output results. Finally, a decision result is generated based on the optimized data, and subsequent operations such as sorting, alarm or shutdown are directly driven through an industrial interface, forming a complete closed loop from image acquisition and intelligent analysis to control execution. Attached Figure Description

[0010] The invention will now be further described with reference to the accompanying drawings.

[0011] Figure 1 This is a flowchart illustrating the steps of an intelligent detection method for bending lines of sheet metal structural parts according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the steps for obtaining bending line position information in an intelligent detection method for bending lines of sheet metal structural parts according to an embodiment of the present invention. Detailed Implementation

[0012] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0013] Example 1 Please see Figure 1 As shown in the embodiment of the present invention, an intelligent detection method for bending lines of sheet metal structural parts includes the following steps: S1: In the dynamic operation scenario of the production line, the structural component images of sheet metal structural components are acquired during the acquisition period and the original image sequence is generated. It is determined whether the structural component images have motion blur. If no motion blur occurs, the structural component images are marked as clear frames; otherwise, they are marked as blurry frames. It is also determined whether to trigger blurry frame clearing processing. Specifically, before starting the bending machine, pre-operation preparations are carried out. The machine body and slider of the bending machine are inspected separately to ensure that there are no cracks in the machine body, no looseness in the slider, no leaks in the hydraulic system, and sufficient lubrication of the double linear guide rails and ball screws of the bending machine beam. The appropriate mold is installed, and the mold gap is adjusted to 1.1-1.2 times the thickness of the sheet metal structural part. After completing the pre-operation preparations, the bending machine is started and the system is initialized. The bending parameters are set, including the thickness, length, material, and bending angle of the sheet metal structural part. The sheet metal structural part is then loaded and the machine enters the dynamic operation scenario of the production line. A dynamic image acquisition system is constructed, consisting of an industrial camera and a stroboscopic light source. In the dynamic operation scenario of the production line, when the sheet metal structural part enters the preset detection station range, the image acquisition of the sheet metal structural part begins. The industrial camera and the stroboscopic light source are driven simultaneously through a hardware-level synchronization circuit to ensure that the pulse width of the stroboscopic light source corresponds precisely to the exposure time of the camera. The structural part images are continuously acquired at a preset sampling frequency. The acquired structural part images are integrated according to the time sequence to obtain the original image sequence until the sheet metal structural part leaves the detection station range, at which point the image acquisition of the sheet metal structural part is judged to be over. The time period for image acquisition of sheet metal structural parts is marked as the acquisition time period. If the current time is within the acquisition time period, the structural part image acquired at the current time is the last structural part image in the original image sequence. The original image sequence is supplemented with structural part images as time progresses within the acquisition time period. The detection station range is determined based on the field of view of the industrial camera. The sampling frequency is set by combining the length of the sheet metal structure and the conveying speed of the production line conveyor roller. The camera exposure time is determined by combining the conveying speed of the production line conveyor roller and the camera pixel equivalent to ensure that the displacement of the sheet metal structure does not exceed 1 pixel during the camera exposure time, thus ensuring the motion freeze effect and avoiding the loss of image details due to insufficient exposure. During the acquisition period, the sharpness of the structural component image acquired at the current moment is evaluated in real time. Specifically, the current structural component image is preprocessed by grayscale conversion, and then the Laplacian operator is applied to perform convolution operation to obtain the second-order differential feature map of the structural component image. The role of the second-order differential feature map is to effectively highlight the edge and detail information in the structural component image. By calculating the variance of all pixel values ​​of the second-order differential feature map, the sharpness evaluation score of the structural component image is obtained. It should be noted that the sharpness assessment score is negatively correlated with the blurriness of the structural component image. The higher the sharpness assessment score, the sharper the edges of the structural component image and the more complete the details are preserved. The sharpness evaluation score of the structural component image is compared with the preset minimum sharpness threshold. If the sharpness evaluation score of the structural component image is higher than the minimum sharpness threshold, it is determined that the structural component image does not have motion blur and is marked as a sharp frame. Otherwise, it is determined that the structural component image has motion blur and is marked as a blurry frame. The minimum sharpness threshold is determined by statistical learning from a sharp frame benchmark database established by a large number of experiments. During the acquisition period, if the first structural component image in the original image sequence is a blurred frame, or if the number of blurred frames in the original image sequence at the current moment is greater than or equal to the preset blur standard, it is determined that the acquisition blur of the dynamic image acquisition system is too high, and a blur alarm is sent to the administrator terminal to notify the administrator to handle it. If the first structural component image in the original image sequence is a clear frame, and the number of blurred frames in the original image sequence at the current moment is less than the preset blur standard, and the structural component image acquired at the current moment is a blurred frame, then blur frame clearing processing is triggered. It should be noted that the purpose of this step is to freeze motion on the dynamic production line using hard synchronization technology, acquire the original image sequence, and perform quality monitoring and classification of each frame of image in real time. Image quality assessment is an active decision-making node in the process, rather than post-analysis, and two processing branches are dynamically triggered based on the assessment results, thus realizing the system's self-awareness. Before starting the bending machine, perform pre-operation preparations, including: I. Pre-power-on checks 1. Inspect the appearance of the equipment: the machine body is not deformed, the screws are not loose, the hydraulic oil pipes are not leaking, and the worktable is free of debris.

[0014] 2. Mold inspection: Confirm that the upper and lower bending dies are securely installed and that the die gap matches the thickness of the sheet material to be processed (generally 1.1 to 1.2 times the sheet material thickness).

[0015] 3. Lubrication and power supply: Add lubricating oil to the slider guide rail and back gauge screw; connect the 380V power supply and turn on the main switch of the electrical box.

[0016] II. Power-on and System Initialization 1. Press the emergency stop button on the side of the equipment (rotate clockwise to reset), then press the "Power Start" button and wait for the CNC system (usually a simplified CNC or DA41 system) to finish loading.

[0017] 2. Click "Start Oil Pump" on the system interface, run the slider 2-3 times under no-load (press "Jig Down / Up"), and observe whether the slider runs smoothly without jamming.

[0018] III. Parameter Settings (Adjust according to processing requirements) 1. Input sheet material parameters: Select "Program Editing" in the system and input the thickness, length and material of the sheet material to be processed (e.g., "2mm steel plate, 1500mm long").

[0019] 2. Set bending angle: Input the target angle (e.g., 90°) using the system numeric keys, and the system will automatically match the corresponding bending depth; if multiple bends are required, you can set "Step 1" and "Step 2" to correspond to different angles respectively.

[0020] 3. Back gauge positioning: Press the "Back gauge jog" key to adjust the back gauge to the required size position for processing (precise positioning can be achieved through the ruler or system values).

[0021] IV. Formal Bending Operation 1. Loading: Place the board material stably on the workbench, align it with the back gauge positioning surface, and ensure that the board material does not shift.

[0022] 2. Start bending: Press the "Start with both hands" button at the same time (safety interlock design), the slider will automatically move down to complete the bending, and after holding the pressure for 1 to 2 seconds, it will automatically move up to reset.

[0023] 3. Material Removal and Inspection: After the slider has fully ascended, remove the sheet material and check whether the bending angle and dimensions meet the requirements. If there is a deviation, fine-tune the backstop position or bending depth and try bending again.

[0024] V. Shutdown and Finishing 1. After processing is complete, first press "Stop Oil Pump", then press "Power Off", and finally press the emergency stop button.

[0025] 2. Clean the metal shavings off the workbench, wipe the mold surface with rust-preventive oil, and turn off the main switch of the electrical box.

[0026] S2: Trigger blur frame clearing processing, use optical flow to estimate motion vector field for blur frame, construct and use deep learning deblurring network, combine motion vector field to restore blur frame, and output corresponding clear frame; If the blur frame clearing process is triggered, the original image sequence is obtained. The blur frame acquired at the current moment is processed within the original image sequence. The blur frame and the adjacent previous frame structural component image are used as input. The two structural component images are converted into grayscale images to reduce computational complexity. Then, a Gaussian filter kernel is used to smooth the two grayscale images to suppress the interference of image noise on the calculation of small amplitude motion, resulting in two preprocessed images. The motion vector of each pixel between two preprocessed images is calculated using the optical flow method. Specifically, the Farneback dense optical flow algorithm is used to approximate the local signal structure of the preprocessed image in the neighborhood of each pixel through a polynomial transformation model, thereby constructing an approximate relationship based on polynomial expansion between the two preprocessed images. By calculating the displacement of the polynomial expansion coefficients between the two preprocessed images, the global dense optical flow field is solved, and a two-dimensional motion vector d=(dx,dy) is calculated for each pixel in the preprocessed image, where dx and dy represent the displacement of the pixel in the x and y directions, respectively. The calculated two-dimensional motion vectors of all pixels are integrated according to their corresponding pixel coordinates to form a matrix composed of two-dimensional motion vectors that is completely consistent with the spatial resolution of the structural component image, and this matrix is ​​marked as the motion vector field. It should be noted that the motion vector field fully quantifies the motion path and amplitude of each pixel in the scene during the entire exposure time between the blurred frame and the adjacent previous frame structure image. A deep learning deblurring network based on an encoder-decoder architecture is constructed. The deep learning deblurring network performs end-to-end image transformation in a feedforward manner. The encoder part is responsible for extracting feature maps of deep features from the input blurred frame and using motion vector fields to perform motion-aware modulation and alignment of the feature maps. The decoder part is responsible for reconstructing the potential sharp image from the modulated and aligned feature map and finally outputting the sharp frame corresponding to the blurred frame. The deep learning deblurring network is trained using a pairing database. The pairing database contains paired data consisting of blurry frames and corresponding clear frames actually collected from the production line, as well as paired data consisting of blurry frames and corresponding clear frames generated by simulating different motion states. This ensures that the deep learning deblurring network can adapt to various blurry scenarios in the production line. It should be noted that the purpose of this step is to solve the motion blur problem that is inevitably faced in dynamic acquisition, providing a reliable image data foundation for subsequent high-precision detection. Even if blurred frames are generated due to instantaneous vibration or other reasons, the system can self-repair, avoiding detection interruption or misjudgment caused by single-frame image quality issues, thus ensuring production cycle time. The optical flow method provides a physically interpretable motion vector field as prior knowledge to guide the deep learning deblurring network, which is more targeted and reliable than a simple end-to-end deblurring network and is more suitable for scenarios with regular motion blur, such as production lines. S3: Based on the original image sequence, the clear frames are processed to generate a clear image sequence. During the acquisition period, the real-time six-degree-of-freedom pose is calculated by combining the constructed digital twin model of the sheet metal structure, and the dynamic region of interest of the bending line is determined in real time and the bending line position information is extracted. All clear frames in the original image sequence and the corresponding clear frames recovered from the blurry frames are collected. All clear frames are then uniformly sorted and integrated according to the original acquisition time sequence to generate a clear image sequence. like Figure 2 As shown, the specific steps for obtaining the bending line position information are as follows; If the current time is within the acquisition period, the last clear frame in the clear image sequence is the current frame; Based on the precise geometric information and 3D geometric information of bending lines of sheet metal structural parts obtained from the product design end, a digital twin model of the sheet metal structural parts is constructed. The current frame is matched with the digital twin model in real time. Specifically, the ORB feature extraction algorithm is used to extract and match key point pairs from the current frame and the digital twin model. The key point pair includes 2D image points of the current frame and corresponding 3D model points of the digital twin model, establishing a correspondence set between 2D image points and 3D model points. ; in, This represents the coordinates of a point in the two-dimensional image in the current frame. This represents the coordinates of the corresponding 3D model point; Based on the correspondence set, by solving the perspective n-point localization problem, the real-time six-DOF pose (R, t) of the workpiece relative to the camera is calculated. The perspective n-point localization problem is then transformed into an optimization problem that minimizes the reprojection error. The cost function is: ; Where R and t are the rotation matrix and translation vector to be solved in the real-time six-DOF pose, respectively, and K represents the intrinsic parameter matrix of the industrial camera. The function represents the coordinates of points in a 3D model. By projecting the real-time six-DOF pose (R, t) and the intrinsic parameter matrix K of the industrial camera onto the two-dimensional image plane, the cost function obtains the optimal solution of the real-time six-DOF pose (R, t) by minimizing the reprojection error of all key point pairs, thus realizing virtual online calibration under dynamic working conditions. Based on the calculated real-time six-degree-of-freedom pose, the three-dimensional geometric information of the bending line in the digital twin model is called. Through coordinate system transformation and perspective projection, the position of the bending line in the digital twin model is accurately projected onto the two-dimensional plane of the current frame. A dynamic region of interest (ROI) that completely surrounds the projected bending line is dynamically predicted and generated according to the projection position. It should be noted that the dynamic region of interest updates in real time as the sheet metal structural parts move. Its function is to significantly reduce the image search range for subsequent processing, effectively eliminate interference from irrelevant backgrounds, and significantly improve processing speed. Within the dynamically active region of interest, a phase-consistent edge detection algorithm, insensitive to changes in illumination and contrast, is employed to accurately extract candidate edge points for the bend line. The phase-consistent edge detection algorithm defines a phase-consistency function in the frequency domain to measure edge intensity. ; Where x represents the pixel position within the dynamic region of interest, and PC(x) is the phase consistency value at pixel position x. and These are the amplitude and local phase at the frequency component ω, respectively. It is the weighted average phase of all frequency components. It is a small constant that avoids division by zero; In the phase consistency function, when the phases of all frequency components reach consistency at pixel position x, the local phase of frequency component ω is... Weighted average phase of all frequency components As they approach equality, the phase coherence value PC(x) approaches its maximum value; For each pixel location, if the phase consistency value of the pixel location is greater than the preset PC threshold, the pixel location is determined to be an edge candidate point. It should be noted that, since the phase coherence function depends on phase rather than amplitude, it has inherent robustness to uniform illumination changes and local shadows. A phase-consistency edge detection algorithm is used to extract candidate edge points within the dynamic region of interest. For the extracted candidate edge points, a cubic polynomial is used to fit the phase consistency value distribution curve along the edge normal direction. The extreme points of the curve are taken as sub-pixel edge coordinates. Isolated noise points are removed from all sub-pixel edge coordinates. The Hough transform is used to connect the discrete sub-pixel edge coordinates into a continuous bend line. The sub-pixel edge coordinate sequence is obtained by integrating according to the connection order and marked as bend line position information. It should be noted that the purpose of this step is to transform the detection reference from a fixed image coordinate system to a model coordinate system that moves with the workpiece through real-time pose estimation. This makes the detection result only concerned with whether the position of the bending line is correct relative to the workpiece itself, and is independent of the specific position of the workpiece on the conveyor belt. This greatly improves processing efficiency and accuracy. The dynamic ROI reduces the search range from the entire image to the area near the bending line, reducing the amount of computation and eliminating background interference. The virtual online calibration based on the digital twin model allows the precise geometric model from the product design to be directly used for online detection, replacing the cumbersome offline calibration with fixed positions in traditional visual inspection, and adapting to the pose changes brought about by the movement of the workpiece. S4: At the end of the acquisition period, the Kalman filter algorithm is introduced to perform temporal filtering and fusion optimization on the bent line position information extracted in the clear image sequence to generate optimized bent line data; Specifically, if the current time is the end of the acquisition period, the clear image sequence already contains all clear frames of the sheet metal structure. Obtain the bending line position information corresponding to all clear frames, and organize the bending line data sequence according to the time sequence. It should be noted that due to mechanical vibration, environmental interference, or image acquisition noise during the dynamic operation of the production line, the bending line position information may experience instantaneous fluctuations or measurement errors. To improve the stability and accuracy of bending line position information, a Kalman filter algorithm is used to perform time-series filtering on the bending line data sequence. Specifically, a state vector for the Kalman filter algorithm is defined, which contains key geometric parameters of the bending line. These key geometric parameters include the starting point coordinates, ending point coordinates, and angle between the bending line and the horizontal direction of the camera image coordinate system in the current camera image coordinate system. Since the sheet metal structural parts move smoothly during transportation, a constant position model is adopted for the state transition model. The constant position model assumes that the state vector remains approximately unchanged between adjacent frames. The process noise covariance is calibrated experimentally based on the stability of the production line operation and is used to characterize the uncertainty of state prediction. A measurement model is constructed to establish the correlation between the state vector and the measured data. Based on the bending line position information extracted from each clear frame in the clear image sequence, key geometric parameters of the bending line are extracted. The measurement vector is defined as the key geometric parameters of the bending line extracted from each clear frame in the clear image sequence, with the same parameter composition as the state vector. The measured data is represented by the measurement vector. The transformation relationship from the state vector to the measurement vector is described by the observation matrix. The observation matrix is ​​set as an identity matrix to characterize the direct correspondence between the state vector and the measurement vector. The measurement noise covariance is determined by statistically analyzing the variance of the bending line position information in the clear image sequence, and is used to quantify the measurement uncertainty introduced by factors such as image processing algorithm error and illumination change. The recursive process of the Kalman filter algorithm includes a prediction step and an update step: In the prediction step, the state vector at the current time is estimated based on the state vector at the previous time step. In the update step, based on the constructed measurement model, the measurement vector at the current time step is compared with the estimated predicted state vector. The measurement vector at the current time step and the estimated predicted state vector at the current time step are fused by calculating the Kalman gain weight to obtain the optimized and updated state estimate vector at the current time step. Starting from the first clear frame in the clear image sequence, the Kalman filter algorithm is applied to the bend line position information of each clear frame in turn. The state estimation vector is continuously corrected through recursive calculation. The clear image sequence is traversed, and the finally updated state estimation vector is marked as the optimized bend line data. It should be noted that the purpose of this step is to effectively smooth out the single-frame measurement fluctuations caused by random errors and instantaneous vibrations in image processing itself, and to obtain optimized bending line data that is closer to the true value. No matter how the single-frame results fluctuate during the detection process, the final result is a unique bending line optimized by the fusion of information from multiple frames, thus avoiding oscillations in the decision result. By embedding control theory into the visual detection process and applying the Kalman filter algorithm to the temporal fusion of the bending line geometric parameters, it is not just a simple smoothing, but an optimal estimation based on the constant position of the system model and measurement uncertainty, thereby improving the overall reliability of the system. S5: Based on the real-time projection results of optimized bending line data and digital twin model, calculate the actual deviation of key geometric parameters of bending line, and make a qualified or unqualified inspection judgment on the bending line of sheet metal structural parts. Specifically, optimizing the bending line data includes the key geometric parameters of the bending line, obtaining the real-time six-degree-of-freedom pose calculated from the last clear frame in the clear image sequence, and projecting the real-time six-degree-of-freedom pose based on the digital twin model to obtain the theoretical values ​​of the key geometric parameters of the bending line under the current pose, which are then marked as theoretical bending line data. The key geometric parameters of the bend line include the coordinates of the start point of the bend line, the coordinates of the end point of the bend line, and the angle between the bend line and the horizontal direction of the camera image coordinate system in the current camera image coordinate system. The coordinates of the start point and the coordinates of the end point of the bend line are integrated into a set and marked as position parameters, and the angle between the bend line and the horizontal direction of the camera image coordinate system is marked as angle parameters. The actual deviations of the key geometric parameters of the bending line are calculated, including positional deviation and angular deviation. The positional deviation is obtained by calculating the Euclidean distance between the positional parameters in the optimized bending line data and the theoretical bending line data, and the angular deviation is obtained by calculating the absolute value of the difference between the angular parameters in the optimized bending line data and the theoretical bending line data. The tolerance range for actual deviations is set based on product design specifications and process requirements. The tolerance range includes position tolerance and angle tolerance. The calculated position deviation and angle deviation are compared with the corresponding tolerance range. Specifically, if both the position deviation and angle deviation are within their respective tolerance ranges, the bending line is judged to be qualified; otherwise, the bending line is judged to be unqualified. If the bending line is deemed unqualified, an out-of-tolerance signal is generated. This signal is fed back to the production line control system in real time via an industrial communication interface. The industrial communication interface uses an industrial Ethernet or fieldbus protocol to ensure low latency and high reliability of signal transmission. The out-of-tolerance signal carries specific actual deviation information. After receiving the out-of-tolerance signal, the production line control system drives the sorting mechanism to sort the unqualified parts to a specific area at the end of the production line. The technical solution of this invention is as follows: During the acquisition period, structural images of sheet metal structural parts are acquired and an original image sequence is generated. It is determined whether motion blur occurs in the structural part images, and the structural part images are marked as clear frames or blurry frames. It is also determined whether blurry frame clearing processing is triggered. If triggered, the motion vector field is estimated for the blurry frames using the optical flow method. A deep learning deblurring network is constructed and used to restore the blurry frames in combination with the motion vector field, and the corresponding clear frames are output. The clear frames are sorted to generate a clear image sequence. During the acquisition period, the real-time six-degree-of-freedom pose is calculated in combination with the constructed digital twin model of the sheet metal structural parts. The dynamic region of interest of the bending line is determined in real time and the bending line position information is extracted. At the end of the acquisition period, the Kalman filter algorithm is introduced to perform temporal filtering fusion optimization on the bending line position information extracted in the clear image sequence to generate optimized bending line data. Based on the real-time projection results of the optimized bending line data and the digital twin model, the actual deviation of the key geometric parameters of the bending line is calculated, and the bending line of the sheet metal structural parts is judged as qualified or unqualified.

[0027] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent detection of bending lines in sheet metal structural parts, characterized in that: include: During the acquisition period, structural component images of sheet metal structural components are acquired and original image sequences are generated. It is determined whether motion blur occurs in the structural component images, and the structural component images are marked as clear frames or blurry frames. It is also determined whether blurry frame clearing processing is triggered. If triggered, the motion vector field is estimated using optical flow for the blurred frame, a deep learning deblurring network is constructed and used, and the blurred frame is restored by combining the motion vector field, and the corresponding clear frame is output. Organize clear frames to generate clear image sequences. During the acquisition period, combine the constructed digital twin model of the sheet metal structure to calculate the real-time six-degree-of-freedom pose, determine the dynamic region of interest of the bending line in real time, and extract the bending line position information. At the end of the acquisition period, a Kalman filter algorithm is introduced to perform temporal filtering and fusion optimization on the bent line position information extracted in the clear image sequence to generate optimized bent line data. Based on the real-time projection results of optimized bending line data and digital twin model, the actual deviation of key geometric parameters of bending line is calculated, and the bending line of sheet metal structural parts is judged as qualified or unqualified.

2. The method according to claim 1, characterized in that: The method for determining whether a structural component image has motion blur is as follows: The structural component image is preprocessed by grayscale conversion, and convolution operation is performed using the Laplacian operator to obtain the second-order differential feature map of the structural component image. The sharpness evaluation score of the structural component image is obtained by calculating the variance of all pixel values ​​in the second-order differential feature map. If the sharpness evaluation score of the structural component image is higher than the minimum sharpness threshold, it is determined that the structural component image does not have motion blur; otherwise, it is determined that motion blur exists.

3. The method of claim 1, wherein the method further comprises: determining a bending line of the sheet metal structure; and determining a bending angle of the bending line. The method for determining whether to trigger blur frame sharpening is as follows: During the acquisition period, if the structural component image acquired at the current moment is a blurry frame, and the first structural component image in the original image sequence is a clear frame, and the number of blurry frames in the original image sequence at the current moment is less than the preset blur standard, then blurry frame clearing processing is triggered.

4. The method of claim 1, wherein the method further comprises: The method for obtaining the motion vector field is as follows: The blurred frame acquired at the current moment and the structural component image of the adjacent previous frame in the original image sequence are used as input. After preprocessing, two preprocessed images are obtained. The Farneback dense optical flow algorithm is used to approximate the local signal structure of the preprocessed image in the neighborhood of each pixel through a polynomial transformation model. Thus, an approximate relationship based on polynomial expansion is constructed between the two preprocessed images. By calculating the displacement of the polynomial expansion coefficients between the two preprocessed images, the global dense optical flow field is solved, and a two-dimensional motion vector is calculated for each pixel in the preprocessed image. The two-dimensional motion vectors of all pixels are integrated according to their corresponding pixel coordinates to form a motion vector field.

5. The intelligent detection method for bending lines of sheet metal structural parts according to claim 4, characterized in that: The specific method for restoring blurred frames is as follows: A deep learning deblurring network based on an encoder-decoder architecture is constructed. The deep learning deblurring network performs end-to-end image transformation in a feedforward manner. The encoder part is responsible for extracting feature maps of deep features from the input blurred frame and using motion vector fields to perform motion-aware modulation and alignment of the feature maps. The decoder part is responsible for reconstructing the potential sharp image from the modulated and aligned feature map and finally outputting the sharp frame corresponding to the blurred frame. The deep learning deblurring network is trained using a pairing database, which contains paired data consisting of blurred frames and corresponding sharp frames actually collected from the production line, as well as paired data consisting of blurred frames and corresponding sharp frames generated by simulating different motion states.

6. The method of claim 1, wherein the method further comprises: The real-time six-degree-of-freedom pose calculation method is as follows: During the acquisition period, the last clear frame in the clear image sequence is the current frame. Based on the precise geometric information and three-dimensional geometric information of the bending line of the sheet metal structural parts obtained from the product design end, a digital twin model of the sheet metal structural parts is constructed. The current frame is matched with the digital twin model in real time. The ORB feature extraction algorithm is used to extract and match key point pairs from the current frame and the digital twin model. The key point pairs include the two-dimensional image points of the current frame and the corresponding three-dimensional model points of the digital twin model. A set of correspondences between two-dimensional image points and three-dimensional model points is established. Based on the correspondence set, the perspective n-point localization problem is solved. A cost function is constructed to transform the perspective n-point localization problem into an optimization problem that minimizes the reprojection error. The cost function obtains the real-time six-degree-of-freedom pose of the optimal solution by minimizing the reprojection error of all key point pairs.

7. The method of claim 6, wherein the method further comprises: determining the bending line of the sheet metal structure based on the bending angle and the bending direction of the sheet metal structure. The method for determining the region of interest dynamically is as follows: Based on the calculated real-time six-degree-of-freedom pose, the three-dimensional geometric information of the bending line in the digital twin model is called. Through coordinate system transformation and perspective projection, the position of the bending line in the digital twin model is accurately projected onto the two-dimensional plane of the current frame. A dynamic region of interest that completely surrounds the projected bending line is dynamically predicted and generated according to the projection position. The dynamic region of interest is updated in real time with the movement of the sheet metal structure.

8. The method of claim 7, wherein the method further comprises: determining the bending line of the sheet metal structure based on the bending angle and the bending direction. The method for extracting the bending line position information is as follows: A phase-consistent edge detection algorithm is used within the dynamic region of interest to extract candidate edge points of the bend line. The phase-consistent edge detection algorithm defines a phase consistency function in the frequency domain to measure the edge intensity. When the phase of all frequency components reaches consistency at the pixel position, the phase consistency value approaches the maximum value. For each pixel position, if the phase consistency value of the pixel position is greater than the preset PC threshold, the pixel position is determined to be an edge candidate point. For the extracted edge candidate points, a phase consistency value distribution curve is fitted with a cubic polynomial along the edge normal direction. The extreme points of the curve are taken as sub-pixel edge coordinates. Isolated noise points are removed from all sub-pixel edge coordinates. The discrete sub-pixel edge coordinates are connected into a continuous bend line using Hough transform. The sub-pixel edge coordinate sequence is obtained by integrating according to the connection order and marked as bend line position information.

9. The intelligent detection method for bending lines of sheet metal structural parts according to claim 1, characterized in that: The optimized method for obtaining bend line data is as follows: At the end of the acquisition period, the position information of the bending line corresponding to all clear frames in the clear image sequence is obtained. The bending line data sequence is obtained by organizing according to the time sequence. The Kalman filter algorithm is used to perform time-series filtering on the bending line data sequence. A state vector containing the key geometric parameters of the bending line is defined. The state transition model and process noise covariance are determined. The measurement vector is defined as the key geometric parameters of the bending line extracted from the clear image sequence. The observation matrix and measurement noise covariance are determined. Starting with the first clear frame in the clear image sequence, the Kalman filter algorithm is applied sequentially to the bend line position information of each clear frame, including a prediction step and an update step. In the prediction step, the state vector at the current time is estimated based on the state vector at the previous time step. In the update step, the measured vector at the current time step is compared with the estimated predicted state vector, and the optimized and updated state estimation vector is obtained by weighted fusion by calculating the Kalman gain. The clear image sequence is traversed, and the finally updated state estimation vector is marked as the optimized bend line data.

10. The method of claim 9, wherein the method further comprises: determining the bending line of the sheet metal structure based on the bending angle and the bending direction of the sheet metal structure. The actual deviation of the key geometric parameters of the bending line is calculated as follows: Optimizing the bending line data includes the key geometric parameters of the bending line, obtaining the real-time six-degree-of-freedom pose calculated from the last clear frame in the clear image sequence, and projecting the real-time six-degree-of-freedom pose based on the digital twin model to obtain the theoretical values ​​of the key geometric parameters of the bending line under the current pose, which are then marked as theoretical bending line data. The key geometric parameters of the bending line are divided into positional parameters and angle parameters. The actual deviations of the key geometric parameters of the bending line are calculated, including positional deviation and angle deviation. The positional deviation is obtained by calculating the Euclidean distance between the positional parameters in the optimized bending line data and the theoretical bending line data. The angle deviation is obtained by calculating the absolute difference between the angle parameters in the optimized bending line data and the theoretical bending line data.

Images