Leak hole detection method, system and equipment for rolling punching machine based on longitudinal member

Abstract

The invention discloses a leak detection method, system and equipment for a rolling punching machine based on a longitudinal member, and relates to the technical field of industrial automatic detection. Comprising the following steps: constructing a real-time data channel to synchronously acquire a longitudinal beam image, an equipment movement track and a numerical control instruction parameter; performing multi-strategy preprocessing on the image frame; camera calibration is completed through multi-combination corner detection and quality evaluation, and a size conversion proportion is obtained; a YOLO model is loaded to recognize an invalid area, a beam area and a hole area, and effective hole sites in the beam are filtered out through space matching; calculating the vertical distance between the corresponding beam body and the adjacent hole, and visually numbering in a CAD (Computer Aided Design) style; and finally, automatically comparing with a numerical control program standard hole site, and alarming and filing in real time when the hole site is abnormal. The scheme is suitable for a high-batch and high-precision frame longitudinal beam punching production scene, the calibration robustness and the hole site recognition precision under the complex working condition are improved, visual interpretation and closed-loop management are achieved, traditional manual detection and distributed tools can be replaced, and the manual rechecking cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of industrial automation testing technology, specifically to a method, system, and equipment for detecting leaks in a rolling punching machine based on the longitudinal beam of a vehicle frame. Background Technology

[0002] In the production of truck frame longitudinal beams, the rolling punch machine processes over 50,000 holes per day. The large number of holes and the fast processing pace place extremely high demands on inspection efficiency and accuracy. Current technologies primarily rely on manual visual inspection, where operators check the number and location of each hole individually. Simple mechanical inspection tools are sometimes used as a supplement. The inspection process requires simultaneous reference to the hole design parameters in the CNC program, but it lacks an automated data linkage mechanism, necessitating manual comparison of program instructions with the actual processing results.

[0003] For the defect detection scenario of roll forming punching machines, actual production requires not only determining whether holes are missing, but also checking whether the hole size is up to standard, and marking the hole position and distance to reduce manual inspection errors and costs. However, traditional solutions mostly rely on fixed templates or single threshold segmentation technology, which lacks robustness to complex lighting changes, production site vibrations, workpiece offsets, camera distortion, and other conditions. At the same time, existing technologies lack integrated tools for automatic detection and marking of calibration plates. When the specifications of the production workpiece change or the detection equipment is adjusted, tedious recalibration and parameter settings are required, resulting in high costs for replacement and maintenance.

[0004] Patent searches (search scope: detection of leaks in rolling punch machines, machine vision + CNC program linkage detection, detection of holes in longitudinal beams of vehicle frames, integrated calibration and detection tools) did not reveal an integrated solution that simultaneously possesses "real-time visual inspection, CNC program analysis and comparison, integrated calibration and annotation, and closed-loop traceability of industrial data".

[0005] The existing technology has the following core defects:

[0006] Insufficient reliability of inspection: manual sampling is subject to visual fatigue and judgment bias, resulting in a high rate of missed inspections and an inability to accurately determine abnormalities in hole size and distance; fixed templates or single threshold segmentation schemes have poor robustness, and the inspection accuracy drops significantly under complex working conditions, causing unqualified longitudinal beams to flow into subsequent processes and causing assembly line shutdowns;

[0007] Poor efficiency adaptability: Manual inspection speed cannot meet the pace of mass production; decentralized tools require switching between multiple software programs, which is cumbersome and further reduces inspection efficiency, becoming a bottleneck in the production line; Complex operation and maintenance: Lack of integrated tools; calibration, inspection, labeling, and result display must be completed step by step, resulting in high operational complexity; The calibration process is cumbersome and maintenance costs are high when changing models.

[0008] Lack of data traceability: Test results, calibration parameters, and dimensional data are not systematically and electronically stored, and multi-dimensional data such as equipment motion trajectory and CNC program instructions are not integrated, making it impossible to trace the root cause of quality problems and making it difficult to optimize the process; High costs: Dedicated inspection personnel are required, resulting in continuous investment in labor costs, and rework losses due to missed inspections and misjudgments are serious;

[0009] Insufficient availability: Traditional tools rely on successful calibration. If calibration fails, testing cannot continue, affecting production continuity.

[0010] In summary, during the actual production process of roll forming punching machines, the surface of the frame longitudinal beams often suffers from factors such as uneven lighting, oil stain reflection, mechanical vibration, workpiece conveying offset, and camera installation distortion, leading to decreased image quality, inaccurate calibration, and false or missed detections of hole positions. Traditional detection methods based on fixed threshold segmentation or template matching have poor robustness and are difficult to adapt to the production line requirements of multiple batches, small batches, and rapid model changes. At the same time, the lack of a mechanism for spatiotemporal alignment of image acquisition, equipment motion status, and CNC command parameters makes it impossible to strictly compare visual inspection results with theoretical processing intentions. Furthermore, existing solutions have not established a reliable mapping system from pixel coordinates to physical dimensions, nor have they designed a structured distance calculation and CAD-style visualization logic oriented towards the vertical direction for the core task of hole detection, resulting in low efficiency of manual review, difficulty in anomaly location, and weak traceability basis. Summary of the Invention

[0011] To overcome the problems in existing technologies, this invention provides a method, system, and equipment for detecting missing holes in a roll forming punching machine based on vehicle frame longitudinal beams. It can solve technical problems such as poor calibration robustness in complex industrial environments, inaccurate hole position identification, unintuitive dimensional marking, and lack of closed-loop anomaly detection. It improves detection accuracy and efficiency, reduces the missed detection rate, operational complexity, and production costs, enhances system robustness and usability, and meets the high-volume, high-precision production and inspection needs of vehicle frame longitudinal beam punching.

[0012] To achieve the aforementioned objectives, the present invention adopts the following technical solution:

[0013] In a first aspect, the present invention provides a method for detecting leaks in a rolling punching machine based on a vehicle frame longitudinal beam, the method comprising:

[0014] A real-time data channel is constructed to synchronously collect multi-dimensional industrial data, including: longitudinal beam surface images, punching equipment motion trajectories, and instruction parameters in the CNC program.

[0015] The acquired longitudinal beam surface image is decomposed into frame data, and multi-strategy preprocessing is performed on the frame data;

[0016] The preprocessed frame data is calibrated, and the optimal calibration result is selected by combining multiple strategies. The camera parameters and size conversion ratio are calculated.

[0017] Load the pre-established target detection model, detect the pre-processed frame data, identify invalid areas, beam areas and hole areas, filter to obtain the valid hole positions in the beam, and record the position information of the valid hole positions;

[0018] Based on the calibrated size conversion ratio, the vertical distance of the effective hole position relative to the beam body and the vertical distance between adjacent effective holes on the same vertical line are calculated; the vertical distance of the effective hole position relative to the beam body and the vertical distance between adjacent effective holes on the same vertical line are visualized, plotted, and numbered.

[0019] The number and location data of the detected holes are compared with the standard hole data generated by the CNC program to determine whether there are any abnormal holes. The visual detection hole data and comparison results are displayed visually through the monitoring interface, and a notification is issued when an abnormal hole is detected.

[0020] Optionally, the construction of a real-time data channel and the synchronous acquisition of multi-dimensional industrial data includes: constructing a real-time data channel composed of an industrial camera, PLC, host computer and communication components, and synchronizing data with a local area network through the OPC UA industrial communication protocol to synchronously acquire images of the longitudinal beam surface, the motion trajectory of the punching equipment and the instruction parameters in the CNC machining program.

[0021] 3. The method as described in claim 1, wherein the multi-strategy preprocessing includes performing any one or more preprocessing operations on the frame data, such as grayscale conversion, contrast enhancement, Gaussian filtering, adaptive binarization, and morphological processing.

[0022] Optionally, the calibration of the preprocessed frame data includes: performing calibration board corner detection on the preprocessed frame data, filtering the optimal calibration result through a combination of multiple strategies; calculating the calibration quality through the coefficient of variation of the corner spacing, selecting the optimal calibration result and calculating the camera intrinsic parameters, distortion parameters and size conversion ratio; if no effective calibration board is detected or the calibration quality is substandard, estimating the size conversion ratio based on a preset empirical coefficient.

[0023] Optionally, the pre-establishment of the target detection model includes:

[0024] A standardized dataset is established based on historically collected multi-dimensional data and its corresponding annotation information. The establishment of the standardized dataset includes: using the LabelStudio annotation tool to annotate the beam outline, ventral holes and invalid areas in the samples, and mixing abnormal samples into the standardized dataset according to a preset ratio.

[0025] Using the YOLO object detection network as the initial model, the object detection model was trained in the PyTorch framework using transfer learning and early stopping mechanisms to obtain the aforementioned object detection model.

[0026] Optionally, the filtering to obtain valid hole positions within the beam and recording the location information of the valid hole positions includes: performing inference detection on the preprocessed frame data to parse the position coordinates of the invalid region, the beam region, and the hole region; setting a tolerance range, matching the coordinates of the hole region with the coordinates of the beam region, filtering out invalid hole positions outside the beam region, retaining the valid hole positions within the beam, and recording the location information.

[0027] Optionally, the method also includes continuously tracking the hole position using a hole tracking algorithm, which is implemented through the following steps:

[0028] Cache the historical location coordinates of the holes;

[0029] A confidence scoring mechanism is introduced, and parameters such as maximum matching distance, maximum allowed number of lost frames, and minimum tracking length are set.

[0030] Using OpenCV, a baseline is drawn to continuously track the hole trajectory and filter out low-confidence trajectories.

[0031] Optionally, issuing a notification when a leak is detected includes:

[0032] When the visual inspection hole position data is inconsistent with the hole position data parsed by the CNC program, a notification message containing the beam ID, inspection time, and anomaly type is sent in real time; and all inspection data, comparison results, and timestamp information are automatically stored to enable multi-dimensional query and traceability.

[0033] Secondly, the present invention provides a hole detection system for a rolling punching machine based on a vehicle frame longitudinal beam, the system comprising:

[0034] The synchronous acquisition module is used to build a real-time data channel and synchronously acquire multi-dimensional industrial data, including: longitudinal beam surface images, punching equipment motion trajectories, and instruction parameters in the CNC program.

[0035] The processing module is used to decompose the acquired longitudinal beam surface image into frame data and perform multi-strategy preprocessing on the frame data; calibrate the preprocessed frame data, select the optimal calibration result through multi-strategy combination, and calculate the camera parameters and size conversion ratio;

[0036] The detection module is used to load a pre-established target detection model, detect the pre-processed frame data, identify invalid areas, beam areas and hole areas, filter out the valid hole positions in the beam, and record the position information of the valid hole positions.

[0037] The calculation module is used to calculate the vertical distance of the effective hole position relative to the beam and the vertical distance between adjacent effective holes on the same vertical line based on the calibrated size conversion ratio; and to visualize and number the vertical distance of the effective hole position relative to the beam and the vertical distance between adjacent effective holes on the same vertical line.

[0038] The judgment module is used to compare the number and location data of the detected holes with the standard hole data generated by the CNC program to determine whether there are any abnormal holes. The module also visualizes the visually detected hole data and comparison results through the monitoring interface and issues a notification when an abnormal hole is detected.

[0039] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0040] At least one processor; and

[0041] A memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of the first aspects.

[0043] Compared with the closest existing technology, the present invention has the following advantages:

[0044] This invention proposes a method, system, and equipment for detecting leaks in a rolling punching machine based on the longitudinal beam of a vehicle frame. By constructing a real-time data channel integrating an industrial camera, PLC, and CNC system, and employing the OPC UA protocol to synchronously acquire multi-source data, it ensures the temporal consistency between images and equipment status. Through multi-strategy preprocessing and multi-marker combination corner detection, combined with a quality assessment mechanism driven by the coefficient of variation, it significantly improves the success rate and accuracy of calibration under complex lighting and distortion conditions. By loading a dedicated YOLO model to identify three types of regions and filtering beam inner holes based on spatial tolerance matching, it effectively suppresses background interference and false detections. Only the critical vertical distance is calculated and visualized in a CAD-style manner, making the detection results intuitive, readable, and easy to judge. Finally, it automatically compares with the standard hole positions in the CNC program and issues real-time alarms, forming a closed loop of "acquisition—detection—calculation—comparison—feedback." This solves the core problems of traditional solutions, such as poor robustness, high false alarm rate, reliance on manual labor, and lack of anomaly traceability, achieving high-precision, high-efficiency, and high-availability fully automatic leak detection. Attached Figure Description

[0045] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0046] Figure 1 This is a flowchart of a method for detecting leaks in a rolling punch machine based on the longitudinal beam of a vehicle frame, provided by the present invention.

[0047] Figure 2 This is a schematic diagram of a hole detection system for a rolling punch machine based on a vehicle frame longitudinal beam, provided by the present invention.

[0048] Figure 3 This is an internal structural diagram of the electronic device provided by the present invention. Detailed Implementation

[0049] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore merely examples, and should not be construed as limiting the scope of protection of the present invention.

[0050] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0051] This invention provides a method, system, and equipment for detecting leaks in a rolling punching machine based on the longitudinal beam of a vehicle frame. Specifically, it relates to an intelligent method and system for detecting leaks in a rolling punching machine used in the production of longitudinal beams of truck frames, which integrates machine vision, camera calibration, CNC program analysis, and industrial data communication technologies.

[0052] Please refer to Figure 1 , Figure 1 Embodiment 1 of the present invention provides a method for detecting leaks in a rolling punching machine based on a vehicle frame longitudinal beam. The method specifically includes the following steps:

[0053] S101, Construct a real-time data channel to synchronously collect multi-dimensional industrial data; the multi-dimensional industrial data includes: longitudinal beam surface image, punching equipment motion trajectory and instruction parameters in CNC program;

[0054] S102, the acquired longitudinal beam surface image is decomposed into frame data, and multi-strategy preprocessing is performed on the frame data; the preprocessed frame data is calibrated, the optimal calibration result is selected by multi-strategy combination, and the camera parameters and size conversion ratio are calculated;

[0055] S103, Load the pre-established target detection model, detect the pre-processed frame data, identify invalid areas, beam areas and hole areas, filter to obtain the effective hole positions in the beam, and record the position information of the effective hole positions;

[0056] S104, based on the calibrated size conversion ratio, calculate the vertical distance of the effective hole position relative to the beam and the vertical distance between adjacent effective holes on the same vertical line; visualize and number the vertical distance of the effective hole position relative to the beam and the vertical distance between adjacent effective holes on the same vertical line;

[0057] S105, compare the detected number and location data of holes with the standard hole data generated by the CNC program to determine whether there is a hole leakage anomaly; and visualize the visual detection hole data and comparison results through the monitoring interface, and issue a notification when a hole leakage anomaly is determined.

[0058] In step S101 above, the construction of a real-time data channel and the synchronous acquisition of multi-dimensional industrial data includes: constructing a real-time data channel composed of an industrial camera, PLC, host computer and communication components, and synchronizing data with the local area network through the OPC UA industrial communication protocol to synchronously acquire longitudinal beam surface images, punching equipment motion trajectories and instruction parameters in CNC machining programs.

[0059] In step S102 above, the multi-strategy preprocessing includes performing any one or more preprocessing operations on the frame data, such as grayscale conversion, contrast enhancement, Gaussian filtering, adaptive binarization, and morphological processing.

[0060] In step S102 above, the calibration of the preprocessed frame data includes: performing calibration board corner detection on the preprocessed frame data, filtering the optimal calibration result through a combination of multiple strategies; calculating the calibration quality through the coefficient of variation of the corner spacing, selecting the optimal calibration result and calculating the camera intrinsic parameters, distortion parameters and size conversion ratio; if no effective calibration board is detected or the calibration quality is substandard, the size conversion ratio is estimated based on a preset empirical coefficient.

[0061] In step S103 above, the pre-establishment of the target detection model includes:

[0062] A standardized dataset is established based on historically collected multi-dimensional data and its corresponding annotation information. The establishment of the standardized dataset includes: using the LabelStudio annotation tool to annotate the beam outline, ventral holes and invalid areas in the samples, and mixing abnormal samples into the standardized dataset according to a preset ratio.

[0063] Using the YOLO object detection network as the initial model, the object detection model was trained in the PyTorch framework using transfer learning and early stopping mechanisms to obtain the aforementioned object detection model.

[0064] In step S103 above, the filtering to obtain the effective hole positions within the beam and recording the position information of the effective hole positions includes: performing inference detection on the preprocessed frame data, parsing to obtain the position coordinates of the invalid region, the beam region, and the hole region; setting a tolerance range, matching the coordinates of the hole region with the coordinates of the beam region, filtering out invalid hole positions outside the beam region, retaining the effective hole positions within the beam, and recording the position information.

[0065] The above embodiments also include using a hole tracking algorithm to continuously track the hole position. The algorithm is implemented through the following steps:

[0066] Cache the historical location coordinates of the holes;

[0067] A confidence scoring mechanism is introduced, and parameters such as maximum matching distance, maximum allowed number of lost frames, and minimum tracking length are set.

[0068] Using OpenCV, a baseline is drawn to continuously track the hole trajectory and filter out low-confidence trajectories.

[0069] In step S105 above, issuing a notification when a leak is detected includes:

[0070] When the visual inspection hole position data is inconsistent with the hole position data parsed by the CNC program, a notification message containing the beam ID, inspection time, and anomaly type is sent in real time; and all inspection data, comparison results, and timestamp information are automatically stored to enable multi-dimensional query and traceability.

[0071] Example 1: To address the challenge of achieving high-precision, high-consistency, traceable, and fully automated longitudinal beam leakage detection in complex industrial environments, Example 1 of this invention provides a leakage detection method for a rolling punching machine based on a vehicle frame longitudinal beam. The method includes: constructing a real-time data channel to synchronously acquire multi-dimensional industrial data; the multi-dimensional industrial data includes: longitudinal beam surface images, the motion trajectory of the punching equipment, and instruction parameters in the CNC program; decomposing the acquired longitudinal beam surface images into frame data and performing multi-strategy preprocessing on the frame data; performing calibration plate corner point detection on the preprocessed frame data, filtering the optimal calibration result through multi-strategy combination, and calculating camera parameters and size conversion ratios; loading a pre-established target detection model to detect the preprocessed frame data, identifying invalid areas and beam areas. The system filters the beam's internal areas to obtain valid hole positions and records their location information. Based on the calibrated dimensional conversion ratio, it calculates the vertical distance between valid hole positions and the beam body, as well as the vertical distance between adjacent valid hole positions on the same vertical line. The system visualizes and numbers the vertical distances between valid hole positions and the beam body, and the vertical distances between adjacent valid hole positions on the same vertical line. It compares the detected hole quantity and location data with standard hole position data generated by the CNC program to determine if any hole leakage is detected. The system then visualizes the visually detected hole position data and comparison results through a monitoring interface, issuing a notification when a hole leakage is detected.

[0072] Step 1: Construct a real-time data channel to synchronously collect multi-dimensional industrial data; the multi-dimensional industrial data includes: longitudinal beam surface images, punching equipment motion trajectories, and instruction parameters in the CNC program.

[0073] The "real-time data channel" refers to a closed-loop data interaction link consisting of an industrial camera, a programmable logic controller (PLC), a host computer, and communication components. This channel uses the OPC Unified Architecture (OPC UA) industrial communication protocol to achieve millisecond-level timestamp alignment and data synchronization within the local area network. The industrial camera continuously acquires images of the longitudinal beam surface at a frame rate of ≥30 fps. The PLC reports the servo axis position, spindle speed, stamping trigger signal, and current process number of the punching equipment in real time. The host computer synchronously parses the CNC machining program (G-code or proprietary format) and extracts instruction parameters related to the hole position, including but not limited to: hole number, target X / Y coordinates, hole diameter, stamping depth, and processing cycle time. All data streams carry a unified high-precision timestamp (nanosecond level) to ensure that image frames strictly correspond to the equipment status and instruction parameters at the same moment, providing a foundation for subsequent spatiotemporal consistency analysis. This channel supports disconnection reconnection and buffered continuation mechanisms. When the network is momentarily interrupted, the local buffer temporarily stores ≤5 seconds of data, which is automatically retransmitted upon recovery to avoid the loss of key frames. Optionally, the industrial camera can be replaced with a smart camera with encoder triggering function, which can synchronize with the PLC stamping action through hardware-level external trigger signal to further reduce timing jitter; the OPC UA protocol can also be replaced by MQTT over TLS or TSN (Time-Sensitive Networking) deterministic Ethernet protocol to adapt to different production line communication architectures.

[0074] Step 2: Decompose the acquired longitudinal beam surface image into frame data, and perform multi-strategy preprocessing on the frame data.

[0075] "Decomposing into frame data" refers to slicing the video stream according to a preset time interval or event triggering conditions: when the input is a video file, the OpenCV cv2.VideoCapture interface is used to obtain a static frame sequence by extracting 1 frame every 20 frames (i.e., the sampling interval is 20), and a global index number and acquisition timestamp are attached to each frame; when the input is a single image or image sequence, it is loaded in natural order according to the file name to form an ordered frame array. The "multi-strategy preprocessing" includes any one or more combinations of grayscale conversion, contrast enhancement, Gaussian filtering, adaptive binarization, and morphological processing. Specifically, the RGB image is first converted to grayscale using a weighted average method. Then, Contrast Limited Adaptive Histogram Equalization (CLAHE) is used to enhance local contrast, with the clipLimit parameter set to 2.0 and the tileGridSize to 8×8. Next, a 3×3 Gaussian kernel is applied for smoothing and noise reduction. Then, a binary mask is generated using the Otsu adaptive thresholding method. Finally, a 3×3 rectangular structuring element is used for closing operations to fill in holes and minor breaks. This preprocessing chain supports dynamic configuration—users can select enabled items through the UI interface, and the system executes them sequentially according to the selected order. It can also be extended to introduce bilateral filtering to replace Gaussian filtering to preserve edges, or use Gamma correction to replace CLAHE to adapt to strong backlighting scenarios.

[0076] Step 3: Perform calibration board corner detection on the preprocessed frame data, select the optimal calibration result through a combination of multiple strategies, and calculate the camera parameters and size conversion ratio.

[0077] The "calibration board corner detection" uses a checkerboard calibration board (physical size of 9×7 internal corners, single grid side length 10mm), and calls the OpenCV function cv2.findChessboardCornersSB to search for sub-pixel level corner coordinates in the preprocessed image; "multi-strategy combination" refers to enabling multiple flag parameter combinations in parallel: including CALIB_CB_ADAPTIVE_THRESH (adaptive threshold enhancement), CALIB_CB_FAST_CHECK (fast initial screening), CALIB_CB_NORMALIZE_IMAGE (normalized brightness), and CALIB_CB_FILTER_QUADS (quadrilateral topology verification); each set of flags runs corner detection independently, outputting a candidate corner set and confidence score. The "selection of the optimal calibration result" is achieved by calculating the coefficient of variation (CV = standard deviation / mean) of the corner spacing in each candidate set: the smaller the CV, the more uniform the corner distribution and the smaller the distortion effect. The quality score is defined as 1 / (1+CV), and the highest score is selected as the optimal result; if the CV of all combinations is greater than 0.15 or no effective corner is detected, the calibration is judged to have failed. The "Calculate Camera Parameters and Size Conversion Ratio" process involves two paths: Upon success, `cv2.calibrateCamera` is called to calculate the camera's intrinsic parameters (fx, fy, cx, cy), distortion coefficients (k1, k2, p1, p2, k3), and extrinsic parameters for each frame based on the optimal corner point set of ≥10 frames. The pixel-to-millimeter conversion ratio α (unit: mm / pixel) is then derived based on the known physical dimensions of the checkerboard pattern. Upon failure, the ratio α' = d_mm / d_pix is ​​estimated based on the pixel distance d_pix between any two adjacent corner points detected in a single frame and the corresponding physical distance d_mm = 10 mm, or a preset empirical coefficient of 0.15 mm / pixel is used as a fallback value. This ratio α is used for the physical dimension conversion of all subsequent pixel coordinates and remains constant throughout the batch processing unless the calibration process is retried.

[0078] Step 4: Load the pre-established target detection model, detect the pre-processed frame data, identify invalid areas, beam areas and hole areas, filter to obtain the valid hole positions in the beam, and record the position information of the valid hole positions.

[0079] The "pre-built target detection model" is a lightweight target detection model based on the YOLOv8s network, trained in the PyTorch framework. The input image size is uniformly scaled to 640×640. A transfer learning strategy is adopted, with COCO pre-trained weights initialized. The model is fine-tuned using a historical dataset labeled with LabelStudio (containing ≥5000 samples including normal beams, ventral holes, fixture occlusions, oil stains, misaligned holes, etc.). An early stopping mechanism (patience=15) is introduced during training. The model terminates if the validation set mAP@0.5 does not improve for 15 consecutive rounds. The final model outputs three types of bounding boxes: category ID 0 is "invalid region" (including fixtures, background, burrs, etc., which are not detected targets), ID 1 is "beam region" (longitudinal beam body outline), and ID 2 is "hole region" (through holes formed by stamping). "Identifying invalid regions, beam regions, and hole regions" refers to parsing the output tensor after model inference and extracting the category, confidence score, normalized coordinates (x_center, y_center, width, height), and original image coordinates of each bounding box. "Filtering to obtain valid hole positions within beams" is achieved through geometric matching: a tolerance range of 20 pixels outside the beam frame is set, and a point-rectangle inclusion judgment is made between the center point coordinates of each hole region and the beam region bounding box. Only when the hole center falls within the extended beam frame is it considered a "valid hole position within beam," and its original image coordinates (x_px, y_px) and corresponding frame index are recorded. For multi-beam side-by-side scenarios, the model can output multiple beam region boxes. The system clusters them according to the x-coordinate and performs hole matching separately to ensure cross-beam isolation processing.

[0080] Step 5: Based on the calibrated size conversion ratio, calculate the vertical distance of the effective hole position relative to the beam and the vertical distance between adjacent effective holes on the same vertical line.

[0081] The "vertical distance of effective holes relative to the beam" refers to the physical distance (unit: mm) obtained by calculating the difference between the y-coordinate of the center point of each effective hole and the y-coordinate of the baseline, using the upper edge of the beam area boundary frame as the baseline, and then multiplying it by the size conversion ratio α. Simultaneously, the vertical distance from the hole center to the lower edge of the beam is calculated, and the smaller of the two values ​​is taken as the "nearest edge distance." The "vertical distance between adjacent effective holes on the same vertical line" requires spatial grouping of the holes: K-means clustering (k=1~5, automatically selecting the optimal number of clusters) is performed based on the x-coordinate of the hole center, with holes within each cluster considered approximately located on the same vertical line; the holes within each cluster are arranged in ascending order of y-coordinate, and the difference in y-coordinate between adjacent holes Δy_px is calculated, multiplied by α to obtain the physical distance Δy_mm; only holes with Δy_mm ≤ 80 mm are considered as valid adjacent relationships, and those exceeding this are considered holes not in the same process sequence; the first hole (the one with the smallest y-coordinate) is only calculated to the upper edge, while the remaining holes are only calculated to the vertical distance from the previous hole. This calculation logic focuses on the vertical dimension, avoiding the amplification of lateral positioning errors, which is in line with the engineering reality that the vertical spacing tolerance is stricter than the lateral tolerance in the punching process.

[0082] Step 6: Visualize and number the vertical distances of the effective holes relative to the beam and the vertical distances between adjacent effective holes on the same vertical line.

[0083] The "Visual Drawing" function uses OpenCV functions such as cv2.line, cv2.arrowedLine, and cv2.putText to implement CAD-style annotations: For each valid hole, a double-arrow dimension line is drawn, starting from the hole center and extending vertically upwards to the upper edge of the beam. A short horizontal line is added to the end of the line, and the text "Edge→Hole XX mm" is displayed in the center. For adjacent holes within the same vertical cluster, a vertical double-arrow line connecting the centers of the two holes is drawn, labeled "Hole→Hole YY mm". All dimension lines are 2px wide, the arrows are 15px long, the font is SimSun, and the font size is 12pt. The "Numbering" function assigns Arabic numerals 1, 2, 3... sequentially from right to left (i.e., descending x-coordinate) according to the hole center x-coordinate. The numbering circle is 20px in diameter, filled in white, and bordered in black, placed 10px to the right of the hole center. The numbering order is consistent with the annotation habits of process drawings, making it easy for operators to quickly identify the correct dimensions. This visualization module supports scaling to a maximum of 800×600 pixels while maintaining the aspect ratio, and automatically adds scrollbars to any areas exceeding this limit; it can also export PNG or SVG vector graphics for offline review.

[0084] Step 7: Compare the detected number and location data of holes with the standard hole data generated by the CNC program to determine whether there are any abnormal holes. Visually display the visually detected hole data and comparison results through the monitoring interface, and issue a notification when an abnormal hole is detected.

[0085] The "standard hole position data generated by the CNC program" is obtained by the host computer parsing G-code or dedicated NC files: all lines containing G81 / G82 drilling cycle instructions are extracted, and the X / Y coordinates, R-plane, Z-depth, and cycle count are parsed. Combined with the workpiece coordinate system origin offset, it is converted into absolute physical coordinates (unit: mm) relative to the beam reference point, and grouped by x-coordinate to form standard vertical hole columns; the number of standard holes, theoretical vertical spacing, and allowable tolerance (±0.3 mm) of each column are included in the comparison rules. The "comparison judgment" adopts a tolerance matching algorithm: for each vertical cluster, the detection hole y-coordinate sequence is matched with the standard hole y-coordinate sequence using Dynamic Time Warping (DTW), and the cumulative distance is calculated; if a standard hole has no matching item in the detection sequence (distance > 1.5 times the theoretical spacing), and the hole is not in the "skipped hole" list marked in the program comments, it is judged as a missing hole anomaly; at the same time, the total deviation (|N_detect − N_standard| ≥ 1) is used as an auxiliary criterion. The monitoring interface, built on Tkinter, uses a Notebook control to display the results of each frame in paginated format: the left Canvas displays the annotated image, and the right Text control outputs a structured text summary, including beam ID, frame time, number of detected holes, number of standard holes, leak hole location number, and anomaly type (e.g., "Missing Hole #3", "Spacing Exceedance Hole #5→#6"). The bottom Statusbar refreshes the processing progress and logs in real time. "Issue Notification" means that when a leak is detected, the system automatically generates a JSON-formatted alarm message, including beam ID, detection timestamp, abnormal hole number, associated device ID, and screenshot URL, and pushes it to the MES system interface via HTTP POST; simultaneously, it triggers an audible and visual alarm (USB buzzer + LED light) and pops up a modal dialog box with a confirmation button on the interface to prevent false alarms from being ignored.

[0086] Through the above steps, a complete technology chain has been achieved, which includes "multi-source synchronous acquisition - robust image preprocessing - adaptive calibration and scale estimation - YOLO semantic detection - geometric filtering - accurate calculation of vertical distance - CAD-style structured annotation - closed-loop comparison of CNC instructions - multi-channel anomaly notification". By constructing a timestamp alignment channel based on OPC UA, the problem of misjudgment caused by the asynchronous nature of image and device status was solved. Through multi-strategy preprocessing and multi-flags corner detection, the calibration success rate under oil stains, low light, and slight motion blur was significantly improved. The YOLO model's end-to-end recognition capability for beam contours and holes, combined with a geometric filtering mechanism with a 20 px tolerance, effectively suppressed interference from invalid areas such as fixtures and shadows. Distance calculation and annotation only in the vertical direction not only meets the key control dimensions of the punching process but also avoids the accumulation of lateral positioning errors. The CAD-style numbering and dimension line design allows operators to complete abnormal positioning within 3 seconds. The experience-based proportion fallback strategy for calibration failures ensures the basic usability of the system in scenarios without a calibration board. Finally, by comparing the visual inspection results with the CNC program instructions item by item and linking the MES and audio-visual equipment, a traceable, responsive, and fully automated closed loop for leak quality control was formed, which greatly reduced the frequency of manual sampling and the misjudgment rate, meeting the high reliability manufacturing requirements of automotive chassis components.

[0087] Example 2: Based on the above examples, this example further provides: constructing a real-time data channel to synchronously collect multi-dimensional industrial data, including: constructing a real-time data channel composed of an industrial camera, PLC, host computer and communication components, and synchronizing data with the local area network through the OPC UA industrial communication protocol to synchronously collect longitudinal beam surface images, punching equipment motion trajectories and instruction parameters in CNC machining programs.

[0088] Step 1: Construct a real-time data channel consisting of an industrial camera, PLC, host computer, and communication components;

[0089] The industrial camera is a global shutter type CMOS industrial camera with a resolution of no less than 1920×1200 and a frame rate of ≥30fps. It is equipped with a telecentric lens and a ring LED light source and is installed directly above the rolling punching station to ensure that the image on the longitudinal beam surface covers the entire punching area and that distortion is controllable. The PLC is a programmable logic controller that supports EtherCAT or PROFINET bus, and the models include, but are not limited to, Siemens S7-1500 series or Beckhoff CX series. It acquires the displacement sensor of the punching hydraulic cylinder, the servo motor encoder, and the pressure switch signal of the pneumatic clamp in real time through a high-speed digital input module, and analyzes and generates the equipment motion trajectory data, including X / Y / Z axis position coordinates, acceleration change sequence, and punch trigger force timestamp. The host computer is equipped with Windows. The industrial control computer, running OS X / 11, is equipped with dual network cards—one connected to the production site's local area network (LAN), and the other connected to the CNC system's RS422 / Ethernet interface. This interface is used to parse G-code or ISO standard CNC program files (.nc / .tap), extracting M-instructions (such as M03 spindle start), G-instructions (such as G01 linear interpolation), coordinate parameters (X / Y / Z values), and process parameters (feed speed F, spindle speed S). Communication components include OPC UA server software (such as Unified Automation UaCPPServer or KEPServerEX), OPC UA client driver modules, and network switches. The switches feature QoS traffic priority tagging, assigning high-priority VLAN tags to image streams to ensure bandwidth. All devices are connected via a gigabit industrial Ethernet network in a star topology. Time synchronization across all nodes uses the IEEE 1588 Precision Time Protocol (PTP), with clock deviation controlled within ±100 ns, providing a nanosecond-level time reference for subsequent multi-source data alignment.

[0090] Step 2: Achieve data synchronization with the local area network via the OPC UA industrial communication protocol;

[0091] OPC UA (Open Platform Communications Unified Architecture) is a service-oriented, cross-platform, secure, and reliable industrial interoperability protocol. Its core comprises an information modeling layer, a service layer, and a transport layer. In this embodiment, the industrial camera encapsulates the image stream into binary data packets conforming to the OPC UA PubSub model via the GigE Vision protocol, carrying image timestamps, exposure parameters, and ROI region identifiers. The PLC, acting as the OPC UA server, maps its collected motion trajectory data to variable nodes in the address space. Each node is named according to the IEC 61850 standard, such as "MotionTrajectory.Position.X" and "MotionTrajectory.Timestamp". The host computer, acting as the OPC UA client, subscribes to all the above nodes through a secure channel (SecurityPolicy: Basic256Sha256, MessageSecurityMode: SignAndEncrypt) and calls the CreateSubscription and Publish services at a fixed period (default 10). (ms) Batch data retrieval; the local area network uses a full-duplex fiber optic link with an end-to-end latency of ≤200 μs and a packet loss rate of <10⁻ 6 All data is cached in the host computer's memory according to a unified timestamp queue. A sliding window mechanism is used to align image frames with the corresponding motion trajectory points and instruction parameter groups at each moment. The window length is set to 50 ms, allowing a maximum timing offset of ±5ms. If this is exceeded, data resampling or interpolation compensation (linear interpolation or cubic spline interpolation) is triggered. This synchronization mechanism does not rely on external triggering hardware, nor does it force the three types of devices to share a physical clock. Logical synchronization is achieved solely through the built-in timestamp semantics of OPC UA and the low latency characteristics of the local area network.

[0092] Step 3: Simultaneously acquire images of the longitudinal beam surface, the motion trajectory of the punching equipment, and the instruction parameters in the CNC machining program;

[0093] The longitudinal beam surface image is an 8-bit grayscale image or a BGR three-channel color image, with each frame appended with EXIF ​​metadata, including acquisition time (UTC+8), camera ID, exposure time, gain value, and lens distortion correction status flag; the punching equipment motion trajectory is a discrete time series, with each trajectory record containing at least six dimensions: timestamp (ns precision), X / Y / Z absolute coordinates (mm), velocity vector (mm / s), acceleration modulus (m / s²), and punch current status (standby / pressing / returning / holding pressure); the instruction parameters in the CNC machining program are structured parsing results, including program segment number (Nxx), G / M code type, coordinate values ​​(X / Y / Z / I / J / K), feed rate ( The parameters F), spindle speed (S), tool number (T), and subroutine call level are all bound to timestamps according to the execution order, forming a traceable instruction execution timeline. These three types of data are uniformly organized in the host computer as "DataFrame" objects. Each DataFrame corresponds to a physical synchronization moment, and its internal fields include image_buffer (image memory pointer), motion_vector (track array), cnc_instruction (instruction structure array), and sync_quality_score (synchronization quality score, calculated based on timestamp variance; a threshold ≥ 0.95 is considered high-reliability synchronization). When the synchronization quality score falls below the threshold, the system automatically activates a backup strategy: freezing the most recent valid synchronization parameters or switching to a hard synchronization mode based on PLC pulse triggering (requires an external photoelectric encoder signal).

[0094] Through the above steps, this application achieves sub-millisecond multi-source data logic synchronization between industrial cameras, PLCs, and host computers without dedicated synchronization hardware, relying on the OPC UA protocol and local area network infrastructure. Because it uses the PubSub mechanism of OPC UA and PTP time synchronization, it solves the problem of image-track-instruction timing misalignment caused by protocol stack delay jitter in traditional Modbus TCP or custom Socket communication; because it parses CNC instruction parameters into a timestamped structured data stream, rather than simply reading static program files, it avoids the technical defect that dynamic parameter changes (such as adaptive feed adjustment) during machining cannot be detected by the system; and because it introduces a synchronization quality scoring and dual-mode backoff mechanism, it can maintain the spatiotemporal consistency of the detection data even in scenarios of network transient congestion or PLC response delay. Therefore, it provides a stable, reliable and verifiable data input foundation for "constructing a real-time data channel and synchronously collecting multi-dimensional industrial data" in claim 1, ensuring that the original data used in subsequent image analysis, hole location identification and size comparison have strict time-series fidelity, fundamentally suppressing false detection (misjudging images that have not reached the punching position as leaks) or missed detection (failure to match the actual punching time due to trajectory lag) caused by data asynchrony, and significantly improving the accuracy of leak detection to over 99.92% (actual measurement data).

[0095] Example 3: Based on the above examples, this example further provides:

[0096] Multi-strategy preprocessing includes performing one or more preprocessing operations on frame data, such as grayscale conversion, contrast enhancement, Gaussian filtering, adaptive binarization, and morphological processing.

[0097] Grayscale conversion refers to converting the original RGB three-channel color image into a single-channel grayscale image using a weighted average method (such as OpenCV's default 0.299R + 0.587G + 0.114B) or a single-channel extraction method (such as directly extracting the green channel). This eliminates color information redundancy, reduces computational dimensionality, and improves the stability of subsequent corner detection and edge recognition. This operation can be replaced by extracting the Y component in the YUV space, or using ITU-R BT.601 standard coefficients for linear weighting. Alternatively, the grayscale stream can be directly output at the industrial camera firmware layer to avoid repeated decoding on the host computer.

[0098] Contrast enhancement refers to addressing the problem of blurred local details caused by uneven lighting, low illumination, or reflective areas. It employs the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm to dynamically stretch grayscale images in blocks. Its core parameters include clipLimit (range 1.0–4.0, typical value 2.0) and tileGridSize (typical value 8×8 pixel grid). By controlling the upper limit of local contrast increase and the granularity of the division, it enhances the clarity of hole edges and calibration plate corners while suppressing noise amplification. This operation can be replaced by Gamma correction (γ=0.6–0.8), Retinex-type algorithms (such as MSRCR), or multi-scale contrast fusion. It can also be combined with white balance pre-compensation to adapt to different light source color temperatures.

[0099] Gaussian filtering refers to using a two-dimensional Gaussian kernel function to perform convolution operations on the image to achieve spatial domain smoothing and denoising. The kernel size used is 3×3 pixels with a standard deviation σ=0.8, which can effectively suppress high-frequency random noise and sensor thermal noise while preserving edge structure information. This operation can be replaced by median filtering (suitable for scenarios dominated by salt and pepper noise), bilateral filtering (balancing edge preservation and noise suppression), or non-local means denoising. The kernel size can also be dynamically adjusted according to the signal-to-noise ratio (e.g., a 5×5 kernel is enabled when SNR < 20 dB).

[0100] Adaptive binarization refers to dynamically calculating the threshold based on the statistical characteristics of the local neighborhood and completing the binary segmentation. Specifically, it uses the cv2.adaptiveThreshold function in OpenCV, selects the ADAPTIVE_THRESH_GAUSSIAN_C method, with a neighborhood size of 11×11 pixels and a constant C=2, so that key edges such as the black and white grid boundaries of the calibration board and the outline of holes can still be stably separated in areas with drastic changes in brightness. This operation can be replaced by the Otsu global thresholding method (suitable for uniformly lit scenes), the Sauvola local thresholding method (specifically for texture document enhancement), or a deep learning-driven semantic binarization network (such as the lightweight branch of U-Net). Morphological closing operations can also be superimposed to bridge the gaps in hole edges caused by threshold fluctuations.

[0101] Morphological processing refers to performing set operations such as dilation, erosion, opening, or closing on binary images using structuring elements. Opening (erosion followed by dilation) is used to eliminate isolated noise and small burrs, while closing (dilation followed by erosion) is used to fill breaks inside holes or connect weak edges. Structuring elements use 3×3 rectangular or cross-shaped kernels. This operation can be replaced by top-hat transformation to enhance bright targets, black-hat transformation to enhance dark targets, or conditional morphology to limit the area of ​​operation. Alternatively, the scale of structuring elements can be constrained based on prior aperture size (e.g., setting the maximum connected region area threshold to 50 pixels to filter fake holes).

[0102] There are clear temporal dependencies and functional complementarities among the five types of preprocessing operations mentioned above: grayscale conversion provides a unified input basis for all subsequent operations; contrast enhancement and Gaussian filtering form an "enhancement-smoothing" synergy, with the former improving the signal-to-noise ratio and the latter suppressing the noise amplification effect introduced by enhancement; adaptive binarization relies on the results of preceding enhancement and smoothing to obtain a stable local threshold; morphological processing acts on the binarized output to perform post-processing restoration of the geometric integrity of the target. Each operation can be enabled individually (e.g., enabling CLAHE only in strong lighting scenarios) or combined as needed—for example, when vibration causes image blurring, a chained process of "Gaussian filtering → CLAHE → adaptive binarization" is used; in low-contrast + high-noise scenarios, a four-level cascade of "CLAHE → Gaussian filtering → adaptive binarization → opening operation" is used; in real-time-constrained scenarios, only a lightweight two-level process of grayscale conversion + adaptive binarization is enabled.

[0103] Through the above steps, this application achieves robust enhancement of the longitudinal beam surface image frame data: grayscale conversion significantly reduces computational load and unifies color response differences; contrast enhancement and Gaussian filtering synergistically improve local discernibility and noise suppression under complex lighting conditions; adaptive binarization ensures stable segmentation of calibration plate corners and hole edges in the light-dark boundary region; morphological processing further corrects segmentation errors, ensuring sufficient structural fidelity in the input images of subsequent corner detection and target recognition modules. Therefore, it solves the technical problems of high calibration failure rate and frequent false and missed hole detections caused by uneven lighting, sensor noise, imaging distortion, and workpiece surface reflection interference in the original image. It improves the adaptability and generalization ability of the multi-strategy preprocessing module to actual production line conditions, providing pre-image quality assurance for calibration quality screening, target detection accuracy, and reliable hole detection.

[0104] Example 4: Based on the above examples, this example further provides:

[0105] The calibration of the preprocessed frame data includes: detecting the corner points of the calibration board in the preprocessed frame data, and selecting the optimal calibration result through a combination of multiple strategies; calculating the calibration quality through the coefficient of variation of the corner point spacing, selecting the optimal calibration result, and calculating the camera intrinsic parameters, distortion parameters, and size conversion ratio; if no effective calibration board is detected or the calibration quality is not up to standard, the size conversion ratio is estimated based on preset empirical coefficients.

[0106] Step 1: Perform calibration board corner detection on the preprocessed frame data, and select the optimal calibration result through a combination of multiple strategies;

[0107] The calibration board is a two-dimensional reference object with a known physical geometry. It can be selected as a checkerboard calibration board, with its inner corner point array arranged in a regular rectangular pattern. The standard specification is 8×6 inner corner points (corresponding to 9×7 squares), and the physical side length of a single square is 10 mm. This size is configurable. The surface of the calibration board is made of a high-contrast matte material to avoid specular reflection interfering with image acquisition.

[0108] Corner detection refers to locating the pixel coordinates of chessboard intersections in a preprocessed grayscale image. It is implemented using the `findChessboardCorners` function from the OpenCV library, but not as a single call; instead, it constructs a multi-strategy combined execution path: first, five preprocessing methods are applied in parallel to the same frame image—original grayscale image, CLAHE contrast-limited adaptive histogram equalization (clipLimit=2.0, tileGridSize=(8,8)), Gaussian filtering followed by CLAHE (GaussianBlur kernel size 3×3), adaptive binarization (blo... (ckSize=11, C=2) Morphological closing operation enhances edge continuity; then, four types of detection flags are enabled for each preprocessing result: CALIB_CB_ADAPTIVE_THRESH (adaptive threshold enhancement), CALIB_CB_FAST_CHECK (fast initial screening skips invalid regions), CALIB_CB_NORMALIZE_IMAGE (normalization improves dynamic range), and CALIB_CB_FILTER_QUADS (filters non-quadrilateral candidates); thus generating a maximum of 20 sets (5 preprocessing × 4 flags) of corner detection candidate results.

[0109] The multi-strategy combination screening of the optimal calibration result refers to selecting the best result after quality evaluation of all candidate results, without relying on any fixed path; the screening process does not introduce human intervention and is completed automatically by the algorithm; each candidate result is cached in the form of an independent corner point set, and then uniformly participates in quality measurement and calibration calculation.

[0110] Step 2: Calculate the calibration quality using the coefficient of variation of the corner spacing, select the optimal calibration result, and calculate the camera intrinsic parameters, distortion parameters, and size conversion ratio;

[0111] Corner spacing refers to the Euclidean distance between adjacent inner corners in the same row or column. For each candidate corner set, the spacing between all adjacent point pairs is extracted along the horizontal and vertical directions to form two sets of distance sequences. The coefficient of variation (CV) is defined as the standard deviation divided by the mean, used to quantify the consistency of the spacing within the corner set. The smaller the CV, the more regular the corner arrangement, the smaller the image distortion, and the higher the detection reliability. The calibration quality Q is calculated according to the formula Q = 1 / (1 + CV), with a value range of (0,1]. Q ≥ 0.95 is considered high quality, and Q < 0.85 is considered low quality. The system traverses all candidate results and selects the one with the largest Q value as the optimal calibration result.

[0112] The calculation of camera intrinsic parameters, distortion parameters, and size conversion ratio refers to solving the camera model parameters using the OpenCV calibrateCamera function based on the optimal corner point set and its corresponding physical coordinates (unit: mm). These parameters include focal lengths fx and fy (unit: pixels), principal point coordinates cx and cy (unit: pixels), and radial distortion coefficients k1, k2, and k3, and tangential distortion coefficients p1 and p2. The size conversion ratio ρ (unit: mm / px) is determined by the ratio of the physical side length L (10 mm) of a single cell on the calibration plate to the average pixel side length l of the corresponding cell in the image, i.e., ρ = L / l. This ratio is used for the linear conversion of all subsequent pixel coordinates to physical coordinates and serves as the benchmark for calculating the vertical distance of the hole.

[0113] Step 3: If no valid calibration plate is detected or the calibration quality is substandard, estimate the size conversion ratio based on preset empirical coefficients;

[0114] "No valid calibration board detected" means that none of the 20 candidate results returned a non-empty corner point set; "Calibration quality substandard" means that the Q value of all candidate results is lower than the preset threshold of 0.85. At this time, the system abandons the calibration operation and enables the rollback mechanism. The preset empirical coefficient is an empirical constant obtained from actual measurement and calibration on the production line. The current value is 0.15 mm / px. This value corresponds to the average pixel physical scale of a typical industrial camera (resolution of 1920×1080, lens focal length of 12 mm, working distance of 600 mm) under standard installation conditions. This coefficient is stored in the configuration file and can be modified by the user according to the actual camera model, lens parameters and installation height. The estimated size conversion ratio ρ′ = 0.15 mm / px, although the accuracy is lower than the calibration result (error of about ±8%), still meets the relative position tolerance requirement (±2 mm) required for hole detection, ensuring that the system can continuously output usable distance data under extreme conditions such as no calibration board, strong reflection, and severe motion blur.

[0115] There is a deterministic logical progression among the various technical features: multi-strategy preprocessing combined with flags constitutes the input layer, providing diverse image representations for corner detection; the corner spacing variation coefficient serves as a quality criterion, establishing a quantitative mapping from image features to calibration reliability; the optimal result selection mechanism directly couples the quality assessment results to the parameter calculation stage, forming a closed-loop feedback; and the empirical coefficient backoff mechanism serves as a safety redundancy, maintaining the system's functional integrity when all preceding stages fail. These three elements work together to form a four-level robust calibration architecture of "detection—evaluation—decision—fault tolerance".

[0116] Through the above-described steps, this application achieves the following: stable acquisition of high-confidence camera calibration parameters even in the complex on-site environment of a rolling punch production line (with vibration, sudden changes in illumination, workpiece displacement, and lens distortion); a quality optimization mechanism driven by the coefficient of variation significantly reduces the risk of misuse of substandard calibrations caused by local noise or distortion; when calibration completely fails, empirical coefficients ensure uninterrupted dimensional conversion capabilities, enabling downstream modules such as hole vertical distance calculation, leak hole comparison, and visual annotation to continue operating; thus solving the problems of inaccurate dimensional measurement and detection interruption caused by "single calibration method being prone to failure, lack of quality assessment, and system shutdown after failure" in the background technology, and improving the engineering robustness, ease of operation and maintenance, and production line adaptability of the entire leak hole detection method.

[0117] Example 5: Based on the above examples, this example further provides:

[0118] The pre-establishment of the object detection model includes: establishing a standardized dataset based on historically collected multi-dimensional data and its corresponding annotation information; establishing the standardized dataset includes: using the LabelStudio annotation tool to annotate the beam contours, ventral holes and invalid regions in the samples, and mixing abnormal samples into the standardized dataset according to a preset ratio; using the YOLO object detection network as the initial model, training it in the PyTorch framework using transfer learning and early stopping mechanisms to obtain the object detection model.

[0119] Among them, "establishing a standardized dataset based on historically collected multi-dimensional data and its corresponding annotation information" refers to collecting raw image data covering the entire working conditions of the real production line, including images of multiple batches of rolling and punching processes under different lighting conditions (such as strong midday light, nighttime supplemental lighting, and side backlighting), different surface conditions (such as oil stain reflection, rust interference, and condensation film), different equipment vibration amplitudes (corresponding to image motion blur of 0.5–3 pixels), and different longitudinal beam models (including variable cross-sections, reinforcing rib structures, and differences between hot-rolled and cold-rolled materials); at least 2000 valid images are collected for each type of working condition, and... The process involves recording auxiliary metadata such as the PLC motion trajectory coordinates, CNC program G-code segments, and instantaneous servo current values ​​of the punching equipment at corresponding moments, forming a multimodal sample set with time-series labels. This dataset is then deduplicated, timestamp aligned, and resolution normalized (uniformly scaled to 1280×720 while maintaining aspect ratio) before serving as the original input for annotation. This step aims to construct basic data resources with scene representativeness, balanced distribution, and physical interpretability, providing data assurance for the subsequent model generalization ability. Optional alternatives include: using Synthetic Data Generation to construct a high-fidelity 3D model of the longitudinal beam using Blender or Unity, simulating different camera poses, lighting angles, and surface textures to generate simulated images with precise pixel-level masks, and mixing them into the real dataset at a 30% ratio; or using a semi-automatic annotation pipeline, first pre-annotating with a lightweight YOLOv5s model, and then manually verifying and correcting it to improve annotation efficiency.

[0120] The phrase "using LabelStudio annotation tools to annotate the beam outline, ventral openings, and invalid areas in the sample" refers to configuring three categories of bounding boxes in the LabelStudio platform. The Box + Polygon hybrid annotation template: For the "Beam" category, polygon annotations are used that closely follow the outer edges of the web and flanges, with ≥12 vertices to accommodate curved longitudinal beam contours; for the "Body Holes and Openings" category, minimum bounding rectangle annotations are used, while also supporting the overlay of polygonal annotations for non-circular holes (such as oblong holes and elliptical holes); for the "Invalid Area" category, arbitrary shape masks are used to cover interfering areas such as burrs, fixture projections, weld slag accumulation, and lens stains; all annotations are bound to attribute fields, including hole type (circular hole / oblong hole / irregular hole), hole diameter (mm), beam segment number, and annotation confidence level (1–5 points); the annotation process is cross-checked by two trained technicians, achieving a single-icon annotation consistency rate of over 98.7%; the purpose of this step is to establish a supervisory signal source with high geometric accuracy, clear semantic definition, and strong traceability; alternative solutions include: using CVAT (Computer Vision Annotation). The tool platform enables collaborative annotation and version management; or it uses instance segmentation to annotate high-value samples (such as typical images of leaks) and outputs pixel-level masks instead of bounding boxes to improve the accuracy of hole edge positioning.

[0121] The "mixing abnormal samples into the standardized dataset at a preset ratio" refers to extracting a subset of images with clear defect characteristics from the actual production line, including five typical anomalies: leaky holes (missing design holes), misaligned holes (positional offset > ±1.5 mm), blocked holes (holes filled with welding slag or sludge), connected holes (adjacent holes fused together, causing contour adhesion), and deformed holes (ellipticity > 0.3). This subset is randomly inserted into the standardized dataset at a ratio of 15%, ensuring that the abnormal samples in each batch of training data cover all five types of defects, with the proportion of a single type of anomaly not less than 2%. Simultaneously, targeted data augmentation is applied to the abnormal samples: random occlusion (Block Size = 32×32 pixels, occlusion rate 10%–20%) is superimposed on leaky hole images; subpixel-level affine shift (±0.8 pixels) is applied to misaligned hole images; and Gaussian noise (σ = 0.05) and local contrast attenuation (CLAHE) are added to blocked hole images. The purpose of this step (clipLimit=1.0) is to force the model to learn the underlying visual representation of defect patterns, avoiding a decrease in the sensitivity to anomaly discrimination due to the dominance of normal samples. Optional alternatives include: using GAN (Generative Adversarial Network) to generate adversarial anomaly samples, such as using CycleGAN to transfer the style of normal hole images to hole-clogging artifacts; or introducing a hard example mining mechanism to dynamically identify the batch of samples with the highest FP (False Positive) rate during training iterations, increase their weight by 1.5 times, and then participate in the next round of training.

[0122] The phrase "using the YOLO object detection network as the initial model" refers to selecting Ultralytics YOLOv8m as the backbone network architecture, with CSPDarknet53 as the backbone, PAN-FPN (PathAggregation Network–Feature Pyramid Network) as the neck, and decoupled network as the head. Head (Decoupled Classification and Regression Head); The input image size is fixed at 640×640, and a joint strategy of Mosaic data augmentation (four-image stitching) and MixUp (image linear mixing) is adopted; The model parameters are initialized using COCO pre-trained weights (yolov8m.pt), which have converged in general object detection tasks and have good feature extraction prior capabilities; The purpose of this step is to accelerate domain adaptation by leveraging large-scale general visual knowledge and reduce the risk of overfitting in small sample scenarios; Optional alternatives include: replacing with YOLOv10 (supporting end-to-end detection without NMS post-processing), RT-DETR (a real-time detector based on Transformer), or PP-YOLOE (a high-precision industrial detection model open sourced by Baidu); or using lightweight variants such as YOLOv8n (nano) to adapt to edge computing devices.

[0123] The phrase "training using transfer learning and early stopping mechanism within the PyTorch framework" refers to constructing the training process in the PyTorch 2.0 environment: freezing the first 5 CSP modules of the backbone network, and only fine-tuning the parameters of the 32 layers in the neck and head regions; using AdamW as the optimizer (initial learning rate 1e−4, weight decay 1e−5), and employing cosine annealing LR for learning rate scheduling; using a weighted combination of CIoU Loss (Complete Intersection over Union) and Focal Loss (weight ratio 0.7:0.3) to enhance gradient response to small target holes and difficult-to-distinguish samples; the early stopping mechanism is set as follows: monitoring the validation set mAP@0.5 metric, if it does not improve for 15 consecutive epochs, training is terminated and rolled back to the checkpoint corresponding to the highest mAP@0.5 value; the training batch size is 16, the maximum number of total epochs is 300, and the training is performed on a single NVIDIA RTX GPU. Training (4090) takes approximately 8.2 hours. This step aims to balance the model's convergence speed with its generalization performance, preventing memory overfitting on limited production data. Alternative options include: using knowledge distillation, with YOLOv8x as the teacher model to guide the training of the YOLOv8m student model; or introducing label smoothing to replace hard labels, setting the true class probability to 0.9 and distributing the remaining classes equally at 0.1 to improve model robustness.

[0124] There are deterministic logical dependencies among the various technical features: multi-dimensional historical data forms the input basis for annotation work; the quality of LabelStudio annotations directly determines the effectiveness of the supervision signal; the proportion of outlier samples affects the model's sensitivity threshold to defects; the selection of the YOLO network restricts the upper limit of feature expression ability; the transfer learning strategy controls the parameter update range; and the early stopping mechanism ensures the stability of the final model on the validation set. The above elements together form a closed-loop model building chain: data diversity supports annotation coverage, annotation accuracy ensures supervision quality, outlier sample injection drives defect perception, network architecture selection determines detection granularity, transfer learning accelerates the convergence process, and the early stopping mechanism locks the optimal generalization point.

[0125] Through the above steps, this application achieves highly customized target detection capabilities for the rolling punching scenario of vehicle frame longitudinal beams: By employing historical multi-dimensional data covering all working conditions and supplemented with LabelStudio fine-grained annotation, the model can accurately distinguish the web body, flange transition zone, hole edges, and various invalid interference areas; by mixing in five typical abnormal samples at a 15% ratio and applying targeted enhancements, the model's recall rate for key defects such as missing holes and misaligned holes is increased to 99.2%, a 12.6 percentage point improvement compared to the general YOLOv8m baseline model; and because it is based on PyTorch... The framework implements transfer learning and early stopping mechanisms, enabling the model to converge with only 2,000 labeled images, shortening the training cycle by 43%. In cross-type longitudinal beam tests, the mAP@0.5 remains above 86.4%, significantly outperforming the same architecture model without transfer training (72.1%). Ultimately, this model becomes the core execution unit for "identifying invalid regions, beam regions, and hole regions" in claim 1, providing structured outputs with high confidence and sub-millimeter positioning accuracy for subsequent steps such as "filtering effective hole positions within beams" and "calculating vertical distances," fundamentally ensuring the reliability and traceability of the hole detection results.

[0126] Example 6: Based on the above examples, this example further provides:

[0127] The process of filtering out valid hole positions within the beam and recording their location information includes: performing inference detection on the preprocessed frame data to parse out the location coordinates of the invalid region, the beam region, and the hole region; setting a tolerance range, matching the coordinates of the hole region with the coordinates of the beam region, filtering out invalid hole positions outside the beam region, retaining valid hole positions within the beam, and recording their location information.

[0128] The "inference detection of preprocessed frame data" refers to calling a pre-trained target detection model (based on the YOLO architecture, trained in the PyTorch framework through transfer learning and early stopping mechanism), taking a single frame of grayscale or enhanced image as input, and outputting the category label, bounding box coordinates (x_min, y_min, x_max, y_max), and confidence score of each detected target. The model outputs three categories: invalid regions (such as clamp occlusion areas, background interference areas, oil stain reflection areas, and other non-beam-related areas), beam regions (i.e., the outline of the frame longitudinal beam, including the overall structure of the web and flange), and hole regions (i.e., through holes formed by stamping, including circular, elliptical, and slightly deformed holes). During model inference, a configurable confidence threshold (default conf=0.25) and a non-maximum suppression threshold (default iou=0.3) are used to balance the detection rate and false detection rate. This step can be replaced by other single-stage target detection models (such as PP-YOLO, RT-DETR) or two-stage models (such as Faster). R-CNN is acceptable as long as its output includes the spatial localization capabilities of the three types of regions mentioned above.

[0129] The process of "analyzing the position coordinates of invalid regions, beam regions, and hole regions" refers to performing structured analysis on the original output of the model: traversing all detection boxes and classifying and storing them according to their category IDs (e.g., class_id=0 corresponds to invalid regions, class_id=1 corresponds to beam regions, and class_id=2 corresponds to hole regions); merging or filtering multiple detection boxes of the same category—for beam regions, only retaining the single bounding box with the largest area and an aspect ratio that conforms to the geometric characteristics of the longitudinal beam (aspect ratio ≥ 5:1) as the reference beam box for subsequent spatial matching; for hole regions, retaining all detection boxes but removing tiny noise boxes with an area smaller than a preset pixel threshold (default 30 px²); this analysis process supports multi-beam scene extensions, and when multiple beam region detection boxes exist, grouping them by X coordinate and processing hole matching within each beam segment independently.

[0130] "Setting tolerance range" refers to expanding the beam region boundary box outward by a fixed pixel distance to form a tolerance-enclosing area. This tolerance range is a two-dimensional rectangular area with its four sides parallel to the original beam frame. The left and right boundaries are each expanded outward by 20 pixels, and the top and bottom boundaries are also expanded outward by 20 pixels. This value can be adjusted according to the actual production line vibration amplitude, camera installation tilt angle deviation, and thermal expansion and contraction deformation of the beam body, with a value range of 10–50 pixels. Alternatively, a non-uniform tolerance strategy can be adopted, such as expanding only the lower edge of the beam frame outward by 30 pixels (to compensate for the hole edge offset caused by burr lifting after punching), while keeping the other directions at 15 pixels. It can also be replaced by a morphological expansion operation based on the beam region mask (such as cv2.dilate, kernel size 5×5) to achieve a tolerance expansion that better fits the beam body contour.

[0131] The "matching of hole region coordinates with beam region coordinates" refers to using the center point coordinates ((x_min+x_max) / 2, (y_min+y_max) / 2) of each hole region detection frame as the judgment point to determine whether it falls within the tolerance containment area. The matching judgment adopts a point-rectangle containment relationship algorithm (a simplified implementation of cv2.pointPolygonTest in OpenCV or direct coordinate comparison). If the x-coordinate of the center point is between the left and right boundaries of the tolerance beam frame and the y-coordinate is between the upper and lower boundaries of the tolerance beam frame, it is considered a successful match. This judgment logic can be replaced with a more robust spatial relationship judgment: for example, calculating the intersection-union ratio (IoU) of the hole frame and the beam frame, and identifying the hole as an inner hole of the beam when IoU≥0.3; or using the minimum circumscribed circle center point + radius tolerance method to adapt to the perspective distortion compensation requirements under tilted camera installation.

[0132] The process of "filtering out invalid holes outside the beam area, retaining valid holes within the beam and recording their position information" refers to performing the aforementioned matching judgment on all hole area detection frames, and then only including successfully matched hole frames in the valid hole set. Each valid hole position records its original pixel coordinates (x_min, y_min, x_max, y_max), center point pixel coordinates (x_c, y_c), normalized relative coordinates (offsets relative to the upper left corner of the beam frame Δx, Δy), and a unique sequence number (arranged in descending order from right to left according to x_c, ensuring that the numbering order is consistent with the physical layout). The position information is stored in the form of a structured dictionary, with keys including "hole_id", "bbox_px", "center_px", and "rel_pos_mm" (subsequently converted by size conversion ratio). This record format supports expansion to JSON or CSV serialization output, facilitating structured comparison with standard hole position data in CNC programs. A timestamp field (corresponding to video frame index or image acquisition time) can also be added to support multi-frame hole position trajectory tracking.

[0133] The above-mentioned technical features work together as follows: inference detection provides the initial positioning basis for three types of regions; analytical operation establishes the beam region as a spatial reference benchmark; the tolerance range design takes into account both image positioning error and physical assembly tolerance; the coordinate matching mechanism realizes strict geometric attribution determination; and the position recording format ensures data consistency for subsequent distance calculation and visualization annotation.

[0134] Through the above steps, this application achieves geometric semantic refinement of the original detection results of the YOLO model. Without changing the model structure and training strategy, it effectively removes non-target interference items such as fixture holes, background holes, and splashed oil fake holes through post-processing logic alone. By introducing a 20-pixel tolerance and dynamically expanding the matching domain based on the beam frame, it solves the problem of misjudging hole edges caused by camera shake, beam conveying offset, and local deformation. By limiting the retention of only holes within the beam frame (including the tolerance) and strictly numbering them, it ensures that the input data relied upon by subsequent steps such as "vertical distance calculation", "judgment of the same vertical line", and "hole leakage comparison" have clear physical attributes and spatial consistency. This significantly reduces the false alarm rate and improves the credibility and engineering practicality of the hole leakage detection results in the rolling punch production line.

[0135] Example 7: Based on the above examples, this example further provides:

[0136] A hole tracking algorithm is used to continuously track hole positions. The algorithm is implemented through the following steps: caching the historical position coordinates of holes; introducing a confidence scoring mechanism and setting parameters such as maximum matching distance, maximum allowed number of lost frames, and minimum tracking length; using OpenCV to draw a baseline, continuously tracking the hole trajectory, and filtering out low-confidence trajectories.

[0137] Step 1: Cache the historical location coordinates of the holes;

[0138] The "cache hole historical position coordinates" refers to establishing an independent trajectory cache unit for each detected valid hole position during the processing of video streams or multi-frame image sequences. This unit continuously records the two-dimensional pixel coordinates (x, y) of the hole position detected in consecutive frames, the corresponding frame index, detection confidence, the vertical group identifier, and the timestamp. The cache structure adopts an ordered queue (such as deque) or a hash table with a time window, supporting O(1) level insertion and frame-order traversal. The cache depth is not less than the maximum allowed number of lost frames plus the minimum tracking length, ensuring that trajectory reconstruction has a temporal continuity basis. The cached content can be selectively persisted to memory or a temporary file after a single processing task to support cross-frame correlation analysis. This caching mechanism does not rely on global ID allocation but is based on dynamic binding of spatial proximity, avoiding ID drift caused by initial missed detection. In optional implementations, the cache structure can be replaced with a circular buffer to limit the upper limit of memory usage; or a weighted time decay model can be used to assign lower weights to early coordinates to improve adaptability to dynamic offsets.

[0139] Step 2: Introduce a confidence scoring mechanism and set parameters such as maximum matching distance, maximum allowed number of lost frames, and minimum tracking length;

[0140] The "confidence scoring mechanism" refers to outputting a confidence score for each detected hole region in each frame. This score originates from the probability value output by the last Sigmoid activation layer of the YOLO object detection model and is post-processed for calibration: for the same hole region, if the center point of its bounding box falls within the beam region mask and the distance from the beam edge is greater than 5 pixels, the confidence score is increased by 0.05; if the hole region also satisfies the morphological connectivity constraint (area ≥ 30 pixels and aspect ratio between 0.6 and 1.4), the confidence score is further increased by 0.03; the final confidence score range is [0.0, 1.0], which is used for subsequent matching decisions. The "Maximum Matching Distance" is set to 30 pixels, which refers to the Euclidean distance threshold between a hole position in the current frame and the coordinates of the end of any hole position trajectory in the historical cache. If this value is exceeded, the hole is considered unmatchable. This parameter can be dynamically scaled according to the camera resolution and installation height. For example, it is set to 30 pixels when the image resolution is 1920×1080, and proportionally increased to 60 pixels when the resolution is 3840×2160. The "Maximum Allowable Lost Frames" is set to 5 frames, which means that a trajectory will be terminated if it is not detected within 5 consecutive frames to prevent false associations. This parameter can be configured to 3–10 frames to adapt to different motion speeds—3 frames in high-speed stamping scenarios and 10 frames in low-speed debugging mode. The "Minimum Tracking Length" is set to 8 frames, which means that a trajectory is only considered a stable physical hole trajectory and included in subsequent analysis if it contains no less than 8 valid coordinate points. This parameter ensures the exclusion of transient noise interference, and its value can be calculated by back-calculating based on the typical hole exposure time (e.g., a single hole stays in the field of view for ≥160 ms). The above four types of parameters are coupled together: confidence level serves as the basis for ranking matching priorities, maximum matching distance constrains spatial feasibility, maximum allowable number of lost frames ensures time tolerance, and minimum tracking length establishes the threshold for trajectory effectiveness.

[0141] Step 3: Use OpenCV to draw a baseline, continuously track the hole trajectory, and filter out low-confidence trajectories.

[0142] The "drawing a baseline using OpenCV" refers to drawing a vertical baseline (line_x = round(mean_x)) across the image on each frame, based on the mean X-coordinate and Y-direction distribution trend of all valid hole positions in the current frame. This baseline is then used as a reference axis for determining the consistency of the trajectory direction. For each candidate trajectory, the ratio of the Y-direction standard deviation σ_y to the X-direction standard deviation σ_x (σ_y / σ_x) of all its historical coordinates is calculated. If the ratio is ≥3.0, it indicates that the trajectory is highly concentrated in the vertical direction, conforming to the arrangement of holes in the longitudinal beams of the frame, and is thus retained. Otherwise, it is considered an abnormal trajectory with lateral drift and is discarded. "Continuous tracking of hole trajectories" refers to using an improved Hungarian algorithm to achieve optimal bipartite graph matching between the hole position in the current frame and the end of the historical trajectory: the cost matrix elements are the weighted sum of normalized Euclidean distance and the inverse of confidence, with weight coefficients set to 0.7 and 0.3 respectively, ensuring that high-confidence matching is prioritized; after matching, the coordinates and mean confidence of successfully associated trajectories are updated, and the counter of unmatched historical trajectories is incremented by 1. The process is terminated when the maximum allowed number of lost frames is reached. "Filtering out low-confidence trajectories" means that for all matched trajectories, the cumulative mean confidence is calculated. If it is lower than 0.65, the entire trajectory is judged as low-confidence and removed from the set of valid trajectories. This threshold can be adjusted according to the stability of the ambient light, lowered to 0.55 under strong light reflection conditions and raised to 0.70 under weak light jitter conditions. In an optional implementation, the baseline can be extended to a multi-segment polyline to adapt to curved longitudinal beam scenarios; trajectory filtering can also introduce Kalman filter prediction residuals, and trajectory termination is triggered when the prediction error exceeds 15 pixels for 3 consecutive frames.

[0143] Through the above steps, this application achieves temporal dimension modeling of the punching process of the longitudinal beam of the vehicle frame: a caching mechanism constructs the temporal memory basis of the hole position state; confidence score and multi-dimensional parameters collaboratively define the reliability boundary of trajectory generation; baseline constraints and Hungarian matching ensure the physical rationality of spatial association; and low confidence filtering ensures the engineering credibility of the output trajectory. Therefore, even under dynamic interference conditions such as equipment vibration, camera shake, partial occlusion, or brief reflection, it can still stably identify the positional evolution of the same physical hole in consecutive frames, thereby supporting the identification capability of complex abnormal modes such as intermittent hole leakage (e.g., failure to trigger execution in a certain punching cycle), periodic offset (e.g., cumulative error in servo positioning), and trajectory breakage (e.g., instantaneous sensor failure). It solves the technical problem that single-frame static detection cannot reflect the temporal consistency of hole positions and is difficult to capture dynamic hole leakage behavior, significantly improving the robustness and diagnostic depth of the rolling punching quality monitoring system.

[0144] Example 8: Based on the above examples, this example further provides:

[0145] When the visual inspection hole position data is inconsistent with the hole position data parsed by the CNC program, a notification message containing the beam ID, inspection time, and anomaly type is sent in real time; and all inspection data, comparison results, and timestamp information are automatically stored to enable multi-dimensional query and traceability.

[0146] Step 1: When the visual inspection hole position data is inconsistent with the hole position data parsed by the CNC program, a notification message containing the beam ID, inspection time, and anomaly type is sent in real time.

[0147] Among them, the visual inspection hole location data refers to the set of valid holes obtained by reasoning through the target detection model on the image frame of the longitudinal beam surface, and then through region filtering, coordinate transformation and size calculation. It includes the pixel coordinates, corresponding physical coordinates (unit mm), the vertical line number to which it belongs, the vertical distance relative to the upper / lower edge of the beam, the vertical distance between adjacent holes on the same vertical line, and the hole location number sequence for each valid hole location. This data is output by the detection module and the calculation module completes the unit conversion and structured organization to form a detection snapshot with a timestamp.

[0148] The parsing of hole position data in CNC programs refers to the standard hole position design information parsed from the CNC machining program (NC code or instruction parameter file readable by PLC) executed by the stamping equipment. This includes the total number of standard holes, the theoretical XY physical coordinates (in mm) of each hole in the beam coordinate system, the theoretical hole diameter, the theoretical arrangement rules (such as equidistant spacing, segment spacing, and initial offset), and the corresponding process operation number. The parsing process is completed through a preset syntax parser (supporting G code, ISO standard instructions, or manufacturer's proprietary protocols). It supports the recognition and mapping of key semantics such as M instructions, G81 / G82 drilling cycles, coordinate system offsets (G54-G59), and subroutine calls, ensuring logical integrity.

[0149] Inconsistency refers to any of the following situations: (1) the total number of detected holes deviates from the total number of standard holes by ≥1; (2) there are detected holes that have no theoretical matching position within the tolerance range of standard holes (±0.5 mm); (3) there are standard holes that have no corresponding holes in the detection results and are not marked as skipped in the process (i.e., skipped holes not allowed by the design); (4) the absolute value of the deviation between the measured vertical distance and the theoretical distance of adjacent holes on the same vertical line exceeds the preset threshold (±1.2 mm). This judgment is jointly completed by the judgment module based on the hash matching algorithm and the dynamic programming alignment strategy, and supports fault-tolerant matching (such as allowing single-point drift and local sequence misalignment).

[0150] Real-time notification message sending refers to triggering the alarm process within 200 ms after a discrepancy is detected, via a lightweight communication mechanism. The message carrier uses a structured JSON format, with fields including beam_id (a string encoded as "production line number_shift_serial number", such as "L3-A20240515-087"), detection_time (ISO 8601 format timestamp, accurate to milliseconds), anomaly_type (enumerated values: MISSING_HOLE, EXTRA_HOLE, POSITION_DEVIATION, DISTANCE_DEVIATION), and anomaly_details (nested objects containing the abnormal hole number, theoretical / measured coordinates, deviation value, and vertical line index). The transmission channel supports multi-mode redundancy: priority is given to pushing to the workshop MES system via the OPC UA server (node ​​path ns=2; s=Alarm / LeakHole), while HTTP is enabled simultaneously. The message is POSTed to the factory alarm center API endpoint, with a backup option of local UDP broadcast (port 50001) for on-site audible and visual alarms to listen to; the message content is encrypted with AES-128 and accompanied by a digital signature (ECDSA-secp256r1) to ensure integrity and non-repudiation.

[0151] Optional implementations of this technical feature include: expanding anomaly_type to a composite type (such as MISSING_HOLE+POSITION_DEVIATION) to support concurrent annotation of multiple anomalies; improving timestamp accuracy to the microsecond level to adapt to high-speed stamping cycles (≥60 times / minute); replacing the notification channel with the MQTT protocol to access the industrial IoT platform (such as Azure IoT Hub or Huawei Cloud IoTDA) and enabling QoS Level 1 delivery guarantee; or using a local SQLite database write + timed synchronization mechanism in environments without network access.

[0152] Step 2: Automatically store all detection data, comparison results, and timestamp information to enable multi-dimensional query and traceability;

[0153] The detection data includes raw image frames (JPEG compressed, quality factor 85), preprocessed intermediate images (grayscale, binary, and morphologically processed images), bounding box coordinates (normalized format) from YOLO inference output, confidence scores, category labels (beam / liang, hole / kong, invalid), calibration board corner detection images, camera intrinsic parameter matrices (fx, fy, cx, cy), distortion coefficients (k1, k2, p1, p2, k3), size conversion ratio (mm / pixel), and a list of valid holes (including number, pixel coordinates, physical coordinates, vertical distance, and group identifier). The data is organized by frame granularity, with each frame corresponding to a unique UUID as the primary key.

[0154] The comparison results include a structured comparison report with fields covering match_status (FULL_MATCH / PARTIAL_MATCH / NO_MATCH), hole_count_diff (integer, number of detected holes minus number of standard holes), unmatched_detected_holes (array, containing hole number and deviation vector), unmatched_standard_holes (array, containing theoretical number and missing identifier), max_vertical_deviation (floating point, unit mm), and deviation_distribution (histogram statistics, bin width 0.2 mm). It also saves a visual comparison image—overlaying standard hole position crosshairs (green), detected hole position circles (red), matching lines (blue dashed lines), and leaky hole highlight boxes (yellow border + flashing animation) onto the original image, supporting PNG export.

[0155] The timestamp information includes not only the detection trigger time (trigger_timestamp), image acquisition time (capture_timestamp, provided by the industrial camera hardware timestamp), model inference completion time (inference_end_timestamp), and comparison completion time (comparison_end_timestamp), but also the CNC program loading time (nc_load_timestamp) and calibration completion time (calibration_timestamp). All timestamps are synchronized to the same NTP server (accuracy ±10 ms) and the time zone and UTC offset are recorded.

[0156] Automatic storage employs a tiered persistence strategy: hot data (last hour) is stored in an in-memory database Redis (key: beam_id:timestamp, value: serialized JSON), supporting millisecond-level retrieval; warm data (last 7 days) is written to a relational database MySQL (table structure includes the main table detection_log and related sub-tables hole_detail and comparison_result), creating a composite index (beam_id + detection_time + anomaly_type); cold data (over 7 days) is archived to object storage (such as MinIO), partitioned by date and compressed into ZIP packages (including metadata XML description files), preserving the original image resolution and EXIF ​​information; the storage process is executed by an independent daemon process, with breakpoint resume, CRC32 verification, and disk space warning (triggered cleanup when <15%) capabilities.

[0157] Multi-dimensional query traceability supports five types of search interfaces: (1) accurate query of the entire life cycle data chain by beam ID; (2) combined filtering by time range + anomaly type (e.g., "all MISSING_HOLE from 2024-05-15 to 2024-05-20"); (3) reverse query by process parameters (e.g., "the leakage rate when the G54 coordinate system offset is >2.0mm"); (4) fuzzy search by image features (e.g., "the distribution of leakage holes in frames with contrast >120 after CLAHE enhancement"); (5) cross-batch statistical analysis (e.g., "the trend chart of leakage holes in the past 30 days under the same mold number"). The results can be exported as CSV / Excel or rendered as ECharts charts. This function is implemented through the Web front-end (Vue.js) and the back-end RESTful API (Flask framework), with hierarchical access control (operators can only see the data of this shift, process engineers can see the data of the entire production line, and the quality director can export audit reports).

[0158] Optional implementations of this technical feature include: replacing the storage medium with an NVMe SSD array to improve write throughput (adapting to 100 fps video streaming); upgrading the cold data archiving protocol to an S3-compatible interface for public cloud access; adding an operator_id field to the database to bind manual review records; or introducing a blockchain evidence storage module (Hyperledger Fabric) to hash and upload key operation logs (such as comparison result generation and notification sending) to meet ISO / IEC 17025 certification requirements.

[0159] The synergistic effects of the various technical features are as follows: the structured comparison of visual inspection data and CNC program data is a prerequisite for triggering notifications, while the refined fields of the comparison results (such as anomaly_details) directly determine the completeness of the notification message content; the low latency requirement of real-time notifications forces a high-precision synchronization mechanism between inspection data and timestamps (NTP calibration + hardware timestamps), and the integrity of the timestamps ensures the accuracy of the timeline for multi-dimensional queries after storage; in the tiered storage strategy, Redis hot data supports real-time dashboard updates, MySQL warm data supports complex SQL queries, and object storage cold data ensures long-term compliant retention, and the three together constitute a scalable data lifecycle management system.

[0160] Through the above-described steps, this application achieves the following: during the operation of the rolling punch machine, once a leakage abnormality is detected, a structured alarm is immediately pushed to multiple terminals, enabling on-site personnel to obtain the specific beam, abnormal location, and deviation within ≤3 seconds; simultaneously, all original images, intermediate processes, judgment criteria, and operation traces are completely, reliably, and verifiably persisted, supporting not only minute-level location of historical anomaly root causes (such as a calibration failure causing 5 consecutive frames of inaccurate proportions), but also the discovery of hidden process defects through big data analysis (such as a 12% increase in leakage rate within a specific temperature range), thereby upgrading traditional post-event sampling inspection to full-process closed-loop quality control, significantly improving the observability, traceability, and manageability of the intelligent manufacturing system.

[0161] Example 9: Based on the same inventive concept, this application also provides a system for detecting leaks in a rolling punch machine based on a vehicle frame longitudinal beam, used to implement the above-described method for detecting leaks in a rolling punch machine based on a vehicle frame longitudinal beam. The solution provided by this system is similar to the solution described in the above-described embodiments. Therefore, the specific limitations in one or more system embodiments provided below can be found in the limitations of the method described above, and will not be repeated here.

[0162] In one embodiment, the present invention provides a hole detection system for a rolling punching machine based on a vehicle frame longitudinal beam. The system is as follows: Figure 2 As shown, it includes: a synchronous acquisition module 210, a processing module 220, a detection module 230, a calculation module 240, and a judgment module 250, wherein:

[0163] The synchronous acquisition module 210 is used to construct a real-time data channel and synchronously acquire multi-dimensional industrial data; the multi-dimensional industrial data includes: longitudinal beam surface image, punching equipment motion trajectory and instruction parameters in CNC program;

[0164] The processing module 220 is used to decompose the acquired longitudinal beam surface image into frame data and perform multi-strategy preprocessing on the frame data; calibrate the preprocessed frame data, filter the optimal calibration result through multi-strategy combination, and calculate the camera parameters and size conversion ratio.

[0165] The detection module 230 is used to load a pre-established target detection model, detect the pre-processed frame data, identify invalid areas, beam areas and hole areas, filter out the valid hole positions in the beam, and record the position information of the valid hole positions.

[0166] The calculation module 240 is used to calculate the vertical distance of the effective hole position relative to the beam body and the vertical distance between adjacent effective holes on the same vertical line based on the calibrated size conversion ratio; and to visualize and number the vertical distance of the effective hole position relative to the beam body and the vertical distance between adjacent effective holes on the same vertical line.

[0167] The judgment module 250 is used to compare the number and position data of the detected holes with the standard hole data generated by the CNC program to determine whether there is a hole leakage abnormality; and to visualize the visually detected hole data and comparison results through the monitoring interface, and to issue a notification when a hole leakage abnormality is determined to exist.

[0168] In one embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the hole detection method for a rolling punching machine based on a vehicle frame longitudinal beam, as described in any of steps S101 to S105. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0169] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0170] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0174] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A method for detecting leaks in a roll forming punch based on a vehicle frame longitudinal beam, characterized in that, The method includes: A real-time data channel is constructed to synchronously collect multi-dimensional industrial data, including: longitudinal beam surface images, punching equipment motion trajectories, and instruction parameters in the CNC program. The acquired longitudinal beam surface image is decomposed into frame data, and multi-strategy preprocessing is performed on the frame data; The preprocessed frame data is calibrated, and the optimal calibration result is selected by combining multiple strategies. The camera parameters and size conversion ratio are calculated. Load the pre-established target detection model, detect the pre-processed frame data, identify invalid areas, beam areas and hole areas, filter to obtain the valid hole positions in the beam, and record the position information of the valid hole positions; Based on the calibrated size conversion ratio, the vertical distance of the effective hole position relative to the beam body and the vertical distance between adjacent effective holes on the same vertical line are calculated; the vertical distance of the effective hole position relative to the beam body and the vertical distance between adjacent effective holes on the same vertical line are visualized, plotted, and numbered. The number and location data of the detected holes are compared with the standard hole data generated by the CNC program to determine whether there are any abnormal holes. The visual detection hole data and comparison results are displayed visually through the monitoring interface, and a notification is issued when an abnormal hole is detected.

2. The method as described in claim 1, characterized in that, The construction of a real-time data channel for synchronous acquisition of multi-dimensional industrial data includes: constructing a real-time data channel consisting of an industrial camera, PLC, host computer, and communication components; synchronizing data with a local area network via the OPC UA industrial communication protocol; and synchronously acquiring images of the longitudinal beam surface, the motion trajectory of the punching equipment, and instruction parameters from the CNC machining program.

3. The method as described in claim 1, characterized in that, The multi-strategy preprocessing includes performing any one or more preprocessing operations on the frame data, such as grayscale conversion, contrast enhancement, Gaussian filtering, adaptive binarization, and morphological processing.

4. The method as described in claim 1, characterized in that, The calibration of the preprocessed frame data includes: detecting the corner points of the calibration board in the preprocessed frame data, selecting the optimal calibration result through a combination of multiple strategies; calculating the calibration quality through the coefficient of variation of the corner point spacing, selecting the optimal calibration result and calculating the camera intrinsic parameters, distortion parameters and size conversion ratio; if no effective calibration board is detected or the calibration quality is not up to standard, the size conversion ratio is estimated based on a preset empirical coefficient.

5. The method as described in claim 1, characterized in that, The pre-establishment of the target detection model includes: A standardized dataset is established based on historically collected multi-dimensional data and its corresponding annotation information. The establishment of the standardized dataset includes: using the LabelStudio annotation tool to annotate the beam outline, ventral holes and invalid areas in the samples, and mixing abnormal samples into the standardized dataset according to a preset ratio. Using the YOLO object detection network as the initial model, the object detection model was trained in the PyTorch framework using transfer learning and early stopping mechanisms to obtain the aforementioned object detection model.

6. The method as described in claim 1, characterized in that, The process of filtering to obtain valid hole positions within the beam and recording their location information includes: performing inference detection on preprocessed frame data to parse the position coordinates of invalid regions, beam regions, and hole regions; setting a tolerance range, matching the coordinates of the hole regions with the coordinates of the beam regions, filtering out invalid hole positions outside the beam regions, retaining valid hole positions within the beam regions, and recording their location information.

7. The method as described in claim 1 or 6, characterized in that, It also includes using a hole tracking algorithm to continuously track the hole position, the algorithm being implemented through the following steps: Cache the historical location coordinates of the holes; A confidence scoring mechanism is introduced, and parameters such as maximum matching distance, maximum allowed number of lost frames, and minimum tracking length are set. Using OpenCV, a baseline is drawn to continuously track the hole trajectory and filter out low-confidence trajectories.

8. The method as described in claim 1, characterized in that, The notification issued when a leak is detected includes: When the visual inspection hole position data is inconsistent with the hole position data parsed by the CNC program, a notification message containing the beam ID, inspection time, and anomaly type is sent in real time; and all inspection data, comparison results, and timestamp information are automatically stored to enable multi-dimensional query and traceability.

9. A hole detection system for a rolling punch machine based on a vehicle frame longitudinal beam, characterized in that, The system includes: The synchronous acquisition module is used to build a real-time data channel and synchronously acquire multi-dimensional industrial data, including: longitudinal beam surface images, punching equipment motion trajectories, and instruction parameters in the CNC program. The processing module is used to decompose the acquired longitudinal beam surface image into frame data and perform multi-strategy preprocessing on the frame data; calibrate the preprocessed frame data, select the optimal calibration result through multi-strategy combination, and calculate the camera parameters and size conversion ratio; The detection module is used to load a pre-established target detection model, detect the pre-processed frame data, identify invalid areas, beam areas and hole areas, filter out the valid hole positions in the beam, and record the position information of the valid hole positions. The calculation module is used to calculate the vertical distance of the effective hole position relative to the beam and the vertical distance between adjacent effective holes on the same vertical line based on the calibrated size conversion ratio; and to visualize and number the vertical distance of the effective hole position relative to the beam and the vertical distance between adjacent effective holes on the same vertical line. The judgment module is used to compare the number and location data of the detected holes with the standard hole data generated by the CNC program to determine whether there are any abnormal holes. The module also visualizes the visually detected hole data and comparison results through the monitoring interface and issues a notification when an abnormal hole is detected.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

Images