A stainless steel bridge punching positioning intelligent control method based on machine vision

Abstract

The present application relates to the technical field of industrial intelligence, and especially relates to a stainless steel bridge frame punching positioning intelligent control method based on machine vision, which comprises the following steps: acquiring visible light and near-infrared light images of the surface of the stainless steel bridge frame and registering; performing multispectral joint denoising on the registered images to generate a high-contrast surface feature map; inputting the feature map into a convolutional neural network to output a geometric contour feature point set and a surface defect mask; matching the feature point set with a pre-stored drawing to calculate the coordinate deviation of the actual pose; planning a collision-free punching path sequence in combination with the defect mask; dynamically setting controller parameters based on the pose deviation to drive the actuator to punch accurately; and analyzing the deformation after punching and dynamically re-planning the subsequent path. The present application suppresses reflection by multispectral fusion, realizes accurate contour and defect recognition by using a multi-task network, combines intelligent path planning with adaptive control to compensate for online deformation, and improves the accuracy, efficiency and intelligent level of punching positioning.

Description

Technical Field

[0001] This invention relates to the technical field of industrial intelligence, and in particular to an intelligent control method for punching and positioning stainless steel cable trays based on machine vision. Background Technology

[0002] With the development of industrial automation and intelligent manufacturing, machine vision technology has been introduced into the punching and positioning process of stainless steel cable trays to improve processing accuracy and efficiency. However, the strong specular reflection of stainless steel surfaces easily produces highlights and glare under complex lighting conditions, resulting in low contrast and blurred edges in the acquired images. Simultaneously, surface contaminants such as oil stains and scratches are mixed with contour features, making traditional image processing algorithms less robust and prone to errors in feature extraction, severely impacting the accuracy of subsequent positioning.

[0003] At the localization and path planning level, existing solutions often separate feature recognition, pose calculation, and path planning. Most systems can only perform coarse contour matching and lack high-precision calculation models for comprehensive pose deviations such as translation, rotation, and deformation. In addition, path planning usually only pursues the shortest travel distance and fails to link with visually detected surface defect areas for intelligent obstacle avoidance, resulting in the quality risk of punching holes at defects, and also lacking the ability to adjust in real time to cope with dynamic deformation caused by stress release during processing.

[0004] Therefore, existing technologies have significant room for improvement. There is an urgent need for an intelligent punching and positioning method that can overcome imaging interference on stainless steel surfaces, achieve high-precision feature recognition and pose calculation, possess intelligent obstacle avoidance path planning, and perform adaptive motion control and online quality feedback. This method would enhance the overall quality, efficiency, and intelligence level of stainless steel cable tray processing. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a machine vision-based intelligent control method for punching and positioning stainless steel cable trays.

[0006] Firstly, the above-mentioned inventive objective of this application is achieved through the following technical solution:

[0007] A machine vision-based intelligent control method for punching and positioning stainless steel cable trays, comprising the following steps:

[0008] Acquire visible light and near-infrared light images of the surface of the stainless steel cable tray, and register the visible light and near-infrared light images to a unified coordinate system;

[0009] Multispectral joint denoising processing was performed on the registered visible light image and near-infrared light image to obtain a high-contrast surface feature map of the cable tray;

[0010] The surface feature map of the cable tray is input into a pre-trained multi-task convolutional neural network model, and the output is the geometric contour feature point set and surface defect mask of the cable tray.

[0011] Based on the geometric contour feature point set and the pre-stored cable tray design drawings, feature matching is performed to calculate the coordinate deviation of the current cable tray's actual pose from its theoretical pose.

[0012] By adopting the above technical solution, and simultaneously acquiring and registering visible and near-infrared images, complementary multispectral information sources are provided for subsequent processing, overcoming the limitation of single-source imaging of stainless steel surfaces being susceptible to high-light interference. Joint denoising of the registered dual-spectral images effectively suppresses specular reflection and random noise, significantly improving the contrast and signal-to-noise ratio of the cable tray surface feature map. By inputting the feature map into a multi-task convolutional neural network, parallel, accurate, and automated recognition of the cable tray's geometric contours and surface defects is achieved, replacing traditional manual visual inspection or simple image processing algorithms, greatly improving recognition accuracy and efficiency. Finally, through feature matching and coordinate calculation, the visual recognition results are transformed into precise pose deviation data, providing a high-precision positioning benchmark for subsequent intelligent punching, thus achieving automation, intelligence, and high precision in the pre-punching positioning process as a whole.

[0013] In a preferred embodiment, this application can be further configured such that: the multispectral joint denoising processing of the registered visible light image and near-infrared light image to obtain a high-contrast cable tray surface feature map specifically includes:

[0014] The registered visible light image is converted from the RGB color space to the HSV color space, and the V luminance channel component is extracted.

[0015] Adaptive histogram equalization is performed on the registered near-infrared image to enhance the surface texture features in the near-infrared image, resulting in an enhanced near-infrared image.

[0016] The V brightness channel component of the enhanced near-infrared image and the visible light image are differentially processed to extract the difference feature map that is not affected by surface reflection.

[0017] The V luminance channel component of the visible light image is weighted and fused with the difference feature map to reconstruct the high-contrast cable tray surface feature map.

[0018] By employing the aforementioned technical solution, the visible light image is converted to HSV space and the V brightness channel is extracted, effectively separating brightness information that is drastically affected by illumination. Next, adaptive histogram equalization is applied to the near-infrared image, enhancing its responsiveness to surface textures and deep defects. Then, through differential operations, leveraging the low surface reflectivity of near-infrared light, the high-brightness components in the visible light V channel are effectively canceled out, extracting a difference map reflecting the intrinsic characteristics of the material. Finally, adaptive weighted fusion balances the sharp edges of the visible light image with the anti-reflective details of the near-infrared difference map, reconstructing a high-contrast feature map with strong illumination robustness. This series of processing steps fundamentally solves the key technical challenges of poor imaging quality and difficult feature extraction on highly reflective materials such as stainless steel, laying a solid image foundation for subsequent high-precision recognition.

[0019] In a preferred embodiment, this application can be further configured as follows: inputting the surface feature map of the cable tray into a pre-trained multi-task convolutional neural network model, and outputting a set of geometric contour feature points and a surface defect mask of the cable tray, specifically including:

[0020] The high-contrast surface feature map of the cable tray is input into the feature extraction backbone of the multi-task convolutional neural network model, and multi-layer convolution and pooling operations are performed to extract the deep semantic feature map.

[0021] The deep semantic feature map is input into the parallel first branch decoder and second branch decoder respectively, wherein the first branch decoder predicts the geometric contour heat map and the second branch decoder predicts the surface defect heat map.

[0022] The geometric contour heatmap and surface defect heatmap are binarized by applying a preset probability threshold to generate an initial contour mask and an initial defect mask.

[0023] Morphological closing operations are performed on the initial contour mask and the initial defect mask to eliminate internal holes, resulting in the final geometric contour feature point set and surface defect mask.

[0024] By employing the above technical solution, deep semantic features are extracted using a shared backbone network, achieving efficient learning of common features for both contour and defect tasks, reducing the number of model parameters and computational overhead. Through a parallel dual-branch decoder structure, the network can focus on its specific tasks—contour heatmap prediction and defect heatmap prediction—avoiding interference between tasks and improving segmentation accuracy. Threshold binarization of the heatmaps transforms the probabilistic output into a deterministic segmentation mask, facilitating subsequent processing. Finally, morphological closing operations eliminate small holes and burrs caused by network prediction uncertainty or image noise, resulting in smooth-bounded, highly connected final contours and defect regions. This solution achieves accurate end-to-end feature extraction, overcoming the shortcomings of traditional algorithms, such as inaccurate segmentation in complex backgrounds and poor generalization ability for defect types.

[0025] In a preferred embodiment, this application can be further configured as follows: the step of performing feature matching based on the geometric contour feature point set and pre-stored cable tray design drawings to calculate the coordinate deviation of the actual pose of the current cable tray relative to the theoretical pose specifically includes:

[0026] Extract the four vertex points of the cable tray and the intersection of the perpendicular bisectors of at least two long sides from the set of geometric contour feature points, and use them as the current actual feature points;

[0027] Read the coordinates of the theoretical feature points corresponding to the current cable tray model from the pre-stored cable tray design drawings, and calculate the perspective transformation matrix by fitting the data using the least squares method based on the correspondence between the actual feature points and the theoretical feature points.

[0028] Substitute the theoretical coordinate set of the current cable tray to be punched into the perspective transformation matrix to calculate the actual processing coordinate set and the pose deviation vector of each coordinate point.

[0029] By adopting the above technical solution, the intersection of the apex and perpendicular bisector of the extracted contour points is selected as matching feature points. These points have the advantages of high stability and easy positioning in CAD drawings, ensuring the reliability of the matching. By reading the coordinates of the theoretical feature points of the corresponding model, the correspondence between the image space and the design drawing space is established. By fitting the perspective transformation matrix using the least squares method, the complete spatial transformation relationship including translation, rotation, scaling and perspective distortion compensation can be solved in the optimal way. The mathematical model is rigorous and the solution accuracy is high. Finally, the theoretical punching coordinates are substituted into the transformation matrix to directly calculate the actual processing coordinates and the pose deviation vector, realizing a seamless and high-precision conversion from visual recognition to machine tool execution coordinates, providing direct data drive for precise punching.

[0030] In a preferred embodiment, this application can be further configured as follows: Substituting the theoretical coordinate set of the current cable tray to be punched into the perspective transformation matrix to calculate the actual processing coordinate set and the pose deviation vector of each coordinate point, the machine vision-based intelligent control method for stainless steel cable tray punching positioning further includes a punching path planning step, specifically including:

[0031] Read the surface defect mask output by the second branch decoder and map the pixel values ​​within the defect mask area to the punching taboo area;

[0032] Based on the actual processing coordinate set, a rasterized cost map is constructed with the optimization objective of avoiding the forbidden zone and minimizing the total movement distance.

[0033] The rasterized cost map is input into the path planning algorithm, and the globally optimal collision-free punching traversal sequence is obtained through iterative search.

[0034] Based on the collision-free punching traversal sequence, the actual processing coordinates and corresponding pose deviation vectors of each punching point in the sequence are extracted sequentially and sent to the punching execution device to perform punching of stainless steel cable trays.

[0035] By adopting the above technical solution, the surface defect mask identified by the neural network is mapped as a punching taboo area, enabling path planning to actively avoid material defect areas. This avoids product strength reduction or unqualified hole shape caused by punching at defective locations from the source, thus improving product quality. By constructing a gridded cost map by combining actual processing coordinates with taboo areas, the complex physical obstacle avoidance problem is transformed into an intuitive graph search problem. By inputting the cost map into the path planning algorithm for global optimization search, a collision-free traversal sequence that visits all punching points and has the shortest total travel distance (or optimal time) can be automatically generated. This significantly reduces the idle travel time of the punching machine actuator and improves processing efficiency. The final optimized sequence directly guides the punching execution, realizing intelligent and efficient processing path.

[0036] In a preferred embodiment, this application can be further configured such that: the delivery to the punching device to perform punching of the stainless steel cable tray specifically includes:

[0037] The pose deviation vector of each punching point in the collision-free punching traversal sequence is read sequentially and decomposed into displacement error components and angle error components.

[0038] The displacement error component and the angle error component are input into the punching execution device, and the proportional coefficient, integral coefficient and derivative coefficient of the PID controller in the punching execution device are dynamically tuned based on fuzzy inference rules.

[0039] The displacement error component and the angle error component are iteratively calculated using the tuned PID controller to generate the motor control compensation amount corresponding to the current punching point.

[0040] The motor control compensation is superimposed on the theoretical number of driving pulses at the current punching point, driving the punching actuator to move to the target punching position.

[0041] By adopting the above technical solution, the pose deviation vector is decomposed into displacement and angle components, which facilitates independent and precise control of different motion axes of the machine tool. By inputting the error components into the fuzzy PID controller and dynamically tuning the PID parameters using fuzzy inference rules, the controller can adaptively adjust the control force according to the magnitude and trend of the error, overcoming the shortcomings of traditional fixed-parameter PID controllers in dealing with nonlinear and time-varying systems, such as slow response, large overshoot, and poor steady-state accuracy. By iteratively calculating and generating motor control compensation, real-time and closed-loop compensation for positioning errors is achieved. Finally, the compensation is superimposed on the theoretical drive command to drive the actuator to move precisely, effectively eliminating system errors introduced by factors such as bridge placement deviation and machine tool mechanical errors, and ensuring the ultimate positioning accuracy of the punching point.

[0042] In a preferred embodiment, this application can be further configured as follows: after extracting the actual processing coordinates and corresponding pose deviation vectors of each punching point in the sequence according to the collision-free punching traversal sequence, and sending them to the punching execution device, the machine vision-based intelligent control method for stainless steel cable tray punching positioning further includes:

[0043] After the punching device moves to the first punching point and completes positioning according to the collision-free punching traversal sequence, the vision system is triggered to perform a secondary scan on the current local area of ​​the cable tray to obtain a high-precision local deformation image.

[0044] Optical flow calculations were performed on the high-precision local deformation image to analyze the micro-displacement vector field on the cable tray surface caused by the release of stamping stress.

[0045] Based on the micro-displacement vector field, the deformation trend of the area where the subsequent punching point is located is predicted, and the passage cost weight of the corresponding area in the rasterized cost map is dynamically updated.

[0046] Based on the preset dynamic path planning algorithm, starting from the current punching point and combining the updated passage cost weight, the traversal sequence of the remaining punching points is locally replanned to generate an optimized punching sequence that is resistant to deformation interference.

[0047] By adopting the above technical solution, secondary image acquisition and quality inspection are performed immediately after each punching, realizing online and real-time quality monitoring of the processing process, replacing the traditional post-processing sampling inspection, and ensuring the quality control of each hole position. By performing sub-pixel edge detection on the local images of the hole positions, the edge positioning accuracy is improved to the sub-pixel level, far exceeding the traditional pixel-level detection, making the measurement results of roundness error and hole diameter tolerance extremely accurate. By comparing the detection results with the qualified threshold in real time, the quality of a single hole can be determined immediately and marked or alarmed, realizing the immediate detection and handling of quality problems. This step forms a rapid "processing-inspection" feedback loop, providing reliable data support for the refined management and quality traceability of the production process.

[0048] Secondly, the above-mentioned inventive objective of this application is achieved through the following technical solutions:

[0049] A machine vision-based intelligent control system for punching and positioning stainless steel cable trays, comprising:

[0050] The cable tray image acquisition and processing module is used to acquire visible light images and near-infrared light images of the surface of stainless steel cable trays, and to register the visible light images and near-infrared light images to a unified coordinate system;

[0051] The surface feature map generation module is used to perform multispectral joint denoising processing on the registered visible light image and near-infrared light image to obtain a high-contrast cable tray surface feature map.

[0052] The contour feature extraction module is used to input the surface feature map of the cable tray into a pre-trained multi-task convolutional neural network model and output the geometric contour feature point set and surface defect mask of the cable tray.

[0053] The cable tray pose determination module is used to perform feature matching based on the set of geometric contour feature points and the pre-stored cable tray design drawings, and calculate the coordinate deviation of the actual pose of the current cable tray relative to the theoretical pose.

[0054] By adopting the above technical solution, and simultaneously acquiring and registering visible and near-infrared images, complementary multispectral information sources are provided for subsequent processing, overcoming the limitation of single-source imaging of stainless steel surfaces being susceptible to high-light interference. Joint denoising of the registered dual-spectral images effectively suppresses specular reflection and random noise, significantly improving the contrast and signal-to-noise ratio of the cable tray surface feature map. By inputting the feature map into a multi-task convolutional neural network, parallel, accurate, and automated recognition of the cable tray's geometric contours and surface defects is achieved, replacing traditional manual visual inspection or simple image processing algorithms, greatly improving recognition accuracy and efficiency. Finally, through feature matching and coordinate calculation, the visual recognition results are transformed into precise pose deviation data, providing a high-precision positioning benchmark for subsequent intelligent punching, thus achieving automation, intelligence, and high precision in the pre-punching positioning process as a whole.

[0055] Thirdly, the above-mentioned objectives of this application are achieved through the following technical solutions:

[0056] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described intelligent control method for punching and positioning stainless steel cable trays based on machine vision.

[0057] Fourthly, the above-mentioned objectives of this application are achieved through the following technical solutions:

[0058] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described intelligent control method for punching and positioning stainless steel cable trays based on machine vision.

[0059] In summary, this application includes at least one of the following beneficial technical effects:

[0060] 1. By simultaneously acquiring and registering visible and near-infrared images, complementary multispectral information sources are provided for subsequent processing, overcoming the limitation of single-source imaging of stainless steel surfaces being susceptible to high-light interference. Joint denoising of the registered dual-spectral images effectively suppresses specular reflection and random noise, significantly improving the contrast and signal-to-noise ratio of the cable tray surface feature map. By inputting the feature map into a multi-task convolutional neural network, parallel, accurate, and automated recognition of the cable tray's geometric contours and surface defects is achieved, replacing traditional manual visual inspection or simple image processing algorithms, greatly improving recognition accuracy and efficiency. Finally, through feature matching and coordinate calculation, the visual recognition results are transformed into precise pose deviation data, providing a high-precision positioning reference for subsequent intelligent punching, thus achieving automation, intelligence, and high precision in the pre-punching positioning process.

[0061] 2. The visible light image is converted to HSV space and the V brightness channel is extracted, effectively separating the brightness information that is drastically affected by illumination. Next, adaptive histogram equalization is applied to the near-infrared image to enhance its responsiveness to surface textures and deep defects. Then, through differential operations, taking advantage of the low surface reflectivity of near-infrared light, the high-gloss components in the visible light V channel are effectively canceled out, extracting a difference map reflecting the intrinsic characteristics of the material. Finally, adaptive weighted fusion is used to balance the sharp edges of the visible light with the anti-reflective details of the near-infrared difference map, reconstructing a high-contrast feature map with strong illumination robustness. This series of processing steps fundamentally solves the key technical challenges of poor imaging quality and difficult feature extraction on highly reflective materials such as stainless steel, laying a solid image foundation for subsequent high-precision recognition.

[0062] 3. By mapping the surface defect mask identified by the neural network to punching forbidden areas, the path planning can proactively avoid areas with material defects, thus preventing product strength reduction or unqualified hole shape caused by punching at defective locations, thereby improving product quality. By constructing a gridded cost map by combining actual processing coordinates with forbidden areas, the complex physical obstacle avoidance problem is transformed into an intuitive graph search problem. By inputting the cost map into the path planning algorithm for global optimization search, a collision-free traversal sequence that visits all punching points and has the shortest total travel distance (or optimal time) can be automatically generated, significantly reducing the idle travel time of the punching machine actuator and improving processing efficiency. The final output optimized sequence directly guides the punching execution, realizing intelligent and efficient processing path.

[0063] 4. Immediately after each punching, secondary image acquisition and quality inspection are performed, enabling online, real-time quality monitoring of the processing process. This replaces traditional post-processing sampling inspection, ensuring controllable quality for each hole. By performing sub-pixel edge detection on local images of the hole positions, the edge positioning accuracy is improved to the sub-pixel level, far exceeding traditional pixel-level detection, resulting in extremely accurate measurements of roundness error and hole diameter tolerance. By comparing the inspection results with the pass threshold in real time, the quality of a single hole can be immediately determined and marked or alarmed, enabling immediate detection and handling of quality problems. This step forms a rapid "processing-inspection" feedback loop, providing reliable data support for refined management and quality traceability of the production process. Attached Figure Description

[0064] Figure 1 This is a flowchart of a machine vision-based intelligent control method for punching and positioning stainless steel cable trays in one embodiment of this application.

[0065] Figure 2 This is a flowchart illustrating the implementation of step S20 in a machine vision-based intelligent control method for punching and positioning stainless steel cable trays, as described in one embodiment of this application.

[0066] Figure 3 This is another implementation flowchart of step S30 in a machine vision-based intelligent control method for punching and positioning stainless steel cable trays in one embodiment of this application.

[0067] Figure 4 This is a flowchart illustrating the implementation of step S40 in a machine vision-based intelligent control method for punching and positioning stainless steel cable trays, as described in one embodiment of this application.

[0068] Figure 5 This is a flowchart illustrating the implementation of the punching path planning step in a machine vision-based intelligent control method for punching and positioning stainless steel cable trays, as described in one embodiment of this application.

[0069] Figure 6 This is a flowchart illustrating the implementation of step S80 in a machine vision-based intelligent control method for punching and positioning stainless steel cable trays, as described in one embodiment of this application.

[0070] Figure 7 This is another implementation flowchart of a machine vision-based intelligent control method for punching and positioning stainless steel cable trays, as described in one embodiment of this application.

[0071] Figure 8 This is a schematic diagram of a machine vision-based intelligent control system for punching and positioning stainless steel cable trays in one embodiment of this application.

[0072] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0073] The present application will be further described in detail below with reference to the accompanying drawings.

[0074] In one embodiment, such as Figure 1 As shown, this application discloses an intelligent control method for punching and positioning stainless steel cable trays based on machine vision, which specifically includes the following steps:

[0075] S10: Acquire visible light and near-infrared light images of the stainless steel cable tray surface, and register the visible light and near-infrared light images to a unified coordinate system.

[0076] In this embodiment, step S10 aims to address the difficulty of feature extraction caused by high reflectivity and oil contamination on stainless steel surfaces. By simultaneously acquiring images in two different wavelength bands—visible and near-infrared—near-infrared light is utilized to provide complementary information sources for subsequent fusion processing, leveraging its strong penetration of surface contaminants and stable reflectivity to metal substrates. Image registration is a crucial preprocessing step to ensure spatial alignment of the two images, laying the foundation for pixel-level fusion and difference operations.

[0077] Specifically, a dual-spectrum industrial camera module is fixedly installed above the punch press worktable. This module integrates a visible light CMOS sensor and a near-infrared CCD sensor, and has undergone rigorous coaxial optical calibration to ensure that the optical axes of the two sensors are parallel and their fields of view overlap. When the stainless steel cable tray is transported to the imaging station, the system synchronously triggers both sensors to acquire images. The acquired visible light image (RGB format) and near-infrared image (grayscale format) are first subjected to distortion correction. Subsequently, a feature point matching algorithm based on SIFT (Scale Invariant Feature Transform) is used to automatically extract stable feature point pairs, such as cable tray corners and edge intersections, from the two images. By calculating the affine transformation matrix (containing translation, rotation, and scaling parameters) between these feature point pairs, the spatial coordinates of the near-infrared image are mapped to the coordinate system of the visible light image, achieving sub-pixel-level accurate registration. After registration, each pixel in the two images corresponds to the same physical location on the cable tray surface.

[0078] S20: Perform multispectral joint denoising processing on the registered visible light image and near-infrared light image to obtain a high-contrast cable tray surface feature map.

[0079] In this embodiment, the core objective of step S20 is to suppress specular reflection (highlights) and random noise on the stainless steel surface, highlighting the geometric contours and key features of the cable tray itself, such as surface defects (scratches and pits). Traditional single-light source image processing struggles to handle complex lighting conditions. This step effectively separates inherent features related to material and texture through multispectral difference and weighted fusion, thus reducing the impact of lighting changes.

[0080] Specifically, firstly, the registered visible light RGB image is converted to the HSV color space. The HSV space separates color information (H, S) from luminance information (V), extracting the V (luminance) channel component. This component mainly contains the brightness and darkness information of the image, but also includes strong reflected light components. Next, adaptive histogram equalization is performed on the registered near-infrared image to enhance its contrast, making subtle surface textures and defects more apparent, resulting in an enhanced near-infrared image. Then, pixel-by-pixel difference operations are performed between the enhanced near-infrared image and the visible light V channel component. Since near-infrared light is less affected by oil and oxide layers and is sensitive to surface micro-morphology, while the visible light V channel is greatly affected by highlights, subtracting the two can significantly weaken the highlight areas, extracting a "difference feature map." This map can better preserve the intrinsic features of the bridge structure that are not affected by reflected light. Finally, an adaptive weighted fusion algorithm is used to fuse the visible light V channel component (weight α) with the difference feature map (weight 1-α). The weight α is dynamically calculated based on the local contrast of the pixels. Higher weights are given to the difference feature map in the highlight area, and the contributions of the two are balanced in the texture-rich area. Finally, a high-contrast, clear-detail, and uniformly lit bridge surface feature map is reconstructed.

[0081] S30: Input the surface feature map of the cable tray into a pre-trained multi-task convolutional neural network model, and output the geometric contour feature point set and surface defect mask of the cable tray.

[0082] In this embodiment, step S30 utilizes a deep learning model to achieve parallel and accurate identification of features and defects. Traditional image processing algorithms (such as edge detection and threshold segmentation) are sensitive to noise and have difficulty distinguishing between real defects and textures. However, this step employs an end-to-end convolutional neural network, which can automatically learn and extract high-level semantic information from the fused feature map, completing both contour localization and defect segmentation tasks in one go, significantly improving recognition accuracy and robustness.

[0083] Specifically, the multi-task convolutional neural network model adopts an encoder-decoder architecture, with two parallel decoder branches connected after the encoder. The encoder uses ResNet-34 as the feature extraction backbone, performing multi-layer convolution and pooling operations on the input cable tray surface feature map to progressively downsample and extract a deep semantic feature map containing global contextual information. This deep semantic feature map is then simultaneously input into the two decoder branches. The first decoder branch (contour decoder) upsamples through a series of transposed convolutional layers, ultimately outputting a "geometric contour heatmap" of the same size as the input image, where the value of each pixel represents the probability that the point belongs to the cable tray contour. The second decoder branch (defect decoder) has a similar structure, outputting a "surface defect heatmap" representing the probability that each pixel belongs to a defect region (such as scratches or dents). During training, labeled binary masks of the cable tray contour and defect regions are used as supervision signals, and a weighted sum of the cross-entropy loss function and the Dice loss function is used as the total loss function for model optimization. During inference, two heatmaps are binarized using a preset probability threshold (e.g., 0.5) to generate initial contour and defect masks. Finally, morphological closing operations are performed on these two initial masks, using a 3x3 circular structuring element to eliminate small holes and burrs caused by noise, resulting in a smooth, connected final geometric contour feature point set (boundary pixel coordinates extracted from the contour mask) and a surface defect mask.

[0084] S40: Based on the geometric contour feature point set and the pre-stored cable tray design drawings, feature matching is performed to calculate the coordinate deviation of the current cable tray's actual pose relative to its theoretical pose.

[0085] In this embodiment, the goal of step S40 is to accurately map the cable tray contour identified in the image coordinate system to the world coordinate system (or machine tool coordinate system) and calculate its pose deviation (including translation, rotation, and possible slight deformation) relative to the ideal position. This is the core of achieving high-precision punching positioning, and its accuracy directly determines the accuracy of subsequent punching coordinates.

[0086] Specifically, firstly, from the set of geometric contour feature points obtained in step S30, the four vertex points of the cable tray are extracted using a corner detection algorithm. Simultaneously, the long straight lines in the contour are detected using Hough transform, and the perpendicular bisectors of at least two long sides are calculated. The intersection of these perpendicular bisectors is then used as auxiliary feature points. These vertex points and the intersection of the perpendicular bisectors together constitute a set of "current actual feature points." Next, the absolute coordinates of the "theoretical feature points" corresponding to the current cable tray model are read from the pre-stored CAD cable tray design drawings. Then, a perspective transformation matrix is ​​calculated using the least squares method. This matrix establishes a mapping relationship between the actual feature point set (image pixel coordinates) and the theoretical feature point set (world coordinates). By solving for the H matrix that minimizes the reprojection error, the translation, rotation, and distortion caused by lens perspective during the placement of the cable tray can be compensated. Finally, the theoretical coordinate set of all punched points in the CAD drawing is substituted into the solved perspective transformation matrix H, and matrix operations are used to obtain the "actual processing coordinates" of each punched point in the current image / world coordinate system. Meanwhile, by comparing the theoretical coordinates with the actual coordinates, the pose deviation vector of each hole point can be calculated. This vector typically includes the X-direction deviation, Y-direction deviation, and rotation angle deviation.

[0087] In this embodiment, by simultaneously acquiring and registering visible and near-infrared light images, complementary multispectral information sources are provided for subsequent processing, overcoming the limitation of single-source imaging of stainless steel surfaces being susceptible to high-light interference. By performing joint denoising processing on the registered dual-spectral images, specular reflection and random noise are effectively suppressed, significantly improving the contrast and signal-to-noise ratio of the cable tray surface feature map. By inputting the feature map into a multi-task convolutional neural network, parallel, accurate, and automated recognition of the cable tray's geometric contour and surface defects is achieved, replacing traditional manual visual inspection or simple image processing algorithms, greatly improving recognition accuracy and efficiency. Finally, through feature matching and coordinate calculation, the visual recognition results are converted into precise pose deviation data, providing a high-precision positioning reference for subsequent intelligent punching, thus realizing the automation, intelligence, and high precision of the pre-punching positioning process as a whole.

[0088] In one embodiment, such as Figure 2 As shown, in step S20, multispectral joint denoising processing is performed on the registered visible light image and near-infrared light image to obtain a high-contrast cable tray surface feature map, specifically including:

[0089] S21: Convert the registered visible light image from the RGB color space to the HSV color space and extract the V luminance channel component.

[0090] In this embodiment, the purpose of this sub-step is to separate the brightness information most directly affected by illumination, preparing for subsequent suppression of specular interference. The three channels (red, green, and blue) in the RGB color space are tightly coupled with brightness information and are severely affected by uneven ambient lighting and surface reflections. The HSV color space decouples color information (hue H, saturation S) from brightness information (lightness V). Extracting the V channel allows for a more direct acquisition of the image's brightness distribution, which includes the specular reflection component that we need to suppress.

[0091] Specifically, the system reads a registered visible light RGB image with dimensions M×N×3. First, a non-linear transformation formula is applied to each pixel to convert the RGB values ​​to HSV space. The transformation formula is as follows: V = max(R, G, B), where R, G, and B are the normalized pixel values. The calculated V channel is an M×N grayscale matrix, where each pixel value represents the brightness intensity of that point. This V channel image will serve as input for subsequent difference operations; it contains rich texture information but also includes strong specular noise.

[0092] S22: Adaptive histogram equalization is performed on the registered near-infrared image to enhance the surface texture features in the near-infrared image, resulting in an enhanced near-infrared image.

[0093] In this embodiment, this sub-step aims to enhance subtle contrast differences in near-infrared images caused by oil, oxide layers, or minor scratches. Near-infrared light exhibits different absorption and reflection characteristics for certain contaminants and microstructures on stainless steel surfaces, but the original images often have low contrast. Adaptive histogram equalization (AHE) is a local contrast enhancement algorithm that dynamically adjusts the transformation function based on the grayscale distribution of local image regions, thereby enhancing details while avoiding excessive noise amplification.

[0094] Specifically, the registered near-infrared grayscale image is read. Adaptive histogram equalization is performed using a sliding window method. A local window size is set (e.g., 64×64 pixels). For each pixel in the image, a grayscale histogram is calculated within the local window centered on that pixel, and a cumulative distribution function (CDF) is constructed based on this histogram. Then, the original grayscale value of the center pixel is mapped to a new grayscale value using this CDF. This process is repeated across the entire image to obtain the enhanced near-infrared image. After this processing, previously inconspicuous surface textures, slight pits, and other defect features are significantly enhanced, providing high-quality input for subsequent extraction of intrinsic features unaffected by visible light reflection.

[0095] S23: Perform a difference operation on the V brightness channel component of the enhanced near-infrared image and the visible light image to extract the difference feature map that is not affected by surface reflection.

[0096] In this embodiment, this sub-step is one of the core innovations of the method, aiming to effectively suppress specular interference on stainless steel surfaces through spectral difference. Its principle is based on a key assumption: near-infrared images are far less affected by surface specular reflection than visible light images, and both have a certain correlation in their responses to the material bulk and deep defects. Therefore, subtracting the enhanced near-infrared image from the specularly contaminated visible light V channel can largely cancel out shared bulk features, while preserving and highlighting the enhanced texture details in the near-infrared image and significantly reducing the specular areas in the V channel.

[0097] Specifically, the enhanced near-infrared image obtained in step S22, denoted as I_NIR_enhanced, is compared with the visible light V channel image extracted in step S21, denoted as I_V, and a pixel-by-pixel difference operation is performed. To ensure data range consistency, the two images are first normalized to the [0, 1] interval. The difference operation formula is: I_diff = |I_NIR_enhanced - I_V|, or a weighted difference is used to retain more effective edge information of the V channel: I_diff = α * I_NIR_enhanced - β * I_V, where α and β are adjustable weight coefficients. The calculated I_diff is the "difference feature map". In this map, areas that are bright in I_V due to strong reflection but not obvious in I_NIR_enhanced have large difference values ​​and are thus effectively suppressed; while the true edges and textures of the cable tray and the enhanced defect features in the near-infrared image are preserved or even enhanced after difference because they respond in both images.

[0098] S24: The V brightness channel component of the visible light image is weighted and fused with the difference feature map to reconstruct the high-contrast bridge surface feature map.

[0099] In this embodiment, the goal of this sub-step is to fuse the information advantages from different spectral sources to generate a final feature map that combines sharp edges (from the V channel) with rich surface details (from the difference feature map). Simple differencing may lose some of the effective geometric contour information in the V channel. Through adaptive weighted fusion, gradient information, which is crucial for contour recognition, can be preserved to the maximum extent while suppressing highlights.

[0100] Specifically, an adaptive weighted fusion strategy based on local contrast is adopted. First, the local contrast maps C_V and C_diff of I_V and I_diff are calculated respectively. For example, by calculating the standard deviation of the small neighborhood around each pixel, for each pixel position (i, j) in the image, its fusion weights w_V(i, j) and w_diff(i, j) are dynamically determined according to its local contrast: w_V(i, j) = C_V(i, j) / (C_V(i, j) + C_diff(i, j) + ε), w_diff(i, j) = 1 - w_V(i, j), where ε is a very small constant used to prevent division by zero errors. Then, a weighted fusion is performed: I_fused(i, j) = w_V(i, j) * I_V(i, j) + w_diff(i, j) * I_diff(i, j). Finally, I_fused is subjected to global contrast stretching, adjusting its grayscale range to [0, 255], resulting in a "high-contrast bridge surface feature map" used as the final input for subsequent neural networks. This image significantly reduces highlights while enhancing edge and surface defect features.

[0101] In one embodiment, such as Figure 3 As shown, in step S30, the surface feature map of the cable tray is input into a pre-trained multi-task convolutional neural network model, and the output is the geometric contour feature point set and surface defect mask of the cable tray, specifically including:

[0102] S31: Input the high-contrast surface feature map of the cable tray into the feature extraction backbone network of the multi-task convolutional neural network model, perform multi-layer convolution and pooling operations, and extract the deep semantic feature map.

[0103] In this embodiment, this sub-step is a crucial step in deep learning feature extraction. The feature extraction backbone network is responsible for automatically learning and extracting highly abstract and semantically meaningful feature representations from the input image. Through multi-layer convolution and non-linear activation, the network can capture complex patterns ranging from low-level edges and textures to high-level shapes and structures, providing a rich feature foundation for subsequent parallel contour and defect decoding.

[0104] Specifically, the multi-task convolutional neural network model uses ResNet-34, pre-trained on a large image dataset, as the backbone network for feature extraction. The input high-contrast bridge surface feature map (size H×W×1, single-channel grayscale image) is first downsampled through a 7×7 convolutional layer and a max-pooling layer. Subsequently, the feature map passes through four stages of ResNet-34, each stage containing multiple residual blocks. Within each residual block, two 3×3 convolution operations are performed, and the ReLU activation function is used to introduce non-linearity. Between stages, convolutional or pooling layers with a stride of 2 are used to halve the feature map size and double the number of channels. After processing by the backbone network, the original H×W image is encoded into a "deep semantic feature map" with a size of (H / 32)×(W / 32)×512. Although the spatial resolution of this feature map is reduced, each feature vector contains rich contextual semantic information of its corresponding original image region.

[0105] S32: Input the deep semantic feature map into the parallel first branch decoder and second branch decoder respectively, wherein the first branch decoder predicts the geometric contour heat map and the second branch decoder predicts the surface defect heat map.

[0106] In this embodiment, this sub-step implements the core of the multi-task learning architecture. Two parallel decoders share the common features extracted by the same encoder (backbone network), but learn to perform their respective tasks—contour localization and defect segmentation—through different upsampling and convolutional paths. This design is more efficient than training two independent networks and enables the backbone network to learn common features beneficial to both tasks.

[0107] Specifically, both the first branch decoder (contour decoder) and the second branch decoder (defect decoder) employ a decoding structure similar to U-Net. They receive deep semantic feature maps from the backbone network, respectively. Each decoder consists of multiple "upsampling modules" cascaded together. Each upsampling module typically includes a transposed convolutional (or bilinear interpolation) layer for 2x upsampling, followed by one or two 3×3 convolutional layers and ReLU activation to refine the features. Furthermore, the decoder uses skip connections to concatenate shallow feature maps of the same spatial scale from the backbone network with the upsampled feature maps, fusing low-level detail information with high-level semantic information to help recover accurate spatial locations. Finally, the first branch decoder outputs a "geometric contour heatmap" of size H×W×1, where each pixel value represents the probability (between 0 and 1) that the point belongs to the bridge outline. The second branch decoder outputs a "surface defect heatmap" of size H×W×1, representing the probability that each pixel belongs to a defect region (such as scratches or dents).

[0108] S33: The geometric contour heatmap and surface defect heatmap are binarized using preset probability thresholds to generate an initial contour mask and an initial defect mask.

[0109] In this embodiment, this sub-step transforms the probability map output by the neural network into a deterministic binary segmentation map, preparing for subsequent morphological processing and feature extraction. The choice of the probability threshold directly affects the accuracy and recall of the segmentation results and needs to be optimized based on the performance on the validation set.

[0110] Specifically, for both the contour heatmap and the defect heatmap, a probability threshold is set for each pixel value in the heatmap. If the pixel value is greater than or equal to the corresponding threshold, the pixel is marked as foreground in the output mask and assigned a value of 1; otherwise, it is marked as background and assigned a value of 0.

[0111] This operation yields a binarized "initial contour mask" and "initial defect mask". Due to potential noise or uncertainty in neural network predictions, these initial masks often contain isolated noise points, small holes, or boundary burrs.

[0112] S34: Perform morphological closing operations on the initial contour mask and the initial defect mask respectively to eliminate internal holes and obtain the final geometric contour feature point set and surface defect mask.

[0113] In this embodiment, this sub-step is intended to post-process the binary segmentation results to smooth the boundaries, fill small internal holes, and break narrow connections, thereby obtaining a cleaner and more complete contour and defect area. Morphological closing operations are particularly suitable for filling small holes and narrow breaks in the foreground area, while basically maintaining the area and shape of the original area.

[0114] Specifically, a structuring element is defined for both the initial contour mask and the initial defect mask, typically a 3×3 or 5×5 circular or square kernel. First, a dilation operation is performed: the structuring element is slid across the mask; if the structuring element intersects with the foreground region of the mask, the output pixel corresponding to the center of the structuring element is set to 1. The dilation operation expands the foreground region. Next, an erosion operation is performed on the dilated result: the structuring element is slid across the image; the center pixel is set to 1 only if the structuring element is completely contained within the dilated foreground region; otherwise, it is set to 0. The erosion operation shrinks the region. This combination of dilation and erosion is called the closing operation. After the closing operation, the small holes caused by thresholding in the initial mask are filled, minor breaks are connected, and the edges become smoother. The processed contour mask is the binary representation of the "final geometric contour feature point set," from which an ordered sequence of contour pixel coordinates can be extracted using an edge tracking algorithm. The processed defect mask is the "final surface defect mask," which can be directly used for subsequent taboo region mapping.

[0115] In one embodiment, such as Figure 4 As shown, in step S40, feature matching is performed based on the geometric contour feature point set and the pre-stored cable tray design drawings to calculate the coordinate deviation of the actual pose of the current cable tray relative to the theoretical pose. Specifically, this includes:

[0116] S41: Extract the four vertex points of the cable tray and the intersection of the perpendicular bisectors of at least two long sides from the set of geometric contour feature points, and use them as the current actual feature points.

[0117] In this embodiment, the goal of this sub-step is to filter out stable, unique feature points from the set of contour points output by the neural network, which are easy to find corresponding theoretical points in CAD drawings. The four vertices of the cable tray are naturally strong feature points. In addition, since cable trays are usually long and narrow, the intersection of the perpendicular bisectors of their long sides (i.e., the point near the geometric center) is also a stable feature that can assist in matching, especially providing redundancy when the vertices are occluded or the imaging is unclear.

[0118] Specifically, firstly, an edge tracking algorithm is applied to the final contour mask obtained in step S34 to obtain an ordered sequence of contour pixels. Then, a corner detection algorithm is used to find the point with the largest curvature on this contour sequence, and the four most significant points are selected as candidate vertex corners. Next, the long side lines in the contour are fitted using the Hough transform or least squares method. For each fitted long side, its perpendicular bisector is calculated. The intersection of at least two (usually the two longest, non-parallel sides) perpendicular bisectors is solved, and this intersection point is the approximate geometric center point of the bridge profile. Finally, the pixel coordinates (u_i, v_i) of the intersection point of the four vertex corners and this perpendicular bisector (a total of 5 points) are used as a set of "current actual feature points". The positions of these points in the actual image reflect the actual pose of the bridge in the current camera field of view.

[0119] S42: Read the coordinates of the theoretical feature points corresponding to the current cable tray model from the pre-stored cable tray design drawings, and calculate the perspective transformation matrix by fitting the data using the least squares method based on the correspondence between the actual feature points and the theoretical feature points.

[0120] In this embodiment, the core of this sub-step is to establish a mathematical mapping relationship between the image pixel coordinate system and the cable tray world coordinate system (or the CAD drawing coordinate system). Due to perspective distortion in camera imaging, this mapping is typically described using a 3x3 perspective transformation matrix (homography). By matching actual feature points with theoretical feature points and using the least squares method to solve for the optimal transformation matrix, the translation, rotation, and perspective distortion caused by the viewing angle of the cable tray can be compensated.

[0121] Specifically, based on the input cable tray model, the corresponding DXF file is retrieved from the pre-stored CAD drawing database. From this DXF file, the world coordinates (X_i, Y_i) of the "theoretical feature points" corresponding to the actual feature points extracted in step S41 are read. For example, the theoretical coordinates of the four corner points can be directly obtained from the rectangular outline of the drawing, and the intersection of the perpendicular bisectors is the geometric center of the rectangle. The correspondence between actual pixel points (u_i, v_i) and theoretical world points (X_i, Y_i) is established. The perspective transformation relationship can be expressed as: [u_i, v_i, 1]^T ~ H * [X_i, Y_i, 1]^T, where H is a 3x3 homography matrix, and ~ indicates equal scale. Using the least squares method, the H matrix that minimizes the sum of squared reprojection errors of all corresponding points is solved. The obtained H matrix contains transformation information such as rotation, translation, scaling, and shearing, establishing a mapping from world coordinates to image coordinates.

[0122] S43: Substitute the theoretical coordinate set of the current cable tray to be punched into the perspective transformation matrix to calculate the actual processing coordinate set and the pose deviation vector of each coordinate point.

[0123] In this embodiment, this sub-step uses the transformation matrix obtained in the previous step to transform the theoretical positions of all holes to be punched to the current image coordinate system, thereby obtaining the "actual processing coordinates" that the punch press needs to execute. At the same time, by comparing the theoretical positions with the actual detected contour positions, the "pose deviation" of each hole position can be calculated for subsequent motion compensation.

[0124] Specifically, from the same CAD design drawing, the theoretical world coordinate set of all points to be punched is read. For each theoretical coordinate point, the inverse matrix of the perspective transformation matrix H obtained in step S42 is used to map it to the current image coordinate system, converting the homogeneous coordinates into two-dimensional coordinates, thus obtaining the "actual processing coordinates" of the hole point in the current field of view.

[0125] In one embodiment, such as Figure 5 As shown, after step S40, the machine vision-based intelligent control method for punching and positioning stainless steel cable trays further includes a punching path planning step, specifically including:

[0126] S50: Read the surface defect mask output by the second branch decoder and map the pixel values ​​within the defect mask area to the punching taboo area.

[0127] In this embodiment, step S50 is a preprocessing step for punching path planning, designed to transform visually identified defective areas into obstacle information that the path planning algorithm can understand. This ensures that the punching point can automatically avoid defective areas on the material surface, preventing reduced product strength or unqualified hole shape caused by punching at defects, thereby improving processing quality and yield.

[0128] Specifically, from the surface defect mask (binary image, defect area pixel value is 1, background is 0) obtained in step S30, all connected regions with a pixel value of 1 are read. For each connected region, its minimum bounding rectangle is calculated, and this rectangular region is marked in the grid map of path planning. During marking, not only the grid cells covered by the rectangle are marked, but also the forbidden region is appropriately expanded according to the type of defect (such as scratches, dents) and a preset safety margin (e.g., expanding outward by 2-3 pixels) to form a "punching forbidden region" that is slightly larger than the actual defect. At the same time, each forbidden region is assigned a "passage cost" weight, which can be set to infinity or an extremely high value to ensure that the path planning algorithm will never cross the region during pathfinding.

[0129] S60: Combining the actual processing coordinate set, with the optimization goal of avoiding the forbidden zone and minimizing the total movement distance, construct a rasterized cost map.

[0130] In this embodiment, step S60 abstracts the actual processing environment into a mathematical model that can be searched by the algorithm. The rasterized cost map discretizes the continuous processing space into a grid, with each grid assigned a cost value reflecting the "cost" of passing through that grid. The purpose of constructing this map is to transform the obstacle avoidance and path optimization problem into a search problem on a weighted graph.

[0131] Specifically, firstly, a two-dimensional grid map is established using the effective stroke of the punch press table as the boundary, with each grid representing a small physical area. Then, the cost value of the grid cells covered by the punching forbidden zone defined in step S50 is set to a maximum value. For grid cells outside the forbidden zone, their base cost value can be set to 1, indicating free passage. Furthermore, to optimize the path, the cost value of grid cells farther from the punch press origin can be slightly increased to guide the path closer to the origin. Finally, all the actual machining coordinates (i.e., the punching points) calculated in step S40 are mapped onto this grid map and marked as target points. At this point, a gridded cost map containing obstacles (forbidden zones), target points (punching points), and passage costs is completed.

[0132] S70: Input the rasterized cost map into the path planning algorithm and iteratively search to obtain the globally optimal collision-free punching traversal sequence.

[0133] In this embodiment, step S70 addresses a variant under obstacle constraints, namely finding a path that visits all punching points (target points) once and only once, avoids all forbidden zones, and has the shortest total travel distance (or time).

[0134] Specifically, this step employs a hybrid strategy combining the A algorithm and the genetic algorithm. First, the A algorithm is used on a rasterized cost map to calculate the shortest collision-free path and its cost between any two punching points, forming a "distance matrix." Then, using "minimizing the total travel distance" as the fitness function, the genetic algorithm optimizes the visiting order (i.e., sequence) of the punching points. Each individual in the genetic algorithm population represents a punching sequence, iteratively evolving through selection, crossover, and mutation. When calculating the fitness of each individual, the cost between adjacent points in the distance matrix is ​​queried and accumulated according to its sequence order to obtain the total cost of the sequence. After a preset number of iterations (e.g., 500 generations), the algorithm converges, outputting the punching point traversal sequence with the minimum cost, i.e., the globally optimal (or near-optimal) collision-free punching sequence.

[0135] S80: Based on the collision-free punching traversal sequence, extract the actual processing coordinates and corresponding pose deviation vectors of each punching point in the sequence, and send them to the punching execution device to perform punching of stainless steel cable trays.

[0136] In this embodiment, step S80 is the connecting step from planning to execution. It transforms the abstract path sequence into specific instructions that can be executed by the motion controller. Sequentially extracting coordinates and deviations ensures that the punching action is performed in the optimized order, thereby saving idle travel time and improving processing efficiency.

[0137] Specifically, following the punching traversal sequence generated in step S70, the control system extracts the "actual processing coordinates" (X, Y) and "pose deviation vector" (ΔX, ΔY, Δθ) of each punching point sequentially from the stored data structure, starting from the first punching point. Then, this data is encapsulated into data frames according to a predefined communication protocol and transmitted in real-time to the host motion controller of the punching actuator via industrial Ethernet. Upon receiving the command, the motion controller adds it to the execution queue. This step ensures that the punching action strictly follows the optimized path, achieving a closed loop from intelligent planning to physical execution.

[0138] In one embodiment, such as Figure 6 As shown, in step S80, the command is sent to the punching device to perform punching of the stainless steel cable tray, specifically including:

[0139] S81: Read the pose deviation vector of each punching point in the collision-free punching traversal sequence in sequence, and decompose it into displacement error components and angle error components.

[0140] In this embodiment, step S81 decouples the pose deviation to prepare for subsequent channel-specific control. Displacement error directly affects the positioning accuracy of the X / Y axes, while angle error affects the attitude of the punch; both need to be processed and compensated separately.

[0141] Specifically, the pose deviation vector is typically represented as (ΔX, ΔY, Δθ). Here, ΔX and ΔY are the displacement error components (unit: millimeters) along the X and Y axes of the machine tool coordinate system. Δθ is the angular error component (unit: radians or degrees) of the bridge frame rotating about the Z axis perpendicular to the worktable plane. The system directly reads this vector and splits it into two independent parts: the displacement error component (ΔX, ΔY) and the angular error component Δθ.

[0142] S82: Input the displacement error component and angle error component into the punching execution device, and dynamically tune the proportional coefficient, integral coefficient and derivative coefficient of the PID controller in the punching execution device based on fuzzy inference rules.

[0143] In this embodiment, step S82 introduces fuzzy control to address the problem that traditional PID parameters are fixed and difficult to adapt to nonlinear and time-varying systems. During the punching process, the dynamic characteristics of the system change with the load and speed. Fuzzy PID can adjust the three parameters Kp, Ki, and Kd online according to the magnitude and trend of the error, achieving better dynamic response and steady-state accuracy.

[0144] Specifically, the motion controller within the punching actuator incorporates a fuzzy inference engine. This inference engine uses the absolute value of the displacement error |E| and the error change rate |EC| as input variables. Based on a preset fuzzy rule base, through fuzzification, inference, and defuzzification processes, it outputs a set of PID parameter corrections (ΔKp, ΔKi, ΔKd) applicable to the current error state in real time. These corrections are added to the initial parameters of the PID controller to obtain the dynamically tuned real-time parameters (Kp', Ki', Kd'). For the angular error Δθ, a separate fuzzy PID controller is used for similar tuning.

[0145] S83: The displacement error component and the angle error component are iteratively calculated using the tuned PID controller to generate the motor control compensation amount corresponding to the current punching point.

[0146] In this embodiment, step S83 is the core of the compensation calculation. Using the PID parameters tuned in the previous step, closed-loop control calculations are performed on the displacement and angle errors to generate the pulse or voltage commands required to drive the servo motor and rotary axis, thereby eliminating the errors.

[0147] Specifically, for the displacement error components (ΔX, ΔY), the position loop PID controllers within the X-axis and Y-axis servo drives use ΔX and ΔY as inputs, respectively. The controller iteratively calculates the error according to the formula Output = Kp' * E + Ki' * ∫E dt + Kd' * dE / dt, where E is the current error, ∫E dt is the error integral, and dE / dt is the error derivative. This calculation is performed once per control cycle (e.g., 1ms), outputting a "motor control compensation amount." This amount may be an additional number of pulses or an analog voltage signal, used to fine-tune the motor position, enabling the worktable to precisely move the bridge to the target position determined by (actual machining coordinates + compensation amount). For the angular error Δθ, the same PID calculation process is performed to compensate for it by the servo motor controlling the rotary axis.

[0148] S84: The motor control compensation amount is superimposed on the theoretical number of driving pulses at the current punching point, and the punching actuator is driven to move to the target punching position.

[0149] In this embodiment, step S84 is the final execution of the compensation. It combines the compensation command calculated by the software with the original theoretical motion command of the system to drive the physical mechanism to complete precise positioning.

[0150] Specifically, when the motion controller generates the movement command (i.e., the theoretical number of drive pulses) to the "actual machining coordinates" of the current punching point, it superimposes the X and Y axis displacement compensation amounts and the rotation axis angle compensation amounts calculated in step S83 onto the corresponding axis commands. For example, if the theoretical command requires 10,000 pulses to be sent on the X axis and the compensation amount is +50 pulses, then 10,050 pulses will ultimately be sent. The superimposed integrated command is sent to the servo drivers of each axis via the bus. The servo drivers drive the motors to execute, causing the punching actuator (which may be a gantry or a robotic arm) to move precisely, ultimately aligning the punch with the "target punching position" after visual positioning and pose compensation. Once in position, the punching action is triggered to complete the punching.

[0151] In one embodiment, such as Figure 7 As shown, after step S80, that is, after extracting the actual processing coordinates and corresponding pose deviation vectors of each punching point in the sequence according to the collision-free punching traversal sequence and sending them to the punching execution device, the intelligent control method for punching and positioning of stainless steel cable trays based on machine vision further includes:

[0152] S801: After the punching execution device moves to the first punching point and completes positioning according to the collision-free punching traversal sequence, the vision system is triggered to perform a secondary scan of the current local area of ​​the cable tray to obtain a high-precision local deformation image.

[0153] In this embodiment, step S801 aims to address the minute deformation of the cable tray caused by the release of stamping stress during processing. This deformation is dynamic and unpredictable, which can cause deviations in the coordinates calculated subsequently based on the initial image. By performing a secondary scan during processing, real-time sensing and compensation of the deformation can be achieved.

[0154] Specifically, once the punch has completed punching at the first planned point and lifted off, the system immediately sends a trigger signal to the vision system. The vision system then controls an industrial camera to quickly capture a high-resolution image of the recently punched local area (e.g., narrowing the field of view to a 200mm x 200mm area around the first punching point), obtaining a "high-precision local deformation image." This scan focuses on the area where deformation may occur, offering faster speed and higher resolution compared to the initial full-frame scan, and is capable of capturing micron-level displacement changes.

[0155] S802: Perform optical flow calculations on the high-precision local deformation image to analyze and obtain the micro-displacement vector field on the cable tray surface caused by the release of stamping stress.

[0156] In this embodiment, step S802 utilizes optical flow from computer vision to quantify local deformation. Optical flow estimates the motion vector of each point in an image by analyzing changes in pixel intensity in a continuous image sequence. By comparing sub-images of the same region in the secondary scan image with those in the initial image, the minute displacement field caused by stamping can be calculated.

[0157] Specifically, from the initial surface feature map of the cable tray, a region corresponding to the current high-precision local deformation image is extracted as a reference image. A dense optical flow algorithm is used to calculate the optical flow field of the current image relative to the reference image. This algorithm calculates a two-dimensional displacement vector (u, v) for each pixel in the image, representing the amount of movement of that pixel from the reference image to the current image in the x and y directions. This vector field reflects the "micro-displacement vector field of the cable tray surface" caused by the release of stamping stress. Noise vectors are removed by filtering to obtain reliable deformation field data.

[0158] S803: Based on the micro-displacement vector field, predict the deformation trend of the area where the subsequent punching point is located, and dynamically update the passage cost weight of the corresponding area in the rasterized cost map.

[0159] In this embodiment, step S803 extrapolates the sensed local deformation to predict and correct the entire processing path. Based on the assumptions of stress propagation and deformation continuity in materials mechanics, the deformation generated by local stamping will affect its surrounding area. By updating the cost map, the path planning can avoid areas that may become unprocessable or have reduced accuracy due to predicted deformation.

[0160] Specifically, spatial analysis is performed on the micro-displacement vector field obtained in step S802, such as calculating the gradient and divergence of the displacement field, to estimate the propagation direction and attenuation degree of deformation. Based on this analysis, a simple deformation prediction model is established to predict the additional positional deviation (δx, δy) that may occur at the subsequent punching point. For areas with large predicted deformation, the grids corresponding to these areas are found in the original rasterized cost map constructed in step S60, and their passage cost weights are dynamically increased. For example, if the grid with an original passage cost of 1 is predicted to have a positioning error that may exceed the tolerance threshold due to deformation, its cost is increased to 10 or 50, so that in subsequent replanning, the guiding path will try to avoid these high-risk areas.

[0161] S804: Based on the preset dynamic path planning algorithm, starting from the current punching point and combining the updated passage cost weight, the traversal sequence of the remaining punching points is locally replanned to generate an optimized punching sequence that is resistant to deformation interference.

[0162] In this embodiment, step S804 involves dynamically adjusting the path during the machining process to address real-time sensed deformation. This is an online replanning strategy that enables the system to cope with uncertain disturbances, improving the robustness of the entire machining process and the final hole positioning accuracy.

[0163] Specifically, dynamic path planning algorithms such as D* Lite or LPA* are employed. The current completed punching position is taken as the new starting point, and all remaining punching points are considered the set of target points that must be traversed. The updated rasterized cost map from step S803 serves as the environmental model. Based on the original plan, the algorithm leverages incremental search to quickly recalculate the optimal collision-free access sequence from the current starting point to all remaining target points. This newly generated sequence, known as the "deformation-resistant optimized punching sequence," considers the deformations that have occurred and their predicted effects, effectively avoiding risk areas where deformation leads to accuracy loss or processing difficulties. The system will then execute subsequent punching operations according to this new sequence.

[0164] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0165] In one embodiment, a machine vision-based intelligent control system for punching and positioning stainless steel cable trays is provided. This machine vision-based intelligent control system for punching and positioning stainless steel cable trays corresponds one-to-one with the machine vision-based intelligent control method for punching and positioning stainless steel cable trays described in the above embodiments. Figure 8 As shown, the machine vision-based intelligent control system for punching and positioning stainless steel cable trays includes a cable tray image acquisition and processing module, a surface feature map generation module, a contour feature extraction module, and a cable tray pose determination module. Detailed descriptions of each functional module are as follows:

[0166] The cable tray image acquisition and processing module is used to acquire visible light images and near-infrared light images of the surface of stainless steel cable trays, and to register the visible light images and near-infrared light images to a unified coordinate system;

[0167] The surface feature map generation module is used to perform multispectral joint denoising processing on the registered visible light image and near-infrared light image to obtain a high-contrast cable tray surface feature map.

[0168] The contour feature extraction module is used to input the surface feature map of the cable tray into a pre-trained multi-task convolutional neural network model and output the geometric contour feature point set and surface defect mask of the cable tray.

[0169] The cable tray pose determination module is used to perform feature matching based on the set of geometric contour feature points and the pre-stored cable tray design drawings, and calculate the coordinate deviation of the actual pose of the current cable tray relative to the theoretical pose.

[0170] Preferred options also include:

[0171] The punching forbidden zone determination module is used to read the surface defect mask output by the second branch decoder and map the pixel values ​​within the defect mask area to the punching forbidden zone.

[0172] The raster map construction module is used to combine the actual processing coordinate set to construct a rasterized cost map with the optimization goal of avoiding the taboo zone and minimizing the total movement distance.

[0173] The punching path sequence planning module is used to input the rasterized cost map into the path planning algorithm and iteratively search to obtain the globally optimal collision-free punching traversal sequence.

[0174] The punching point determination module is used to extract the actual processing coordinates and corresponding pose deviation vector of each punching point in the sequence according to the collision-free punching traversal sequence, and send them to the punching execution device to perform punching of stainless steel cable trays.

[0175] Preferred options also include:

[0176] The local image acquisition module is used to trigger the vision system to perform a secondary scan of the current local area of ​​the cable tray after the punching execution device moves to the first punching point and completes the positioning according to the collision-free punching traversal sequence, so as to acquire a high-precision local deformation image.

[0177] The local graphics analysis module is used to perform optical flow calculations on the high-precision local deformation image to analyze and obtain the micro-displacement vector field on the cable tray surface caused by the release of stamping stress.

[0178] The weight update module is used to predict the deformation trend of the area where the subsequent punching point is located based on the micro-displacement vector field, and dynamically update the passage cost weight of the corresponding area in the rasterized cost map.

[0179] The path replanning module, based on a preset dynamic path planning algorithm, uses the current punching point as the starting point and combines it with the updated passage cost weights to locally replan the traversal sequence of the remaining punching points, generating an optimized punching sequence that is resistant to deformation interference.

[0180] Specific limitations regarding the machine vision-based intelligent control system for punching and positioning stainless steel cable trays can be found in the above description of the intelligent control method for punching and positioning stainless steel cable trays based on machine vision, and will not be repeated here. Each module in the aforementioned machine vision-based intelligent control system for punching and positioning stainless steel cable trays can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the electronic device, or stored in the memory of the electronic device as software, so that the processor can call and execute the corresponding operations of each module.

[0181] In one embodiment, an electronic device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores the database. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a machine vision-based intelligent control method for punching and positioning stainless steel cable trays.

[0182] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0183] Acquire visible light and near-infrared light images of the surface of the stainless steel cable tray, and register the visible light and near-infrared light images to a unified coordinate system;

[0184] Multispectral joint denoising processing was performed on the registered visible light image and near-infrared light image to obtain a high-contrast surface feature map of the cable tray;

[0185] The surface feature map of the cable tray is input into a pre-trained multi-task convolutional neural network model, and the output is the geometric contour feature point set and surface defect mask of the cable tray.

[0186] Based on the geometric contour feature point set and the pre-stored cable tray design drawings, feature matching is performed to calculate the coordinate deviation of the current cable tray's actual pose from its theoretical pose.

[0187] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0188] Acquire visible light and near-infrared light images of the surface of the stainless steel cable tray, and register the visible light and near-infrared light images to a unified coordinate system;

[0189] Multispectral joint denoising processing was performed on the registered visible light image and near-infrared light image to obtain a high-contrast surface feature map of the cable tray;

[0190] The surface feature map of the cable tray is input into a pre-trained multi-task convolutional neural network model, and the output is the geometric contour feature point set and surface defect mask of the cable tray.

[0191] Based on the geometric contour feature point set and the pre-stored cable tray design drawings, feature matching is performed to calculate the coordinate deviation of the current cable tray's actual pose from its theoretical pose.

[0192] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0193] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0194] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A machine vision-based intelligent control method for punching and positioning stainless steel cable trays, characterized in that, The intelligent control method for punching and positioning stainless steel cable trays based on machine vision includes the following steps: Acquire visible light and near-infrared light images of the surface of the stainless steel cable tray, and register the visible light and near-infrared light images to a unified coordinate system; Multispectral joint denoising processing was performed on the registered visible light image and near-infrared light image to obtain a high-contrast surface feature map of the cable tray; The surface feature map of the cable tray is input into a pre-trained multi-task convolutional neural network model, and the output is the geometric contour feature point set and surface defect mask of the cable tray. Based on the geometric contour feature point set and the pre-stored cable tray design drawings, feature matching is performed to calculate the coordinate deviation of the current cable tray's actual pose from its theoretical pose.

2. The intelligent control method for punching and positioning stainless steel cable trays based on machine vision according to claim 1, characterized in that, The process of performing multispectral joint denoising on the registered visible light and near-infrared light images to obtain a high-contrast cable tray surface feature map specifically includes: The registered visible light image is converted from the RGB color space to the HSV color space, and the V luminance channel component is extracted. Adaptive histogram equalization is performed on the registered near-infrared image to enhance the surface texture features in the near-infrared image, resulting in an enhanced near-infrared image. The V brightness channel component of the enhanced near-infrared image and the visible light image are differentially processed to extract the difference feature map that is not affected by surface reflection. The V luminance channel component of the visible light image is weighted and fused with the difference feature map to reconstruct the high-contrast cable tray surface feature map.

3. The intelligent control method for punching and positioning stainless steel cable trays based on machine vision according to claim 1, characterized in that, The step of inputting the surface feature map of the cable tray into a pre-trained multi-task convolutional neural network model and outputting the geometric contour feature point set and surface defect mask of the cable tray specifically includes: The high-contrast surface feature map of the cable tray is input into the feature extraction backbone of the multi-task convolutional neural network model, and multi-layer convolution and pooling operations are performed to extract the deep semantic feature map. The deep semantic feature map is input into the parallel first branch decoder and second branch decoder respectively, wherein the first branch decoder predicts the geometric contour heat map and the second branch decoder predicts the surface defect heat map. The geometric contour heatmap and surface defect heatmap are binarized by applying a preset probability threshold to generate an initial contour mask and an initial defect mask. Morphological closing operations are performed on the initial contour mask and the initial defect mask to eliminate internal holes, resulting in the final geometric contour feature point set and surface defect mask.

4. The intelligent control method for punching and positioning stainless steel cable trays based on machine vision according to claim 3, characterized in that, The step of performing feature matching based on the geometric contour feature point set and the pre-stored cable tray design drawings to calculate the coordinate deviation of the current cable tray's actual pose from its theoretical pose specifically includes: Extract the four vertex points of the cable tray and the intersection of the perpendicular bisectors of at least two long sides from the set of geometric contour feature points, and use them as the current actual feature points; Read the coordinates of the theoretical feature points corresponding to the current cable tray model from the pre-stored cable tray design drawings, and calculate the perspective transformation matrix by fitting the data using the least squares method based on the correspondence between the actual feature points and the theoretical feature points. Substitute the theoretical coordinate set of the current cable tray to be punched into the perspective transformation matrix to calculate the actual processing coordinate set and the pose deviation vector of each coordinate point.

5. The intelligent control method for punching and positioning stainless steel cable trays based on machine vision according to claim 4, characterized in that, The method for intelligent control of stainless steel cable tray punching positioning, based on machine vision, further includes a punching path planning step, specifically: Substituting the theoretical coordinate set of the current cable tray to be punched into the perspective transformation matrix to calculate the actual processing coordinate set and the pose deviation vector of each coordinate point. Read the surface defect mask output by the second branch decoder and map the pixel values ​​within the defect mask area to the punching taboo area; Based on the actual processing coordinate set, a rasterized cost map is constructed with the optimization objective of avoiding the forbidden zone and minimizing the total movement distance. The rasterized cost map is input into the path planning algorithm, and the globally optimal collision-free punching traversal sequence is obtained through iterative search. Based on the collision-free punching traversal sequence, the actual processing coordinates and corresponding pose deviation vectors of each punching point in the sequence are extracted sequentially and sent to the punching execution device to perform punching of stainless steel cable trays.

6. The intelligent control method for punching and positioning stainless steel cable trays based on machine vision according to claim 5, characterized in that, The process of sending the data to the punching device to perform punching of the stainless steel cable tray specifically includes: The pose deviation vector of each punching point in the collision-free punching traversal sequence is read sequentially and decomposed into displacement error components and angle error components. The displacement error component and the angle error component are input into the punching execution device, and the proportional coefficient, integral coefficient and derivative coefficient of the PID controller in the punching execution device are dynamically tuned based on fuzzy inference rules. The displacement error component and the angle error component are iteratively calculated using the tuned PID controller to generate the motor control compensation amount corresponding to the current punching point. The motor control compensation is superimposed on the theoretical number of driving pulses at the current punching point, driving the punching actuator to move to the target punching position.

7. The intelligent control method for punching and positioning stainless steel cable trays based on machine vision according to claim 5, characterized in that, After extracting the actual processing coordinates and corresponding pose deviation vectors of each punching point in the non-collision punching traversal sequence according to the sequence and sending them to the punching execution device, the machine vision-based intelligent control method for punching and positioning stainless steel cable trays further includes: After the punching device moves to the first punching point and completes positioning according to the collision-free punching traversal sequence, the vision system is triggered to perform a secondary scan on the current local area of ​​the cable tray to obtain a high-precision local deformation image. Optical flow calculations were performed on the high-precision local deformation image to analyze the micro-displacement vector field on the cable tray surface caused by the release of stamping stress. Based on the micro-displacement vector field, the deformation trend of the area where the subsequent punching point is located is predicted, and the passage cost weight of the corresponding area in the rasterized cost map is dynamically updated. Based on the preset dynamic path planning algorithm, starting from the current punching point and combining the updated passage cost weight, the traversal sequence of the remaining punching points is locally replanned to generate an optimized punching sequence that is resistant to deformation interference.

8. A machine vision-based intelligent control system for punching and positioning stainless steel cable trays, characterized in that, The machine vision-based intelligent control system for punching and positioning stainless steel cable trays includes: The cable tray image acquisition and processing module is used to acquire visible light images and near-infrared light images of the surface of stainless steel cable trays, and to register the visible light images and near-infrared light images to a unified coordinate system; The surface feature map generation module is used to perform multispectral joint denoising processing on the registered visible light image and near-infrared light image to obtain a high-contrast cable tray surface feature map. The contour feature extraction module is used to input the surface feature map of the cable tray into a pre-trained multi-task convolutional neural network model and output the geometric contour feature point set and surface defect mask of the cable tray. The cable tray pose determination module is used to perform feature matching based on the set of geometric contour feature points and the pre-stored cable tray design drawings, and calculate the coordinate deviation of the actual pose of the current cable tray relative to the theoretical pose.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent control method for punching and positioning stainless steel cable trays based on machine vision as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent control method for punching and positioning stainless steel cable trays based on machine vision as described in any one of claims 1 to 7.

Images