Robot automatic labeling system and method applied to complex industrial environment

Abstract

The invention belongs to the field of robot automation control, and discloses a robot automatic labeling system and method applied to a complex industrial environment. The system comprises a labeling actuator module, a mechanical arm module, a visual perception module and a control module, wherein the visual perception module adopts a depth camera and is used for collecting multiple groups of checkerboard calibration plate images and collecting target object images; the depth camera is fused with an internal reference calibration and error correction unit, a target identification unit and a pose calculation unit. The system combines an internal reference calibration and error correction unit to complete camera internal reference calibration and hand-eye calibration to establish a high-precision coordinate system mapping relation, and a visual perception module adopts a preprocessing subunit and a positioning subunit to solve the precise three-dimensional pose of a labeling position. The control module is integrated with the mixed visual servo control unit to guarantee that the mechanical arm moves stably and accurately, the labeling efficiency is greatly improved compared with manual labeling, and the high-precision positioning performance is achieved.

Description

Technical Field

[0001] This invention belongs to the technical field of robot automation control and visual servoing, and particularly relates to an automatic labeling system and method for robots applied in complex industrial environments. Background Technology

[0002] In intelligent manufacturing production lines, automatic labeling is a crucial step in product information identification and traceability. Currently, robot-based automatic labeling systems using machine vision have become mainstream. However, their commonly adopted two-stage vision paradigm of "coarse positioning followed by fine correction" has inherent flaws, hindering further improvements in accuracy and stability. Specifically, the mainstream technical process typically involves: first, using general object detection networks such as YOLO and Faster R-CNN to identify the product, outputting a rectangular bounding box containing the product and background. This bounding box is coarsely positioned, unable to provide accurate product-background boundaries, and its coordinates are easily affected by lighting and occlusion, resulting in jitter. Subsequently, within the bounding box area, traditional image processing algorithms (such as Canny edge detection and Hough transform) are used to extract geometric features for pose calculation. This method is highly dependent on image quality and extremely sensitive to interference such as uneven lighting, product reflection, and background texture. The feature extraction is unstable, often leading to edge breakage or false detections, resulting in significant errors and fluctuations in the final calculated 3D pose, failing to meet sub-millimeter labeling accuracy requirements. In addition, existing solutions typically separate the vision module from the control module, treating the vision results as a one-time guidance signal. This lacks real-time, high-frequency closed-loop feedback capabilities during robot movement and cannot compensate for robot positioning drift or minor workpiece movements.

[0003] Therefore, existing technologies suffer from the following core pain points: Insufficient visual perception accuracy: Relying on "region-level" detection boxes and "fragile" manual features, they cannot provide stable and accurate pixel-level contour information; Poor system robustness: Traditional image processing algorithms are sensitive to environmental changes, resulting in poor stability in complex industrial scenarios; Low control closed-loop bandwidth: Visual perception is not deeply integrated into the real-time control loop, making it difficult to achieve dynamic correction and high-precision servo control. Therefore, it is urgent to construct a collaborative control system to systematically solve the core problems of accurate positioning, smooth movement, and reliable adhesion in automated labeling under complex industrial scenarios. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a robot automatic labeling system and method for use in complex industrial environments, which solves the technical defects of existing machine vision-based robot automatic labeling systems, such as insufficient positioning accuracy, weak environmental anti-interference ability, and poor real-time control closed-loop performance, resulting in poor adaptability to complex industrial scenarios.

[0005] To achieve the above objectives, according to one aspect of the present invention, a robotic automatic labeling system for use in complex industrial environments is provided, comprising: Labeling actuator module; The robotic arm module includes a six-axis collaborative robotic arm, the end of which is connected to the labeling actuator module; The visual perception module, employing a depth camera, is connected to the end effector of the six-axis collaborative robotic arm. The depth camera integrates an intrinsic parameter calibration and error correction unit, a target recognition unit, and a pose calculation unit. It acquires multiple sets of chessboard calibration board images and target object images, and calculates the 3D point cloud data of the target object image based on the back projection of the target recognition unit's output, using depth values ​​and camera intrinsic parameters. The intrinsic parameter calibration and error correction unit calibrates the depth camera's intrinsic parameters using the chessboard calibration board images and corrects the target object image using a dynamic distortion correction algorithm. The target recognition unit includes a preprocessing subunit and a localization subunit. The preprocessing subunit preprocesses the corrected target object image using Gaussian filtering for noise reduction and HSV color enhancement. The localization subunit processes the preprocessed target object image using U... 2 -Net semantic segmentation network performs region recognition and then uses Canny operator edge detection to determine the boundary; the pose calculation unit includes a pose calculation subunit and an error compensation subunit. The pose calculation subunit is used to calculate the three-dimensional pose of the target image in the depth camera coordinate system by solving the three-dimensional point cloud data of the target image. The error compensation subunit is used to update the process noise covariance and observation noise covariance of the three-dimensional pose of the target image through Kalman filtering and output the optimized three-dimensional pose of the target image. The control module is connected to other modules and integrates a hand-eye matrix conversion unit and a hybrid vision servo control unit. The hand-eye matrix conversion unit is used to convert the output of the pose calculation unit into a transformation matrix between the coordinate system of the labeling actuator module and the coordinate system of the target object. The hybrid vision servo control unit is used to fuse the output of the hand-eye matrix conversion unit with a PID parameter self-tuning algorithm and Kalman filtering to output the motion speed command of the six-axis collaborative robot arm.

[0006] Preferably, the Gaussian filtering is implemented using the following formula: ; in, Represented as the current pixel The weight value at the location; x , y These represent the horizontal and vertical distances of the corrected target image pixels from the center of the convolution kernel, respectively. This is expressed as the standard deviation, taken as 2.

[0007] Preferably, the boundary is determined by Canny operator edge detection, specifically: the target object image after the output of the U²-Net semantic segmentation network is used to calculate the gradient intensity through the following formula , and then compared with the determined high and low thresholds T high and T low . If M>T high , it is determined that the pixel belongs to the boundary and needs to be retained. If M<Tlow, it is determined that the pixel does not belong to the boundary and is not retained. If T low <M<T high , it is determined whether there are retained pixels around it. If there are retained pixels around, it is determined that the pixel still needs to be retained; then the retained pixels are connected to their adjacent retained pixels to obtain the boundary of the target object image; ; Among them, represents the gradient intensity of the current pixel, T high is 180, T low is 100; represents the partial derivative of the pixel value in the horizontal direction; represents the partial derivative of the pixel value in the vertical direction.

[0008] Preferably, the six-axis collaborative robotic arm also integrates a joint angle feedback unit and a singularity detection module. The joint angle feedback unit is used to collect the joint angle information of each joint in the six-axis collaborative robotic arm, and the singularity detection module is used to construct a closed-loop correction mechanism according to the joint angle information collected by the six-axis collaborative robotic arm and the control module.

[0009] Preferably, the depth camera is used to collect multiple groups of checkerboard calibration board images at different orientations and distances; the internal parameter calibration and error correction unit is used to solve the internal parameter matrix of the depth camera based on the checkerboard calibration board images according to Zhang Zhengyou calibration method in the Open CV vision library.

[0010] Preferably, the pose solving sub-unit is used to perform voxel grid downsampling and statistical outlier removal on the three-dimensional point cloud data corresponding to the optimized depth camera, then perform quadratic surface fitting by the least squares method to obtain the normal vector at the center point of the cloth, and then construct a coordinate system in combination with the optimal straight line of the product edge, and output the three-dimensional pose of the target image in the depth camera coordinate system.

[0011] Preferably, the control module is connected to other modules through heterogeneous communication links respectively.

[0012] Preferably, the labeling actuator module includes a push rod, a vacuum suction cup, and a label adsorption platform. The end of the push rod is connected to the vacuum suction cup via a spring, and the label adsorption platform is provided with a linear support rounded corner and a waste paper recycling cylinder.

[0013] Preferably, the process is based on the Ubuntu operating system.

[0014] According to another aspect of the present invention, an automatic labeling method for robots applied in complex industrial environments is provided, comprising the following steps: S1: Acquire multiple sets of chessboard calibration board images and target object images; S2: Use the chessboard calibration board image to calibrate the intrinsic parameters of the depth camera, and then combine the target image with the dynamic distortion correction algorithm to correct the image; S3: The target image corrected in S2 is preprocessed using Gaussian filtering for noise reduction and HSV color enhancement. Then, the preprocessed target image is processed using U... 2 -Net semantic segmentation network performs region identification and then Canny operator edge detection to determine the boundary; S4: Based on the output of S3, the three-dimensional point cloud data of the target object image is calculated by back projection based on the depth value and camera intrinsic parameters; S5: Solve the three-dimensional point cloud data of the target object image to obtain the three-dimensional pose of the target image in the depth camera coordinate system, and then update the process noise covariance and observation noise covariance of the three-dimensional pose of the target image through Kalman filtering, and output the optimized three-dimensional pose of the target image. S6: Convert the output of S5 into a transformation matrix between the coordinate system of the labeling actuator module and the coordinate system of the target object, and then combine it with the PID parameter self-tuning algorithm and Kalman filtering to output the motion speed command of the six-axis collaborative robotic arm.

[0015] S7: Apply labels according to the robotic arm's movement speed command.

[0016] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages: 1. This invention provides an automated labeling system for robots applied in complex industrial environments. It utilizes a labeling actuator module to achieve stable label pickup and automatic waste paper recycling. Combined with intrinsic parameter calibration and error correction units, as well as hand-eye calibration, it establishes high-precision coordinate system mapping by completing camera intrinsic parameters. Furthermore, through a visual perception module employing preprocessing and positioning subunits, it accurately calculates the three-dimensional pose of the labeling position, improving anti-interference capabilities. Finally, a hybrid visual servo control unit in the control module rapidly adjusts the position and pose of the robotic arm to achieve accurate and rapid labeling. Simultaneously, real-time capture of visual information enables higher precision, ensuring smooth and precise robotic arm movement. On one hand, compared to an open-loop control system directly using a hand-eye matrix, this invention utilizes a visual perception module to transform into a more reliable closed-loop control system, ensuring normal operation under complex environments and conditions, and improving tracking accuracy and robustness. On the other hand, the visual perception module combines the advantages of position-based and image-based visual servoing in a hybrid visual servoing technology, which is beneficial for improving labeling speed and accuracy during automated labeling. In summary, this system overcomes the pain points of traditional labeling technology, such as low positioning accuracy, weak anti-interference ability, and insufficient motion stability. It significantly improves labeling efficiency compared to manual methods, has high-precision positioning performance, is suitable for automated production lines in complex industrial scenarios, effectively reduces labor costs, ensures consistent labeling quality, and has strong engineering application value.

[0017] 2. The Gaussian filtering in this invention preferably employs a distance-weighted averaging mechanism. The weights decrease exponentially with increasing distance between the pixel and the center of the convolution kernel. The preferred value of σ in the formula is 2, indicating a moderate coverage area for the convolution kernel, effectively smoothing out common interferences such as noise and lighting in industrial environments. Simultaneously, it avoids the edge "halo" phenomenon that may occur with mean filtering, thus providing a foundation for subsequent U... 2 -Net's salient feature extraction preserves key boundary information.

[0018] 3. The present invention is preferably performed in U 2 Based on the region priors provided by -Net semantic segmentation, a dual-threshold hysteresis thresholding strategy (T) is employed. high =180, T low =100) accurately extracts the continuous geometric boundaries of the target object, achieving a leap in accuracy from semantic region to geometric contour, providing high-confidence boundary point cloud constraints for the pose calculation unit, and working in conjunction with Kalman filtering to output stable and accurate target 3D pose in dynamic robotic arm operation scenarios.

[0019] 4. The system of the present invention only requires one robot. Compared with the method that requires two robots in a relatively stable environment to determine the positional relationship between the robot and the positioning base station through a series of methods, it can avoid errors caused by many external factors, ensure the consistency of labeling quality, and has strong engineering application value. Attached Figure Description

[0020] Figure 1 This is a system composition block diagram illustrating an example of the present invention.

[0021] Figure 2 A schematic diagram of the method flow of this invention.

[0022] Figure 3 This is a schematic diagram of the system communication process as an example of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0024] To achieve the above objectives, such as Figure 1 As shown, the present invention adopts the following technical solution: This embodiment presents a robotic automatic labeling system for complex industrial environments. It possesses high anti-interference capabilities and motion stability, and can be widely applied to automated production lines in complex industrial settings such as textiles, food, and chemicals. In particular, it can achieve high-precision, high-efficiency automatic labeling of cylindrical products of different sizes, providing core technological support for the automation upgrade of industrial production. The system mainly includes: Labeling actuator module; The robotic arm module uses a six-axis collaborative robotic arm, with its end flange rigidly connected to the labeling actuator module. The visual perception module, with a depth camera at its core, is connected to the end effector of a six-axis collaborative robotic arm. The depth camera integrates an intrinsic parameter calibration and error correction unit, a target recognition unit, and a pose calculation unit. It is used to acquire multiple sets of chessboard calibration board images, acquire target object images, and calculate the 3D point cloud data of the target object image based on the back projection of the target recognition unit's output using depth values ​​and camera intrinsic parameters. The intrinsic parameter calibration and error correction unit is used to calibrate the depth camera's intrinsic parameters using the chessboard calibration board images and to correct the target object image using a dynamic distortion correction algorithm. The target recognition unit includes a preprocessing subunit and a localization subunit. The preprocessing subunit preprocesses the corrected target object image using Gaussian filtering for noise reduction and HSV color enhancement, while the localization subunit uses U... 2- The -Net semantic segmentation network performs region recognition and then uses the Canny operator to detect the edge to determine the boundary; the pose calculation unit includes a pose calculation subunit and an error compensation subunit. The pose calculation subunit is used to calculate the three-dimensional pose of the target image in the depth camera coordinate system from the three-dimensional point cloud data of the target image. The error compensation subunit is used to update the process noise covariance and observation noise covariance of the three-dimensional pose of the target image through Kalman filtering and output the optimized three-dimensional pose of the target image. The control module connects to other modules for data interaction and command scheduling. It integrates a hand-eye matrix conversion unit and a hybrid vision servo control unit. The hand-eye matrix conversion unit converts the output of the pose calculation unit into a transformation matrix between the coordinate system of the labeling actuator module and the target object's coordinate system. The hybrid vision servo control unit fuses the output of the hand-eye matrix conversion unit with a PID parameter self-tuning algorithm and Kalman filtering to output the motion speed command for the six-axis collaborative robotic arm. However, without considering process covariance, the results may exhibit lag and blinding phenomena: when the fabric position actually changes, the filter may mistakenly identify this change as an observation error and refuse to update the coordinates, causing the filtered coordinates to lag behind the actual position; if the object moves too fast, the filter may completely lock into the old position, no longer responding to any new observation data, leading to tracking loss. If the observation covariance is not considered, the error and jitter of the result will be directly reflected. The filter will directly output the camera's raw measurement value, and the identified cloth coordinate axis will shake violently on the screen. Tiny jitter of 1-2 pixels per frame will be received completely, resulting in high-frequency oscillation of the calculated three-dimensional pose, which cannot be used to drive the robot arm to move smoothly.

[0025] In some implementations, the original U 2 The -Net semantic segmentation network is not specifically optimized for industrial-grade, fine-grained segmentation. It may produce artifacts or blurred boundaries when dealing with complex industrial backgrounds and low-contrast images. Directly using U... 2The -Net semantic segmentation network treats all features equally, lacking a selective enhancement mechanism for fabrics in industrial environments. Background noise may be indiscriminately amplified. Therefore, a preprocessing subunit is used to preprocess the corrected target image by applying Gaussian filtering for noise reduction and HSV color enhancement. Gaussian filtering, through weighted averaging of convolutional kernels, effectively suppresses high-frequency interference introduced by sensor noise and uneven lighting common in industrial environments. While preserving key edge information such as fabric texture, it smooths the background area, reducing the risk of background noise being indiscriminately amplified by the network. HSV color enhancement converts the image to a color space with separate hue, saturation, and brightness. By independently adjusting the saturation and brightness channels, it enhances the contrast between the target object (such as areas with fabric defects) and the background, compensating for the problem of indistinct target features in low-contrast images. At the same time, it avoids color shifts that may be caused by global adjustments in the RGB space, making the preprocessed image more conducive to U.S. 2 -Net's encoding and decoding structure focuses on fine-grained target boundaries, thereby reducing artifacts and boundary blurring, and improving segmentation accuracy and robustness in industrial scenarios.

[0026] Gaussian filtering is achieved using the following formula: ; in, Represented as the current pixel The weight value at the location; x , y These represent the horizontal and vertical distances of the corrected target image pixels from the center of the convolution kernel, respectively. σ is represented as the standard deviation. A larger σ indicates a stronger smoothing effect but a more blurred image. In this scheme, a σ of 2 is preferred, meaning the convolution kernel has a moderate coverage area, effectively smoothing common industrial environments such as noise, artificial textures caused by uneven lighting, and high-frequency interference from fine fly waste on the fabric surface. Simultaneously, because the smoothing characteristic of the Gaussian kernel is gradual, the warp and weft textures of the fabric itself, as well as the edges of defective areas, are not excessively destroyed during the smoothing process, avoiding the edge "bleeding" phenomenon that may occur with mean filtering, thus providing a foundation for subsequent U... 2 -Net's salient feature extraction preserves key boundary information.

[0027] In some implementations, U 2The output of the -Net semantic segmentation network is a pixel-level classification result, that is, to judge whether each pixel belongs to the "target object (fabric defect)" or the "background", and the output form is usually a binary mask. However, this mask only represents the region attribution and does not clarify the precise geometric boundary of the target object. Based on this, the Canny operator extracts the edges of the mask region as continuous, single-pixel-width closed contours by calculating the gradient intensity, achieving an accuracy improvement from region recognition to boundary localization, providing geometric constraints for subsequent pose calculation. Determine the boundary through the Canny operator edge detection, specifically: For the U 2 -Judge whether the pixel point needs to be retained for the target object image after the output of the -Net semantic segmentation network, and then connect the retained pixel points with their adjacent retained pixel points to obtain the boundary of the target object image; ; Among them, represents the gradient intensity of the current pixel point, and T high is 180, and T low is 100; represents the partial derivative of the pixel value in the horizontal direction; represents the partial derivative of the pixel value in the vertical direction.

[0028] Specifically, calculate the gradient intensity through the above formula, and then compare it with the determined high and low thresholds T high and T low . If M>T high , judge that this pixel point belongs to the boundary and needs to be retained. If M<Tlow, judge that this pixel point does not belong to the boundary and is not retained. If T low <M<T high , then judge whether there are retained pixel points around it. If there are retained pixel points around, judge that this pixel point still needs to be retained; then connect the retained pixel points with their adjacent retained pixel points to obtain the boundary of the target object image.

[0029] In some embodiments, the six-axis collaborative robotic arm also integrates a joint angle feedback unit and a singularity detection module. The joint angle feedback unit is used to collect the joint angle information in the six-axis collaborative robotic arm, providing real-time and accurate joint motion data feedback for the control system; the singularity detection module is used to construct a closed-loop correction mechanism according to the joint angles collected by the six-axis collaborative robotic arm and the control module. The motion instruction reception and execution are realized through the corresponding function package of the robotic arm, and it has high-precision repeat positioning performance. At the same time, the hybrid vision servo control unit is used to convert the output of the hand-eye matrix conversion unit and the pose data collected by the control module through the vision system at high frequency, and combine the joint angle feedback information of the robotic arm to construct a double closed-loop correction mechanism, continuously adjusting the motion trajectory to ensure that the labeling position has high-precision positioning performance.

[0030] The kinematic modeling and singular solution avoidance unit establishes the kinematic model of the robotic arm and completely avoids singular solutions in planar operations through a triple strategy including optimal inverse solution screening, high-density trajectory interpolation, and work plane mapping. The triple singular solution avoidance strategy includes: ① writing an inverse solution function containing a joint angle optimal solution screening program segment; ② using high-density interpolation in trajectory planning to control the amplitude of joint angle abrupt changes; ③ transforming the work space through work plane mapping and achieving complete avoidance of singular solutions in planar operations through end effector attitude compensation.

[0031] In some implementations, a depth camera is used to acquire multiple sets of chessboard calibration board images at different orientations and distances according to the Zhang Zhengyou calibration method. Simultaneously, an intrinsic parameter calibration and error correction unit implements camera intrinsic parameter calibration based on the OpenCV library. Using the Zhang Zhengyou calibration method, multiple complete chessboard calibration board images are captured from multiple angles within a normal working distance range, and the initial intrinsic parameter matrix is ​​solved using a calibration function. Furthermore, a dynamic distortion correction algorithm is combined to eliminate imaging errors caused by changes in ambient light and lens temperature fluctuations, ultimately enabling high-precision intrinsic parameter calibration.

[0032] In some implementations, the target recognition unit of the visual perception module uses U 2 The -Net semantic segmentation network enables the recognition of cylindrical product regions. Combined with multi-step processing including HSV color enhancement, Gaussian filtering noise reduction, and Canny edge detection, it achieves accurate 3D pose calculation of the labeling location. The visual perception module's U... 2 The U²-Net semantic segmentation network boasts high segmentation accuracy and is adaptable to various lighting conditions and partially occluded industrial scenarios. It enhances anti-interference capabilities through a multi-step image processing workflow: first, the RGB image is converted to the HSV color space to improve color robustness; then, Gaussian filtering is applied for noise reduction; subsequently, the U²-Net semantic segmentation network performs pixel-level segmentation on the preprocessed image, outputting a product region segmentation mask to achieve accurate separation of the target product from the background; finally, edge detection using the Canny operator highlights product edge features, determining the boundaries of the corrected target image.

[0033] To ensure the stable adaptation of the algorithm in the labeling system, a dataset covering various industrial scenarios was constructed, encompassing different lighting conditions, partially occluded scenarios, and cylindrical products with various common diameter ranges. After training, the model has high segmentation accuracy, high success rate in recognizing partially occluded products, and significantly improved resistance to lighting interference compared to single visual recognition methods.

[0034] In some implementations, the pose solving subunit of the pose solving unit is used to solve the 3D pose of the target image in the depth camera coordinate system. Specifically, this includes: performing voxel mesh downsampling, statistical outlier removal, and surface smoothing preprocessing on the 3D point cloud data corresponding to the optimized depth camera; then, using the least squares method to fit a quadratic surface, solving for the surface parameters, and calculating the normal vector at the product center point (as the z-axis vector of the product body coordinate system), achieving high-precision fitting performance. Simultaneously, the optimal straight line of the product edge is extracted from the edge features of the RGB image through straight line Hough detection (as the x-axis vector of the product body coordinate system); combining the x and z-axis vectors to construct the product body coordinate system, and then solving the 3D pose of the labeling position in the camera coordinate system. To improve the adaptability of the pose solving results to the robotic arm motion, the coordinate mapping error compensation algorithm is optimized, enabling the solved pose data to more accurately match the robotic arm motion coordinate system.

[0035] In some implementations, the control module integrates a hand-eye matrix conversion unit and a hybrid vision servo control unit. The control module converts the labeling pose data in the camera coordinate system using a hand-eye matrix. The specific hand-eye matrix is ​​obtained by iteratively solving multiple sets of non-coaxial pose data to obtain the transformation matrix between the coordinate system of the labeling actuator module and the coordinate system of the target object, which has high-precision conversion performance. Then, the hybrid vision servo control unit integrates a PID parameter self-tuning algorithm and Kalman filtering. The PID parameters are dynamically adjusted according to the distance between the robotic arm and the target position, outputting smooth robotic arm movement speed commands.

[0036] In some implementations, the control module is built on an industrial computer running Ubuntu and deployed with software environments such as PyTorch, OpenCV, and the robotic arm control SDK. This module interacts with the robotic arm, vision camera, and labeling actuator via heterogeneous communication links for data exchange and command scheduling, integrating core algorithms such as kinematics calculation and visual servo control. The control module connects to other modules via these heterogeneous communication links to work collaboratively. These heterogeneous communication links are constructed using an operating system pseudo-terminal combined with network tools. Pose data is transmitted in character format, and a buffer reset and data verification mechanism ensures transmission integrity and provides low-latency transmission performance.

[0037] In some implementations, the labeling actuator module is a modular device installed at the end of a robotic arm, integrating a push rod, a vacuum suction cup, and a label adsorption platform. The end of the push rod is connected to the vacuum suction cup via a spring. Its main control board uses an STM32 chip, a high-performance microcontroller, and achieves signal linkage with the control module through the robotic arm's I / O port. The buffer spring can adapt to the surface curvature of various common diameter cylindrical products, adaptively adjusting the labeling pressure to ensure uniform labeling pressure, adapting to cylindrical products, and preventing label wrinkling or detachment. The label adsorption platform is equipped with linear support rounded corners and a waste paper recycling bin. The linear support rounded corners on the surface of the label adsorption platform reduce the label adhesion area, and the outer waste paper recycling bin automatically collects the peeled backing paper, resulting in a high label adsorption success rate.

[0038] The following example illustrates a labeling actuator module with a multi-layered modular design, offering high rigidity and ease of maintenance. From top to bottom, it includes: an independent camera clamping mechanism for mounting a depth camera, which can be disassembled as needed; a drive element mounting layer, securing the main power supply and air pump of the labeling actuator module's main control board, providing power and control for the entire actuator; an electrical signal processing and communication element mounting layer, integrating a step-down module and an I / O signal receiving module for signal conversion and communication; a control element mounting layer, integrating an STM32-based main control board and solenoid valve drive circuit, receiving and executing control commands; a push rod drive layer, containing an electric push rod with a series buffer spring and vacuum suction cup at its end. The push rod can extend and retract vertically, and the buffer spring adapts to the curvature of the product surface for flexible label application; and a label adsorption platform using a scissor-lift structure to accommodate different height requirements. The label adsorption platform surface is machined with an array of breathable micropores and linear support rounded corners, forming a negative pressure zone after connection with the air pump. This stabilizes and pre-fixes individual labels, while special surface treatment ensures uniform label peeling force and a high adsorption success rate. The waste backing paper recycling component is located on one side of the label adsorption platform. It can automatically roll up the peeled label backing paper, keeping the work area clean.

[0039] Furthermore, this embodiment also provides a labeling method for the above-mentioned robotic automatic labeling system applied in complex industrial environments, comprising the following steps: S1: Acquire multiple sets of chessboard calibration board images and target object images; S2: Use the checkerboard calibration board image and Zhang Zhengyou calibration method to calibrate the intrinsic parameters of the depth camera, and then combine the target image with the dynamic distortion correction algorithm to correct the image. S3: The target image corrected in S2 is preprocessed by Gaussian filtering for noise reduction and HSV color enhancement. Then, the U²-Net semantic segmentation network performs pixel-level segmentation on the preprocessed image and outputs a product region segmentation mask to achieve accurate separation of the target product from the background. Finally, the Canny operator edge detection is used to highlight the edge features of the product and determine the boundary of the corrected target image. S4: Based on the output of S3, the three-dimensional point cloud data of the target object image is calculated by back projection based on the depth value and camera intrinsic parameters; S5: Solve the three-dimensional point cloud data of the target image to obtain the three-dimensional pose of the target image in the depth camera coordinate system, and then update the process noise covariance and observation noise covariance of the three-dimensional pose of the target image through Kalman filtering, and output the optimized three-dimensional pose of the target image. S6: Converts the output of S5 into a transformation matrix between the coordinate system of the labeling actuator module and the coordinate system of the target object, and then integrates the PID parameter self-tuning algorithm and Kalman filtering to output the motion speed command of the six-axis collaborative robotic arm.

[0040] S7: Apply labels according to the robotic arm's movement speed command.

[0041] Combination Figures 2-3 The following is an example of a system workflow: System initialization and calibration: Each hardware module performs a power-on self-test, and the heterogeneous communication link is established. The control module guides the completion of camera intrinsic parameter calibration (using multiple sets of checkerboard images, providing high-precision performance) and hand-eye matrix calibration (using multiple sets of non-coaxial pose data, providing high-precision conversion performance). The tag adsorption platform automatically adjusts to the preset height.

[0042] Tag extraction: The control module plans the robotic arm's movement path, moving it directly above the label suction platform. Then, the following sequence is executed: the labeling actuator's push rod extends, causing the vacuum suction cup to contact the label; the air pump and solenoid valve are activated, generating negative pressure to firmly suck up the label; the push rod retracts, lifting the label. Simultaneously, the peeled backing paper is automatically wound into the recycling bin.

[0043] Visual positioning and pose calculation: A depth camera captures images of a cylindrical product on a conveyor belt. The acquired RGB images undergo HSV color space conversion and Gaussian filtering for noise reduction. The preprocessed images are then input into a U²-Net semantic segmentation network, which outputs a precise pixel-level mask for the product, accurately separating the product from the background. Finally, Canny edge detection is used to highlight the product's edge features and determine the boundaries of the corrected target image. Based on the output of the above steps, back-projection calculations based on depth values ​​and camera intrinsic parameters yield the 3D point cloud data of the target image. Combining the product mask with the corresponding 3D point cloud data, after point cloud filtering and downsampling, the least squares method is used to fit the cylindrical surface of the product, and a product coordinate system is constructed to calculate the 3D pose of the labeling point relative to the camera. This pose data is immediately fed into a Kalman filter for smoothing and prediction optimization. The optimized target pose is then transformed to the robotic arm's base coordinate system using a hand-eye matrix. The hybrid vision servo controller dynamically adjusts the PID parameters (proportional, integral, and derivative coefficients) at a high frequency based on the current pose error, and, combined with the prediction results of the Kalman filter, calculates the smooth movement speed command for the robotic arm's end effector. During the movement of the robotic arm, its joint angle feedback and visual pose feedback form a double closed loop, which corrects the trajectory deviation in real time and ensures that the terminal approaches the target marking point with high precision.

[0044] Hybrid vision servo control and labeling motion: The control module converts the optimized target pose to the robotic arm's base coordinate system using a hand-eye matrix. The hybrid vision servo control unit dynamically adjusts the PID parameters based on the current pose error and integrates Kalman filter results to generate smooth motion speed commands, driving the robotic arm towards the target point. Simultaneously, the vision system continuously provides pose feedback at high frequencies, forming a double closed loop with joint angle feedback to correct the motion trajectory in real time. After the robotic arm completes positioning, it sends a labeling command to the labeling actuator via the I / O port. The push rod drives the suction cup to flexibly attach the label to the product surface, after which the air pump stops, and the suction is released.

[0045] Reset and Loop: After labeling is completed, the push rod retracts, and the robotic arm drives the actuator back to standby above the label adsorption platform, waiting for the next work instruction to complete a single labeling cycle.

[0046] In summary, through the collaboration of the above modules, the system operates according to a closed-loop logic of "high-precision visual positioning - smooth motion servoing - adaptive end-efficiency labeling", achieving high labeling efficiency and enabling efficient, accurate, and reliable labeling of multi-size cylindrical products in complex industrial environments.

[0047] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automated labeling system for robots applied in complex industrial environments, characterized in that, include: Labeling actuator module; The robotic arm module includes a six-axis collaborative robotic arm, the end of which is connected to the labeling actuator module. The visual perception module, employing a depth camera, is connected to the end effector of the six-axis collaborative robotic arm. The depth camera integrates an intrinsic parameter calibration and error correction unit, a target recognition unit, and a pose calculation unit. It acquires multiple sets of chessboard calibration board images and target object images, and calculates the 3D point cloud data of the target object image based on the back projection of the target recognition unit's output, using depth values ​​and camera intrinsic parameters. The intrinsic parameter calibration and error correction unit calibrates the depth camera's intrinsic parameters using the chessboard calibration board images and corrects the target object image using a dynamic distortion correction algorithm. The target recognition unit includes a preprocessing subunit and a localization subunit. The preprocessing subunit preprocesses the corrected target object image using Gaussian filtering for noise reduction and HSV color enhancement. The localization subunit processes the preprocessed target object image using U... 2 -Net semantic segmentation network performs region recognition and then uses Canny operator edge detection to determine the boundary; the pose calculation unit includes a pose calculation subunit and an error compensation subunit. The pose calculation subunit is used to calculate the three-dimensional pose of the target image in the depth camera coordinate system by solving the three-dimensional point cloud data of the target image. The error compensation subunit is used to update the process noise covariance and observation noise covariance of the three-dimensional pose of the target image through Kalman filtering and output the optimized three-dimensional pose of the target image. The control module is connected to other modules and integrates a hand-eye matrix conversion unit and a hybrid vision servo control unit. The hand-eye matrix conversion unit is used to convert the output of the pose calculation unit into a transformation matrix between the coordinate system of the labeling actuator module and the coordinate system of the target object. The hybrid vision servo control unit is used to fuse the output of the hand-eye matrix conversion unit with a PID parameter self-tuning algorithm and Kalman filtering to output the motion speed command of the six-axis collaborative robot arm.

2. The robotic automatic labeling system for complex industrial environments according to claim 1, characterized in that, The Gaussian filtering is achieved using the following formula: ； in, Represented as the current pixel The weight value at the location; x , y These represent the horizontal and vertical distances from the center of the convolution kernel to the pixels of the corrected target image; This is expressed as the standard deviation, taken as 2.

3. The robotic automatic labeling system for complex industrial environments according to claim 1, characterized in that, The boundary determination via Canny operator edge detection specifically involves calculating the gradient intensity of the target image output from the U²-Net semantic segmentation network using the following formula. Then, it is compared with the determined high and low thresholds T. high and T low Comparison, if M > T high If the pixel is determined to be a boundary point, it needs to be retained. <T low If T determines that the pixel does not belong to the boundary, it is not retained. low <M<T high If there are already retained pixels around it, then it is determined whether the pixel still needs to be retained. Then, the retained pixels are connected to their adjacent retained pixels to obtain the boundary of the target image; ； in, T represents the gradient intensity of the current pixel. high For 180, T low It is 100; Represented as the partial derivative of the pixel value in the horizontal direction; It is represented as the partial derivative of the pixel value in the vertical direction.

4. The robotic automatic labeling system for complex industrial environments according to claim 1, characterized in that, The six-axis collaborative robotic arm also integrates a joint angle feedback unit and a singular solution detection module. The joint angle feedback unit is used to collect the angle information of each joint in the six-axis collaborative robotic arm, and the singular solution detection module is used to construct a closed-loop correction mechanism with the control module based on the angle information of each joint collected by the six-axis collaborative robotic arm.

5. The robotic automatic labeling system for complex industrial environments according to claim 1, characterized in that, The depth camera is used to acquire multiple sets of chessboard calibration board images from different orientations and distances; the intrinsic parameter calibration and error correction unit is used to solve the intrinsic parameter matrix of the depth camera based on the chessboard calibration board images using the Zhang Zhengyou calibration method and the OpenCV vision library.

6. The robotic automatic labeling system for complex industrial environments according to claim 1, characterized in that, The pose solving subunit is used to perform voxel mesh downsampling and statistical outlier removal on the 3D point cloud data corresponding to the depth camera after optimization. Then, the least squares method is used to perform quadratic surface fitting to obtain the normal vector at the center point of the fabric. Finally, a coordinate system is constructed by combining the optimal straight line of the product edge, and the 3D pose of the target image in the depth camera coordinate system is output.

7. The robotic automatic labeling system for complex industrial environments according to claim 1, characterized in that, The control module is connected to other modules via heterogeneous communication links.

8. The robotic automatic labeling system for complex industrial environments according to claim 1, characterized in that, The labeling actuator module includes a push rod, a vacuum suction cup, and a label adsorption platform. The end of the push rod is connected to the vacuum suction cup via a spring. The label adsorption platform is equipped with a linear support rounded corner and a waste paper recycling cylinder.

9. The robotic automatic labeling system for complex industrial environments according to claim 1, characterized in that, The program is based on the Ubuntu operating system.

10. A robotic automatic labeling method for use in complex industrial environments, characterized in that, Includes the following steps: S1: Acquire multiple sets of chessboard calibration board images and target object images; S2: Use the chessboard calibration board image to calibrate the intrinsic parameters of the depth camera, and then combine the target image with the dynamic distortion correction algorithm to correct the image; S3: The target image corrected in S2 is preprocessed using Gaussian filtering for noise reduction and HSV color enhancement. Then, the preprocessed target image is processed using U... 2 -Net semantic segmentation network performs region identification and then Canny operator edge detection to determine the boundary; S4: Based on the output of S3, the three-dimensional point cloud data of the target object image is calculated by back projection based on the depth value and camera intrinsic parameters; S5: Solve the three-dimensional point cloud data of the target object image to obtain the three-dimensional pose of the target image in the depth camera coordinate system, and then update the process noise covariance and observation noise covariance of the three-dimensional pose of the target image through Kalman filtering, and output the optimized three-dimensional pose of the target image. S6: Converts the output of S5 into a transformation matrix between the coordinate system of the labeling actuator module and the coordinate system of the target object, and then integrates the PID parameter self-tuning algorithm and Kalman filtering to output the motion speed command of the six-axis collaborative robotic arm. S7: Apply labels according to the movement speed command of the six-axis collaborative robotic arm.

Images