Improved yolov8n and double-calibration visual detection-based heavy-load robot hoisting control method

Abstract

The application provides a heavy-load mechanical arm hoisting control method based on improved YOLOv8n and double-calibration visual detection, and belongs to the technical field of heavy-load mechanical arm hoisting control; in order to realize the recognition accuracy, real-time operation, millimeter-level positioning, dynamic obstacle avoidance and heavy-load safety protection of the mechanical arm during hoisting control, a 3D camera is installed at the extended end of the mechanical arm, color images and depth images of the working area are acquired in real time, a laser radar is installed at the base of the mechanical arm, and distance data of the workshop environmental obstacles are collected; the 3D camera and the laser radar send the collected information to an edge computing processor, the edge computing processor performs image processing, coordinate conversion and point cloud three-dimensional modeling in parallel through the built-in target detection model, and sends the processed data to an industrial computer; the industrial computer communicates with the control end of the mechanical arm based on the grabbing point information, and controls the movement of the mechanical arm based on the issued point instruction and trajectory correction data; the application is applied to mechanical arm hoisting control.

Description

Technical Field

[0001] This invention provides a heavy-duty robotic arm hoisting control method based on improved YOLOv8n and dual-calibration visual detection, belonging to the field of heavy-duty robotic arm hoisting control technology. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing, robotic arms are increasingly widely used in scenarios such as heavy-duty material handling and assembly. Traditional heavy-duty robotic arm grasping control mainly relies on teach programming or fixed-point control, which requires pre-setting the grasping position. The system lacks environmental perception and adaptive capabilities. In actual production environments, due to the randomness of the placement of materials to be handled and assembled, the diversity of their shapes, and the uncertainty of ambient lighting, traditional heavy-duty robotic arm control methods often fail to meet the needs of flexible production.

[0003] In recent years, with the development of computer vision and artificial intelligence technologies, vision-guided robotic arm positioning technology has gradually become a research hotspot. Existing vision-guided positioning systems mainly employ 2D vision detection or simple 3D vision detection schemes, but they still have many shortcomings in heavy-duty scenarios. Traditional 2D vision detection methods can only acquire planar images, lacking depth dimensions, and cannot construct 3D models of the weights, nor can they predict obstacles along the lifting path. Currently, common 3D vision detection solutions lack edge lightweighting optimization, resulting in large parameters in deep learning models, lag in embedded device operation, and an inability to output position data in real time. Furthermore, currently used heavy-duty robotic arms also suffer from the following problems in positioning and grasping heavy objects:

[0004] Low visual positioning accuracy: Traditional 2D vision cannot acquire depth information, resulting in inaccurate calculation of grasping points; existing camera calibration and hand-eye calibration methods have cumulative errors and cannot meet the requirements of heavy-duty precision grasping.

[0005] Low target recognition efficiency: Traditional image processing methods are not robust in complex industrial environments, and deep learning models are computationally intensive and difficult to run in real time on embedded devices;

[0006] Lack of dynamic obstacle avoidance capability: Conventional path planning uses the cargo lifting point, i.e. the center of gravity, as the trajectory planning. The three-dimensional environment of the scene, the three-dimensional spatial posture of the cargo, and the lifting operation path are not fully adapted and safed. There is a significant safety risk of heavy object collision.

[0007] The safety mechanism for heavy-duty hoisting is lacking: the commonly used vision, radar and robotic arm control lacks linkage emergency response capabilities, has no space limit protection, and the safety guarantee for autonomous operation throughout the entire process is insufficient.

[0008] Several related technologies have attempted to address the aforementioned issues. For example, patent CN109676635A discloses a vision-based robotic arm positioning and grasping method. This method acquires the 3D information of the target object using an RGB-D camera and utilizes a point cloud processing algorithm for target recognition and pose estimation. While this method can acquire depth information, its target recognition algorithm is based on traditional feature extraction methods, resulting in limited accuracy in complex industrial scenarios. Another patent, CN110948520A, proposes a deep learning-based robotic arm positioning and grasping pose detection method. This method uses a convolutional neural network to identify the target object and predict the grasping point. Although it employs deep learning technology, its camera calibration method is relatively simple, resulting in low hand-eye calibration accuracy and significant positioning errors in heavy-duty precision grasping scenarios. Finally, patent CN111890351A discloses an industrial robot vision guidance system that achieves 3D reconstruction and target localization through binocular vision. While this system has good 3D perception capabilities, its computational complexity is high, making it difficult to meet industrial real-time requirements, and it is not optimized for heavy-duty scenarios.

[0009] In summary, there is an urgent need to develop a hoisting control solution that balances recognition accuracy, real-time computation, millimeter-level positioning, dynamic obstacle avoidance, and heavy-load safety protection. Summary of the Invention

[0010] To address the technical problems existing in the background art, the present invention provides a heavy-duty robotic arm lifting control method based on improved YOLOv8n and dual-calibration vision detection, comprising the following control steps:

[0011] Step 1: Install a 3D camera at the extension end of the robotic arm to acquire color and depth images of the work area in real time, and install a LiDAR at the base of the robotic arm to collect distance data of obstacles in the workshop environment.

[0012] Step 2: Control the 3D camera and LiDAR to send the acquired information to the edge computing processor. The edge computing processor performs image processing, coordinate transformation, and point cloud 3D modeling in parallel through the built-in target detection model, and sends the processed data to the industrial control computer.

[0013] Step 3: The industrial control computer communicates with the robotic arm control terminal via the Modbus protocol based on the grasping point information, and controls the movement of the robotic arm based on the issued point commands and trajectory correction data;

[0014] Step 4: The industrial control computer sends the robotic arm's movement data to the display, enabling manual control, task assignment, and operation monitoring.

[0015] The target detection model used in step 2 is specifically the YOLOv8n model, which improves the network architecture through multi-module collaborative improvement and loss function optimization. The multi-module collaborative improvement of the network architecture adopts a hierarchical collaborative optimization strategy with lightweight depth-separable convolution, local deformable convolution distortion correction, and full-process CA coordinate attention anti-interference weighting.

[0016] The specific method for lightweight full-network depthwise separable convolution is as follows:

[0017] All conventional convolutions in the YOLOv8n backbone and neck network were replaced and modified, and the original standard CBS units and the two-level standard convolutions inside the Bottleneck were uniformly replaced with DW-CBS depth-separable convolution units.

[0018] The DW-CBS depthwise separable convolutional unit decouples the spatial convolution and channel fusion operations of traditional convolution, dividing them into two levels of operations: depthwise convolution and pointwise convolution.

[0019] Deep convolution independently completes single-channel spatial texture feature extraction, adapting to the acquisition of basic workpiece features under alternating light and dark environments;

[0020] Pointwise 1×1 convolution enables cross-channel information fusion, supplementing channel interaction capabilities;

[0021] When simultaneously modifying the Bottleneck module of the network, the original residual topology and short-circuit connection logic of Bottleneck are retained, and only the two-level convolution inside the module is replaced with DW-CBS.

[0022] The specific method for local deformable convolution distortion correction is as follows:

[0023] A key layer local embedding deformable convolution strategy is adopted, with the embedding position limited to: after the output of the SPPF-DW module at the end of the backbone network and before the input of the PAN neck network. Only a small number of deformable convolution modules are embedded, and the entire network is not replaced.

[0024] The Deformable Convolution Deform-Conv strategy is adopted to predict the X and Y two-dimensional offsets of each sampling point through the pre-offset branch, and complete adaptive dynamic sampling by combining bilinear interpolation to fit the distorted contour and incomplete shape of the workpiece.

[0025] The specific full-process CA coordinate attention anti-interference weighting strategy is as follows:

[0026] CA coordinate attention modules are embedded layer by layer at the output of each C2f-DW transformation module in the backbone network. The CA coordinate attention module integrates channel dimension weight encoding and spatial position coordinate encoding to capture workpiece spatial position information.

[0027] The optimization and improvement of the loss function includes:

[0028] The original YOLOv8n loss function defines the total loss as a weighted sum of three parts: classification loss, confidence loss, and bounding box regression loss. The formula for calculating the total loss of the original YOLOv8n loss function is as follows:

[0029] ;

[0030] To replace the CIoU bounding box loss in the original YOLOv8n loss function, a hybrid NWD-CIoU loss function is adopted, while optimizing the weight ratio of each loss to adapt to workpiece detection scenarios with lighting distortion.

[0031] When replacing the original CIoU bounding box loss with NWD+CIoU weighted fusion loss, the Gauss-Walsh distance (NWD) loss is introduced to compensate for the insensitivity of IoU to non-overlapping and distorted targets, while retaining the convergence advantage of CIoU distance, including:

[0032] The CIoU loss is calculated using the following formula:

[0033] ;

[0034] in: Euclidean distance from the center point The distance is the diagonal of the circumscribed rectangle. The aspect ratio consistency coefficient;

[0035] The NWD Gauss-Walsh distance loss is calculated using the following formula:

[0036] ;

[0037] ;

[0038] The fusion improvement loss is calculated using the following formula:

[0039] ;

[0040] Among them, hyperparameters , ;

[0041] The total loss of the improved YOLOv8n loss function is calculated using the following formula:

[0042] .

[0043] Step 2, the coordinate transformation, specifically employs the Zhang Zhengyou and Tsai-Lenz dual-calibration coordinate transformation method, including:

[0044] Using the intrinsic parameters of the Zhang Zhengyou camera for calibration, the robotic arm drives the end-effector 3D camera to acquire 20 sets of chessboard calibration board images under different pitch and rotation poses. The sub-pixel coordinates of the chessboard corner points in each image are extracted, and a set of corresponding equations between pixel coordinates and calibration board world coordinates is established. The correlation coefficient is solved by the least squares method to eliminate the depth calculation error caused by lens deformation and establish a precise pixel-camera coordinate system mapping relationship.

[0045] The Tsai-Lenz hand-eye calibration method was used to perform the camera coordinate system to robot arm base coordinate system transformation, including:

[0046] First, control the robotic arm to switch between 20 sets of non-repeating end-effector TCP poses, and under each pose, the 3D camera will capture images of the fixed calibration plate.

[0047] For each set of data, obtain two sets of transformation matrices, including the camera relative to the calibration board extrinsic parameter matrix. The relative end-effector TCP matrix of the robotic arm base ;

[0048] Constructing a standard hand-eye equation The homogeneous transformation matrix is ​​solved using Tsai's two-step method. This will yield the rotation matrix representing the camera relative to the end effector of the robotic arm. Translation vector ;

[0049] Coordinate transformation is performed using a base coordinate system; the transformation formula is as follows:

[0050] ;

[0051] in Using the camera's three-dimensional coordinate system, the center of the hoisted object is determined. Three-dimensional hoisting reference point in the coordinate system of the robotic arm.

[0052] The specific method for performing point cloud 3D modeling in step 2 is as follows:

[0053] The 3D camera outputs a pixel depth map, which is matched with the bounding box of the suspended object recognized by YOLO. The effective depth point cloud within the bounding box is extracted, and invalid background points are removed.

[0054] Based on the calibration intrinsic parameter matrix, the two-dimensional depth points are converted into a three-dimensional point set in the camera coordinate system. Then, pass-through filtering is used to remove noise points at ultra-far / ultra-near distances, and statistical filtering is used to remove isolated noise points.

[0055] The random sampling consensus algorithm is used to fit each surface of the hoisting object and solve for information such as the three-dimensional coordinates of the hoisting object's center, the hoisting object's height, length, and thickness.

[0056] By fusing the 3D model of the hoisted object with the point cloud of the overall workshop environment collected by LiDAR, an integrated 3D map of the workshop space containing the heavy object and obstacles is generated.

[0057] Step 3 specifically controls the movement of the robotic arm based on hoisting path planning and dynamic obstacle avoidance algorithms, including:

[0058] Using the starting point of weight hoisting and the ending point of unloading as constraints, an initial hoisting path is generated based on the path planning algorithm, and a safety lifting threshold for the hook is forcibly set to avoid ground tool protrusions.

[0059] The lidar provides real-time dynamic obstacle avoidance correction. The lidar is set to output a 360° environmental ranging data every 50ms. If a new obstacle is detected within 1.2m of the path, local path replanning is immediately initiated.

[0060] The working condition constraint optimization involves smoothing the planned trajectory through interpolation, limiting the maximum acceleration and deceleration of the robotic arm, and suppressing inertial swaying during cargo transfer.

[0061] Define a dedicated space limit frame for hoisting; automatically reduce speed if the hoisting exceeds the boundary.

[0062] The movement trajectory is corrected in real time. The industrial control computer continuously compares the planned trajectory with the actual TCP coordinates of the robotic arm. When the coordinate deviation exceeds 2mm, the computer corrects the movement point by point and issues motion commands to prevent the goods from deviating or colliding.

[0063] The present invention has the following advantages over the prior art:

[0064] I. Improved Hardware Architecture: This invention constructs an integrated safety hoisting system that combines a 3D end-effector camera, a base-mounted LiDAR, an edge AI development kit, and a heavy-duty robotic arm. It employs dual visual data sources to achieve 3D modeling of heavy objects and global obstacle avoidance, and is equipped with a hardware-connected tiered emergency stop module to adapt to the transfer operations of hoisted objects of corresponding dimensions. This achieves integration of visual perception, environmental detection, and the hardware of the actuators, improving the overall hardware integration of the hoisting system.

[0065] II. Improvements to the target detection algorithm:

[0066] This invention structurally improves the network, employing depthwise separable convolutions in the backbone for lightweight transformation, effectively reducing the number of model parameters and computational load while maintaining the original lightweight characteristics of YOLOv8n, making it adaptable to the interference conditions of alternating light and dark environments in industrial settings. Deformable convolutions are locally embedded in key layers, with only a limited increase in computational overhead, effectively correcting contour distortion and positioning offset caused by lighting distortion. A lightweight CA coordinate attention module is embedded in the backbone layers, introducing almost no increase in latency, strengthening weak features of distorted workpieces, constraining spatial positional deviations, and synergistically improving the recall rate and positioning accuracy of workpieces under complex conditions. Simultaneously, this invention optimizes the loss function, overcoming the shortcomings of traditional IoU loss in regressing workpieces with lighting distortion, partial occlusion, and incomplete contours. The loss function is adapted to the adaptive distortion features extracted by deformable convolutions, effectively reducing the positioning offset error of distorted workpieces, enhancing the learning ability of difficult samples under complex lighting conditions, and reducing the probability of missed and false detections. The loss function weights are adapted to the lightweight network, without increasing the model training burden, and there is no additional computational overhead during the model inference stage, meeting the real-time detection requirements of edge devices.

[0067] III. Improvement of hierarchical linkage safety control: This invention realizes real-time monitoring of three data sources: vision, radar, and robotic arm. It forms a dual protection through software limit speed reduction and machine stop and physical hardware emergency stop, and builds a multi-layer safety protection mechanism adapted to heavy-duty hoisting, effectively avoiding collision risks in heavy-duty operations and improving the safety and operational reliability of heavy-duty robotic arm hoisting operations. Attached Figure Description

[0068] The present invention will be further described below with reference to the accompanying drawings:

[0069] Figure 1 This is a schematic diagram of the structure of the heavy-duty robotic arm hoisting control system based on the improved YOLOv8n and dual-calibration vision detection of the present invention;

[0070] Figure 2 This is a flowchart of the steps of the heavy-duty robotic arm hoisting control method based on the improved YOLOv8n and dual-calibration vision detection of the present invention;

[0071] Figure 3 This is a schematic diagram of the structure of the DW-CBS module used in an embodiment of the present invention;

[0072] Figure 4 This is a schematic diagram of the Deform-Conv module used in an embodiment of the present invention;

[0073] Figure 5 This is a schematic diagram of the CA coordinate attention module used in an embodiment of the present invention. Detailed Implementation

[0074] like Figure 1 and Figure 2 As shown, this invention provides a heavy-duty robotic arm lifting control scheme based on improved YOLOv8n and dual-calibrated visual detection. Specifically, it employs machine vision, lightweight deep learning object detection, Zhang Zhengyou and Tsai-Lenz dual calibration, 3D point cloud modeling, autonomous planning of heavy-duty lifting paths, and real-time obstacle avoidance safety control technology. It is specifically designed for unmanned and high-safety lifting and transportation scenarios, integrating edge AI computing and multi-axis heavy-duty robotic arm collaborative control to achieve autonomous detection and positioning, path planning, and active obstacle avoidance throughout the entire process of heavy object lifting.

[0075] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a hoisting and positioning method that integrates 3D vision, LiDAR calibration, lightweight YOLOv8n improved network, dual calibration algorithm, 3D point cloud modeling, heavy-load dedicated path planning, dynamic obstacle avoidance and graded safety shutdown. This method enables autonomous, accurate and safe transfer of hoisted objects, retaining only two manual operations: hooking and unhooking, while the rest of the process is fully automated.

[0076] To achieve the above functions, the system hardware architecture adopted in this invention is as follows:

[0077] The system mainly consists of a 3D camera, LiDAR, edge computing AI development kit, industrial control computer, multi-axis heavy-duty robotic arm, emergency stop control module, and human-machine interface. The 3D camera, an Orbbec Gemini 435Le model, is mounted at the end of the heavy-duty robotic arm to acquire real-time color and depth images of the work area. The Airy LiDAR, mounted on the robotic arm base, can perform 360° horizontal and 90° vertical scanning, collecting real-time distance data to obstacles in the workshop environment as a data source for coordinate calibration and obstacle avoidance. The edge computing AI development kit, equipped with a high-performance processor and neural network acceleration unit, is used to deploy a deep learning object detection model, performing image processing, object detection, and point cloud 3D modeling in parallel. The industrial control computer, as the system's central control unit, receives the grasping point information calculated by the vision module and communicates with the multi-axis heavy-duty robotic arm via the Modbus protocol to complete point location distribution and trajectory correction. The emergency stop module is directly connected to the robotic arm controller and vision system; upon manual triggering, it immediately locks all motion axes of the robotic arm and cuts off power in stages.

[0078] In target detection, this invention improves the YOLOv8n model in two core ways: multi-module collaborative improvement of the network architecture and optimization of the loss function. All improvements are designed around the industrial deployment requirements of extreme lightweighting, no additional latency burden, strong resistance to light and shadow interference, and high-precision positioning. The structural improvements to the network architecture employ a hierarchical collaborative optimization strategy: lightweighting with fully depthwise separable convolutions + locally deformable convolutional distortion correction + full-process CA coordinate attention weighting to resist interference. This strategy does not compromise the native YOLOv8n network topology and inference speed, and specifically addresses the challenge of detection under complex lighting conditions.

[0079] like Figure 3 As shown, in order to adapt to the real-time inference requirements of edge devices, this invention adopts a lightweight modification of the entire network depth separable convolution DW-CBS. Specifically, all conventional convolutions in the YOLOv8n backbone network and neck network are replaced and modified, and the original standard CBS (Conv+BN+SiLU) units and the two-level standard convolutions inside Bottleneck are uniformly replaced with DW-CBS depth separable convolution units.

[0080] The depthwise separable convolution used in this invention decouples the spatial convolution and channel fusion operations of traditional convolution, and divides them into two levels of operations: depthwise convolution and pointwise convolution. Depthwise convolution independently completes the extraction of single-channel spatial texture features, which is suitable for the acquisition of basic features of workpieces in alternating light and dark environments. Pointwise 1×1 convolution realizes cross-channel information fusion and completes the channel interaction capability.

[0081] Compared to standard convolution, the above improvements significantly reduce the number of parameters and computational cost, fully retaining the lightweight attributes of YOLOv8n without additional computational overhead; multi-channel independent extraction of texture features provides strong illumination stability, making it suitable for industrial environments with alternating light and dark interference; simultaneously, this invention modifies the Bottleneck network module: retaining the original residual topology and short-circuit connection logic of Bottleneck, only replacing the two-level convolution within the module with DW-CBS, achieving global lightweighting while ensuring stable gradient backpropagation.

[0082] like Figure 4As shown, to address the shortcomings of depthwise separable convolution in adapting to workpiece contours distorted by light and shadow, and to avoid the problems of increased computing power and excessive inference latency caused by replacing deformable convolution with a full network, this invention adopts a strategy of locally embedding deformable convolutions in key layers. The embedding position is strictly limited to: after the output of the SPPF-DW module at the end of the backbone network and before the input of the PAN neck network. Only a small number of deformable convolution modules are embedded, without replacing the entire network. The deformable convolution strategy adopted abandons the traditional fixed grid sampling mode of convolution. It predicts the X and Y two-dimensional offsets of each sampling point through a pre-offset branch, and completes adaptive dynamic sampling by combining bilinear interpolation. It can accurately fit the distorted contours and incomplete shapes of workpieces caused by strong light overexposure, dark light shadows, and surface reflections.

[0083] This method, through local fixed-point deployment, has a very small increase in computing power and hardly affects the real-time inference speed of edge devices. It adopts special correction for target contour distortion and positioning offset caused by illumination distortion, which greatly improves the positioning accuracy of deformed workpieces and can adapt to complex working conditions where the shape of industrial hoisting workpieces is not fixed and the light and shadow interference changes randomly.

[0084] like Figure 5 As shown, in order to solve the problem of complex backgrounds and light and shadow noise obscuring the effective features of the workpiece, this invention embeds CA coordinate attention modules layer by layer at the output end of each C2f-DW transformation module of the backbone network. The CA coordinate attention module integrates channel dimension weight encoding and spatial position coordinate encoding capabilities, which can accurately capture the spatial position information of the workpiece, adaptively enhance the effective features of the target, and suppress shadows, reflections, and invalid noise from cluttered backgrounds.

[0085] Based on this embedding strategy, the module is extremely lightweight, with no significant increase in parameters or latency after embedding, making it suitable for edge deployment. Through multi-level feature weighting optimization, weak features of distorted workpieces are strengthened, improving target recall under complex lighting conditions. Furthermore, by constraining spatial position deviations and combining with deformable convolution, the accuracy of workpiece positioning is further improved.

[0086] The optimizations and improvements made to the loss function in this invention include:

[0087] To address the shortcomings of the original YOLOv8n loss function: the total loss of the original YOLOv8n is composed of three weighted components: classification loss, confidence loss, and bounding box regression loss, calculated as follows:

[0088] ;

[0089] The shortcomings of native loss in industrial scenarios include: native CIoU loss ( It only focuses on distance and overlap, which is insufficient for constraining difficult-to-regress workpieces with light and shadow distortion and local missing parts; it has poor adaptability to the problem of incomplete target feature samples and imbalance of positive and negative samples caused by shadow occlusion and reflection; it lacks constraints on angle and shape consistency, and the predicted box of distorted workpieces is prone to offset, drift and poor fit.

[0090] To address the aforementioned shortcomings, this invention replaces the original CIoU bounding box loss with a hybrid NWD-CIoU loss function, while simultaneously optimizing the weight ratio of the three types of losses to adapt to workpiece detection scenarios with illumination distortion. Specifically:

[0091] When replacing the original CIoU bounding box loss with NWD+CIoU weighted fusion loss, the main focus is on the failure of IoU calculation for distorted, incomplete, and shadow-occluded workpieces. Gauss-Walsh distance NWD loss is introduced to compensate for the insensitivity of IoU to non-overlapping and distorted targets, while retaining the convergence advantage of CIoU distance. The specific process is as follows:

[0092] The CIoU loss is calculated using the following formula:

[0093] ;

[0094] in: Euclidean distance from the center point The distance is the diagonal of the circumscribed rectangle. The aspect ratio consistency coefficient;

[0095] The NWD Gauss-Walsh distance loss is calculated using the following formula:

[0096] ;

[0097] ;

[0098] The present invention incorporates improved loss calculation formula:

[0099] ;

[0100] Among them, hyperparameters , .

[0101] The improved principle of the above calculation is as follows: for normal and clear workpieces, the overlap of the bounding box and the center distance are quickly converged by CIoU; for workpieces with shadow occlusion, reflection distortion, and incomplete contours, the IoU failure problem is avoided by using NWD Gaussian distribution measurement, thereby improving the regression accuracy of difficult samples; and the distortion features extracted by the deformable convolution of this invention are adapted to achieve deep matching between features and loss function.

[0102] When optimizing the classification loss and confidence loss, the original BCE classification and BCE confidence loss are retained. The loss weights are fine-tuned for complex lighting scenarios, and the confidence supervision weights for difficult samples are increased to suppress false detections of reflective backgrounds and missed detections of shadow targets, so that the model can maintain high confidence output for workpieces with weak features.

[0103] The final total loss of the improved YOLOv8n loss function is calculated using the following formula:

[0104] .

[0105] The above-mentioned loss function improvement strategy solves the problem of traditional IoU failing to regress workpieces with light and shadow distortion, local occlusion, and incomplete contours. By matching deformable convolution adaptive distortion features, the positioning offset error of distorted workpieces is greatly reduced. At the same time, the learning ability of difficult samples with complex lighting is enhanced, significantly reducing the false negative rate and false positive rate. The weight ratio is adapted to the lightweight network, which does not increase the training burden and has no additional computation during the inference stage.

[0106] To eliminate lens distortion, this invention employs Zhang Zhengyou's camera intrinsic parameter calibration method. First, a heavy-duty robotic arm is controlled to drive an end-effector 3D camera to acquire 20 sets of chessboard calibration board images under different pitch and rotation poses. Then, the sub-pixel coordinates of the chessboard corner points in each image are extracted, and a set of corresponding equations between pixel coordinates and calibration board world coordinates is established. Finally, the correlation coefficient is solved using the least squares method, thereby eliminating the depth calculation error caused by lens distortion and establishing a precise pixel-to-camera coordinate system mapping relationship.

[0107] The next step will be to use Tsai-Lenz hand-eye calibration to complete the transformation from the camera coordinate system to the robotic arm's base coordinate system. First, the robotic arm will be controlled to switch between 20 sets of non-repeating end-effector TCP poses. Under each pose, the 3D camera will capture images of a fixed high-precision calibration board. Next, for each set of data, two transformation matrices will be obtained: the camera's extrinsic parameter matrix relative to the calibration board. The relative end-effector TCP matrix of the robotic arm base Then construct the standard hand-eye equation. The homogeneous transformation matrix is ​​solved using Tsai's two-step method. This will yield the rotation matrix representing the camera relative to the end effector of the robotic arm. Translation vector Finally, coordinate transformation is performed using the base coordinate system: (in The center of the hoisted object is obtained from the three-dimensional coordinates of the camera coordinate system. Three-dimensional hoisting reference point in the coordinate system of the robotic arm.

[0108] This invention, when performing 3D point cloud modeling for hoisted objects, primarily uses a pixel depth map output from a 3D camera, matches it with the bounding box of the hoisted object recognized by YOLO, extracts the effective depth point cloud within the bounding box, and removes invalid background points. Based on the calibration intrinsic parameter matrix, the 2D depth points are converted into a 3D point set in the camera coordinate system. Pass-through filtering is then used to remove noise points at extremely far / near distances, and statistical filtering is used to remove isolated noise points. A random sampling consensus algorithm is used to fit each surface of the hoisted object, solving for information such as the 3D coordinates of the hoisted object's center, its height, length, and thickness. Finally, the 3D model of the hoisted object is fused with the global environmental point cloud of the workshop collected by LiDAR to generate an integrated 3D map of the workshop space containing heavy objects and obstacles.

[0109] The heavy-load dedicated hoisting path planning strategy and dynamic obstacle avoidance algorithm provided by this invention include:

[0110] Using the starting point of weight hoisting and the ending point of unloading as constraints, an initial hoisting path is generated based on a path planning algorithm. A safety lifting threshold for the hook is forcibly set (30cm above the load for safety) to avoid protrusions in the ground fixtures. Real-time dynamic obstacle avoidance correction is achieved using LiDAR, which outputs 360° environmental ranging data every 50ms. If a new obstacle is detected within 1.2m of the path, local path replanning is immediately initiated. Heavy-load condition constraint optimization involves smoothing the planned trajectory through interpolation, limiting the maximum acceleration and deceleration of the robotic arm, and suppressing inertial swaying during the transfer of a 100kg load. A dedicated space limit frame for hoisting is defined, and the speed is automatically reduced if the boundary is exceeded. Real-time trajectory correction logic involves the industrial control computer continuously comparing the planned trajectory with the actual TCP coordinates of the robotic arm. When the coordinate deviation exceeds 2mm, point-by-point corrections are made and motion commands are issued to prevent the load from deviating and colliding.

[0111] The Modbus communication and graded safety hoisting full-process control scheme adopted in this invention includes:

[0112] Communication and Interaction: The edge AI kit outputs the weight position and 3D model data to the industrial control computer; the industrial control computer acts as a Modbus master station and sends coordinate, attitude, speed, and acceleration commands to the heavy-duty robotic arm via RS485 / Ethernet;

[0113] Standardized hoisting: The system initializes by loading calibration parameters, AI model and surrounding environment map, and the robotic arm returns to zero; after manual hooking, the equipment automatically completes image acquisition, target detection, 3D modeling, and path planning, and the robotic arm lifts and transports the heavy object; the LiDAR scans and avoids obstacles in real time throughout the process, stops when it reaches the unloading point, and automatically returns to the standby position when the hook is manually released;

[0114] Multi-level safety protection: Automatic speed reduction and shutdown when exceeding the lifting limit area; immediate locking of movement when the tilt angle of the lifted object is too large; physical emergency stop buttons are set up on site, which directly cut off the power shaft of the robotic arm when pressed, and the vision system stops the trajectory output simultaneously; the display shows the three-dimensional model of the weight, the lifting trajectory, and the distribution of radar obstacles in real time, and pop-up sound and light warnings for abnormal status.

[0115] This invention employs a lightweight adaptive distortion-aware network architecture, performing a lightweight modification of the entire YOLOv8n network using depthwise separable convolutions to ensure low computational overhead and lighting stability. Deformable convolutions are embedded in key network layers, and a lightweight CA coordinate attention module is integrated into the backbone layer. This adaptively corrects workpiece contour distortion and positional deviations, making it suitable for complex lighting and irregular workpiece detection scenarios in industry, balancing inference speed and positioning accuracy. Furthermore, an optimized loss function adapted to distorted workpieces is used, overcoming the failure of traditional IoU loss regression. A dedicated loss function is designed to match the specific workpiece shape. The adaptive distortion features of deformable convolution effectively reduce the positioning error of distorted workpieces, enhance the learning ability of difficult samples with complex lighting and occlusion, reduce the false negative and false positive rates, and are compatible with lightweight networks without additional training and inference overhead. In addition, this invention also adopts a multi-linkage hierarchical safety protection and control mechanism. By building a real-time monitoring system with triple data linkage of vision, radar and robotic arm, combined with a dual protection structure of software limit speed reduction and physical hardware emergency stop, a multi-level safety protection mechanism is constructed, which is effectively adapted to heavy-load hoisting conditions and can significantly improve the safety and operational stability of heavy-load hoisting operations.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A heavy-duty robotic arm lifting control method based on improved YOLOv8n and dual-calibration vision inspection, characterized in that: The control steps include the following: Step 1: Install a 3D camera at the extension end of the robotic arm to acquire color and depth images of the work area in real time, and install a LiDAR at the base of the robotic arm to collect distance data of obstacles in the workshop environment. Step 2: Control the 3D camera and LiDAR to send the acquired information to the edge computing processor. The edge computing processor performs image processing, coordinate transformation, and point cloud 3D modeling in parallel through the built-in target detection model, and sends the processed data to the industrial control computer. Step 3: The industrial control computer communicates with the robotic arm control terminal via the Modbus protocol based on the grasping point information, and controls the movement of the robotic arm based on the issued point commands and trajectory correction data; Step 4: The industrial control computer sends the robotic arm's movement data to the display, enabling manual control, task assignment, and operation monitoring.

2. The heavy-duty robotic arm lifting control method based on improved YOLOv8n and dual-calibration vision detection according to claim 1, characterized in that: The target detection model used in step 2 is specifically the YOLOv8n model, which improves the network architecture through multi-module collaborative improvement and loss function optimization. The multi-module collaborative improvement of the network architecture adopts a hierarchical collaborative optimization strategy with lightweight depth-separable convolution, local deformable convolution distortion correction, and full-process CA coordinate attention anti-interference weighting.

3. The heavy-duty robotic arm hoisting control method based on improved YOLOv8n and dual-calibration vision detection according to claim 2, characterized in that: The specific method for lightweight full-network depthwise separable convolution is as follows: All conventional convolutions in the YOLOv8n backbone and neck network were replaced and modified, and the original standard CBS units and the two-level standard convolutions inside the Bottleneck were uniformly replaced with DW-CBS depth-separable convolution units. The DW-CBS depthwise separable convolutional unit decouples the spatial convolution and channel fusion operations of traditional convolution, dividing them into two levels of operations: depthwise convolution and pointwise convolution. Deep convolution independently completes single-channel spatial texture feature extraction, adapting to the acquisition of basic workpiece features under alternating light and dark environments; Pointwise 1×1 convolution enables cross-channel information fusion, supplementing channel interaction capabilities; When simultaneously modifying the Bottleneck module of the network, the original residual topology and short-circuit connection logic of Bottleneck are retained, and only the two-level convolution inside the module is replaced with DW-CBS.

4. The heavy-duty robotic arm hoisting control method based on improved YOLOv8n and dual-calibration vision detection according to claim 2, characterized in that: The specific method for local deformable convolution distortion correction is as follows: A key layer local embedding deformable convolution strategy is adopted, with the embedding position limited to: after the output of the SPPF-DW module at the end of the backbone network and before the input of the PAN neck network. Only a small number of deformable convolution modules are embedded, and the entire network is not replaced. The Deformable Convolution Deform-Conv strategy is adopted to predict the X and Y two-dimensional offsets of each sampling point through the pre-offset branch, and complete adaptive dynamic sampling by combining bilinear interpolation to fit the distorted contour and incomplete shape of the workpiece.

5. The heavy-duty robotic arm hoisting control method based on improved YOLOv8n and dual-calibration vision detection according to claim 2, characterized in that: The specific full-process CA coordinate attention anti-interference weighting strategy is as follows: CA coordinate attention modules are embedded layer by layer at the output of each C2f-DW transformation module in the backbone network. The CA coordinate attention module integrates channel dimension weight encoding and spatial position coordinate encoding to capture workpiece spatial position information.

6. The heavy-duty robotic arm lifting control method based on improved YOLOv8n and dual-calibration vision detection according to claim 2, characterized in that: The optimization and improvement of the loss function includes: The original YOLOv8n loss function defines the total loss as a weighted sum of three parts: classification loss, confidence loss, and bounding box regression loss. The formula for calculating the total loss of the original YOLOv8n loss function is as follows: ； To replace the CIoU bounding box loss in the original YOLOv8n loss function, a hybrid NWD-CIoU loss function is adopted, while optimizing the weight ratio of each loss to adapt to workpiece detection scenarios with lighting distortion. When replacing the original CIoU bounding box loss with NWD+CIoU weighted fusion loss, the Gauss-Walsh distance (NWD) loss is introduced to compensate for the insensitivity of IoU to non-overlapping and distorted targets, while retaining the convergence advantage of CIoU distance, including: The CIoU loss is calculated using the following formula: ； in: Euclidean distance from the center point The distance is the diagonal of the circumscribed rectangle. The aspect ratio consistency coefficient; The NWD Gauss-Walsh distance loss is calculated using the following formula: ； ； The fusion improvement loss is calculated using the following formula: ； Among them, hyperparameters , ; The total loss of the improved YOLOv8n loss function is calculated using the following formula: 。 7. The heavy-duty robotic arm lifting control method based on improved YOLOv8n and dual-calibration vision detection according to claim 2, characterized in that: Step 2, the coordinate transformation, specifically employs the Zhang Zhengyou and Tsai-Lenz dual-calibration coordinate transformation method, including: Using the intrinsic parameters of the Zhang Zhengyou camera for calibration, the robotic arm drives the end-effector 3D camera to acquire 20 sets of chessboard calibration board images under different pitch and rotation poses. The sub-pixel coordinates of the chessboard corner points in each image are extracted, and a set of corresponding equations between pixel coordinates and calibration board world coordinates is established. The correlation coefficient is solved by the least squares method to eliminate the depth calculation error caused by lens deformation and establish a precise pixel-camera coordinate system mapping relationship. The Tsai-Lenz hand-eye calibration method was used to perform the camera coordinate system to robot arm base coordinate system transformation, including: First, control the robotic arm to switch between 20 sets of non-repeating end-effector TCP poses, and under each pose, the 3D camera will capture images of the fixed calibration plate. For each set of data, obtain two sets of transformation matrices, including the camera-to-calibration-board extrinsic parameter matrix. The relative end-effector TCP matrix of the robotic arm base ; Constructing a standard hand-eye equation The homogeneous transformation matrix is ​​solved using Tsai's two-step method. This will yield the rotation matrix representing the camera relative to the end effector of the robotic arm. Translation vector ; Coordinate transformation is performed using a base coordinate system; the transformation formula is as follows: ； in Using the camera's three-dimensional coordinate system, the center of the hoisted object is determined. Three-dimensional hoisting reference point in the coordinate system of the robotic arm.

8. The heavy-duty robotic arm lifting control method based on improved YOLOv8n and dual-calibration vision detection according to claim 7, characterized in that: The specific method for performing point cloud 3D modeling in step 2 is as follows: The 3D camera outputs a pixel depth map, which is matched with the bounding box of the suspended object recognized by YOLO. The effective depth point cloud within the bounding box is extracted, and invalid background points are removed. Based on the calibration intrinsic parameter matrix, the two-dimensional depth points are converted into a three-dimensional point set in the camera coordinate system. Then, pass-through filtering is used to remove noise points at ultra-far / ultra-near distances, and statistical filtering is used to remove isolated noise points. The random sampling consensus algorithm is used to fit each surface of the hoisting object and solve for information such as the three-dimensional coordinates of the hoisting object's center, the hoisting object's height, length, and thickness. By fusing the 3D model of the hoisted object with the point cloud of the overall workshop environment collected by LiDAR, an integrated 3D map of the workshop space containing the heavy object and obstacles is generated.

9. The heavy-duty robotic arm lifting control method based on improved YOLOv8n and dual-calibration vision detection according to claim 8, characterized in that: Step 3 specifically controls the movement of the robotic arm based on hoisting path planning and dynamic obstacle avoidance algorithms, including: Using the starting point of weight hoisting and the ending point of unloading as constraints, an initial hoisting path is generated based on the path planning algorithm, and a safety lifting threshold for the hook is forcibly set to avoid ground tool protrusions. The lidar provides real-time dynamic obstacle avoidance correction. The lidar is set to output a 360° environmental ranging data every 50ms. If a new obstacle is detected within 1.2m of the path, local path replanning is immediately initiated. The working condition constraint optimization involves smoothing the planned trajectory through interpolation, limiting the maximum acceleration and deceleration of the robotic arm, and suppressing inertial swaying during cargo transfer. Define a dedicated space limit frame for hoisting; automatically reduce speed if the hoisting exceeds the boundary. The movement trajectory is corrected in real time. The industrial control computer continuously compares the planned trajectory with the actual TCP coordinates of the robotic arm. When the coordinate deviation exceeds 2mm, the computer corrects the movement point by point and issues motion commands to prevent the goods from deviating or colliding.

Images