A real-time grabbing method for semi-disordered multi-pile castings in a restricted material frame based on an RGB-D camera

Abstract

This invention discloses a real-time grasping method for semi-disordered multi-stack castings within a confined space based on an RGB-D camera. The method includes: acquiring RGB images, depth frames, and point cloud data of the castings using an RGB-D camera; after image enhancement and annotation, training a model based on an improved YOLOv8 model to obtain a concentric outer circle detection model for the castings; achieving target tracking by combining an improved StrongSORT and NSA Kalman filter; extracting the center point of the concentric outer circle based on the detection results; obtaining three-dimensional measurement data of the castings through point cloud mapping and statistical filtering; fusing camera IMU acceleration and angular velocity information using complementary filtering; dynamically fusing point cloud and camera IMU data through an adaptive unscented Kalman filter fusion processor to obtain updated three-dimensional position and attitude data of the castings; and transmitting the data to a robotic arm control system via a PLC to achieve real-time and precise grasping of the castings. This invention enables high-precision identification and positioning of castings in confined environments, improving the stability and real-time performance of multi-stack grasping.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a method for real-time grasping of semi-disordered multi-stack castings within a confined material frame based on an RGB-D camera. Background Technology

[0002] Casting is a crucial component manufacturing technology in the aerospace, automotive, and construction machinery industries. However, the casting process inevitably introduces various defects such as porosity, sand holes, and cracks. For small and medium-sized enterprises, removing these defects relies heavily on manual labor. However, manual defect removal suffers from inconsistent quality and low efficiency. Therefore, adopting vision-assisted grasping technology is an inevitable choice.

[0003] Currently, vision-assisted grasping mainly uses RGB-D cameras for assisted grasping. The main methods for grasping and localization include point cloud registration, fusion calibration using depth maps, target recognition and prediction grasping using neural networks, and localization methods such as matching point clouds with CAD model point clouds.

[0004] For example, Chinese patent application number 202210611118.2 uses an RGB-D camera to segment and process the point cloud of the pipe segment. Then, point cloud smoothing and normal estimation are performed, the point clouds of the template pipe segment and the pipe segment to be grasped are registered, and the pose of the pipe segment to be grasped is identified and estimated to guide the assembly machine in grasping. Chinese patent application number 202110534173.1 uses a neural network to segment the image to obtain the 3D point cloud of the target workpiece, then matches it with the point cloud of a virtual CAD model to achieve accurate grasping. The two point cloud matching and registration methods provided in the above applications have some limitations, such as the need to prepare the CAD drawing of the object to be grasped in advance, and potential instability when dealing with complex environments or scenes with large changes.

[0005] For example, Chinese patent application number 202310829290.X uses a hybrid architecture of Transformer and CNN to construct a deep learning network, calibrating and fusing information from different modalities of RGB and Depth images to extract target feature information for grasping; Chinese patent application number 202010361914.6 utilizes deep learning for the positioning and grasping of irregular workpieces. Although the methods mentioned in the above two applications employ advanced intelligent deep learning networks to increase the accuracy of positioning and grasping, the complexity of the deep learning networks prevents the robotic arm from grasping actively and quickly, thus affecting the efficiency of automated production in factories. Summary of the Invention

[0006] This invention provides a real-time grasping method for semi-disordered multi-stack castings within a confined material frame based on an RGB-D camera, aiming to solve the problems of high computational complexity and large error that are common in traditional grasping methods.

[0007] To solve the above problems, the technical solution adopted by the present invention is as follows: A first aspect of the present invention provides a method for real-time grasping of semi-disordered multi-stacking castings within a confined material frame based on an RGB-D camera, comprising the following steps: Step S1: Acquire RGB images, depth frames, and point cloud data of the casting; preprocess and label the acquired data; use the processed data to train an improved YOLOv8 detection model to achieve high-precision recognition of concentric circle features of the casting, thus obtaining the concentric circle detection model of the casting. Step S2: Apply the concentric circle detection model obtained in step S1 to the real-time acquired casting image to detect the concentric circle region of the casting and extract its center point pixel coordinates; use the improved StrongSORT temporal tracking algorithm to perform continuous frame association on the center point to obtain stable center point temporal tracking results, reduce jitter and false detection, and improve detection accuracy. Step S3: Combine the center point pixel coordinates obtained in step S2 with the depth information, perform spatial back projection through the intrinsic parameter calibration parameters, and calculate the corresponding three-dimensional coordinates of the center point; extract the local point cloud region with the coordinates as the center, and perform voxel mesh downsampling and statistical outlier removal on the point cloud of the region to obtain refined local three-dimensional point cloud measurement data of the casting. Step S4: Use the IMU module built into the RGB-D camera to read the three-axis linear acceleration and angular velocity data; fuse the acceleration and gyroscope information through a complementary filtering algorithm to obtain a stable and reliable IMU attitude estimation result; Step S5: Input the refined 3D point cloud measurement data obtained in Step S3 and the IMU attitude estimation results in Step S4 into the Adaptive Unscented Kalman Filter (AUKF) fusion module. Calculate the high-precision pose of the casting by fusing multi-source information. During the fusion process, AUKF automatically adjusts the noise parameters based on the observation residuals to achieve adaptive optimization of the filtering parameters, thereby improving the accuracy of pose estimation and the robustness of the system. Output the final 3D pose data of the casting. Step S6: Transmit the three-dimensional pose data of the casting output in step S5 to the PLC control system. The motion planning module generates control instructions for the robotic arm, driving the robotic arm to perform real-time grasping operations on the casting within the confined material frame.

[0008] As one possible implementation of the first aspect of the present invention, an RGB-D camera is used to acquire RGB images, depth frames and point cloud data of the casting; the preprocessing operations include random rotation, scaling, brightness / contrast perturbation, occlusion synthesis and depth noise simulation of the image.

[0009] As one possible implementation of the first aspect of the present invention, in step S1, the concentric feature module detects the concentric circles of the casting by fitting an ellipse at the edge points within the bounding box using the least squares method.

[0010] As a possible implementation of the present invention, in step S1, a concentric feature module and a concentric loss module are integrated on the basis of the traditional YOLOv8 network structure to enhance the model's ability to detect concentric circle features of castings; the concentric feature module is embedded in the feature extraction stage of the YOLOv8 backbone network to enhance the extraction and representation of circular and concentric geometric features; the concentric loss module introduces a concentric constraint loss function in the model output stage to constrain the deviation between the predicted circle and the real circle in terms of center position and radius parameters, thereby improving the concentric circle detection accuracy.

[0011] As one possible implementation of the first aspect of the present invention, in step S1, the concentricity loss module consists of standard YOLOv8 loss and concentricity loss. L conc The model is composed of two components, in which the concentric loss is used to measure the deviation between the predicted circle and the real circle in terms of center coordinates and radius. The contribution ratio of the concentric loss to the main loss is adjusted by the weighting factor λ to achieve joint optimization of model parameters, thereby improving the fitting accuracy between the center point of the detection box and the center of the real concentric circle.

[0012] As a possible embodiment of the first aspect of the present invention, the method for calculating the three-dimensional coordinates of the corresponding center point is as follows: Step S3.1: Calculate the depth pixel coordinates of the concentric outer circle of the casting; Step S3.2: Map the point cloud onto the color frame and calculate the 3D point cloud in the depth frame; Step S3.3: Reshape the vertex data into a 3D array, extract the 3D camera coordinates of the pixels corresponding to the center of the bounding box, and return them as the final three-dimensional coordinates of the center point.

[0013] As one possible implementation of the first aspect of the present invention, step S5 includes the following specific steps: Step S5.1: Define the state transition formula and observation formula of the adaptive Kalman filter and obtain the Sigma point set and its weights; Step S5.2: Obtain the weighted average based on the Sigma points; Step S5.3: Dynamically adjust the covariance of process and observation noise based on the consistency of observations to dynamically suppress error amplification and ensure data stability and accuracy; Step S5.4: Based on Kalman gain K k To update the state and covariance of the depth camera; Step S5.5: Obtain the final updated 3D data.

[0014] In a second aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device enables the above-described real-time casting gripping method.

[0015] A third aspect of the present invention provides a computer-readable storage medium for storing a computer program that, when run on a computer, causes the computer to perform the above-described real-time casting gripping method.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention improves the YOLOv8 model by integrating the concentric feature module (CFM) and the concentric loss function (CFM). L conc This technology is specifically designed to improve the detection accuracy of concentric outer circles in semi-disordered multi-layer castings. CFM effectively improves target detection capabilities in complex scenarios by optimizing the fitting of ellipses using least squares optimization, solving the instability problem of traditional point cloud registration or CAD model matching methods in complex environments.

[0017] (2) This invention improves the StrongSORT algorithm by introducing a concentric similarity term into the association cost matrix. S conc This method employs Kalman filtering for motion prediction and the Hungarian algorithm for correlation, combining IoU and appearance cost to enhance the initial 2D tracking accuracy of concentric outer circular objects. This improvement allows the tracking process to better adapt to semi-random stacked scenarios, improving the robustness and accuracy of target matching and overcoming the limitations of traditional methods in dynamic scenes.

[0018] (3) This invention uses an adaptive unscented Kalman filter fusion converter, which dynamically adjusts the covariance of process and observation noise by fusing 3D measurement data from an RGB-D camera and IMU data, effectively suppressing the amplification of depth measurement errors caused by ground vibration and other environmental factors. This method significantly improves the stability and accuracy of depth measurement and reduces the risk of collision between the robotic arm and the casting.

[0019] (4) This invention achieves faster and more accurate real-time grasping operations by combining an improved YOLOv8 model, StrongSORT tracking, and adaptive unscented Kalman filtering design. This overcomes the problem of slow grasping speed and reduced factory automation efficiency caused by the high computational complexity of traditional deep learning networks. The entire process, from data acquisition to robotic arm grasping, has been optimized to ensure efficient industrial applications. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall process of the method of the present invention; Figure 2 This is the model framework of the improved StrongSoRT tracking algorithm of this invention; Figure 3 This is a schematic diagram of the process for calculating point cloud 3D data in this invention; Figure 4 This is a schematic diagram of the adaptive Kalman filter fusion process of the present invention; Detailed Implementation To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] To address the challenges of precise positioning, large errors, and complex calculations in existing robotic arms for grasping castings, this invention provides a real-time grasping method for semi-disordered multi-stacking castings within a confined material frame based on an RGB-D camera. This method effectively solves the aforementioned problems. Furthermore, in situations where heavy machinery is operating on-site and causing ground vibrations, traditional methods often face increased depth measurement errors and reduced depth accuracy, easily leading to collisions between the robotic arm and components. The following solution proposed in this invention enables more stable, accurate, and rapid real-time grasping operations. Specifically, combined with... Figure 1 The method of the present invention includes the following steps: Step S1: Use an RGB-D camera to simultaneously acquire RGB images, depth frames, and point cloud data of the casting in a semi-disordered stacked state. Perform denoising, registration, and annotation processing on the acquired data. Use the processed data to train an improved YOLOv8 detection model to achieve high-precision recognition of concentric circle features of the casting, and obtain the concentric circle detection model of the casting. Specifically, the RGB-D camera used in this step is an Intel RealSense D435i, which provides synchronized RGB images and depth frames, with a resolution option of 640×480 or 1280×720.

[0022] Data acquisition scenario: Semi-disordered stacked castings within a confined material frame; acquiring RGB images and depth data of the castings from multiple perspectives and with varying degrees of occlusion; simultaneously recording IMU data and synchronizing with timestamps.

[0023] Labeling method: Use LabelImg extension to support depth mask. For each casting, label: 2D bounding box (bbox), true center and radius of concentric outer circles, and depth mask (used for training depth histogram and 3D measurement).

[0024] Data augmentation: random rotation, scaling, brightness / contrast perturbation, occlusion synthesis, and depth noise simulation.

[0025] The preprocessed and labeled data is used to train the detection model. First, an improved YOLOv8 model is trained to accurately acquire the 3D data of the semi-disordered stacked castings in this invention. The improved YOLOv8, while maintaining the original network detection framework (Backbone, Neck, Head), has undergone structural optimization specifically for the concentric circle geometric features of the castings. A concentric feature module is introduced into the backbone network, and the conventional loss function is changed to a concentric loss function. These improvements enhance the model's ability to identify and locate circular and concentric structures. In contrast, the conventional YOLOv8 is mainly designed for rectangular bounding box detection and is difficult to accurately represent the geometric constraints of concentric circles.

[0026] The concentric feature module used in this invention detects concentric outer circles by fitting ellipses to the inner edge points of the bounding box using least-squares optimization. The concentricity loss function combines standard YOLO loss (such as bounding box regression). L bbox ,Classification L cls Objectivity L obj ) and concentric constraint loss term L conc The concentricity constraint loss term is used to constrain the deviation between the predicted circle and the true circle in terms of center position and radius parameters. Specifically:

[0027] In the above formula, L total Here, λ is the concentricity loss function, and λ is the weighting factor, which is determined according to the specific detection task. In Embodiment 1 of this invention, it is specifically set to 0.5. L conc This is the concentricity constraint loss term, used to measure the deviation between the predicted result and the ideal concentricity; c i = ( x i ,y i ) is the first i The predicted center of each detection circle; c gt = ( x gt , y gt () is the center of the real circle; r i For the first i The predicted radius of each detection circle; r gt The radius of the actual circle; N This represents the total number of objects to be detected in a single image.

[0028] This invention adds a concentric constraint loss term to the original loss function of YOLOv8. Lconc This constraint is used to constrain the deviation between the predicted circle and the real circle in terms of center position and radius parameters. Through geometric consistency optimization, this constraint enables the model to learn the distribution law of concentric features of the casting during training, thereby significantly improving the detection accuracy and stability of concentric circle features and enhancing detection robustness in complex lighting, occlusion, and stacking environments.

[0029] Step S2: Apply the concentric circle detection model of the casting obtained in step S1 to the real-time acquired casting image to detect the concentric circle region of the casting and extract the pixel coordinates of its center point; use the improved StrongSORT temporal tracking algorithm to perform continuous frame association on the center point to obtain stable center point temporal tracking results and reduce jitter and false detection. This invention improves the traditional StrongSORT time-series tracking algorithm to address the issues of target drift and mismatch that easily occur under complex working conditions such as castings stacked within a confined material frame, occlusion, and changes in lighting. Specifically, the improved StrongSORT uses Kalman filtering for motion prediction and the Hungarian algorithm for association, combining IoU and appearance costs.

[0030] In this step, the association cost matrix C is modified to include concentric similarity terms. S conc This is used to enhance the matching accuracy of concentric outer circular objects during the tracking process. Wherein:

[0031] In the above formula, w 1, w 2, w The weight is 3, which is determined according to the specific detection task. In Embodiment 1 of the present invention, the weights are 0.4, 0.4 and 0.2 respectively. d appThe cosine distance is embedded in the appearance (now greatly enhanced). S conc Measure the predicted trajectory i and detection j Geometric similarity between them; s c and s r This is the variance parameter.

[0032] like Figure 2 As shown, the input module receives multi-source data from an RGB-D camera, including RGB frames, depth histograms, and YOLO detection results. By extracting joint features from the RGB and depth images, the fusion of two-dimensional visual information and three-dimensional depth features is achieved. Based on the traditional StrongSORT correlation matrix, this paper introduces concentric circle geometric constraint features to enhance the spatial consistency of target matching. The improved cost matrix comprehensively considers three indicators: IoU, appearance similarity, and concentricity similarity, thus more accurately reflecting the correspondence between targets. Subsequently, based on the improved correlation cost matrix, the Hungarian algorithm is used to achieve optimal target matching between frames. Finally, the matching results are input into the extended Kalman filter module to predict and update the target motion state, outputting stable target tracking results.

[0033] Step S3: Combine the center point pixel coordinates obtained in step S2 with the casting depth information, perform spatial back projection through the camera's intrinsic calibration parameters, and calculate the corresponding center point three-dimensional coordinates; extract the local point cloud region centered on these coordinates, and perform voxel mesh downsampling and outlier removal on the point cloud region to obtain refined casting local three-dimensional point cloud measurement data. Specifically, such as Figure 3 As shown, by projecting the center of the tracked object onto 3D space, a local point cloud is extracted. The point cloud data is then mapped onto a color frame using the `map_to()` method of the point cloud object, associating each point in the point cloud with a corresponding pixel in the color image. Subsequently, through depth image mapping, the point cloud data calculated from the depth frame is reconstructed into a three-dimensional array containing the three-dimensional coordinates of each point in the point cloud.

[0034] Step S4: Use the IMU module built into the RGB-D camera to read the three-axis linear acceleration and angular velocity data; fuse the acceleration and gyroscope information through a complementary filtering algorithm to obtain a stable and reliable IMU attitude estimation result for the camera; Among them, complementary filtering is used to obtain IMU data. i x , i y , i z They represent the updatedx , y and z Axis attitude angle. This can be expressed by the following formula:

[0035]

[0036]

[0037] In the above formula, α These are the filter coefficients, used to adjust the weights of the gyroscope and accelerometer data. These are gyroscope pairs x , y and z Predicted update of axis attitude angle; a x , a y and a z They are x , y and z Linear acceleration of the axis; g x , g y and g z They are x , y and z Angular velocity of the axis; Δ t It is the time interval of the measurement.

[0038] Step S5: Input the refined 3D point cloud measurement data obtained in Step S3 and the IMU attitude estimation results of the camera in Step S4 into the Adaptive Unscented Kalman Filter (AUKF) fusion module. Calculate the high-precision pose of the casting by fusing multi-source information. During the fusion process, AUKF automatically adjusts the noise parameters according to the observation residuals to achieve adaptive optimization of the filtering parameters, thereby improving the accuracy of pose estimation and the robustness of the system. Output the final 3D pose data of the casting. The specific steps for fusion are as follows: Step S5.1: Define the state transition formula and observation formula of the adaptive Kalman filter and obtain the Sigma point set and its weights, i.e.:

[0039]

[0040] In the two formulas above, is the state estimation based on the depth camera. State covariance To generate a set of Sigma points These points can approximate the probability distribution of the depth camera's state, where n It is the dimension of state. It is an adjustable parameter. The first term represents the square root of the covariance matrix. i The column, and for each Sigma point, requires the weights for calculating the mean and covariance, where and There are two adjustable parameters.

[0041] Step S5.2: Obtain the weighted average based on the Sigma points, i.e.:

[0042]

[0043]

[0044] To obtain the observed Sigma point from the Sigma point through the transformation of the observation formula, and This is a weighted average based on Sigma points.

[0045] Step S5.3: Dynamically adjust the covariance of process and observation noise based on the consistency of observations to dynamically suppress error amplification and ensure data stability and accuracy. The adjustment formula is as follows, where... For the first k Time-prior state covariance P zz The covariance matrix of the observation noise, P xz The cross-covariance matrix between states and observations:

[0046]

[0047]

[0048] In the formula, Q k It is the covariance of process noise. P k and P k-1 These are the current and previous state covariances, respectively. After each update, the Adaptive Kalman Filter (AUKF) performs an adjustment step, dynamically updating the process noise covariance by observing the prediction error. R k It is the covariance of the observation noise, and Qk Similarly, AUKF will undergo an adjustment process after each update.

[0049] Step S5.4: Based on Kalman gain K k To update the state and covariance of the depth camera, i.e.:

[0050]

[0051]

[0052] In the above formula, according to the Kalman gain K k This is used to update the state and covariance of the depth camera, minimizing the error between the predicted and actual observations, and finally obtaining the updated depth value.

[0053] Step S5.5: Obtain the final updated 3D data, i.e.: After the fusion process is completed, the fused state vector can be obtained through x. Since the depth information is the first element (or the 0th index) in the state vector, the fused depth value is obtained through x[0], while x[1] and x[2] are the updated x-axis coordinates and y-axis coordinates.

[0054] Combination Figure 4 It can be seen that during the fusion process, state initialization... This includes depth information of the casting, camera acceleration, and camera angular velocity, and observation initialization. The measurement values ​​of point cloud depth measurement are included. The state covariance is initialized to describe the uncertainty of state estimation. The updated process noise covariance and measurement covariance are used for the next iteration. The updated state x is the latest fused depth information, x coordinate information, y coordinate information, acceleration and angular velocity. Among them, x[0] is the fused depth value, while x[1] and x[2] are the updated x-axis coordinates and y-axis coordinates.

[0055] Step S6: Transmit the three-dimensional pose data of the casting output in step S5 to the PLC control system. The motion planning module generates control instructions for the robotic arm, driving the robotic arm to perform real-time grasping operations on the casting within the confined material frame.

[0056] In this invention, in the local coordinate system of the casting, the X-axis points to the front of the casting, the Y-axis points to the left side of the casting, and the Z-axis points to the top of the casting; while in the global coordinate system used by the robotic arm control system, the X-axis points directly in front of the robotic arm, the Y-axis points directly to the left side of the robotic arm, and the Z-axis points directly above the robotic arm.

[0057] Generate the grasp pose (grab point, normal vector, grasp direction) based on the principal axis estimated from the 3D point cloud and the surface normal.

[0058] Generate robotic arm path points: grasping posture, descent height, gripper opening, etc.

[0059] The capture task is sent to the PLC via an industrial protocol as a structured message, with fields including: target ID, x, y, z, roll, pitch, yaw, timestamp, confidence level, and priority.

[0060] Example 1 Combination Figure 1-Figure 4 This embodiment provides a method for real-time grasping of semi-disordered multi-stacking castings within a confined material frame based on an RGB-D camera, including the following steps: Step S1: Use an RGB-D camera to synchronously acquire RGB images, depth frames, and point cloud data of the casting. Perform denoising, registration, and annotation on the acquired data. Use the processed data to train an improved YOLOv8 detection model to achieve high-precision recognition of concentric circle features of the casting, and obtain the concentric circle detection model of the casting. Data Acquisition: Image data of semi-disordered multi-stack castings within a confined space was acquired using an RGB-D camera (such as an Intel RealSense D435i). Acquisition parameters: RGB image resolution 1280×720 pixels, depth frame resolution 640×480 pixels, frame rate 30 fps. RGB images, depth frames, and point cloud data were acquired simultaneously, and denoising (using bilateral filtering) and registration (aligning the depth image with the RGB image) were performed.

[0061] Data augmentation and annotation: To improve the model's generalization ability, the acquired images were augmented, including: random horizontal flipping (probability 0.5), random rotation (angle range ±15°), brightness adjustment (scale 0.8-1.2), and addition of Gaussian noise (standard deviation 0.01). The augmented images were labeled using Labelimg, and the concentric outer circles of the casting were labeled as "circle". A .txt file and a .xml file were generated for each image.

[0062] In this embodiment, the training environment using the improved YOLOv8 was as follows: CPU: i7-13700KF, GPU: RTX4080 (16G), IDE: PyTorch, Python version: 3.8, CUDA version: 11.6, training times: 50 epochs. The resulting concentric circle detection model for castings (weight file: best.pt) achieved a mAP@0.5 of 95.3% on the test set.

[0063] The improved YOLOv8 model of this invention is trained using the processed data to obtain a concentric circle detection model for castings.

[0064] Step S2: Apply the concentric circle detection model of the casting obtained in step S1 to the real-time acquired casting image to detect the concentric circle region of the casting and extract its center point pixel coordinates; use the improved StrongSORT temporal tracking algorithm to perform continuous frame association on the center point pixel coordinates to obtain stable center point temporal tracking results and reduce jitter and false detections.

[0065] Specifically, the pre-trained concentric circle detection model for castings is first loaded, and inference is performed on the real-time video stream (1280×720 pixels) to obtain the bounding boxes of the concentric circles of the castings. For each detected bounding box, the center point pixel coordinates are calculated as follows:

[0066] In the above formula, These represent the coordinates of the upper left and lower right corners of the concentric outer circles of the casting, respectively.

[0067] Subsequently, an improved StrongSORT temporal tracking algorithm is used to correlate the center point pixel coordinates across consecutive frames, obtaining stable center point temporal tracking results and reducing jitter and false detections. The specific process is as follows: Step S2.1: Use the concentric circle detection model to identify the concentric outer circle detection frame of the casting; Step S2.2: Input the concentric outer circle detection frame of the casting into the improved StrongSORT for tracking training; Step S2.2.1: Based on historical trajectory data, predict the concentric outer circle trajectory of the casting in the current frame; Step S2.2.2: Use the NSA Kalman filter algorithm to confirm the prediction results; Step S2.2.3: Detect the concentric outer circle of the casting in the current frame and correlate it with the previously predicted trajectory data; Step S2.2.4: Query the status of the associated matching; If the association match is successful, the trajectory is updated and the next step is performed; if the trajectory does not match the detection result, it is deleted according to the set maximum trajectory; if the detection result does not match the previous trajectory, a new trajectory is created, and the NSA Kalman filter algorithm is used to predict and track the new trajectory.

[0068] Step S3: Combine the center point pixel coordinates obtained in step S2 with the depth information of the casting, perform spatial back projection through the camera's intrinsic calibration parameters, calculate the corresponding three-dimensional coordinates of the center point, extract the local point cloud region with the coordinates as the center, and perform voxel mesh downsampling and statistical outlier removal on the point cloud of the region to obtain refined local three-dimensional point cloud measurement data of the casting. The specific operation of step S3 is as follows: Step S3.1: Calculate the depth pixel coordinates of the concentric outer circle of the casting, as follows:

[0069] In the above formula, These represent the coordinates of the upper left and lower right corners of the concentric outer circles of the casting, respectively. Step S3.2: Map the point cloud onto the color frame and calculate the 3D point cloud in the depth frame; like Figure 4 As shown, the point cloud data is mapped onto the color frame using the map_to() method of the point cloud object, so that each point in the point cloud is associated with the corresponding pixel in the color image. Subsequently, through depth image mapping, the point cloud data calculated from the depth frame is reshaped into a three-dimensional array with the shape (480, 640, -1), which contains the three-dimensional coordinates of each point in the point cloud.

[0070] Step S3.3: Reshape the vertex data into a 3D array, extract the 3D camera coordinates of the pixels corresponding to the center of the bounding box, and return them as the final three-dimensional coordinates of the center point.

[0071] Step S3.4: In this embodiment, local point cloud extraction and processing involves extracting point cloud data within a cubic region with a side length of 0.1 meters, centered on the three-dimensional coordinates of the center point. Voxel downsampling uses the VoxelGrid filter from the PCL library, with the voxel size set to 0.005 meters to reduce point cloud density and improve processing efficiency. Statistical outlier removal uses the StatisticalOutlierRemoval filter from PCL, with settings... k Nearest neighbor number k=55 Standard deviation multiplier α=1.7 .

[0072] Elimination criteria: Point P i and k The average distance of its nearest neighbors exceeds the global average distance. m add α Double standard deviation s At that time, that is d (Pi,neighbors)>μ+α sIf the outliers are removed, then the outliers are removed. In the experiment, 1347 outliers were successfully removed, the sparsity of the point cloud was reduced to 0.003, and key features were preserved.

[0073] Step S4: Use the IMU module built into the RGB-D camera to read the three-axis linear acceleration and angular velocity data; fuse the acceleration and gyroscope information through a complementary filtering algorithm to obtain a stable and reliable IMU attitude estimation result for the camera.

[0074] In this embodiment, IMU data is acquired by reading IMU data at a frequency of 200Hz using the RGB-D camera's SDK (Intel RealSense SDK 2.0): triaxial linear acceleration. (a x ,a y ,a z ) (Unit: m / s²) and triaxial angular velocity (oh x ,oh y ,oh z ) (Unit: rad / s).

[0075] In this embodiment, complementary filtering is performed to obtain IMU data, and the filter coefficients are... If we set the weight to 0.65, then the weight of the accelerometer is 0.65, and the weight of the gyroscope is 0.35.

[0076] Using complementary filtering to process sensor data can more effectively suppress noise and improve data stability in vibrating environments. Through continuous updates, complementary filtering can dynamically adjust its dependence on measurement and integral data, thereby enabling real-time estimation of the system state.

[0077] Step S5: Input the refined 3D point cloud measurement data obtained in Step S3 and the IMU attitude estimation results of the camera in Step S4 into the Adaptive Unscented Kalman Filter (AUKF) fusion module. Calculate the high-precision pose of the casting by fusing multi-source information. During the fusion process, AUKF automatically adjusts the noise parameters according to the observation residuals to achieve adaptive optimization of the filtering parameters, thereby improving the accuracy of pose estimation and the robustness of the system. Output the final 3D pose data of the casting.

[0078] The output of this embodiment is: a high-precision casting six-degree-of-freedom pose (position and attitude), with a position error of less than 2mm and an attitude error of less than 1°.

[0079] Step S6: Transmit the three-dimensional pose data of the casting output in step S5 to the PLC control system. The motion planning module generates control instructions for the robotic arm, driving the robotic arm to perform real-time grasping operations on the casting within the confined material frame.

[0080] Pose data is sent to the PLC (Siemens S7-1200) using the Modbus TCP / IP protocol. The data frame includes: register address mapped to pose value. (X、Y、Z、φ、θ、ψ) Each value occupies 2 registers (32-bit floating-point numbers).

[0081] After the PLC parses the data, coordinate transformation is performed: the pose in the camera coordinate system is transformed to the robot arm base coordinate system (using the hand-eye calibration matrix).

[0082] The motion planning module generates the joint trajectory of the robotic arm (Yuejiang CR3 robotic arm) based on the target pose, taking into account collision avoidance and path optimization.

[0083] Robotic arm drive: Sends control commands to the robotic arm controller to execute the grasping action. The grasping cycle is less than 0.5 seconds, enabling real-time operation.

[0084] Finally, in the experimental results, this embodiment tested semi-randomly stacked castings within a confined material frame (1m×1m×0.5m), achieving a success rate of 96.7% in grasping and an average processing frame rate of 25 fps, meeting the real-time requirements of industry.

[0085] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time grabbing method for semi-disordered multi-pile castings within a limited magazine based on an RGB-D camera, characterized in that, Includes the following steps: Step S1: Obtain the RGB image, depth frame, and point cloud data of the casting, and preprocess and label the data; use the processed data to train the improved YOLOv8 detection model to achieve high-precision recognition of the concentric circle features of the casting, and obtain the concentric circle detection model of the casting; the improved YOLOv8 includes: introducing a concentric feature module into its backbone network, and changing the loss function to the concentric loss function; Step S2: Apply the concentric circle detection model of the casting obtained in step S1 to the image of the casting to be tested, detect the concentric circle region of the casting and extract the pixel coordinates of its center point; Step S3: Combine the center point pixel coordinates obtained in step S2 with the casting depth information, perform spatial back projection through the camera's intrinsic calibration parameters, and calculate the corresponding center point three-dimensional coordinates; extract the local point cloud region centered on the center point three-dimensional coordinates, and sample and remove the point cloud in the point cloud region to obtain the refined casting local three-dimensional point cloud measurement data. Step S4: Use the IMU module built into the RGB-D camera to read the three-axis linear acceleration and angular velocity data; fuse the acceleration and gyroscope information through a complementary filtering algorithm to obtain the camera's IMU attitude estimation results; Step S5: Input the refined 3D point cloud measurement data obtained in step S3 and the IMU attitude estimation results of the camera in step S4 into the adaptive unscented Kalman filter fusion module for processing. During the fusion process, the adaptive unscented Kalman filter automatically adjusts the noise parameters according to the observation residuals to achieve adaptive optimization of the filter parameters and output the final 3D pose data of the casting. Step S6: Transmit the three-dimensional pose data of the casting output in step S5 to the PLC control system to generate control instructions for the robotic arm, and drive the robotic arm to perform real-time grasping operation on the casting within the confined material frame.

2. The real-time casting gripping method according to claim 1, characterized in that, In step S1, an RGB-D camera is used to acquire RGB images, depth frames, and point cloud data of the casting; preprocessing operations include random rotation, scaling, brightness / contrast perturbation, occlusion synthesis, and depth noise simulation of the images.

3. The real-time casting gripping method according to claim 1, characterized in that, In step S1, the introduced concentric feature module detects concentric outer circles by fitting ellipses at the edge points within the bounding box using least squares optimization.

4. The real-time casting gripping method according to claim 1, characterized in that, In step S1, the concentricity loss function combines the standard YOLO loss and the concentricity constraint loss term, as detailed below: In the above formula, L total Here, λ is the concentricity loss function, and λ is the weighting factor. L bbox For the bounding box regression loss term, L cls For categorized loss items, L obj This is the loss term related to object nature; L conc This is the concentricity constraint loss term, used to measure the deviation between the predicted result and the ideal concentricity; c i =( x i , y i ) is the first i The prediction center of each detection circle; c gt =( x gt , y gt () is the center of the real circle; r i For the first i The predicted radius of each detection circle; r gt The radius of the actual circle; N This represents the total number of objects to be detected in a single image.

5. The real-time casting gripping method according to any one of claims 1-4, characterized in that, In step S2, the improved StrongSORT temporal tracking algorithm is used to perform continuous frame association on the center point pixel coordinates to obtain the center point temporal tracking results. The improved StrongSORT uses Kalman filtering for motion prediction and Hungarian algorithm for association, combining IoU cost and appearance cost.

6. The real-time casting gripping method according to any one of claims 1-4, characterized in that, Step S3, the method for calculating the corresponding three-dimensional coordinates of the center point is as follows: Step S3.1: Calculate the depth pixel coordinates of the concentric outer circle of the casting; Step S3.2: Map the point cloud onto the color frame and calculate the 3D point cloud in the depth frame; Step S3.3: Reshape the vertex data into a 3D array, extract the 3D camera coordinates of the corresponding pixels at the center of the bounding box, and return them as the final three-dimensional coordinates of the center point.

7. The real-time casting gripping method according to claim 6, characterized in that, In step S3, voxel grid downsampling and outlier removal are performed on all point cloud data within the point cloud region.

8. The real-time casting gripping method according to any one of claims 1-4, characterized in that, In step S5, the specific steps of fusion are as follows: Step S5.1: Define the state transition formula and observation formula of the adaptive Kalman filter and obtain the Sigma point set and its weights; Step S5.2: Obtain the weighted average based on the Sigma points; Step S5.3: Dynamically adjust the covariance of process and observation noise based on the consistency of observations to dynamically suppress error amplification and ensure data stability and accuracy; Step S5.4: Based on Kalman gain K k To update the state and covariance of the depth camera; Step S5.5: Obtain the final updated 3D data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it enables the electronic device to implement the real-time casting grasping method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the real-time casting gripping method as described in any one of claims 1-8.

Images