Collaborative robot target identification and grabbing attitude planning system

Abstract

The invention discloses a target recognition and grabbing posture planning system for a collaborative robot, and belongs to the technical field of mechanical arms. According to material posture information provided by a sensing module, the optimal grabbing posture is intelligently matched from a preset grabbing strategy library by fusing mechanical arm movement track parameters and target material physical characteristics; the initial accuracy of the grabbing scheme is ensured, in the grabbing process, the camera continuously collects the material poses in real time, and the offset is accurately calculated by comparing the real-time poses with the expected poses. Once material deviation is detected, a visual and force closed-loop adjusting mechanism is triggered immediately, a force-position hybrid control model is utilized, position deviation is synchronously corrected, and grabbing force parameters are dynamically adjusted, so that the grabbing stability and precision are remarkably improved, the grabbing failure rate is effectively reduced, manual intervention and reset time are greatly shortened, and the grabbing efficiency is improved. Therefore, the production efficiency is comprehensively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arms, and more particularly to a collaborative robot target recognition and grasping posture planning system. Background Art

[0002] Currently, the industrial automation field generally adopts a model in which robotic arms and fixed feeding tables work together. For example, a material plate is fixedly mounted on a workbench, and the robotic arm uses visual positioning to sequentially grab workpieces from the material trough. When the material is exhausted, the machine must be stopped to replace the material plate, resulting in increased waiting time for the machine tool and a loss of production efficiency of approximately 15%-22%. Some systems use cylinders to push individual material plates in or out of the work area, but the robotic arm must completely stop operating during the replacement process, making continuous feeding impossible. This is particularly affected in high-frequency production scenarios, such as automotive parts processing, where average daily downtime can even account for 18% of total working hours. A large number of machine tool stations are forced to idle due to material changes, seriously restricting the production rhythm.

[0003] In addition to the problem of feeding continuity, there are also problems with the grasping efficiency of the robotic arm. At present, most robotic arm grippers are integrated with cameras for real-time identification of material posture. However, in scenarios where materials are stacked or adjacent workpieces obstruct each other, traditional systems often need to adjust the robotic arm posture multiple times to complete positioning. This directly leads to a single grasping cycle being extended to 3-5 seconds, significantly slowing down the production rhythm.

[0004] When the robotic gripper approaches the material, its body easily blocks the camera's field of view, rendering the real-time positioning function ineffective. In this case, the system must rely on a pre-programmed path for grasping. During the production process, materials inevitably deviate from the preset position due to factors such as vibration. This deviation directly leads to a grasping failure rate as high as 12%. After a failure, manual intervention is required to reset the gripper, further increasing time and manpower costs.

[0005] In terms of space utilization of the feeding table, the existing design also has shortcomings. Although the single-layer feeding table has a simple structure, it cannot effectively utilize the vertical space, resulting in a very limited material capacity per unit area, which is difficult to adapt to high production capacity requirements; and some companies use stacked feeding racks to increase capacity. Although this can increase the material storage capacity, it will greatly expand the range of motion of the robot arm, and the positioning error will increase due to the extension of the motion path, which in turn affects the grasping accuracy and stability.

[0006] In summary, it can be seen that the current collaborative mode between robotic arms and feeding systems has shortcomings in multiple dimensions such as feeding continuity, grasping efficiency, and space utilization. These problems are superimposed on each other and jointly restrict the improvement of the efficiency of industrial automation. Summary of the Invention

[0007] In response to the problems existing in the prior art, the purpose of the present invention is to provide a collaborative robot target recognition and grasping posture planning system, which can match the optimal grasping posture from the preset grasping strategy library according to the material posture information provided by the perception module, combined with the robot arm motion trajectory parameters and the physical properties of the target material. During the grasping process, the material posture is collected in real time by the camera, and the offset is calculated by comparing the real-time posture with the expected posture, and the vision and force closed-loop adjustment mechanism is triggered when the material is offset. Through the force-position hybrid control model, the position deviation can be corrected and the grasping force parameters can be dynamically adjusted synchronously to ensure that the robot arm can complete the grasping task stably and accurately even in complex environments. This control mechanism greatly improves the stability and accuracy of grasping, reduces the grasping failure rate, reduces manual intervention and reset time, and further improves production efficiency.

[0008] To solve the above problems, the present invention adopts the following technical solutions.

[0009] A collaborative robot target recognition and grasping posture planning system includes a cabinet, a first workbench and a second workbench are slidably provided on the cabinet, the second workbench can slide through from below the first workbench, and material plates are installed on the first and second workbench, and the upper surfaces of the material plates are provided with evenly distributed material slots. A robotic arm is also installed on the cabinet, and the robotic arm is used to clamp materials on the material plates. A camera is installed at the clamping end of the robotic arm, and is used to adjust the grasping posture of the robotic arm according to data collected by the camera; It also includes: The perception module is used to capture material images through deep vision; output the material's six-degree-of-freedom pose based on a deep learning model; and trigger the spinor compensation mechanism to adjust the end-point pose when visual occlusion exceeds the threshold; The planning module is used to integrate the material pose, workbench position, and robot arm state to construct the state space; it uses reinforcement learning to generate collision-free trajectories and optimizes the path with a multi-objective reward function; and it uses dynamic obstacle prediction to avoid the moving parts of the workbench; The control module is used to match the optimal grasping posture based on the trajectory and material characteristics; trigger vision and force closed-loop adjustments when the material deviates; and dynamically optimize grasping parameters using a force-position hybrid model; The scheduling module is used to switch to the second workbench when the material on the first workbench is exhausted; based on the time series prediction, the pre-notification of material replacement is triggered in the preset window; Mapping module, used to reconstruct the workspace model through global scanning; solve the coordinate system transformation relationship after the workbench is changed; The optimization module is used to collect operating data to build a training set; optimize control parameters in a virtual environment and inject them into the system in a closed loop.

[0010] Furthermore, the perception module further includes: The camera shoots the target material, collects and outputs the original image including its three-dimensional structure, surface texture and surrounding environment; The pose estimation model receives the original image and processes it through the rotation-invariant feature extraction layer, focusing on the visible area, filtering out occlusion interference, and extracting stable features that are not affected by rotation. Based on this, it outputs the material's six-degree-of-freedom pose information and visual occlusion degree assessment results; The occlusion assessment result is compared with a preset threshold. If the threshold is exceeded, the robot arm's end-of-arm posture adaptive adjustment mechanism is triggered. This mechanism combines posture information, occlusion assessment, and kinematic constraints to calculate the optimal posture compensation to avoid occlusion through a screw motion model. The robotic arm adjusts the posture of the end gripping end based on the optimal posture compensation amount, allowing the camera to avoid occlusion and realign with the material; after adjustment, the camera captures the image again, and the above process is repeated until the occlusion level is lower than the threshold, ensuring that the robotic arm accurately obtains material information to complete gripping.

[0011] Furthermore, the planning module also includes: Collect data such as material posture, workbench dynamic position, and robot arm joint status, and form a standardized multi-dimensional fusion data set through time synchronization and format unification; Based on the fused data set, the core dimensions and motion boundaries of the robot arm, workbench, and material are determined and integrated into a structured model that describes the overall state. Based on the real-time position and physical size of the workbench in the planned state space, its space occupation area in the future period is predicted to form a dynamic space-time map of obstacles; Combine path efficiency, energy consumption index and safety margin, and integrate them into a comprehensive reward function according to weights as the trajectory optimization goal; Using the planning state space as the environment, dynamic obstacle prediction as the constraint, and the reward function as the goal, the algorithm generates the optimal collision-free trajectory. Verify whether the trajectory complies with the robot arm joint limits and worktable dynamics, provide feedback and adjust when conflicts occur, and finally output a trajectory that complies with actual constraints.

[0012] Furthermore, the control module further includes: Combining the robot arm's motion trajectory parameters with the target material's physical characteristics, the preset grasping strategy library is input, and the matching algorithm is used to select the optimal grasping point, gripping angle, and initial grasping force range, and output the specific optimal grasping posture parameters. The robotic arm performs gripping according to the optimal gripping posture. At the same time, the camera collects the material's posture in real time, compares the real-time posture with the expected posture, and calculates the offset. If the threshold is exceeded, the closed-loop force control adjustment is triggered, and the posture offset and the initial direction of the force control adjustment are output. Receive the posture offset and initial adjustment direction, combine it with the real-time position of the end of the robot arm, enable the force-position hybrid control model, synchronously correct the position deviation and dynamically adjust the grasping force parameters, and output the optimized grasping force parameters; The robot arm trajectory, the real-time status and physical dimensions of the workbench are collected, and a kinematic model is established to predict the three-dimensional occupied area of ​​the workbench within a preset time period in the future. An occupied area map with a time series is output as a constraint feedback to the first three steps to ensure obstacle avoidance during the grasping process.

[0013] Furthermore, the scheduling module also includes: The camera at the gripping end of the robotic arm captures the image of the material plate, identifies whether there is material in the material trough, and counts the remaining quantity. When the material is exhausted, an exhaustion signal and the current position information of the first workbench are output; Receive the material exhaustion signal and position information, control the cylinder to push the first workbench away and the second workbench to the preset clamping position, and output the second workbench activation signal and arrival confirmation information after the switching is completed.

[0014] Based on the activation signal of the second workbench, the camera continuously collects the material status of the two workbenches, integrates historical data with current information, and forms a structured data set containing image detection data and time series features; Based on this dataset, we divide the dataset into training and validation sets, design a model with a time series processing layer, adjust the parameters until the prediction error meets the standard, and output the trained model. The real-time material data of the second workbench detected by the camera is input into the model. Combined with the historical trend and the current gripping frequency, the material consumption trend and the estimated exhaustion time in the future preset period are predicted and the prediction results are output; Receive the prediction result, compare the remaining consumption time with the preset time window, and generate a pre-notification trigger signal and the predicted exhaustion time point if the conditions are met; Through sound and light alarms or terminals, workers are prompted to replace the material plate of the first workbench in time, and feedback notification status is provided to ensure that the plate replacement is completed before the material on the second workbench is exhausted.

[0015] Furthermore, the mapping module further includes: Use the scanning device to perform a full-scale scan of all objects in the workspace, collect regional point cloud data containing three-dimensional coordinates from multiple angles along the preset path, and output multiple sets of original point cloud data with viewpoint labels; Receive raw point cloud data, align multi-view data through feature matching, remove noise and redundant information, merge into a complete point cloud dataset in a unified coordinate system, and output the pre-processed workspace point cloud dataset; Based on the pre-processed point cloud dataset, a global point cloud model containing the three-dimensional shapes of all objects in the workspace is generated through surface reconstruction. The point cloud clusters of each key component are clearly marked and the global point cloud model is output. Identify the fixed feature points of the workbench from the global point cloud model, determine their original three-dimensional coordinates in the global coordinate system through point cloud analysis, and output the original coordinate set of each workbench feature point; After the workbench position is changed, the image coordinates of the same feature point are captured and extracted through the camera, and converted into new three-dimensional coordinates in the global coordinate system based on the global point cloud scale, and the new coordinate set of the feature point is output; The original coordinate set and the new coordinate set of the feature points are input into the visual calibration algorithm to calculate the translation and rotation parameters after the workbench position changes, integrate them into the coordinate system transformation relationship, and output them for subsequent path planning and pose calculation.

[0016] Furthermore, the optimization module also includes: The sensor network collects operational data such as robot arm movement, joint force, workbench switching, and gripping success rate. After cleaning and labeling, the data is classified by type and time sequence, divided into training and validation sets, and outputs a structured system operation status training data set. The training dataset is input into a virtual simulation environment built to scale with physical models. A reinforcement learning strategy is deployed, and a reward function is designed with the goals of improving efficiency, reducing energy consumption, and minimizing collisions. This drives the simulation system to iteratively adjust control parameters. The optimal parameter combination is evaluated and retained based on the validation set, and the optimized control parameter set is output. Import the optimized parameters into the physical control system to replace the original parameters, conduct a trial run and record the new operation data, compare the effect with the simulation expectation, and if there is a deviation, supplement the data and return to the first iteration. After meeting the standards, output the upgraded stable physical control system to complete the closed-loop upgrade.

[0017] Furthermore, the perception module further includes: The camera collects images of the target material under different lighting, rotation, occlusion, and workbench positions. The actual six-degree-of-freedom pose is recorded with high-precision equipment. The degree of occlusion is manually annotated to form a labeled dataset that associates the original image, pose, and occlusion degree and outputs it. Based on the convolutional neural network, an image enhancement module is added to the front end, and a rotation-invariant feature extraction layer is designed. The back end is divided into two branches to output six-degree-of-freedom pose parameters and occlusion degree assessment results, and output the network structure containing the extraction layer; The dataset is input into the network, trained using a joint loss function, and parameters are iteratively adjusted until the validation set indicators meet the standards. Occlusion samples are fine-tuned to enhance feature extraction capabilities, and a pose estimation model with preliminary training is output. The compression model meets the real-time requirements. Through interface adaptation, the model receives camera image input, outputs standardized pose information and occlusion assessment results, and outputs a deployable pose estimation model.

[0018] Furthermore, the perception module further includes: Collect physical parameters such as the length and range of motion of each joint of the robotic arm, as well as the installation coordinate offset of the camera relative to the end effector, to form and output a set of basic kinematic parameters; Based on the parameters and according to the screw theory, the mapping relationship between the end effector posture and joint angle is established, and the basic screw motion model is output; Control the robotic arm to move along a preset trajectory, record the actual end-point pose through a camera, compare it with the theoretical calculated value of the basic model, adjust the parameters to reduce the error, and output a calibrated screw motion model; By embedding logic in the model and inputting the material pose and occlusion assessment results, the model analyzes the difference between the current camera pose and the unobstructed pose, calculates the optimal pose compensation to avoid occlusion, and outputs a model that can be directly used for robot arm pose adjustment.

[0019] Furthermore, the control module further includes: Collect the robot arm's structural parameters, force and position sensor data, and the physical properties of the material. Through experiments, record the associated data in different grasping scenarios to form a labeled force and position control data set and output it. Construct a dual closed-loop control framework. The position loop calculates position compensation using proportional, integral, and differential algorithms, while the force loop calculates force compensation using an impedance control algorithm. A weight distribution module is designed to dynamically adjust the output ratio of the two loops based on material characteristics, resulting in a control framework that includes dual closed-loop and weight modules. The dataset is input into the model, with the goal of minimizing position deviation and force fluctuation and maximizing grasping success rate. The position loop proportional coefficient, force loop impedance parameters, and weight threshold are iteratively optimized. Fine-tuning is performed for special scenarios, and the optimized parameter set and preliminary model are output. A real-time feedback module is added to automatically adjust the weights of the position and force loops when the material posture deviation or force value exceeds the limit is detected. The interface adapts to receive inputs such as posture deviation and outputs optimized gripping force parameters and position compensation instructions, forming a model that can respond in real time. The model is embedded in the robotic arm control system, and the position accuracy and force control stability are verified through physical grasping experiments. When problems arise, the optimized parameters are returned, and finally a force-position hybrid control model that meets both accuracy and safety is output.

[0020] Compared with the prior art, the present invention has the following beneficial effects: (1) This solution matches the optimal grasping posture from the preset grasping strategy library based on the material posture information provided by the perception module, combined with the robot arm motion trajectory parameters and the physical characteristics of the target material. During the grasping process, the material posture is collected in real time through the camera, and the offset is calculated by comparing the real-time posture with the expected posture. When the material is offset, the vision and force closed-loop adjustment mechanism is triggered. Through the force-position hybrid control model, the position deviation can be corrected and the grasping force parameters can be dynamically adjusted simultaneously to ensure that the robot arm can complete the grasping task stably and accurately even in complex environments. This control mechanism greatly improves the stability and accuracy of grasping, reduces the grasping failure rate, reduces manual intervention and reset time, and further improves production efficiency.

[0021] (2) When the material on the first workbench is exhausted, this solution can automatically switch to the second workbench to continue working, and at the same time prompt the worker to replace the material plate on the first workbench, thus realizing continuous material feeding without stopping the machine, greatly reducing the waiting time of the machine tool, and improving production efficiency. It is suitable for high-frequency production scenarios, effectively solving the problem of idle workstations caused by material replacement, and promoting the acceleration of production rhythm. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0023] Figure 1 This is an appearance view of the overall structure of the present invention; Figure 2 It is a structural diagram of the camera of the present invention; Figure 3 This is a diagram showing the material trough of the present invention; Figure 4 It is a display diagram of the cylinder of the present invention; Figure 5 Flowchart of a collaborative robot target recognition and grasping posture planning system.

[0024] Description of the numbers in the figure: 1. Cabinet; 2. First workbench; 3. Second workbench; 4. Robotic arm; 5. Camera; 6. Material plate; 7. Material trough; 8. Cylinder; 9. Safety grille. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] See also Figures 1 to 5 A collaborative robot target recognition and grasping posture planning system includes a cabinet 1, on which a first workbench 2 and a second workbench 3 are slidingly arranged, and the second workbench 3 can slide through from under the first workbench 2, and the first workbench 2 and the second workbench 3 are both installed with material plates 6, and the upper surface of the material plate 6 is provided with evenly distributed material troughs 7. A robotic arm 4 is also installed on the cabinet 1, and the robotic arm 4 is used to clamp the material on the material plate 6. A camera 5 is installed at the clamping end of the robotic arm 4, which is used to adjust the grasping posture of the robotic arm 4 according to the data collected by the camera 5.

[0027] When in use, first install the material plate 6 on the first workbench 2 and the second workbench 3, and fill the material to be processed in the material trough 7. In the initial state, the second workbench 3 is close to the robotic arm 4, and the robotic arm 4 uses the mechanical clamp on the robotic arm 4 to clamp the materials on the second workbench 3 one by one. The cabinet 1 is placed next to the machine tool, and the robotic arm 4 will put the clamped materials onto the processing table of the machine tool, providing automatic loading service for the machine tool. After the materials on the second workbench 3 are clamped, the corresponding cylinder 8 will push the second workbench 3 away from the robotic arm 4, and at the same time, the corresponding cylinder 8 will push the first workbench 2 close to the robotic arm 4. At this time, the robotic arm 4 clamps the materials on the first workbench 2 to provide loading service for the machine tool. When the robotic arm 4 is clamping the materials on the first workbench 2 for loading, the worker can remove the material plate 6 on the second workbench 3 for refilling, or can directly refill the materials into the material trough 7 in the material plate 6 without removing it. After the material is fully loaded, such as when the material on the first workbench 2 is used up, the second workbench 3 is controlled to move closer to the robotic arm 4, while the first workbench 2 moves away from the robotic arm 4, and refilling is performed, thereby achieving non-stop loading and significantly improving processing efficiency. It should be noted that two sets of cylinders 8 are installed within the cabinet 1: one set of cylinders 8 is used to control the movement of the first workbench 2, and the other set of cylinders 8 is used to control the movement of the second workbench 3. One end of the first and second workbench 2, 3 extends into the cabinet 1, and the extensions are connected to the corresponding cylinders 8. This allows the corresponding cylinders 8 to move the corresponding first and second workbench 2, 3 when they are extended or retracted. A safety grille 9 is also installed on the cabinet 1. During daily production operations, when workers need to remove the material plate 6 or perform loading operations on the left side of the safety grille 9, direct contact between the worker and the robotic arm 4 is effectively prevented. If the robotic arm 4 collides or contacts a human body during high-speed operation, it could cause serious physical injury to the worker. The presence of the safety grille 9 reduces the probability of this danger. For example, without the safety grille 9, a worker might accidentally touch the operating robot arm 4 while removing a material plate 6 or loading a material due to negligence or limited operating space. This could result in minor injuries to the worker's hands or body, such as abrasions or sprains; in more serious cases, it could lead to fractures, limb pinching, or even more serious injuries.

[0028] In some embodiments of the present invention, the system further includes: a perception module for capturing material images through deep vision; outputting the material's six-degree-of-freedom pose based on a deep learning model; and triggering a spinor compensation mechanism to adjust the end pose when visual occlusion exceeds a threshold. The planning module is used to integrate the material pose, workbench position, and robot arm state to construct the state space; it uses reinforcement learning to generate collision-free trajectories and optimizes the path with a multi-objective reward function; and it uses dynamic obstacle prediction to avoid the moving parts of the workbench; The control module is used to match the optimal grasping posture based on the trajectory and material characteristics; trigger vision and force closed-loop adjustments when the material deviates; and dynamically optimize grasping parameters using a force-position hybrid model; The scheduling module is used to switch to the second workbench 3 when the material on the first workbench 2 is exhausted; based on the time series prediction, the pre-notification of material replacement is triggered in the preset window; Mapping module, used to reconstruct the workspace model through global scanning; solve the coordinate system transformation relationship after the workbench is changed; The optimization module is used to collect operating data to build a training set; optimize control parameters in a virtual environment and inject them into the system in a closed loop.

[0029] The perception module also includes: The camera 5 first shoots the target material, collects and outputs the original image including its three-dimensional structure, surface texture and surrounding environment; The pose estimation model receives the original image and processes it through the rotation-invariant feature extraction layer, focusing on the visible area, filtering out occlusion interference, and extracting stable features that are not affected by rotation. Based on this, it outputs the material's six-degree-of-freedom pose information and visual occlusion degree assessment results; The occlusion assessment result is compared with a preset threshold. If the threshold is exceeded, the adaptive posture adjustment mechanism of the end-end of the manipulator 4 is triggered. This mechanism combines the posture information, occlusion assessment and kinematic constraints, and calculates the optimal posture compensation amount for the camera 6 to avoid occlusion through the screw motion model; The robot arm 4 adjusts the posture of the end gripping end according to the optimal posture compensation amount, allowing the camera 6 to avoid occlusion and realign with the material; after adjustment, the camera 6 captures the image again, and the above process is repeated until the occlusion level is lower than the threshold, ensuring that the robot arm 4 accurately obtains material information to complete the gripping.

[0030] By adopting the above technical solution, camera 5 serves as the initial visual perception device to shoot the target material, capturing the original image including the three-dimensional structure of the material, such as outline and size; surface texture, such as lines and color distribution; and surrounding environment, such as the edge of the workbench and other objects, to provide basic visual data for subsequent processing.

[0031] Next, after receiving the raw image output by camera 5, the pose estimation model uses a rotation-invariant feature extraction layer to focus on the visible areas of the material, such as unobstructed surfaces, while filtering out interference from areas obscured by other objects, such as the sides of the material obscured by the edge of the workbench. This layer also specifically encodes features such as the material's edges and textures, ensuring that the extracted features are unaffected by the material's rotational state. For example, even after the material is flipped, the features remain reliably recognized. Based on these stable features, the model calculates the material's six-degree-of-freedom pose information, namely its three-dimensional spatial position and three-dimensional rotation angle. It also analyzes the proportion of occluded areas in the image and outputs a visual occlusion assessment, such as if the occluded area accounts for 30% of the total material area.

[0032] The occlusion assessment result is then compared with a preset threshold, such as 20%. If the assessment result exceeds the threshold (e.g., 30% > 20%), the adaptive posture adjustment mechanism at the end of robot arm 4 is triggered. This mechanism combines the material's six-degree-of-freedom pose information output by the pose estimation model (i.e., the material's position), the occlusion assessment result (i.e., the position and degree of occlusion), and the robot arm's own kinematic constraints, such as the range of joint motion and maximum rotation angle. The adjustment solution is calculated using a screw motion model, which simultaneously considers the coordinated compensation of translation and rotation, ultimately determining the optimal posture compensation for camera 6 to avoid the occluded area, such as a 5-centimeter right translation and a 10-degree rotation.

[0033] Finally, robotic arm 4 adjusts the posture of its gripping tip based on the optimal posture compensation, moving camera 6 to a new position, avoiding the previously obstructed area and realigning it with the material. Once the adjustment is complete, camera 6 captures another image, which is then fed back into the posture estimation model, repeating the process. This closed-loop cycle continues until the occlusion assessment falls below a preset threshold, such as 15% to 20%. This ensures that robotic arm 4 can obtain clear and complete material information, providing an accurate basis for subsequent gripping actions.

[0034] The construction of the pose estimation model includes: The camera 5 is used to collect images of the target material under different lighting, rotation, occlusion and workbench positions. The actual six-degree-of-freedom pose is recorded with high-precision equipment, and the degree of occlusion is manually annotated to form a labeled dataset associated with the original image, pose and occlusion degree and output it; Based on the convolutional neural network, an image enhancement module is added to the front end, and a rotation-invariant feature extraction layer is designed. The back end is divided into two branches to output six-degree-of-freedom pose parameters and occlusion degree assessment results, and output the network structure containing the extraction layer; The dataset is input into the network, trained using a joint loss function, and parameters are iteratively adjusted until the validation set indicators meet the standards. Occlusion samples are fine-tuned to enhance feature extraction capabilities, and a pose estimation model with preliminary training is output. The compression model meets the real-time requirements. Through interface adaptation, the model receives camera 6 image input, outputs standardized pose information and occlusion assessment results, and outputs a deployable pose estimation model.

[0035] By adopting the above technical solution, during the dataset construction phase, the model must be provided with a rich set of training samples. Camera 5 is used to capture images of the target material in a variety of scenarios, including varying lighting intensities, such as strong light and shadow; material rotation angles, such as flipping from 0° to 360°; occlusion levels, such as partial occlusion by the workbench edge and occlusion by other stacked materials; and the positions of the first and second workbenches, ensuring that the samples cover all scenarios that may occur in actual work. Furthermore, high-precision measurement equipment, such as a laser tracker, is used to simultaneously record the actual 6DOF pose of the material in each image, serving as the ground truth label for pose prediction. The images are manually annotated with the ratio of the occluded area to the total material area, e.g., 30% occlusion is labeled as 0.3, which serves as the occlusion level label. Finally, the original image, 6DOF pose, and occlusion level are linked to form a structured, labeled dataset, providing the data foundation for model training.

[0036] With the convolutional neural network as the basic framework, an image enhancement module is added to the front end. By performing random rotation, scaling, local occlusion simulation and other processing on the original image, the model's adaptability to changes in the input image is enhanced, such as coping with slight offsets in the camera shooting angle. The core layer designs a rotation-invariant feature extraction layer. By performing special encoding on features such as the edge contour and surface texture of the material, such as mapping the features to a spherical coordinate system, the influence of the rotation direction is eliminated, so that the extracted features are not affected by the material's own rotation state. For example, after the material is flipped, the key features can still be stably identified. The back end is divided into two parallel outputs: one is a step-by-step regression calculation through the fully connected layer to output six-degree-of-freedom posture parameters; the other is an analysis of the occlusion area features through the classification layer to output the occlusion degree assessment result, forming a network structure that can handle posture estimation and occlusion judgment at the same time.

[0037] The constructed dataset is proportionally divided into training and validation sets, which are then fed into the designed network architecture. Training is performed using a joint loss function: one loss constrains the deviation between the model's predicted pose and the actual pose recorded by high-precision equipment, such as positional and angular deviations, to ensure pose estimation accuracy. The other loss constrains the deviation between the occlusion level output by the model and the manually annotated level, improving the reliability of occlusion judgment. Network parameters, such as convolution kernel weights and fully connected layer coefficients, are continuously adjusted through backpropagation until the pose prediction error and occlusion assessment accuracy on the validation set meet preset standards. For example, the average positional deviation of the pose prediction error is less than 2 mm, and the occlusion assessment accuracy is greater than 90%, which meets the preset standards. For a subset of samples with severe occlusion, such as images with an occlusion rate exceeding 50%, intensive training is performed to adjust the network's focus on occlusion features, enhancing the model's feature extraction capabilities in complex occlusion scenarios. The model is then output as a preliminarily trained model.

[0038] Finally, the non-core redundant layers in the network are simplified, reducing the number of parameters. For example, quantizing 32-bit parameters to 16 bits reduces computational time and ensures that the model can achieve millisecond-level inference in the robotic arm control system. Simultaneously, interface adaptation is performed so that the model can directly receive raw image data output by camera 6 and output standardized six-degree-of-freedom pose information according to the system protocol. This includes 3D coordinates and angles in a unified format, as well as occlusion assessment results, such as quantized values ​​between 0 and 1. This ultimately forms a pose estimation model that can be directly deployed in the actual system, providing accurate visual feedback for the robotic arm's posture adjustment.

[0039] The construction of the spinor motion model includes: Collect physical parameters such as the length and range of motion of each joint of the robot arm 4, and the installation coordinate offset of the camera 5 relative to the end effector, to form a basic kinematic parameter set and output it; Based on the parameters and according to the screw theory, the mapping relationship between the end effector posture and joint angle is established, and the basic screw motion model is output; Control the robot arm 4 to move along the preset trajectory, record the actual end position through the camera 5, compare it with the theoretical calculated value of the basic model, adjust the parameters to reduce the error, and output the calibrated screw motion model; Logic is embedded in the model. After inputting the material pose and occlusion assessment results, the difference between the current camera 5 pose and the unobstructed pose is analyzed, the optimal pose compensation to avoid occlusion is calculated, and the output is a model that can be directly used for the posture adjustment of the robot arm 4.

[0040] By adopting the above technical solution, the physical structural parameters of the robot arm 4 are collected in advance through a laser rangefinder, including the length of the connecting rod between each joint, such as the distance from the shoulder to the elbow and from the elbow to the wrist; the range of motion of each joint, such as the maximum angle of rotation of the shoulder joint; the type of joint, such as a rotary joint or a translational joint, etc.; at the same time, the installation position parameters of the camera 5 on the end effector of the robot arm 4 are recorded, that is, the three-dimensional coordinate offset of the center of the camera 5 relative to the end effector reference point, such as a 2 cm offset along the X-axis and a 1 cm offset along the Y-axis. These parameters are integrated to form a basic kinematic parameter set.

[0041] Based on the parameter set, the model is constructed according to the screw theory. The screw theory can simultaneously describe the translation of the end of the robot arm, such as forward, backward, left and right movement; and rotational movement, such as rotation around an axis. The movement of each joint is regarded as a screw, which includes the direction and magnitude of movement. By integrating the screw information of each joint, the model establishes a mapping relationship between the end effector posture and each joint angle. For example, when the shoulder joint rotates 30 degrees and the elbow joint rotates 20 degrees, the model can calculate the specific position and orientation of the end effector. The final output of the screw motion basic model can directly deduce the end posture by inputting the joint angle, or reversely deduce the required joint motion by the target end posture.

[0042] The robot arm 4 is controlled to move along a preset trajectory, such as moving the end effector from point A to point B along a straight line or rotating 90 degrees around a fixed point. Meanwhile, the actual position of the end effector, such as the position change of a marker, is captured in real time by camera 5. The actual position is compared with the theoretical position calculated by the basic model of the screw motion to identify deviations, such as if the theoretical movement is 10 cm but the actual movement is only 9.5 cm. Model parameters are adjusted based on the deviations, such as correcting for minor errors in the connecting rod length, adjusting the theoretical length from 10 cm to 9.8 cm. Joint stiffness coefficients are optimized to compensate for motion lag caused by joint clearance. Iterations are repeated until the deviation between the theoretical calculation and the actual movement falls below a preset threshold, such as less than 0.5 mm. The calibrated screw motion model is then output.

[0043] The planning module also includes: Collect data such as material posture, workbench dynamic position, and the status of the four joints of the robotic arm, and form a standardized multi-dimensional fusion data set through time synchronization and format unification; Based on the fused data set, the core dimensions and motion boundaries of the robot arm 4, workbench, and material are determined and integrated into a structured model that describes the overall state; Based on the real-time position and physical size of each workbench in the planning state space, its space occupation area in the future period is predicted to form a dynamic space-time map of obstacles; Combine path efficiency, energy consumption index and safety margin, and integrate them into a comprehensive reward function according to weights as the trajectory optimization goal; Using the planning state space as the environment, dynamic obstacle prediction as the constraint, and the reward function as the goal, the algorithm generates the optimal collision-free trajectory. Verify whether the trajectory complies with the limits of the robot's four joints and the dynamics of the worktable, provide feedback and make adjustments when conflicts occur, and ultimately output a trajectory that complies with the actual constraints.

[0044] By adopting the above technical solution, we first obtain the three-dimensional position of the material, the current coordinates of the first workbench 2 and the second workbench 3, and the status information of each joint of the robotic arm 4. Then, we unify the data format of this information, such as setting the coordinate system 1 as the global coordinate system. Finally, we form a standardized fusion data set containing multi-dimensional information of the material, workbench, and robotic arm 4, providing a consistent data basis for subsequent analysis.

[0045] The spatial pose of the robot arm's end effector, the angular range of motion of each joint, the real-time position of the workbench, and the location parameters of the target material are extracted from the data. At the same time, the motion boundaries of each dimension are clarified, such as the maximum rotation angle of the robot arm's joints and the movement range of the workbench. These dimensions and boundaries are integrated into a structured model to fully describe the overall motion state of the system at any time, providing a comprehensive picture of the environment for trajectory planning.

[0046] Then, dynamic obstacle modeling is performed based on the planned state space, such as the real-time position and physical size of each workbench, and a kinematic model is established to simulate its motion law. The position change trajectory of the workbench in the future time period is predicted through the model, and the three-dimensional occupied area of ​​each workbench in space at each moment is calculated based on the size, such as the cube range marked by the coordinate frame, to form a dynamic obstacle spatiotemporal map containing a time series, and to clarify the area that the robot arm 4 needs to avoid in the future time period.

[0047] With trajectory optimization as the goal, three key metrics are integrated: Path efficiency, where shorter gripping tasks are completed and shorter end-point travel distances result in higher rewards; Energy consumption, where larger joint rotation angles and faster speeds result in higher energy consumption, with rewards deducted if thresholds are exceeded; and Safety margin, where greater minimum distances between the robot arm and obstacles, such as the workbench, result in higher safety rewards, with penalties triggered if distances fall below the safety threshold. Weights are assigned to the three metrics based on actual needs, such as prioritizing safety over efficiency. This results in a comprehensive reward function, which serves as the core criterion for evaluating trajectory quality.

[0048] Based on the above foundation, reinforcement learning is used to generate optimal collision-free trajectories. The planned state space serves as the algorithm's environment, and the spatiotemporal map of dynamic obstacles serves as motion constraints, such as requiring that trajectories not enter predicted occupied areas. A comprehensive reward function serves as the learning objective. The algorithm explores the robot's possible motion paths in the state space. Each candidate trajectory is scored using the reward function, prioritizing high-reward trajectories. Trajectories are also checked in real time to see if they overlap with the future occupied areas of dynamic obstacles. If a collision is detected, the trajectory is eliminated. Through multiple rounds of iterative learning, trajectory parameters are gradually optimized, ultimately outputting an optimal collision-free trajectory that balances path efficiency, low energy consumption, and a high safety margin.

[0049] Finally, trajectory verification and adaptation are performed, comparing the generated trajectory with the actual constraints. The following checks are performed to ensure that the trajectory is within the joint motion limits of the robot arm 4, such as whether the joint angles do not exceed the maximum range, and whether it can adapt to the dynamic movement of the worktable, such as whether the trajectory time matches the changes in the worktable position. If there is a conflict, such as a joint angle exceeding the limit, the problem is fed back to the previous step, and the weight of the reward function or the boundary parameters of the state space are adjusted to regenerate the trajectory. Verification and adjustment are repeated until the trajectory fully meets the actual operational constraints, and finally a directly executable robot arm motion trajectory is output.

[0050] The control module also includes combining the motion trajectory parameters of the robot arm 4 and the physical characteristics of the target material, inputting the preset grasping strategy library, screening the optimal grasping point, clamping angle and initial grasping force range through the matching algorithm, and outputting the specific optimal grasping posture parameters; The robot arm 4 performs gripping according to the optimal gripping posture. At the same time, the camera 6 collects the material posture in real time, compares the real-time posture with the expected posture, and calculates the offset. If the offset exceeds the threshold, the closed-loop force control adjustment is triggered, and the posture offset and the initial direction of the force control adjustment are output. Receive the posture offset and initial adjustment direction, combine it with the real-time position of the end of the robot arm 4, enable the force-position hybrid control model, synchronously correct the position deviation and dynamically adjust the grasping force parameters, and output the optimized grasping force parameters; The robot arm 4 trajectory, the real-time status and physical dimensions of the workbench are collected, and a kinematic model is established to predict the three-dimensional occupied area of ​​the workbench within a preset time period in the future. An occupied area map with a time series is output as a constraint feedback to the first three steps to ensure obstacle avoidance during the grasping process.

[0051] Using the above technical solution, the robot arm 4 first obtains trajectory parameters, such as the spatial coordinates of the end effector's approach to the material, the angle of approach, and the target material's physical characteristics, such as its regular shape, weight, surface smoothness, and the presence of vulnerable areas. This information is then input into a pre-configured grasping strategy library, which stores optimal solutions for different materials. These include grasping points, such as the center of gravity or anti-slip contact area; gripping angles, such as the angle with the material's axis; and initial gripping force ranges, such as 5-10N for lightweight materials and 20-30N for heavy materials. Using a matching algorithm, such as one based on shape similarity or weight range, the library selects the solution that best matches the current material's characteristics and trajectory parameters. The library then outputs the optimal grasping posture parameters, such as grasping point coordinates, a 60-degree gripping angle, and an initial force of 10-15N.

[0052] The robotic arm 4 performs the gripping action based on the output optimal gripping posture. Simultaneously, a camera 6 mounted at the end captures real-time footage of the material in the grip. Image recognition is used to extract the material's real-time pose, three-dimensional position, and rotation angle. The real-time pose is compared with the expected pose, and the offset is calculated (e.g., a position deviation of 3 mm or an angular deviation of 8 degrees). If the offset exceeds a preset threshold (e.g., a position deviation greater than 2 mm or an angular deviation greater than 5 degrees), a closed-loop force control adjustment mechanism based on visual feedback is triggered, outputting the specific pose offset data and the initial direction of the force control adjustment, such as applying a compensating force in the opposite direction of the offset.

[0053] Then, force-position hybrid control optimizes the gripping force. The output posture offset and initial adjustment direction are received, and combined with the real-time position data from the end of the robot arm 4, such as its relative distance to the material, the force-position hybrid control model is activated. This model uses position control to correct the spatial position of the robot arm's end, such as fine-tuning the joint angle to reduce position deviation by 3 mm. Furthermore, based on real-time data from the force feedback sensor at the gripping end, such as the current gripping force of 12 N, the gripping force parameters are dynamically adjusted. If the material is detected to be slipping due to insufficient friction, the gripping force in the corresponding area is increased, for example to 14 N. If the material contacts a vulnerable area, the pressure is reduced, for example to 10 N. By balancing position accuracy and gripping force, optimized gripping force parameters are output, such as the force distribution between each gripper finger and the 10-14 N force retention range.

[0054] Finally, the motion trajectory of the robot arm 4 is collected, such as the path planning for the next 5 seconds. The position change of the workbench within the preset time period in the future is predicted through the kinematic model, and the three-dimensional occupied area of ​​the workbench in space at each moment, such as the rectangular range marked by the coordinate box, is calculated. The occupied area map with a time series is output. This map is used as a constraint condition. For example, if it is predicted that the workbench will move to a certain area in 1 second, the trajectory adjustment range of the robot arm 4 will be limited to avoid entering the area, or the matching results of the grasping strategy library will be corrected to avoid grasping angles that conflict with the workbench, ensuring that there is no collision between the robot arm, material and workbench during the entire grasping process, realizing full process optimization from posture planning to dynamic adjustment, and ensuring that the robot arm 4 completes material clamping accurately and safely.

[0055] The construction of the force-position hybrid control model includes: Collect the structural parameters, force and position sensor data of the robot arm 4 and the physical properties of the material. Through experiments, record the associated data in different grasping scenarios, form a labeled force and position control data set and output it; Construct a dual closed-loop control framework. The position loop calculates position compensation using proportional, integral, and differential algorithms, while the force loop calculates force compensation using an impedance control algorithm. A weight distribution module is designed to dynamically adjust the output ratio of the two loops based on material characteristics, resulting in a control framework that includes dual closed-loop and weight modules. The dataset is input into the model, with the goal of minimizing position deviation and force fluctuation and maximizing grasping success rate. The position loop proportional coefficient, force loop impedance parameters, and weight threshold are iteratively optimized. Fine-tuning is performed for special scenarios, and the optimized parameter set and preliminary model are output. A real-time feedback module is added to automatically adjust the weights of the position and force loops when the material posture deviation or force value exceeds the limit is detected. The interface adapts to receive inputs such as posture deviation and outputs optimized gripping force parameters and position compensation instructions, forming a model that can respond in real time. The model is embedded in the control system of the robotic arm 4, and the position accuracy and force control stability are verified through physical grasping experiments. When problems exist, the optimized parameters are returned, and finally a force-position hybrid control model that meets both accuracy and safety is output.

[0056] By adopting the above technical solution, the structural parameters of the robot arm 4 are collected in advance, such as joint lengths, joint stiffness, and end effector size; force feedback sensor data, such as real-time gripping force and torque changes during gripping; and position sensor data, such as the three-dimensional coordinates of the end effector and joint angles. The physical properties of the target material, such as weight, surface friction coefficient, and force thresholds for vulnerable areas, are also recorded. A large number of grasping experiments are then conducted to simulate different scenarios. These include, for example, the force required to compensate for a 1mm position offset when grasping lightweight materials; the adjustment effect when the force exceeds the threshold when grasping fragile materials; and the correlation between position deviation and force deviation when grasping rigid materials. These correspondences between position deviation, force deviation, and adjustment effects are annotated and integrated into a labeled force-position control dataset, providing a data foundation for model training.

[0057] With the goals of precise positioning and stable force control, a dual closed-loop control framework was established: the position loop is responsible for correcting spatial position deviations, receiving the difference between the end effector's real-time position and the target position, and calculating the position compensation using proportional, integral, and differential algorithms. For example, if the position is 2 mm to the left, an instruction to adjust the position to the right by 2 mm is output to ensure that the end effector can accurately align with the material. The force loop is responsible for stabilizing the clamping force, receiving the difference between the real-time force value of the force sensor and the target force value, and calculating the force compensation using an impedance control algorithm. For example, if the current force is 3N less than the target, an instruction to increase the force by 3N is output to prevent sudden force changes that could damage or cause the material to slip. A weight distribution module was also designed to dynamically adjust the output ratio of the two loops based on the material's characteristics. For example, when grasping rigid materials, the position loop's weight is increased to prioritize alignment accuracy; when grasping fragile materials, the force loop's weight is increased to prioritize limiting the force value. Ultimately, a control framework consisting of dual closed-loop and weighted modules is formed.

[0058] Parameters are adjusted iteratively with three goals in mind: minimize position deviation, such as controlling it within 0.5 mm; minimize force fluctuation range, such as within ±1N; and maximize grasping success rate, such as over 99%. Adjust the proportional coefficient of the position loop to affect the reaction speed of position adjustment. A coefficient that is too large will cause oscillation, while a coefficient that is too small will cause slow adjustment. The impedance parameter of the force loop affects the softness of force value changes. Inappropriate parameters will cause the force value to fluctuate. There is also the threshold value of weight distribution. For example, the force control weight threshold for fragile materials is set to 70%. For special scenarios, such as increasing friction when the material slips, and reducing clamping force to prevent collision when approaching the workbench, the corresponding samples are extracted separately for parameter fine-tuning, and the optimized parameter set and preliminary model are finally output.

[0059] To enable the model to respond in real time to changes during the grasping process, a real-time feedback module was added. Upon receiving information about a material's positional deviation, such as a 3mm positional deviation, the module automatically increases the weight of the position loop to accelerate position correction. When force values ​​are detected exceeding a safe range, such as when a fragile material's force reaches 110% of the threshold, the module prioritizes increasing the weight of the force loop to forcibly limit force growth. Furthermore, through interface adaptation, the model can receive previously output data such as positional offsets and initial adjustment directions, and directly output optimized grasping force parameters and position compensation instructions, such as a 5-degree rotation plus a 1mm translation, creating a model that can respond to dynamic changes in real time.

[0060] The scheduling module also includes capturing an image of the material plate through the camera 5 at the gripping end of the robotic arm 4, identifying whether there is material in the material slot 7 and counting the remaining quantity. When the material is exhausted, an exhaustion signal and the current position information of the first workbench 2 are output; Receive the material exhaustion signal and position information, control the cylinder to push the first workbench 2 away and the second workbench 3 to the preset clamping position, and output the second workbench 3 activation signal and arrival confirmation information after the switching is completed.

[0061] Based on the activation signal of the second workbench 3, the camera 5 continuously collects the material status of the two workbenches, integrates historical data with current information, and forms a structured data set containing image detection data and time series features; Based on this dataset, we divide the dataset into training and validation sets, design a model with a time series processing layer, adjust the parameters until the prediction error meets the standard, and output the trained model. The real-time material data of the second workbench 3 detected by the camera 5 is input into the model, and the historical trend and the current gripping frequency are combined to predict the material consumption trend and the estimated exhaustion time in the future preset period, and the prediction result is output; Receive the prediction result, compare the remaining consumption time with the preset time window, and generate a pre-notification trigger signal and the predicted exhaustion time point if the conditions are met; The worker is prompted to replace the material plate of the first workbench 2 in time through sound and light alarm or terminal, and feedback notification status is given to ensure that the plate replacement is completed before the material of the second workbench 3 is exhausted.

[0062] By adopting the above technical solution, it can be seen that the camera 5 at the gripping end of the robotic arm 4 is responsible for real-time monitoring. If the first worktable 2 is close to the robotic arm 4 at this time, the camera 5 continuously captures the image of the material plate on the first worktable 2. Using image recognition technology, it determines whether there is material in each material slot 7. For example, by detecting whether the material slot 7 area in the image has features that match the material outline, such as a specific shape or color block, the remaining material quantity is counted. When the camera 5 detects that all material slots 7 are empty of material features, that is, the material is exhausted, the system generates a material exhaustion signal and simultaneously records and outputs the current spatial position information of the first worktable 2.

[0063] After receiving the material exhaustion signal and the position of the first workbench 2, the system activates the workbench switching mechanism, driving the corresponding cylinders. One set of cylinders pushes the first workbench 2 along the slide rails, away from the working area of ​​the robot arm 4, until it reaches the standby position. Simultaneously, another set of cylinders unlocks the fixing device of the second workbench 3 and pushes it along the slide rails to the preset gripping position, aligning it with the working range of the robot arm 4. When the second workbench 3 reaches the designated position, the system outputs an activation signal for the second workbench 3, indicating that it has entered the working state, paving the way for subsequent material gripping.

[0064] Based on the activation signal from the second workbench 3, camera 5 begins synchronously collecting material status data from both workbench locations. For the first workbench 2, it records its current empty state and historical gripping records, such as the total number of materials on a single material plate 6 and the duration of consumption. For the second workbench 3, it captures changes in material quantity before and after each gripping in real time, identifies reduced material slots 7 through image comparison, and timestamps each gripping. It also integrates historical data, such as gripping intervals at different time periods and the corresponding consumption rate for material types, as well as current information, such as the initial material quantity on the second workbench 2 and the real-time gripping frequency of the robotic arm 4. This data is structured according to the correlation between time, material quantity, and gripping frequency, forming a dataset containing image detection results and time series features, providing a training foundation for the prediction model.

[0065] Subsequently, a material consumption forecasting model was constructed using this dataset. The dataset was proportionally divided into training and validation sets. A model architecture was designed that included a time series processing layer to capture cyclical patterns in the data, such as differences in pick-up frequency during peak hours in the morning and evening. The model was trained on the training set to learn historical consumption trends, such as the hourly consumption of 10 items of a certain material. The model parameters were then repeatedly adjusted based on the validation set, such as optimizing sensitivity to cyclical characteristics, until the forecast error fell below a preset standard, such as the difference between actual and predicted consumption. Finally, the trained forecasting model was output.

[0066] After the model is put into operation, camera 5 monitors the remaining material on second workbench 3 in real time, identifies the number of slots currently filled with material through image recognition, and inputs this real-time data into the prediction model. The model combines historical consumption trends with the current gripping frequency to calculate material consumption trends for a preset future time period, such as a curve showing the decrease in remaining material over time. It then infers the estimated time when second workbench 3 will be depleted and outputs a specific prediction result.

[0067] Based on the prediction results, a pre-notification trigger is determined. The difference between the estimated exhaustion time and the current time is compared with a preset time window. If the remaining time is less than the window value, such as exhaustion in 10 minutes, a pre-notification trigger signal is generated with the specific predicted exhaustion time point, such as 10:30, to ensure workers have sufficient preparation time.

[0068] Finally, the notification module responds to the trigger signal, emitting a warning tone and flashing lights through the workshop's audio and visual alarms. Simultaneously, a text notification is displayed on the worker's terminal. For example, if the material plate at Workbench 1 needs to be replaced by 10:30, the system simultaneously sends a notification to the control center, ensuring that managers are aware of the progress. Upon receiving the notification, the worker completes the replacement of the material plate at Workbench 1, 2, before Workbench 2, 3, runs out of material. This allows the switching mechanism to immediately activate Workbench 1, 2, with its newly restocked material, when Workbench 2, 3, becomes empty, ensuring an uninterrupted material supply process.

[0069] The mapping module also includes scanning all objects in the workspace using a scanning device, collecting regional point cloud data containing three-dimensional coordinates from multiple angles according to a preset path, and outputting multiple sets of original point cloud data with viewpoint labels; Receive raw point cloud data, align multi-view data through feature matching, remove noise and redundant information, merge into a complete point cloud dataset in a unified coordinate system, and output the pre-processed workspace point cloud dataset; Based on the pre-processed point cloud dataset, a global point cloud model containing the three-dimensional shapes of all objects in the workspace is generated through surface reconstruction. The point cloud clusters of each key component are clearly marked and the global point cloud model is output. Identify the fixed feature points of the workbench from the global point cloud model, determine their original three-dimensional coordinates in the global coordinate system through point cloud analysis, and output the original coordinate set of each workbench feature point; After the workbench position is changed, the image coordinates of the same feature point are captured and extracted by camera 5, and converted into new three-dimensional coordinates in the global coordinate system based on the global point cloud scale, and the new coordinate set of the feature point is output; The original coordinate set and the new coordinate set of the feature points are input into the visual calibration algorithm to calculate the translation and rotation parameters after the workbench position changes, integrate them into the coordinate system transformation relationship, and output them for subsequent path planning and pose calculation.

[0070] By adopting the above technical solution, the workspace is first subjected to all-round data collection through a laser scanner, and the scanning head is moved according to a preset path to photograph all objects in the workspace from different angles, including the first workbench 2, the second workbench 3, the robotic arm 4, the material plate 6, the surrounding equipment, etc., to capture the three-dimensional coordinate information of the surface of each object, such as the X, Y, and Z axis distance of a certain point from the scanning origin. Finally, multiple groups of original point cloud data with perspective marks are output, each group of data corresponds to a scanning angle, and the position information of the angle is marked.

[0071] Next, the raw point cloud data is preprocessed and spliced. After receiving multiple sets of raw point clouds, the data from different perspectives are aligned using feature matching technology: common features of the same objects in each perspective are identified, such as the right-angled edges of the workbench and the circular marks of the robotic arm joints. The spatial position of each set of data is adjusted based on these features to ensure that they are aligned in the same reference frame; at the same time, noise points and redundant data are filtered out, and data from all perspectives are merged into a complete point cloud dataset in a unified coordinate system. The preprocessed workspace point cloud dataset is output to provide a clean and coherent data foundation for subsequent modeling.

[0072] Based on the preprocessed point cloud dataset, an algorithm constructs a continuous topological structure of the object's surface, such as connecting discrete point clouds into surfaces to restore the object's three-dimensional form. This generates a global point cloud model encompassing the three-dimensional forms of all objects within the workspace. Clustering and labeling techniques are used within the model to distinguish the point clouds of different key components. For example, point cloud clusters representing the first worktable 2, the second worktable 3, and the base of the robotic arm 4 are marked with different colors or labels. This clearly illustrates the spatial distribution of each object, and the global point cloud model is output as a three-dimensional map of the entire workspace.

[0073] Subsequently, the original coordinates of the characteristic points of the workbenches, such as the first workbench 2 and the second workbench 3, are extracted from the global point cloud model. Fixed characteristic points on their surfaces are identified. These points are unique, immutable markers of the workbenches, such as the vertices of the workbench corners and the centers of circular calibration targets mounted on the surface. Point cloud analysis techniques, such as calculating the geometric centers and edge intersections of point cloud clusters, are used to determine the three-dimensional coordinates of these characteristic points in the global coordinate system. For example, if a characteristic point is 100cm, 50cm, and 80cm from the origin, this forms the original coordinate set of each workbench's characteristic points, which serves as a benchmark for subsequent position comparisons.

[0074] When the position of the workbench changes due to switching or adjustment, such as the first workbench 2 moves to the standby position and the second workbench 3 moves to the clamping position, new coordinate acquisition is started, and the surface of the workbench after movement is photographed by the camera 5 at the clamping end of the robotic arm 4, and the same fixed feature points as before are identified and extracted, such as the same calibration target, and the two-dimensional coordinates of these points in the image, such as pixel positions; combined with the scale information of the global point cloud model, such as how many pixels in the image correspond to 1 cm in the point cloud model, the image coordinates are converted into new three-dimensional coordinates in the global coordinate system, the new coordinate set of the feature points is output, and the position of the workbench after movement is recorded.

[0075] Finally, a visual calibration algorithm calculates the coordinate transformation relationship. The original and new coordinate sets of the feature points are input into the algorithm. By comparing the position differences of the corresponding feature points, the algorithm calculates the translation parameters of the worktable after the change in position, such as a 30cm movement along the X axis and a 5cm movement along the Y axis. It also calculates the rotation parameters, such as a 5-degree rotation around the Z axis. These parameters are integrated into a complete coordinate transformation relationship, that is, the conversion rule from the original position to the new position. Once output, this transformation relationship can be directly used for the path planning and pose calculation of the robot arm 4. For example, it adjusts the motion trajectory based on the new position of the worktable and corrects the coordinates of the material relative to the robot arm, ensuring that the robot arm can still accurately locate the target after the worktable moves.

[0076] The optimization module also includes collecting operational data such as the robot arm 4's motion, joint forces, workbench switching, and gripping success rate through a sensor network. After cleaning and labeling, the data is classified by type and time sequence, divided into training and validation sets, and outputs a structured system operation status training data set. The training dataset is input into a virtual simulation environment built to scale with physical models. A reinforcement learning strategy is deployed, and a reward function is designed with the goals of improving efficiency, reducing energy consumption, and minimizing collisions. This drives the simulation system to iteratively adjust control parameters. The optimal parameter combination is evaluated and retained based on the validation set, and the optimized control parameter set is output. Import the optimized parameters into the physical control system to replace the original parameters, conduct a trial run and record the new operation data, compare the effect with the simulation expectation, and if there is a deviation, supplement the data and return to the first iteration. After meeting the standards, output the upgraded stable physical control system to complete the closed-loop upgrade.

[0077] By employing the above technical solution, a training dataset for the system's operational status is first constructed. A sensor network, such as encoders for the manipulator's four joints, position sensors for the workstations, force sensors at the gripping end, and a camera 5 for monitoring gripping results, is deployed. This data is then collected in real time from multiple dimensions, including the motion trajectory of each of the manipulator's four joints, such as the rotation angle and speed of each joint; joint force conditions, such as load magnitude and overload; workstation switching time, such as the time it takes to switch from the first workstation 2 to the second workstation 3; and the material gripping success rate, such as the ratio of successful gripping attempts to total attempts. After collection, the data is processed. A cleaning step removes outliers and fills in missing entries. A labeling step associates parameters with operational performance, such as labeling gripping failures as negative examples for the corresponding parameter combination, and labeling fast, collision-free workstation switching as positive examples. The data is then divided into training and validation sets in a 7:3 ratio, categorized by data type (e.g., motion, force control, and time series), and by hour or shift. The data is then divided into training and validation sets in a 7:3 ratio. This ultimately creates a structured training dataset for the system's operational status, providing a foundation for subsequent parameter optimization.

[0078] Next, control parameters are optimized in a virtual simulation environment, and the resulting training dataset is fed into the virtual simulation environment. This environment, built at a 1:1 scale with the physical system, includes a three-dimensional model of the robot arm 4, the first and second workbenches 2 and 3, and the material. It also includes modules that simulate physical properties, such as the friction coefficient of the robot arm 4's joints, the inertia of the workbenches, and the weight and hardness of the material, replicating the physical system's operational scenarios. Deploying a reinforcement learning strategy within this environment hinges on designing a reward function that matches the objective: positive rewards are awarded if the robot arm completes a gripping operation in a shorter time than the historical average, joint energy consumption is below a set threshold, and there are no collisions with the workbenches during movement; negative penalties are imposed otherwise. Driven by training data, the simulation system repeatedly simulates the operation process under different parameter combinations, such as adjusting the robot arm joint movement speed, gripping force, and the workbench switching trigger threshold. Reinforcement learning strategies are used to iteratively optimize through trial and error. After each simulation, the parameter effects are evaluated based on the reward function score, and high-scoring parameter combinations are retained and further fine-tuned. A validation set is also used to verify the generalization ability of the parameters, such as whether they are still effective in new scenarios not involved in training. Ultimately, the parameter combination with the highest reward value is selected, and the optimized control parameter set is output, such as the optimal joint movement speed range, gripping force threshold, and workbench switching trigger conditions. The role of the virtual environment is to avoid the risks of directly testing parameters in the physical system, such as collision damage to the equipment, while efficiently completing thousands of iterative tests.

[0079] The final output of the optimized control parameter set is imported into the parameter configuration module according to the interface protocol of the physical control system, replacing the original default parameters. For example, the default speed of the robot arm's four joints is adjusted from 50° / s to the optimized 65° / s. The physical system is then started for a trial run, and the real-time monitoring module (e.g., camera 5) records operational data under the new parameters, including whether the gripping success rate is improved, the average gripping time is shortened, the joint energy consumption is reduced, and the workstation switching is smoother. The trial run data is compared with the expected results from the virtual simulation. For example, if the simulation predicts a 10% increase in the gripping success rate, if there is a discrepancy, such as a 3% increase in the physical model, which falls short of the simulation expectation, the newly collected trial run data is added to the training dataset from step one. The process then returns to step one for cleansing and annotation, and the parameters are then re-entered into the virtual environment for iterative parameter optimization. If the trial run results are consistent with the simulation expectations, such as if the success rate, energy consumption, and collision rate all meet the standards, the parameters are confirmed to be valid, and the upgraded physical control system is output. At this point, the system has achieved improved operational performance, completing the closed-loop upgrade.

[0080] The above are only preferred embodiments of the present invention; however, the scope of protection of the present invention is not limited thereto. Any person skilled in the art who, within the technical scope disclosed by the present invention, makes equivalent substitutions or modifications based on the technical solutions and improved concepts of the present invention shall be covered by the scope of protection of the present invention.

Claims

1. A collaborative robot target recognition and grasping posture planning system, comprising a table cabinet (1), characterized in that: A first workbench (2) and a second workbench (3) are slidably provided on the cabinet (1), and the second workbench (3) can slide through from below the first workbench (2). Material plates (6) are installed on both the first workbench (2) and the second workbench (3), and the upper surface of the material plate (6) is provided with evenly distributed material troughs (7). A robotic arm (4) is also installed on the cabinet (1), and the robotic arm (4) is used to clamp the material on the material plate (6). A camera (5) is installed at the clamping end of the robotic arm (4) for adjusting the gripping posture of the robotic arm (4) according to data collected by the camera (5); It also includes: The perception module is used to capture material images through deep vision; output the material's six-degree-of-freedom pose based on a deep learning model; and trigger the spinor compensation mechanism to adjust the end-point pose when visual occlusion exceeds the threshold; The planning module is used to integrate the material pose, workbench position, and robot arm state to construct the state space; it uses reinforcement learning to generate collision-free trajectories and optimizes the path with a multi-objective reward function; and it uses dynamic obstacle prediction to avoid the moving parts of the workbench; The control module is used to match the optimal grasping posture based on the trajectory and material characteristics; trigger vision and force closed-loop adjustments when the material deviates; and dynamically optimize grasping parameters using a force-position hybrid model; A scheduling module, configured to switch to a second workbench (3) when the material on the first workbench (2) is exhausted; and trigger a pre-notification of material replacement in a preset window based on a time series prediction; Mapping module, used to reconstruct the workspace model through global scanning; solve the coordinate system transformation relationship after the workbench is changed; The optimization module is used to collect operating data to build a training set; optimize control parameters in a virtual environment and inject them into the system in a closed loop.

2. A collaborative robot target recognition and grasping posture planning system according to claim 1, characterized in that: The perception module also includes: The camera (5) shoots the target material, collects and outputs the original image including its three-dimensional structure, surface texture and surrounding environment; The pose estimation model receives the original image and processes it through the rotation-invariant feature extraction layer, focusing on the visible area, filtering out occlusion interference, and extracting stable features that are not affected by rotation. Based on this, it outputs the material's six-degree-of-freedom pose information and visual occlusion degree assessment results; Comparing the occlusion assessment result with a preset threshold, if the threshold is exceeded, triggering the end-position adaptive adjustment mechanism of the manipulator (4), which combines the posture information, occlusion assessment and kinematic constraints, and calculates the optimal posture compensation for the camera (6) to avoid occlusion through the spinor motion model; The robotic arm (4) adjusts the posture of the end gripping end according to the optimal posture compensation amount, so that the camera (6) avoids occlusion and realigns with the material; after adjustment, the camera (6) collects images again, and the above process is repeated until the occlusion level is lower than the threshold, ensuring that the robotic arm (4) accurately obtains material information to complete gripping.

3. A collaborative robot target recognition and grasping posture planning system according to claim 2, characterized in that: The planning module also includes: Collect data such as material posture, workbench dynamic position and robot arm (4) joint status, and form a standardized multi-dimensional fusion data set through time synchronization and format unification; Based on the fused data set, the core dimensions and motion boundaries of the robot arm (4), workbench, material, etc. are determined and integrated into a structured model that describes the overall state; Based on the real-time position and physical size of the workbench in the planned state space, its space occupation area in the future period is predicted to form a dynamic space-time map of obstacles; Combine path efficiency, energy consumption index and safety margin, and integrate them into a comprehensive reward function according to weights as the trajectory optimization goal; Using the planning state space as the environment, dynamic obstacle prediction as the constraint, and the reward function as the goal, the algorithm generates the optimal collision-free trajectory. Verify whether the trajectory complies with the joint limits of the robot arm (4) and the dynamics of the workbench, provide feedback and adjust when conflicts occur, and finally output a trajectory that complies with the actual constraints.

4. A collaborative robot target recognition and grasping posture planning system according to claim 3, characterized in that: The control module also includes: Combine the motion trajectory parameters of the robot arm (4) with the physical characteristics of the target material, input the preset grasping strategy library, screen out the optimal grasping point, clamping angle and initial grasping force range through the matching algorithm, and output the specific optimal grasping posture parameters; The robot arm (4) performs gripping according to the optimal gripping posture, and at the same time collects the material posture in real time through the camera (6), compares the real-time posture with the expected posture to calculate the offset, and triggers closed-loop force control adjustment if it exceeds the threshold, outputs the posture offset and the initial direction of the force control adjustment; Receive the posture offset and the initial adjustment direction, combine with the real-time position of the end of the robot arm (4), enable the force-position hybrid control model, synchronously correct the position deviation and dynamically adjust the grasping force parameters, and output the optimized grasping force parameters; The trajectory of the robotic arm (4), the real-time status of the workbench and its physical dimensions are collected, and a kinematic model is established to predict the three-dimensional occupied area of ​​the workbench within a preset time period in the future. The occupied area map with a time series is output as a constraint feedback to the first three steps to ensure obstacle avoidance during the grasping process.

5. The collaborative robot target recognition and grasping posture planning system according to claim 4, characterized in that: The scheduling module also includes: The camera (5) at the gripping end of the robotic arm (4) captures an image of the material plate, identifies whether there is material in the material slot (7) and counts the remaining quantity, and outputs an exhaustion signal and current position information of the first workbench (2) when the material is exhausted; Receive the material exhaustion signal and position information, control the cylinder to push the first workbench (2) away and the second workbench (3) to the preset clamping position, and output the second workbench (3) activation signal and arrival confirmation information after the switching is completed; Based on the activation signal of the second workbench (3), the material status of the two workbenches is continuously collected through the camera (5), and historical data and current information are integrated to form a structured data set containing image detection data and time series features; Based on this dataset, we divide the dataset into training and validation sets, design a model with a time series processing layer, adjust the parameters until the prediction error meets the standard, and output the trained model. Input the real-time material data of the second workbench (3) detected by the camera (5) into the model, combine the historical trend with the current gripping frequency, predict the material consumption trend and the estimated exhaustion time in the future preset period, and output the prediction result; Receive the prediction result, compare the remaining consumption time with the preset time window, and generate a pre-notification trigger signal and the predicted exhaustion time point if the conditions are met; The worker is prompted to replace the material plate of the first workbench (2) in time through an audible and visual alarm or a terminal, and feedback notification status is given to ensure that the plate replacement is completed before the material of the second workbench (3) is exhausted.

6. The collaborative robot target recognition and grasping posture planning system according to claim 5, characterized in that: The mapping module also includes: Use the scanning device to perform a full-scale scan of all objects in the workspace, collect regional point cloud data containing three-dimensional coordinates from multiple angles along the preset path, and output multiple sets of original point cloud data with viewpoint labels; Receive raw point cloud data, align multi-view data through feature matching, remove noise and redundant information, merge into a complete point cloud dataset in a unified coordinate system, and output the pre-processed workspace point cloud dataset; Based on the pre-processed point cloud dataset, a global point cloud model containing the three-dimensional shapes of all objects in the workspace is generated through surface reconstruction. The point cloud clusters of each key component are clearly marked and the global point cloud model is output. Identify the fixed feature points of the workbench from the global point cloud model, determine their original three-dimensional coordinates in the global coordinate system through point cloud analysis, and output the original coordinate set of each workbench feature point; After the workbench position is changed, the image coordinates of the same feature point are captured and extracted by the camera (5), and converted into new three-dimensional coordinates in the global coordinate system in combination with the global point cloud scale, and the new coordinate set of the feature point is output; The original coordinate set and the new coordinate set of the feature points are input into the visual calibration algorithm to calculate the translation and rotation parameters after the workbench position changes, integrate them into the coordinate system transformation relationship, and output them for subsequent path planning and pose calculation.

7. The collaborative robot target recognition and grasping posture planning system according to claim 6, characterized in that: The optimization module also includes: The sensor network is used to collect the operation data of the robot arm (4), such as movement, joint force, workbench switching, and gripping success rate. After cleaning and labeling, the data are classified by type and time sequence, divided into training set and verification set, and the structured system operation status training data set is output; The training dataset is input into a virtual simulation environment built to scale with physical models. A reinforcement learning strategy is deployed, and a reward function is designed with the goals of improving efficiency, reducing energy consumption, and minimizing collisions. This drives the simulation system to iteratively adjust control parameters. The optimal parameter combination is evaluated and retained based on the validation set, and the optimized control parameter set is output. Import the optimized parameters into the physical control system to replace the original parameters, conduct a trial run and record the new operation data, compare the effect with the simulation expectation, and if there is a deviation, supplement the data and return to the first iteration. After meeting the standards, output the upgraded stable physical control system to complete the closed-loop upgrade.

8. The collaborative robot target recognition and grasping posture planning system according to claim 7, characterized in that: The perception module also includes: The camera (5) is used to collect images of the target material under different lighting, rotation, occlusion and workbench positions, and the actual six-degree-of-freedom posture is recorded with high-precision equipment. The degree of occlusion is manually marked to form a labeled data set associated with the original image, posture and occlusion degree and output it; Based on the convolutional neural network, an image enhancement module is added to the front end, and a rotation-invariant feature extraction layer is designed. The back end is divided into two branches to output six-degree-of-freedom pose parameters and occlusion degree assessment results, and output the network structure containing the extraction layer; The dataset is input into the network, trained using a joint loss function, and parameters are iteratively adjusted until the validation set indicators meet the standards. Occlusion samples are fine-tuned to enhance feature extraction capabilities, and a pose estimation model with preliminary training is output. The compression model meets the real-time requirements. Through interface adaptation, the model receives camera (6) image input, outputs standardized pose information and occlusion assessment results, and outputs a deployable pose estimation model.

9. The collaborative robot target recognition and grasping posture planning system according to claim 8, characterized in that: The perception module also includes: Collect physical parameters such as the length and range of motion of each joint of the robotic arm (4), and the installation coordinate offset of the camera (5) relative to the end effector, to form a kinematic basic parameter set and output it; Based on the parameters and according to the screw theory, the mapping relationship between the end effector posture and joint angle is established, and the basic screw motion model is output; Controlling the robotic arm (4) to move along a preset trajectory, recording the actual end position through a camera (5), comparing it with the theoretical calculated value of the basic model, adjusting the parameters to reduce the error, and outputting a calibrated spinor motion model; The logic is embedded in the model. After inputting the material posture and occlusion assessment results, the difference between the current camera (5) posture and the unobstructed posture is analyzed, the optimal posture compensation to avoid occlusion is calculated, and the output is a model that can be directly used for the posture adjustment of the robot arm (4).

10. The collaborative robot target recognition and grasping posture planning system according to claim 4, characterized in that: The control module also includes: Collect the structural parameters, force and position sensor data of the robot arm (4) and the physical properties of the material, record the associated data under different grasping scenarios through experiments, form a labeled force and position control data set and output it; Construct a dual closed-loop control framework. The position loop calculates position compensation using proportional, integral, and differential algorithms, while the force loop calculates force compensation using an impedance control algorithm. A weight distribution module is designed to dynamically adjust the output ratio of the two loops based on material characteristics, resulting in a control framework that includes dual closed-loop and weight modules. The dataset is input into the model, with the goal of minimizing position deviation and force fluctuation and maximizing grasping success rate. The position loop proportional coefficient, force loop impedance parameters, and weight threshold are iteratively optimized. Fine-tuning is performed for special scenarios, and the optimized parameter set and preliminary model are output. A real-time feedback module is added to automatically adjust the weights of the position and force loops when the material posture deviation or force value exceeds the limit is detected. The interface adapts to receive inputs such as posture deviation and outputs optimized gripping force parameters and position compensation instructions, forming a model that can respond in real time. The model is embedded in the control system of the robotic arm (4), and the position accuracy and force control stability are verified through physical grasping experiments. When problems exist, the optimized parameters are returned, and finally a force-position hybrid control model that meets the requirements of accuracy and safety is output.

Images