A cooperative robot target recognition and grasping pose planning system

Abstract

The application discloses a kind of collaborative robot target identification and grasping posture planning system, belong to mechanical arm technical field, the present application is provided by the material pose information according to perception module, fusion mechanical arm motion trajectory parameter and target material physical characteristics, from preset grasping strategy library, the optimal grasping posture is intelligently matched, ensure the initial accuracy of grasping scheme, in the process of grasping, camera continuously real-time acquisition material pose, by comparing real-time pose and expected pose, accurately calculate offset.Once detecting material offset, immediately trigger vision, force closed-loop adjustment mechanism, using force position hybrid control model, synchronous correction position deviation and dynamically adjust grasping force parameter, this kind of control not only significantly improves the stability and precision of grasping, but also effectively reduces the failure rate of grasping, greatly reduces artificial intervention and reset time, to comprehensively improve production efficiency.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, and more specifically, to a collaborative robot target recognition and grasping posture planning system. Background Technology

[0002] Currently, industrial automation commonly employs a collaborative operation model involving robotic arms and fixed feeding stations. For example, material trays are fixedly mounted on a worktable, and the robotic arm uses vision positioning to sequentially grab workpieces from the material trough. When the material is depleted, the machine must be stopped to replace the material tray, leading to increased machine tool downtime and a production efficiency loss of approximately 15%-22%. Some systems use cylinders to move individual material trays into or out of the work area, but the robotic arm must completely stop operating during the replacement process, making continuous feeding impossible. This impact is particularly pronounced in high-frequency production scenarios, such as automotive parts processing, where daily downtime can account for up to 18% of total working hours. Many machine tool stations are forced to remain idle due to material replacement, severely restricting production rhythm.

[0003] In addition to the issue of continuous material supply, the gripping efficiency of robotic arms is also problematic. Currently, most robotic arm grippers integrate cameras to identify the material's position and orientation in real time. However, in scenarios where materials are stacked or adjacent workpieces are obstructed by each other, traditional systems often need to adjust the robotic arm's orientation multiple times to complete the positioning. This directly leads to a single gripping cycle being extended to 3-5 seconds, significantly slowing down the production cycle.

[0004] When the robotic arm's gripper approaches the material, its body can easily obstruct the camera's view, causing the real-time positioning function to fail. In this case, the system can only rely on the pre-programmed path for gripping. However, during the production process, the material inevitably deviates from the preset position due to factors such as vibration. This deviation directly leads to a gripping failure rate as high as 12%. After a failure, manual intervention is required to reset the gripper, further increasing time costs and manpower investment.

[0005] In terms of space utilization of the feeding platform, the existing design also has shortcomings. Although the single-layer feeding platform has a simple structure, it cannot effectively utilize vertical space, resulting in a very limited material capacity per unit area, which is difficult to meet the demand for high production capacity. While some companies use stacked feeding racks to increase capacity, although this can increase the amount of material stored, it will greatly expand the range of motion of the robotic arm and increase the positioning error due to the extended movement path, which will affect the grasping accuracy and stability.

[0006] In summary, the current collaborative model between robotic arms and material feeding systems has shortcomings in multiple dimensions, including material feeding continuity, grasping efficiency, and space utilization. These problems are compounded and collectively restrict the improvement of industrial automation efficiency. Summary of the Invention

[0007] To address the problems existing in the prior art, the present invention aims to provide a collaborative robot target recognition and grasping posture planning system. This system can match the optimal grasping posture from a pre-set grasping strategy library based on material pose information provided by the perception module, combined with the robotic arm's motion trajectory parameters and the physical characteristics of the target material. During the grasping process, the system uses a camera to collect the material pose in real time, compares the real-time pose with the expected pose to calculate the offset, and triggers a vision and force closed-loop adjustment mechanism when the material deviates. Through a force-position hybrid control model, it can simultaneously correct positional deviations and dynamically adjust grasping force parameters, ensuring that the robotic arm can stably and accurately complete grasping tasks even in complex environments. This control mechanism greatly improves the stability and accuracy of grasping, reduces the grasping failure rate, reduces manual intervention and reset time, and further improves production efficiency.

[0008] To solve the above problems, the present invention adopts the following technical solution.

[0009] A collaborative robot target recognition and grasping posture planning system includes a cabinet with a first workbench and a second workbench slidably mounted on it. The second workbench can slide through from under the first workbench. Material plates are installed on both the first and second workbench. The upper surface of the material plates has evenly distributed material grooves. A robotic arm is also installed on the cabinet. The robotic arm is used to grasp materials on the material plates. A camera is installed at the grasping end of the robotic arm to adjust the grasping posture of the robotic arm based on the data collected by the camera.

[0010] This also includes:

[0011] The perception module is used to acquire material images through deep vision; output the material's six-degree-of-freedom pose based on a deep learning model; and trigger a spin compensation mechanism to adjust the end effector pose when visual occlusion exceeds the threshold.

[0012] The planning module is used to construct a state space by integrating material pose, workbench position, and robotic arm state; it uses reinforcement learning to generate collision-free trajectories and optimizes the path with a multi-objective reward function; and it avoids moving parts of the workbench through dynamic obstacle prediction.

[0013] The control module is used to match the optimal grasping posture based on the trajectory and material characteristics; triggers visual and force closed-loop adjustments when the material deviates; and dynamically optimizes the grasping parameters using a force-position hybrid model.

[0014] The scheduling module is used to switch to the second workbench when the material on the first workbench is exhausted; it triggers a material replacement pre-notification within a preset window based on time-series prediction.

[0015] The mapping module is used to reconstruct the workspace model through global scanning and solve the coordinate system transformation relationship after the workbench is changed.

[0016] The optimization module is used to collect runtime data to build a training set; optimize control parameters in a virtual environment and inject them into the system in a closed loop.

[0017] Furthermore, the sensing module also includes:

[0018] The camera captures images of the target material, collecting and outputting raw images containing its three-dimensional structure, surface texture, and surrounding environment.

[0019] The pose estimation model receives the original image, processes it through a rotation-invariant feature extraction layer, focuses on the visible area, filters out occlusion interference, extracts stable features that are not affected by rotation, and outputs the material's six-degree-of-freedom pose information and visual occlusion assessment results.

[0020] Compare the occlusion assessment results with the preset threshold. If the threshold is exceeded, the robotic arm end-effector posture adaptive adjustment mechanism is triggered. This mechanism combines pose information, occlusion assessment and kinematic constraints, and calculates the optimal pose compensation amount to make the camera avoid occlusion through a screw motion model.

[0021] The robotic arm adjusts the posture of its end gripper based on the optimal pose compensation amount, allowing the camera to avoid obstruction and re-align with the material. After adjustment, the camera acquires images again, and the above process is repeated until the degree of obstruction is below the threshold, ensuring that the robotic arm accurately obtains material information to complete the gripping.

[0022] Furthermore, the planning module also includes:

[0023] Collect data such as material pose, dynamic position of worktable and joint status of robotic arm, and form a standardized multi-dimensional fusion dataset through time synchronization and format unification;

[0024] Based on the fusion dataset, the core dimensions and motion boundaries of the robotic arm, workbench, and materials are determined and integrated into a structured model that describes the overall state.

[0025] Based on the real-time position and physical dimensions of the workbench in the planned state space, predict its space occupancy area in the future period and form a dynamic obstacle spatiotemporal map;

[0026] Combining path efficiency, energy consumption indicators, and safety margins, these are integrated into a comprehensive reward function based on weights, which serves as the trajectory optimization objective.

[0027] Using the planning state space as the environment, dynamic obstacle prediction as the constraint, and the reward function as the objective, the algorithm generates the optimal collision-free motion trajectory.

[0028] Verify whether the trajectory conforms to the limits of the robotic arm joints and the dynamics of the worktable, provide feedback and adjustments when conflicts occur, and finally output a trajectory that conforms to the actual constraints.

[0029] Furthermore, the control module also includes:

[0030] Combining the motion trajectory parameters of the robotic arm with the physical characteristics of the target material, the system inputs a preset grasping strategy library, uses a matching algorithm to filter out the optimal grasping point, gripping angle, and initial grasping force range, and outputs the specific optimal grasping posture parameters.

[0031] The robotic arm performs gripping in the optimal grasping posture, while the camera collects the material's posture in real time. The offset is calculated by comparing the real-time posture with the expected posture. If the offset exceeds the threshold, a closed-loop force control adjustment is triggered, and the posture offset and the initial direction of the force control adjustment are output.

[0032] Receive the pose offset and initial adjustment direction, combine them with the real-time position of the robotic arm end effector, enable the force-position hybrid control model, simultaneously correct the position deviation and dynamically adjust the gripping force parameters, and output the optimized gripping force parameters.

[0033] Collect the robotic arm trajectory, real-time status and physical dimensions of the workbench, establish a kinematic model to predict the three-dimensional occupied area of ​​the workbench within a preset time period, and output an occupied area map with time series as a constraint feedback to the first three steps to ensure obstacle avoidance during the grasping process.

[0034] Furthermore, the scheduling module also includes:

[0035] The robotic arm uses a camera at the gripping end to capture images of the material board, identify whether there is material in the material trough and count the remaining quantity. When the material is exhausted, it outputs an exhaustion signal and the current position information of the first worktable.

[0036] Upon receiving the material depletion signal and position information, the system controls the cylinder to push the first worktable away and the second worktable to the preset clamping position. After the switch is completed, the system outputs the second worktable activation signal and the arrival confirmation information.

[0037] Based on the activation signal of the second workbench, the material status of the two workbench is continuously collected by the camera, and historical data and current information are integrated to form a structured dataset containing image detection data and time series features.

[0038] Based on this dataset, we divide it into training and validation sets, design a model with a time series processing layer, train and adjust the parameters until the prediction error reaches the target, and output the trained model.

[0039] Input the real-time material data of the second workbench detected by the camera into the model, combine historical trends and current gripping frequency, predict the material consumption trend and expected depletion time for a future preset period, and output the prediction results.

[0040] Receive the prediction results, compare the remaining consumption time with the preset time window, and if the conditions are met, generate a pre-notification trigger signal and the predicted exhaustion time point;

[0041] Workers are prompted to replace the material tray on the first workbench in a timely manner through audible and visual alarms or terminal prompts, and the status is reported back to ensure that the tray replacement is completed before the material on the second workbench is exhausted.

[0042] Furthermore, the mapping module also includes:

[0043] The scanning device performs a full-range scan of all objects in the workspace, collects regional point cloud data containing three-dimensional coordinates from multiple angles along a preset path, and outputs multiple sets of raw point cloud data with viewpoint markings.

[0044] Receive raw point cloud data, align multi-view data through feature matching, remove noise and redundant information, merge into a complete point cloud dataset under a unified coordinate system, and output a preprocessed workspace point cloud dataset.

[0045] Based on the preprocessed point cloud dataset, a global point cloud model containing the three-dimensional shape of all objects in the workspace is generated through surface reconstruction, the point cloud clusters of each key component are clearly marked, and the global point cloud model is output.

[0046] Identify fixed feature points of the workbench from the global point cloud model, determine their original three-dimensional coordinates in the global coordinate system through point cloud analysis, and output the original coordinate set of each workbench feature point;

[0047] After the workbench position is changed, the image coordinates of the same feature point are captured by the camera and extracted. The coordinates are then converted into new 3D coordinates in the global coordinate system by combining the global point cloud scale, and a new set of feature point coordinates is output.

[0048] The original coordinate set and the new coordinate set of feature points are input into the visual calibration algorithm to calculate the translation and rotation parameters after the change of the workbench position. These parameters are then integrated into a coordinate system transformation relationship, and the output is used for subsequent path planning and pose calculation.

[0049] Furthermore, the optimization module also includes:

[0050] The system collects operational data such as robotic arm movement, joint forces, worktable switching, and gripping success rate through a sensor network. After cleaning and labeling, the data is classified by type and time sequence, and divided into training and validation sets to output a structured system operation status training dataset.

[0051] The training dataset is input into a virtual simulation environment built according to the entity scale. A reinforcement learning strategy is deployed. A reward function is designed with the goal of improving efficiency, reducing energy consumption, and reducing collisions. The simulation system is driven to iteratively adjust the control parameters. The optimal parameter combination is evaluated and retained in combination with the validation set, and the optimized control parameter set is output.

[0052] The optimized parameters are imported into the physical control system to replace the original parameters. The system is then tested and the new running data is recorded. The results are compared with the simulation expectations. If there is a deviation, the data is supplemented and the system is returned to the first iteration. Once the target is met, the upgraded stable physical control system is output, and the closed-loop upgrade is completed.

[0053] Furthermore, the sensing module also includes:

[0054] Images of the target material under different lighting, rotation, occlusion and workbench positions are captured by a camera. The actual six-degree-of-freedom pose is recorded by high-precision equipment. The degree of occlusion is manually labeled to form a labeled dataset that associates the original image, pose and degree of occlusion and outputs it.

[0055] Based on a convolutional neural network, an image enhancement module is added to the front end, a rotation-invariant feature extraction layer is designed, and the back end is divided into two branches to output six-degree-of-freedom pose parameters and occlusion evaluation results, and outputs the network structure containing the extraction layer.

[0056] The dataset is input into the network, trained using a joint loss function, and the parameters are iteratively adjusted until the validation set metrics meet the requirements. Fine-tuning is performed on occluded samples to enhance feature extraction capabilities, and the preliminarily trained pose estimation model is output.

[0057] The compressed model meets real-time requirements. Through interface adaptation, the model can receive camera image input, output standardized pose information and occlusion evaluation results, and output a deployable pose estimation model.

[0058] Furthermore, the sensing module also includes:

[0059] Collect physical parameters such as the length and range of motion of each joint of the robotic arm, as well as the installation coordinate offset of the camera relative to the end effector, to form a set of basic kinematic parameters and output them;

[0060] Based on the parameters, the mapping relationship between the end effector pose and the joint angle is established according to the screw theory, and the basic model of screw motion is output.

[0061] The robotic arm is controlled to move along a preset trajectory. The actual end-effector pose is recorded by a camera, compared with the theoretical calculation value of the basic model, and the parameters are adjusted to reduce the error. The calibrated screw motion model is then output.

[0062] The logic is embedded in the model. After inputting the material pose and occlusion evaluation results, it analyzes the difference between the current camera pose and the unoccluded pose, calculates the optimal pose compensation amount to avoid occlusion, and outputs a model that can be directly used for robotic arm posture adjustment.

[0063] Furthermore, the control module also includes:

[0064] Collect the structural parameters of the robotic arm, force and position sensor data, and material physical properties. Record the correlation data under different grasping scenarios through experiments to form a labeled force and position control dataset and output it.

[0065] A dual-closed-loop control framework is constructed. The position loop calculates the position compensation amount using proportional, integral and derivative algorithms, while the force loop calculates the force compensation amount using an impedance control algorithm. A weight allocation module is designed to dynamically adjust the output ratio of the two loops according to the material characteristics, and the output is a control framework containing the dual closed loops and the weight module.

[0066] Input the dataset into the model with the goal of minimizing position deviation, force fluctuation and maximizing the grasping success rate. Iteratively optimize the position loop proportional coefficient, force loop impedance parameter and weight threshold, fine-tune for special scenarios, and output the optimized parameter set and preliminary model.

[0067] A real-time feedback module is added. After detecting material pose deviation or force value exceeding the limit, the weights of the position loop and force loop are automatically adjusted. The model receives inputs such as pose deviation through interface adaptation and outputs optimized grasping force parameters and position compensation instructions to form a model that can respond in real time.

[0068] The model is embedded into the robotic arm control system, and the position accuracy and force control stability are verified through physical grasping experiments. If problems are found, the optimized parameters are returned, and the final output is a force-position hybrid control model that meets both accuracy and safety requirements.

[0069] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0070] (1) Based on the material pose information provided by the sensing module, this solution combines the motion trajectory parameters of the robotic arm and the physical characteristics of the target material to match the optimal grasping posture from the preset grasping strategy library. During the grasping process, the material pose is collected in real time by the camera, and the offset is calculated by comparing the real-time pose with the expected pose. When the material deviates, the vision and force closed-loop adjustment mechanism is triggered. Through the force-position hybrid control model, the position deviation can be corrected simultaneously and the grasping force parameters can be dynamically adjusted to ensure that the robotic arm can complete the grasping task stably and accurately in complex environments. This control mechanism greatly improves the stability and accuracy of grasping, reduces the grasping failure rate, reduces manual intervention and reset time, and further improves production efficiency.

[0071] (2) When the material on the first workbench is exhausted, this solution can automatically switch to the second workbench to continue the operation. At the same time, it prompts the workers to replace the material plate on the first workbench, realizing continuous material supply without stopping the machine. This greatly reduces the machine tool waiting time, improves production efficiency, is suitable for high-frequency production scenarios, effectively solves the problem of idle workstations caused by material replacement, and promotes the acceleration of production pace. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0073] Figure 1 This is an overall structural appearance view of the present invention;

[0074] Figure 2 This is a schematic diagram of the structure of the camera in this invention;

[0075] Figure 3 This is a diagram illustrating the material tank of the present invention;

[0076] Figure 4 This is a diagram showing the cylinder part of the present invention;

[0077] Figure 5 This is a flowchart of a collaborative robot target recognition and grasping posture planning system.

[0078] Explanation of the labels in the diagram:

[0079] 1. Cabinet; 2. First workbench; 3. Second workbench; 4. Robotic arm; 5. Camera; 6. Material board; 7. Material trough; 8. Cylinder; 9. Safety grille. Detailed Implementation

[0080] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0081] Please see Figures 1 to 5 A collaborative robot target recognition and grasping posture planning system includes a cabinet 1, on which a first workbench 2 and a second workbench 3 are slidably mounted. The second workbench 3 can slide under the first workbench 2. Material plates 6 are installed on both the first workbench 2 and the second workbench 3. The upper surface of the material plates 6 has evenly distributed material grooves 7. A robotic arm 4 is also installed on the cabinet 1. The robotic arm 4 is used to grasp the material on the material plates 6. A camera 5 is installed at the grasping end of the robotic arm 4 to adjust the grasping posture of the robotic arm 4 according to the data collected by the camera 5.

[0082] In operation, the material plate 6 is first installed on the first workbench 2 and the second workbench 3. The material slot 7 is filled with the material to be processed. Initially, the second workbench 3 is close to the robotic arm 4. The robotic arm 4 uses its mechanical chuck to pick up the material from the second workbench 3 one by one. The cabinet 1 is placed next to the machine tool. The robotic arm 4 will place the picked-up material onto the machine tool's processing table, providing automatic feeding service for the machine tool. After the material on the second workbench 3 is picked up, the corresponding cylinder 8 will push the second workbench 3 away from the robotic arm 4. At the same time, the corresponding cylinder 8 will push the first workbench 2 closer to the robotic arm 4. At this time, the robotic arm 4 will pick up the material on the first workbench 2 to provide feeding service for the machine tool. While the robotic arm 4 is picking up the material on the first workbench 2 for feeding, the worker can either remove the material plate 6 from the second workbench 3 for replenishment, or leave it in place and directly add material to the material slot 7 inside the material plate 6. After the material is filled, if the material on the first workbench 2 is used up, the second workbench 3 is moved closer to the robotic arm 4, and the first workbench 2 is moved away from the robotic arm 4 to perform a replenishment operation, thus achieving non-stop material feeding and greatly improving processing efficiency. It should be noted that two sets of cylinders 8 are installed inside the cabinet 1. One set of cylinders 8 is used to control the movement of the first workbench 2, and the other set is used to control the movement of the second workbench 3. One end of the first workbench 2 and the second workbench 3 extends into the cabinet 1, and their extensions are connected to the corresponding cylinders 8. Thus, when the corresponding cylinders 8 extend or retract, they can drive the corresponding first workbench 2 and second workbench 3 to move. A safety grille 9 is also installed on the cabinet 1. During daily production operations, when workers need to disassemble material plates 6 or perform material loading operations on the left side of the safety grille 9, it effectively prevents direct contact between workers and the robotic arm 4. If the robotic arm 4 collides or comes into contact with a human body while operating at high speed, it may cause serious bodily injury to the worker. The presence of the safety grille 9 reduces the probability of such danger. For example, without safety grilles 9, workers focused on disassembling material trays 6 or loading materials may accidentally touch the operating robotic arm 4 due to carelessness or limited operating space. Such accidental contact may result in minor injuries such as abrasions or sprains to the worker's hand or body; or more serious injuries such as fractures, limb crushing, or even more severe disabilities.

[0083] In some embodiments of the present invention, it further includes: a sensing module, used to acquire material images through depth vision; output the material's six-degree-of-freedom pose based on a deep learning model; and trigger a spin compensation mechanism to adjust the end-effector pose when visual occlusion exceeds a threshold;

[0084] The planning module is used to construct a state space by integrating material pose, workbench position, and robotic arm state; it uses reinforcement learning to generate collision-free trajectories and optimizes the path with a multi-objective reward function; and it avoids moving parts of the workbench through dynamic obstacle prediction.

[0085] The control module is used to match the optimal grasping posture based on the trajectory and material characteristics; triggers visual and force closed-loop adjustments when the material deviates; and dynamically optimizes the grasping parameters using a force-position hybrid model.

[0086] The scheduling module is used to switch to the second workbench 3 when the material on the first workbench 2 is exhausted; and to trigger a material replacement pre-notification within a preset window based on time-series prediction.

[0087] The mapping module is used to reconstruct the workspace model through global scanning and solve the coordinate system transformation relationship after the workbench is changed.

[0088] The optimization module is used to collect runtime data to build a training set; optimize control parameters in a virtual environment and inject them into the system in a closed loop.

[0089] The sensing module also includes:

[0090] Camera 5 first captures and outputs raw images of the target material, including its three-dimensional structure, surface texture, and surrounding environment;

[0091] The pose estimation model receives the original image, processes it through a rotation-invariant feature extraction layer, focuses on the visible area, filters out occlusion interference, extracts stable features that are not affected by rotation, and outputs the material's six-degree-of-freedom pose information and visual occlusion assessment results.

[0092] Compare the occlusion assessment results with the preset threshold. If the threshold is exceeded, the end-effector posture adaptive adjustment mechanism of the robotic arm 4 is triggered. This mechanism combines pose information, occlusion assessment and kinematic constraints, and calculates the optimal pose compensation amount to make the camera 5 avoid occlusion through a screw motion model.

[0093] The robotic arm 4 adjusts the posture of its end gripper according to the optimal pose compensation amount, so that the camera 5 avoids obstruction and re-aligns with the material; after adjustment, the camera 5 collects images again, and the above process is repeated until the degree of obstruction is lower than the threshold, ensuring that the robotic arm 4 accurately obtains material information to complete the gripping.

[0094] By adopting the above technical solution, camera 5 serves as the initial visual perception device to capture images of the target material, including the material's three-dimensional structure, such as its outline and size; surface texture, such as its patterns and color distribution; and the surrounding environment, such as the edge of the workbench and other objects, providing basic visual data for subsequent processing.

[0095] Next, after receiving the raw image output from camera 5, the pose estimation model focuses on the visible area of ​​the material, such as the unobstructed surface, through a rotation-invariant feature extraction layer, filtering out interference information from parts obscured by other objects, such as the side of the material obscured by the edge of the workbench. Simultaneously, this layer specially encodes the material's edges, textures, and other features, ensuring that the extracted features are unaffected by the material's own rotation state; for example, features can still be stably identified even after the material is flipped. Based on these stable features, the model calculates the material's six-degree-of-freedom pose information, namely its three-dimensional spatial position and three-dimensional rotation angle. It also analyzes the proportion of obscured areas in the image and outputs a visual occlusion assessment result, such as an obscured area accounting for 30% of the total material area.

[0096] Subsequently, the occlusion assessment result is compared with a preset threshold, for example, 20%. If the assessment result exceeds the threshold, such as 30% > 20%, the adaptive adjustment mechanism of the robotic arm 4's end effector is triggered. The mechanism combines the material's six-DOF pose information output by the pose estimation model, i.e., clarifying the material's position; the occlusion assessment result, i.e., clarifying the occlusion position and degree; and the robotic arm 4's own kinematic constraints, such as joint range of motion and maximum rotation angle. The adjustment scheme is calculated through a screw motion model, which can simultaneously consider the coordinated compensation of translation and rotation, and finally obtains the optimal pose compensation amount that allows the camera 5 to avoid the occluded area, such as translating 5 cm to the right and rotating 10 degrees.

[0097] Finally, robotic arm 4 adjusts the posture of its end effector based on the optimal pose compensation, moving camera 5 to a new position to avoid the original occlusion area and re-align it with the material. After adjustment, camera 5 acquires images again, and the new images are re-input into the pose estimation model, repeating the above process. This closed-loop cycle continues until the occlusion assessment result is lower than a preset threshold, such as 15% < 20%, ensuring that robotic arm 4 can obtain clear and complete material information, providing accurate basis for subsequent gripping actions.

[0098] The construction of the pose estimation model includes:

[0099] Images of the target material under different lighting, rotation, occlusion and workbench positions are acquired by camera 5. The actual six-degree-of-freedom pose is recorded by high-precision equipment, and the degree of occlusion is manually labeled to form a labeled dataset that associates the original image, pose and degree of occlusion and output it.

[0100] Based on a convolutional neural network, an image enhancement module is added to the front end, a rotation-invariant feature extraction layer is designed, and the back end is divided into two branches to output six-degree-of-freedom pose parameters and occlusion evaluation results, and outputs the network structure containing the extraction layer.

[0101] The dataset is input into the network, trained using a joint loss function, and the parameters are iteratively adjusted until the validation set metrics meet the requirements. Fine-tuning is performed on occluded samples to enhance feature extraction capabilities, and the preliminarily trained pose estimation model is output.

[0102] The compressed model meets real-time requirements. Through interface adaptation, the model can receive image input from camera 5, output standardized pose information and occlusion evaluation results, and output a deployable pose estimation model.

[0103] By adopting the above technical solution, during the dataset construction phase, rich training samples need to be provided for the model. Images of the target material in diverse scenarios are pre-collected using camera 5, covering different lighting intensities, such as strong light and shadow; the material's own rotation angle, such as 0° to 360° rotation; the degree of occlusion, such as partial occlusion by the edge of the workbench or occlusion by other stacked materials; and the state at different positions on the first and second workbenches, ensuring that the samples cover all scenarios that may occur in actual work. Simultaneously, high-precision measurement equipment such as a laser tracker is used to synchronously record the actual six-degree-of-freedom pose of the material in each image, serving as the ground truth label for pose prediction. The proportion of the occluded area to the total area of ​​the material in the image is manually labeled; for example, 30% occlusion is labeled as 0.3, serving as the label for the degree of occlusion. Finally, the original images, six-degree-of-freedom poses, and occlusion degrees are correlated to form a structured labeled dataset, providing a data foundation for model training.

[0104] Based on a convolutional neural network framework, an image enhancement module is added to the front end. This module processes the original image through random rotation, scaling, and local occlusion simulation to enhance the model's adaptability to changes in the input image, such as handling slight shifts in the camera's shooting angle. The core layer is a rotation-invariant feature extraction layer. This layer uses special encoding techniques to extract features such as material edge contours and surface textures, mapping these features to a spherical coordinate system to eliminate the influence of rotation direction. This ensures that the extracted features are unaffected by the material's own rotation; for example, key features can still be stably identified even after the material is flipped. The back end has two parallel outputs: one branch performs stepwise regression calculations through a fully connected layer to output six-DOF pose parameters; the other branch analyzes occlusion region features through a classification layer to output an occlusion degree assessment result, forming a network structure that can simultaneously handle pose estimation and occlusion judgment.

[0105] The constructed dataset is proportionally divided into training and validation sets, and the designed network structure is input. A joint loss function is used for training: one part of the loss constrains the deviation between the model's predicted pose and the actual pose recorded by a high-precision device, such as positional and angular deviations, ensuring the accuracy of pose estimation; the other part of the loss constrains the deviation between the occlusion level output by the model and the manually labeled level, improving the reliability of occlusion judgment. Network parameters, such as convolutional kernel weights and fully connected layer coefficients, are continuously adjusted through backpropagation until the pose prediction error and occlusion assessment accuracy on the validation set reach a preset standard. For example, the average positional deviation of the pose prediction error is less than 2 mm, and the occlusion assessment accuracy is higher than 90%, to meet the preset standard. For a severely occluded sample subset, such as images with an occlusion rate exceeding 50%, separate reinforcement training is performed to adjust the network's focus on occlusion features, enhancing the model's feature extraction capability in complex occlusion scenarios, and outputting the pre-trained model.

[0106] Finally, non-core redundant layers in the network are simplified, and the number of parameters is reduced, such as quantizing 32-bit parameters to 16-bit, reducing computation time and ensuring that the model can achieve millisecond-level inference in the robotic arm control system. Simultaneously, interface adaptation is performed, enabling the model to directly receive raw image data from camera 5 and output standardized six-DOF pose information according to the system protocol, such as uniformly formatted 3D coordinates and angle values, and occlusion assessment results, such as quantized values ​​between 0 and 1. This ultimately forms a pose estimation model that can be directly deployed to a real-world system, providing accurate visual feedback for the robotic arm's posture adjustment.

[0107] The construction of the spinor motion model includes:

[0108] Collect physical parameters such as the length and range of motion of each joint of the robotic arm 4, and the installation coordinate offset of the camera 5 relative to the end effector, to form a set of basic kinematic parameters and output them;

[0109] Based on the parameters, the mapping relationship between the end effector pose and the joint angle is established according to the screw theory, and the basic model of screw motion is output.

[0110] The robotic arm 4 is controlled to move along a preset trajectory. The actual end-effector pose is recorded by the camera 5, compared with the theoretical calculation value of the basic model, and the parameters are adjusted to reduce the error. The calibrated spinor motion model is then output.

[0111] The logic is embedded in the model. After inputting the material pose and occlusion evaluation results, it analyzes the difference between the current 5-position pose of the camera and the pose without occlusion, calculates the optimal pose compensation amount to avoid occlusion, and outputs a model that can be directly used for the 4-position adjustment of the robotic arm.

[0112] By adopting the above technical solution, the physical structural parameters of the robotic arm 4 are collected in advance using a laser rangefinder, including the length of the links between each joint, such as the distance from the shoulder to the elbow and from the elbow to the wrist; the range of motion of each joint, such as the maximum angle that the shoulder joint can rotate; the joint type, such as a rotary joint or a translational joint; and the installation position parameters of the camera 5 on the end effector of the robotic arm 4, that is, the three-dimensional coordinate offset of the center of the camera 5 relative to the reference point of the end effector, such as an offset of 2 cm along the X-axis and an offset of 1 cm along the Y-axis. These parameters are integrated to form a set of basic kinematic parameters.

[0113] Based on the parameter set, a model is constructed using screw theory. Screw theory can simultaneously describe the translational motion of the robotic arm's end effector, such as forward, backward, left, and right movements; and rotational motion, such as rotation around an axis. The motion of each joint is considered a screw, containing both direction and magnitude. By integrating the screw information of each joint, the model establishes a mapping relationship between the end effector's pose and the angles of each joint. For example, when the shoulder joint rotates 30 degrees and the elbow joint rotates 20 degrees, the model can calculate the specific position and orientation of the end effector. The final output screw motion model can directly derive the end effector pose from the input joint angles, or inversely deduce the required joint motion from the target end effector pose.

[0114] The robotic arm 4 is controlled to move along a preset trajectory, such as moving its end effector from point A to point B along a straight line or rotating 90 degrees around a fixed point. Simultaneously, camera 5 captures the actual pose of the end effector in real time, such as changes in the position of marker points. The actual pose is compared with the theoretical pose calculated by the screw motion model to identify deviations, such as a theoretical movement of 10 cm but an actual movement of only 9.5 cm. Based on these deviations, model parameters are adjusted, such as correcting minor errors in the link length, adjusting the theoretical length from 10 cm to 9.8 cm. The joint stiffness coefficient is optimized to compensate for motion lag caused by joint clearance. This process is iterated until the deviation between the theoretical calculation and the actual movement is below a preset threshold, such as less than 0.5 mm, at which point the calibrated screw motion model is output.

[0115] The planning module also includes:

[0116] Collect data such as material pose, dynamic position of workbench and status of 4 joints of robotic arm, and form a standardized multi-dimensional fusion dataset through time synchronization and format unification;

[0117] Based on the fusion dataset, the core dimensions and motion boundaries of the robotic arm 4, workbench, and materials are determined and integrated into a structured model describing the overall state.

[0118] Based on the real-time position and physical dimensions of each workbench in the planned state space, predict the space occupied by them in the future period and form a dynamic obstacle spatiotemporal map.

[0119] Combining path efficiency, energy consumption indicators, and safety margins, these are integrated into a comprehensive reward function based on weights, which serves as the trajectory optimization objective.

[0120] Using the planning state space as the environment, dynamic obstacle prediction as the constraint, and the reward function as the objective, the algorithm generates the optimal collision-free motion trajectory.

[0121] Verify whether the trajectory conforms to the limits of the four joints of the robotic arm and the dynamics of the worktable. Feedback and adjustments are made when there is a conflict, and the final output is a trajectory that conforms to the actual constraints.

[0122] By adopting the above technical solution, the three-dimensional position of the material, the current coordinates of the first workbench 2 and the second workbench 3, and the status information of each joint of the robotic arm 4 are first obtained. The data format of this information is unified, such as the coordinate system 1 being the global coordinate system. Finally, a standardized fusion dataset containing multi-dimensional information of the material, workbench, and robotic arm 4 is formed, providing a consistent data foundation for subsequent analysis.

[0123] The spatial pose of the robotic arm's end effector, the angular range of motion of each joint, the real-time position of the worktable, and the position parameters of the target material are extracted from the data. At the same time, the motion boundaries of each dimension are defined, such as the maximum rotation angle of the robotic arm joints and the movement range of the worktable. These dimensions and boundaries are integrated into a structured model to fully describe the overall motion state of the system at any time, providing a complete picture of the environment for trajectory planning.

[0124] Then, dynamic obstacle modeling is performed based on the planned state space, such as the real-time position and physical size of each workbench. A kinematic model is established to simulate its motion law. The model predicts the position change trajectory of the workbench in the future time period. Then, combined with the size, the three-dimensional occupied area of ​​each workbench in the space at each moment is calculated, such as the cube range marked by the coordinate frame, forming a dynamic obstacle spatiotemporal map containing the time series, and clarifying the area that the robotic arm 4 needs to avoid in the future time period.

[0125] With trajectory optimization as the goal, three key indicators are integrated: a path efficiency indicator, which rewards higher performance for shorter completion times and shorter end-effector distances; an energy consumption indicator, which rewards higher performance for larger joint rotation angles and faster speeds, with rewards deducted if a threshold is exceeded; and a safety margin indicator, which rewards higher performance for larger minimum distances between the robotic arm and obstacles such as workbenches, with penalties triggered if the distance is less than a safety threshold. Weights are assigned to these three indicators according to actual needs, with safety weighted higher than efficiency, forming a comprehensive reward function that serves as the core standard for evaluating trajectory quality.

[0126] Based on the above, an optimal collision-free trajectory is generated through reinforcement learning. The planning state space serves as the algorithm's environment, the spatiotemporal map of dynamic obstacles acts as motion constraints (e.g., the trajectory cannot enter predicted occupied areas), and the comprehensive reward function serves as the learning objective. The algorithm explores possible motion paths for the robotic arm in the state space. For each candidate trajectory generated, its score is calculated using the reward function, prioritizing high-reward trajectories. Simultaneously, it checks in real-time whether the trajectory overlaps with the future occupied areas of dynamic obstacles; if a collision occurs, the trajectory is eliminated. Through multiple rounds of iterative learning, the trajectory parameters are gradually optimized, ultimately outputting an optimal collision-free motion trajectory that balances path efficiency, low energy consumption, and high safety margin.

[0127] Finally, trajectory verification and adaptation are performed, comparing the generated trajectory with the actual constraints: checking whether the trajectory is within the joint range of the robotic arm 4 (e.g., the joint angles do not exceed the maximum range) and whether it can adapt to the dynamic movement of the worktable (e.g., the trajectory time matches the changes in the worktable position). If there are conflicts, such as joint angles exceeding limits, the problem is fed back to the preceding steps, the weights of the reward function or the boundary parameters of the state space are adjusted, and the trajectory is regenerated; this verification and adjustment is repeated until the trajectory fully conforms to the actual operating constraints, and finally, an executable robotic arm motion trajectory is output.

[0128] The control module also includes a pre-set grasping strategy library that combines the motion trajectory parameters of the robotic arm 4 with the physical characteristics of the target material, selects the optimal grasping point, gripping angle and initial grasping force range through a matching algorithm, and outputs the specific optimal grasping posture parameters.

[0129] The robotic arm 4 performs gripping in the optimal grasping posture, while the camera 5 collects the material's posture in real time. It compares the real-time posture with the expected posture to calculate the offset. If the offset exceeds the threshold, it triggers closed-loop force control adjustment and outputs the posture offset and the initial direction of force control adjustment.

[0130] Receive the pose offset and initial adjustment direction, combine them with the real-time position of the robotic arm's end effector, enable the force-position hybrid control model, synchronously correct the position deviation and dynamically adjust the gripping force parameters, and output the optimized gripping force parameters.

[0131] Collect the robotic arm's 4-axis trajectory, the real-time status of the workbench, and its physical dimensions. Establish a kinematic model to predict the 3D occupied area of ​​the workbench within a preset time period in the future. Output an occupied area map with a time series as a constraint feedback to the first three steps to ensure obstacle avoidance during the grasping process.

[0132] By adopting the above technical solution, the motion trajectory parameters of the robotic arm 4 are first obtained, such as the spatial coordinates of the end effector reaching the material, the angle of approach to the material, and the physical characteristics of the target material, such as whether the shape is regular, the weight, whether the surface is smooth, and whether there are vulnerable parts. This information is input into a pre-set gripping strategy library, which stores the optimal solutions for different materials, including the gripping point, such as the center of gravity position or anti-slip contact area; the gripping angle, such as the angle with the material axis; and the initial gripping force range, such as 5-10N for light materials and 20-30N for heavy materials. Through a matching algorithm, such as searching based on shape similarity and weight range, the solution that best matches the current material characteristics and trajectory parameters is selected from the library, and the specific optimal gripping posture parameters are output, such as the gripping point coordinates, a 60-degree gripping angle, and an initial force of 10-15N.

[0133] The robotic arm 4 performs a gripping action based on the output optimal grasping posture. Simultaneously, the camera 5 installed at the end effector captures real-time images of the material in the gripping state. Image recognition is used to extract the material's real-time pose, 3D position, and rotation angle. The real-time pose is compared with the expected pose to calculate the offset (e.g., a position deviation of 3 mm or an angle deviation of 8 degrees). If the offset exceeds a preset threshold (e.g., a position deviation > 2 mm or an angle deviation > 5 degrees), a closed-loop force control adjustment mechanism based on visual feedback is triggered. This mechanism outputs specific pose offset data and the initial direction of force control adjustment, such as a compensating force in the opposite direction of the offset.

[0134] Then, the gripping force is optimized through force-position hybrid control. The system receives the output pose offset and initial adjustment direction, and combines this with real-time position data from the robotic arm's end effector, such as the relative distance to the material, to activate the force-position hybrid control model. This model corrects the spatial position of the robotic arm's end effector through position control, such as fine-tuning joint angles to reduce positional deviation by 3 mm. Furthermore, based on real-time data from the force feedback sensor at the gripping end, such as a current gripping force of 12 N, it dynamically adjusts the gripping force parameters: if the material is detected to be slipping due to insufficient friction, the gripping force in the corresponding area is increased, such as to 14 N; if it contacts a vulnerable area, the pressure is reduced, such as to 10 N. By balancing positional accuracy and gripping force, the optimized gripping force parameters are output, such as the force distribution of each gripping finger and the 10-14 N force holding range.

[0135] Finally, the motion trajectory of robotic arm 4 is collected, such as the path planning for the next 5 seconds. The position change of the workbench within the preset time period is predicted by the kinematic model, and the three-dimensional occupied area of ​​the workbench in space at each moment is calculated, such as the range of a cuboid marked with a coordinate frame. The occupied area map with time series is output. This map is used as a constraint condition. For example, if it is predicted that the workbench will move to a certain area in 1 second, the trajectory adjustment range of robotic arm 4 will be limited to avoid entering that area, or the matching results of the grasping strategy library will be corrected to avoid grasping angles that conflict with the workbench. This ensures that there is no collision between the robotic arm, the material and the workbench during the entire grasping process, realizing the whole process optimization from posture planning to dynamic adjustment, and ensuring that robotic arm 4 can accurately and safely complete the material grasping.

[0136] The construction of the force-position hybrid control model includes:

[0137] Collect structural parameters, force and position sensor data of the robotic arm 4 and physical properties of materials. Record the correlation data under different grasping scenarios through experiments to form a labeled force and position control dataset and output it.

[0138] A dual-closed-loop control framework is constructed. The position loop calculates the position compensation amount using proportional, integral and derivative algorithms, while the force loop calculates the force compensation amount using an impedance control algorithm. A weight allocation module is designed to dynamically adjust the output ratio of the two loops according to the material characteristics, and the output is a control framework containing the dual closed loops and the weight module.

[0139] Input the dataset into the model with the goal of minimizing position deviation, force fluctuation and maximizing the grasping success rate. Iteratively optimize the position loop proportional coefficient, force loop impedance parameter and weight threshold, fine-tune for special scenarios, and output the optimized parameter set and preliminary model.

[0140] A real-time feedback module is added. After detecting material pose deviation or force value exceeding the limit, the weights of the position loop and force loop are automatically adjusted. The model receives inputs such as pose deviation through interface adaptation and outputs optimized grasping force parameters and position compensation instructions to form a model that can respond in real time.

[0141] The model is embedded into the control system of the robotic arm 4. The position accuracy and force control stability are verified through physical grasping experiments. When problems exist, the optimization parameters are returned, and the final output is a force-position hybrid control model that meets the requirements of accuracy and safety.

[0142] By adopting the above technical solution, the structural parameters of the robotic arm 4 are collected in advance, such as the length of each joint, joint stiffness, and end effector size; force feedback sensor data, such as real-time gripping force and torque changes during gripping; and position sensor data, such as the three-dimensional coordinates of the end effector and the angles of each joint. Simultaneously, the physical properties of the target material are recorded, such as weight, surface friction coefficient, and force thresholds of vulnerable parts. Then, a large number of gripping experiments are conducted to simulate different scenarios: for example, the force value required to compensate for a 1 mm positional deviation when gripping lightweight materials, the adjustment effect when the force value exceeds the threshold when gripping vulnerable materials, and the correlation between position deviation and force deviation when gripping rigid materials. After labeling these correspondences of position deviation, force deviation, and adjustment effect, they are integrated into a labeled force-position control dataset, providing a data foundation for model training.

[0143] With the goals of precise positioning and stable force control, a dual-closed-loop control framework is constructed: the position loop is responsible for correcting spatial position deviations, receiving the difference between the real-time position of the end effector and the target position, and calculating the position compensation amount through proportional, integral, and differential algorithms. For example, if the position is 2 mm to the left, an instruction to adjust 2 mm to the right is output to ensure that the end effector can accurately align with the material. The force loop is responsible for stabilizing the clamping force, receiving the difference between the real-time force value of the force sensor and the target force value, and calculating the force compensation amount using an impedance control algorithm. For example, if the current force is 3 N less than the target, an instruction to increase it by 3 N is output to avoid sudden changes in force value that could damage or slip off the material. A weight allocation module is also designed to dynamically adjust the output ratio of the two loops according to the material characteristics. For example, when gripping rigid materials, the weight of the position loop is increased to prioritize alignment accuracy; when gripping fragile materials, the weight of the force loop is increased to prioritize limiting the force value. This ultimately forms a control framework that includes dual closed loops and a weighting module.

[0144] The parameters are iteratively adjusted with three objectives: minimizing positional deviation (e.g., within 0.5 mm); minimizing force fluctuation (e.g., within ±1 N); and maximizing the grasping success rate (e.g., above 99%). The proportional coefficient of the position loop is adjusted to affect the response speed of position adjustment; an excessively large coefficient will cause oscillation, while an excessively small coefficient will result in sluggish adjustment. The impedance parameter of the force loop affects the smoothness of force value changes; inappropriate parameters will cause the force value to fluctuate wildly. The threshold for weight allocation is also adjusted; for example, the force control weight threshold for fragile materials is set to 70%. For special scenarios, such as increasing friction when materials slip or reducing clamping force to prevent collisions when approaching the worktable, corresponding samples are extracted for parameter fine-tuning. Finally, the optimized parameter set and preliminary model are output.

[0145] To enable the model to respond to changes during the grasping process in real time, a real-time feedback module is added: when it receives material pose offset information, such as a 3 mm position deviation, the module automatically increases the position loop weight to accelerate position correction; when it detects that the force value exceeds the safe range, such as the force on fragile materials reaching 110% of the threshold, the module prioritizes increasing the force loop weight to forcibly limit the force value increase. Simultaneously, through interface adaptation, the model can receive previously output pose offset, initial adjustment direction, and other data, directly outputting optimized grasping force parameters and position compensation commands, such as a 5-degree rotation plus a 1 mm translation, forming a model that can respond to dynamic changes in real time.

[0146] The scheduling module also includes taking images of the material plate 6 by the camera 5 at the gripping end of the robotic arm 4, identifying whether there is material in the material trough 7 and counting the remaining quantity, and outputting an exhaustion signal and the current position information of the first workbench 2 when the material is exhausted.

[0147] Upon receiving the material depletion signal and position information, the system controls the cylinder to push the first worktable 2 away from the second worktable 3 to the preset clamping position. After the switching is completed, the system outputs the activation signal of the second worktable 3 and the arrival confirmation information.

[0148] Based on the activation signal of the second workbench 3, the material status of the two workbench is continuously collected by the camera 5, and historical data and current information are integrated to form a structured dataset containing image detection data and time series features.

[0149] Based on this dataset, we divide it into training and validation sets, design a model with a time series processing layer, train and adjust the parameters until the prediction error reaches the target, and output the trained model.

[0150] Input the real-time material data of the second workbench 3 detected by camera 5 into the model, combine historical trends and current gripping frequency, predict the material consumption trend and expected depletion time for the future preset period, and output the prediction results.

[0151] Receive the prediction results, compare the remaining consumption time with the preset time window, and if the conditions are met, generate a pre-notification trigger signal and the predicted exhaustion time point;

[0152] The system uses audible and visual alarms or terminals to prompt workers to replace the material board 6 on the first workbench 2 in a timely manner, and provides feedback on the status to ensure that the board replacement is completed before the material on the second workbench 3 is exhausted.

[0153] By adopting the above technical solution, it can be seen that the camera 5 at the gripping end of the robotic arm 4 undertakes the task of real-time monitoring. If the first worktable 2 is close to the robotic arm 4 at this time, the camera 5 continuously captures images of the material plate 6 on the first worktable 2. Image recognition technology is used to determine whether there is material in each material slot 7. For example, by detecting whether there are features in the material slot 7 area in the image that match the outline of the material, such as a specific shape or color block, the remaining material quantity can be counted. When the camera 5 detects that there are no material features in all material slots 7, that is, the material is exhausted, the system will generate a material exhaustion signal, and at the same time record and output the current spatial position information of the first worktable 2.

[0154] After receiving the material depletion signal and the position information of the first workbench 2, the system activates the workbench switching mechanism, driving the corresponding cylinders. One set of cylinders pushes the first workbench 2 along the slide rail away from the working area of ​​the robotic arm 4 until it reaches the standby position. Simultaneously, another set of cylinders unlocks the fixing device of the second workbench 3, pushing it along the slide rail to the preset gripping position, aligning it with the working range of the robotic arm 4. When the second workbench 3 reaches the designated position, the system outputs an activation signal for the second workbench 3, indicating that it has entered the working state, providing a basis for subsequent material gripping.

[0155] Based on the activation signal of the second workbench 3, the camera 5 begins to synchronously collect material status data from both workbenches. For the first workbench 2, it records its current empty material status and historical gripping records, such as the total number of materials on the material plate 6 in a single gripping session and the consumption time. For the second workbench 3, it captures the changes in the amount of material before and after each gripping session in real time, identifies the reduced material slots 7 through image comparison, and marks the timestamp of each gripping session. Simultaneously, it integrates historical data, such as gripping intervals at different times and the consumption rate corresponding to different material types, as well as current information, such as the initial material quantity on the second workbench 2 and the real-time gripping frequency of the robotic arm 4. This data is structured according to the correlation between time, material quantity, and gripping frequency, forming a dataset containing image detection results and temporal features, providing a training basis for the prediction model.

[0156] Subsequently, a material consumption prediction model was constructed using this dataset. The dataset was proportionally divided into training and validation sets, and a model architecture including a time series processing layer was designed. This layer can capture periodic patterns in the data, such as the difference in sampling frequency during morning and evening peak hours. The model learns historical consumption trends through the training set, such as consuming 10 units of a certain type of material per hour. The model parameters are repeatedly adjusted using the validation set, such as optimizing the sensitivity to periodic features, until the prediction error is lower than a preset standard, such as the difference between the actual consumption and the predicted consumption. Finally, the trained prediction model is output.

[0157] After the model is put into use, camera 5 monitors the remaining material quantity of the second workbench 3 in real time, identifies the number of material troughs with material in the image, and inputs this real-time data into the prediction model. The model combines historical consumption trends and current gripping frequency to calculate the material consumption trend for a future preset period, such as the curve of the remaining material decreasing over time, and calculates the expected depletion time of the second workbench 3, outputting specific prediction results.

[0158] Based on the prediction results, a pre-notification trigger judgment is made. The difference between the estimated exhaustion time and the current time is compared with a preset time window: if the remaining time is less than the window value, such as the predicted exhaustion time being 10 minutes, a pre-notification trigger signal is generated, along with the specific predicted exhaustion time, such as 10:30, to ensure that workers have sufficient preparation time.

[0159] Finally, the notification module responds to the trigger signal, issuing a prompt sound and flashing lights through the workshop's audible and visual alarms. Simultaneously, a text notification is displayed on the worker's terminal, such as the requirement to replace material plate 6 on the first workbench before 10:30 AM. The system also simultaneously sends a notification status update to the control center, ensuring management is aware of the progress. Upon receiving the notification, workers replace material plate 6 on the first workbench 2 before the material on the second workbench 3 runs out. This ensures the switching mechanism can immediately activate the newly replenished first workbench 2 when the second workbench 3 is empty, guaranteeing an uninterrupted material supply process.

[0160] The mapping module also includes scanning all objects in the workspace from all angles using a scanning device, collecting point cloud data of the region containing three-dimensional coordinates from multiple angles along a preset path, and outputting multiple sets of raw point cloud data with viewpoint markings.

[0161] Receive raw point cloud data, align multi-view data through feature matching, remove noise and redundant information, merge into a complete point cloud dataset under a unified coordinate system, and output a preprocessed workspace point cloud dataset.

[0162] Based on the preprocessed point cloud dataset, a global point cloud model containing the three-dimensional shape of all objects in the workspace is generated through surface reconstruction, the point cloud clusters of each key component are clearly marked, and the global point cloud model is output.

[0163] Identify fixed feature points of the workbench from the global point cloud model, determine their original three-dimensional coordinates in the global coordinate system through point cloud analysis, and output the original coordinate set of each workbench feature point;

[0164] After the workbench position is changed, the image coordinates of the same feature point are captured and extracted by camera 5. Combined with the global point cloud scale, the coordinates are converted into new three-dimensional coordinates in the global coordinate system, and the new coordinate set of the feature point is output.

[0165] The original coordinate set and the new coordinate set of feature points are input into the visual calibration algorithm to calculate the translation and rotation parameters after the change of the workbench position. These parameters are then integrated into a coordinate system transformation relationship, and the output is used for subsequent path planning and pose calculation.

[0166] By adopting the above technical solution, the workspace is collected in all directions in advance by a laser scanner. The scanning head moves along a preset path to photograph all objects in the workspace from different angles, including the first workbench 2, the second workbench 3, the robotic arm 4, the material board 6, and surrounding equipment. The three-dimensional coordinate information of the surface of each object is captured, such as the X, Y, and Z axis distances of a certain point from the scanning origin. Finally, multiple sets of raw point cloud data with viewpoint markings are output. Each set of data corresponds to a scanning angle and marks the position information of that angle.

[0167] Next, the raw point cloud data is preprocessed and stitched together. After receiving multiple sets of raw point clouds, the data from different perspectives are aligned using feature matching technology: common features of the same objects in each perspective are identified, such as the right-angled edges of the workbench and the circular marks of the robotic arm joints. The spatial positions of each set of data are adjusted based on these features to ensure that they are aligned under the same reference frame. At the same time, noise points and redundant data are filtered out, and the data from all perspectives are merged into a complete point cloud dataset under a unified coordinate system. The preprocessed workspace point cloud dataset is output, providing a clean and coherent data foundation for subsequent modeling.

[0168] Based on the preprocessed point cloud dataset, algorithms are used to construct the continuous topological structure of object surfaces, such as connecting discrete point clouds into surfaces to reconstruct the three-dimensional shape of objects, generating a global point cloud model containing the three-dimensional shapes of all objects within the workspace. The model uses clustering and labeling techniques to distinguish the point clouds of different key components; for example, different colors or labels are used to mark the point cloud clusters of the first workbench 2, the second workbench 3, and the robotic arm 4 base, clearly presenting the spatial distribution of each object. The output global point cloud model serves as a three-dimensional map of the entire workspace.

[0169] Subsequently, the original coordinates of feature points of the worktables are extracted from the global point cloud model, such as the first worktable 2 and the second worktable 3. Fixed feature points on their surfaces are identified. These points are unique and immutable markers of the worktables, such as the vertices of the worktable corners and the center of the circular calibration targets mounted on the surface. Through point cloud analysis techniques, such as calculating the geometric center and edge intersections of point cloud clusters, the three-dimensional coordinates of these feature points in the global coordinate system are determined. For example, if a feature point is located at X=100cm, Y=50cm, and Z=80cm from the origin, the original coordinate set of each worktable feature point is formed, serving as a benchmark for subsequent position comparison.

[0170] When the worktable changes position due to switching or adjustment, such as the first worktable 2 moving to the standby position and the second worktable 3 moving to the gripping position, a new coordinate acquisition is initiated. The camera 5 at the gripping end of the robotic arm 4 captures the surface of the worktable after the movement, identifies and extracts the same fixed feature points as before, like a calibration target, and obtains the two-dimensional coordinates of these points in the image, such as pixel positions. Combining the scale information of the global point cloud model, such as how many pixels 1cm in the point cloud model corresponds to in the image, the image coordinates are converted into new three-dimensional coordinates in the global coordinate system, and the new coordinate set of feature points is output, recording the position of the worktable after the movement.

[0171] Finally, a visual calibration algorithm is used to calculate the coordinate system transformation relationship. The original and new coordinate sets of the feature points are input into the algorithm. By comparing the positional differences of corresponding feature points, the algorithm calculates the translation parameters after the worktable position change, such as a 30cm movement along the X-axis and a 5cm movement along the Y-axis. Simultaneously, it calculates the rotation parameters, such as a 5-degree rotation around the Z-axis. These parameters are then integrated into a complete coordinate system transformation relationship, i.e., the transformation rule from the original position to the new position. This transformation relationship, once output, can be directly used for path planning and pose calculation of the robotic arm 4. For example, it can adjust the motion trajectory and correct the coordinates of the material relative to the robotic arm based on the new position of the worktable, ensuring that the robotic arm can still accurately locate the target after the worktable moves.

[0172] The optimization module also includes collecting operational data such as the movement of the robotic arm 4, joint forces, workbench switching, and gripping success rate through a sensor network. After cleaning and labeling, the data is classified by type and time sequence, and training and validation sets are divided to output a structured system operation status training dataset.

[0173] The training dataset is input into a virtual simulation environment built according to the entity scale. A reinforcement learning strategy is deployed. A reward function is designed with the goal of improving efficiency, reducing energy consumption, and reducing collisions. The simulation system is driven to iteratively adjust the control parameters. The optimal parameter combination is evaluated and retained in combination with the validation set, and the optimized control parameter set is output.

[0174] The optimized parameters are imported into the physical control system to replace the original parameters. The system is then tested and the new running data is recorded. The results are compared with the simulation expectations. If there is a deviation, the data is supplemented and the system is returned to the first iteration. Once the target is met, the upgraded stable physical control system is output, and the closed-loop upgrade is completed.

[0175] By adopting the above technical solution, firstly, a system operation status training dataset is constructed, and a sensor network is deployed, such as encoders for the four joints of the robotic arm, position sensors for the worktable, force sensors for the gripping end, and cameras 5 monitoring the gripping results. Multi-dimensional operation data is collected in real time, including the motion trajectory of each joint of the robotic arm, such as the rotation angle and speed of each joint; the force conditions on the joints, such as the load size and whether there is overload; the time for worktable switching, such as the time taken to switch from the first worktable 2 to the second worktable 3; and the material gripping success rate, such as the ratio of successful gripping to the total number of attempts. After collection, the data is processed. The cleaning process removes outliers and fills in missing items; the labeling process associates parameters with operational effects, such as labeling gripping failures as negative samples of the corresponding parameter combinations and labeling rapid, collision-free worktable switching as positive samples. Subsequently, the data is categorized by data type, such as motion, force control, and time series, and further divided by hour or shift, into training and validation sets in a 7:3 ratio. Finally, a structured system operation status training dataset is output, providing a data foundation for subsequent parameter optimization.

[0176] Next, the control parameters are optimized in a virtual simulation environment. The output training dataset is input into the virtual simulation environment, which is built at a 1:1 scale to the physical system. It includes a robotic arm 4, a first workbench 2 and a second workbench 3, a 3D model of the material, and modules simulating physical properties, such as the friction coefficient of the robotic arm 4 joints, the inertia of the workbench movement, and the weight and hardness of the material, thus replicating the operating scenario of the physical system. A reinforcement learning strategy is deployed in the environment. The core of this strategy is to design a reward function that matches the target: if the robotic arm completes the gripping in a time shorter than the historical average, the joint energy consumption is lower than a set threshold, and there is no collision with the workbench during the movement, a positive reward is given; otherwise, a negative penalty is given. Driven by training data, the simulation system repeatedly simulates the operation process under different parameter combinations, such as adjusting the joint movement speed of the robotic arm, the magnitude of the gripping force, and the trigger threshold for worktable switching. It iterative optimization through trial and error is achieved using a reinforcement learning strategy: after each simulation, the parameter effectiveness is evaluated based on the reward function score, high-scoring parameter combinations are retained and further fine-tuned, and the generalization ability of the parameters is verified using a validation set, such as whether they remain effective in new scenarios not involved in the training. Finally, the parameter combination with the highest reward value is selected, and the optimized set of control parameters is output, such as the optimal range of joint movement speed, gripping force threshold, and trigger conditions for worktable switching. The virtual environment avoids the risks of directly testing parameters in a physical system, such as collision damage to the equipment, while efficiently completing thousands of iterative tests.

[0177] The final output of the optimized control parameter set is imported into the parameter configuration module according to the interface protocol of the physical control system, replacing the original default parameters. For example, the default speed of the four joints of the robotic arm is adjusted from 50° / s to the optimized 65° / s. The physical system is started for trial operation, and the operation data under the new parameters is recorded through a real-time monitoring module (such as camera 5): including whether the gripping success rate has improved, whether the average gripping time has shortened, whether the joint energy consumption has decreased, and whether the workbench switching is smoother. The trial operation data is compared with the expected effect in the virtual simulation. For example, if the simulation predicts a 10% increase in gripping success rate, and there is a deviation, such as the success rate in the physical system only increasing by 3%, which is lower than the simulation expectation, the newly collected trial operation data is added to the training dataset in the first step, and the system is returned to the first step to clean and label it again, and then input into the virtual environment to iteratively optimize the parameters. If the trial operation effect is consistent with the simulation expectation, such as the success rate, energy consumption, and collision rate all meeting the standards, the parameters are confirmed to be effective, and the upgraded physical control system is output. At this time, the system has better operating performance, and the closed-loop upgrade is completed.

[0178] The above are merely preferred embodiments of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concept, should be covered within the scope of protection of the present invention.

Claims

1. A collaborative robot target recognition and grasping posture planning system, comprising a cabinet (1), characterized in that: The cabinet (1) is slidably provided with a first workbench (2) and a second workbench (3). The second workbench (3) can slide through from under the first workbench (2). Material plates (6) are installed on both the first workbench (2) and the second workbench (3). The upper surface of the material plate (6) is provided with uniformly distributed material grooves (7). The cabinet (1) is also equipped with a robotic arm (4). The robotic arm (4) is used to grip the material on the material plate (6). A camera (5) is installed at the gripping end of the robotic arm (4) to adjust the gripping posture of the robotic arm (4) according to the data collected by the camera (5). This also includes: The perception module is used to acquire material images through deep vision; output the material's six-degree-of-freedom pose based on a deep learning model; and trigger a spin compensation mechanism to adjust the end effector pose when visual occlusion exceeds the threshold. The planning module is used to construct a state space by integrating material pose, workbench position, and robotic arm state; it uses reinforcement learning to generate collision-free trajectories and optimizes the path with a multi-objective reward function; and it avoids moving parts of the workbench through dynamic obstacle prediction. The control module is used to match the optimal grasping posture based on the trajectory and material characteristics; triggers visual and force closed-loop adjustments when the material deviates; and dynamically optimizes the grasping parameters using a force-position hybrid model. The scheduling module is used to switch to the second workbench (3) when the material on the first workbench (2) is exhausted; it also triggers a material replacement pre-notification in a preset window based on time-series prediction, and specifically includes: The robotic arm (4) grips the camera (5) to capture the image of the material plate (6), identifies whether there is material in the material tank (7) and counts the remaining quantity. When the material is exhausted, it outputs an exhaustion signal and the current position information of the first workbench (2). Receive material depletion signal and position information, control the cylinder to push the first worktable (2) away and the second worktable (3) to the preset clamping position, and output the second worktable (3) activation signal and position confirmation information after the switch is completed; Based on the activation signal of the second workbench (3), the material status of the two workbench is continuously collected by the camera (5), and historical data and current information are integrated to form a structured dataset containing image detection data and time series features; Based on this dataset, we divide it into training and validation sets, design a model with a time series processing layer, train and adjust the parameters until the prediction error reaches the target, and output the trained model. Input the real-time material data of the second workbench (3) detected by the camera (5) into the model, combine the historical trend and the current gripping frequency, predict the material consumption trend and the expected depletion time in the future preset period, and output the prediction result; Receive the prediction results, compare the remaining consumption time with the preset time window, and if the conditions are met, generate a pre-notification trigger signal and the predicted exhaustion time point; The sound and light alarm or terminal prompts the workers to replace the material board (6) on the first workbench (2) in time, and the notification status is fed back to ensure that the board replacement is completed before the material on the second workbench (3) is exhausted. The mapping module is used to reconstruct the workspace model through global scanning and solve the coordinate system transformation relationship after the workbench is changed. The optimization module is used to collect runtime data to build a training set; optimize control parameters in a virtual environment and inject them into the system in a closed loop.

2. The collaborative robot target recognition and grasping posture planning system according to claim 1, characterized in that, The perception module also includes: The camera (5) captures images of the target material, collects and outputs original images containing its three-dimensional structure, surface texture and surrounding environment; The pose estimation model receives the original image, processes it through a rotation-invariant feature extraction layer, focuses on the visible area, filters out occlusion interference, extracts stable features that are not affected by rotation, and outputs the material's six-degree-of-freedom pose information and visual occlusion assessment results. Compare the occlusion assessment results with the preset threshold. If the threshold is exceeded, the end posture adaptive adjustment mechanism of the robotic arm (4) is triggered. This mechanism combines pose information, occlusion assessment and kinematic constraints, and calculates the optimal pose compensation amount for the camera (5) to avoid occlusion through the screw motion model. The robotic arm (4) adjusts the posture of the end gripper according to the optimal pose compensation amount, so that the camera (5) avoids the obstruction and re-aligns with the material; after adjustment, the camera (5) collects images again, and repeats the above process until the obstruction level is lower than the threshold, so as to ensure that the robotic arm (4) accurately obtains material information to complete the gripping.

3. The collaborative robot target recognition and grasping posture planning system according to claim 2, characterized in that, The planning module also includes: Data on material pose, workbench dynamic position and robotic arm (4) joint status are collected and, after time synchronization and format unification, a standardized multi-dimensional fusion dataset is formed. Based on the fusion dataset, the core dimensions and motion boundaries of the robotic arm (4), workbench, and materials are determined and integrated into a structured model describing the overall state. Based on the real-time position and physical dimensions of the workbench in the planned state space, predict its space occupancy area in the future period and form a dynamic obstacle spatiotemporal map; Combining path efficiency, energy consumption indicators, and safety margins, these are integrated into a comprehensive reward function based on weights, which serves as the trajectory optimization objective. Using the planning state space as the environment, dynamic obstacle prediction as the constraint, and the reward function as the objective, the algorithm generates the optimal collision-free motion trajectory. Verify whether the trajectory conforms to the joint limits of the robotic arm (4) and the dynamics of the worktable. Feedback and adjustment are provided when there is a conflict. Finally, the trajectory that conforms to the actual constraints is output.

4. The collaborative robot target recognition and grasping posture planning system according to claim 3, characterized in that, The control module also includes: Combining the motion trajectory parameters of the robotic arm (4) with the physical characteristics of the target material, inputting the preset gripping strategy library, filtering out the optimal gripping point, gripping angle and initial gripping force range through the matching algorithm, and outputting the specific optimal gripping posture parameters; The robotic arm (4) performs gripping in the optimal gripping posture, and at the same time collects the material posture in real time through the camera (5), compares the real-time posture with the expected posture to calculate the offset. If it exceeds the threshold, it triggers closed-loop force control adjustment and outputs the posture offset and the initial direction of force control adjustment. Receive the pose offset and initial adjustment direction, combine the real-time position of the end of the robotic arm (4), enable the force-position hybrid control model, synchronously correct the position deviation and dynamically adjust the gripping force parameters, and output the optimized gripping force parameters; Collect the trajectory of the robotic arm (4), the real-time status and physical dimensions of the workbench, establish a kinematic model to predict the three-dimensional occupied area of ​​the workbench within a preset time period in the future, output a map of the occupied area with time series, and use it as a constraint feedback to the first three steps to ensure obstacle avoidance during the grasping process.

5. The collaborative robot target recognition and grasping posture planning system according to claim 4, characterized in that, The mapping module also includes: The scanning device performs a full-range scan of all objects in the workspace, collects regional point cloud data containing three-dimensional coordinates from multiple angles along a preset path, and outputs multiple sets of raw point cloud data with viewpoint markings. Receive raw point cloud data, align multi-view data through feature matching, remove noise and redundant information, merge into a complete point cloud dataset under a unified coordinate system, and output a preprocessed workspace point cloud dataset. Based on the preprocessed point cloud dataset, a global point cloud model containing the three-dimensional shape of all objects in the workspace is generated through surface reconstruction, the point cloud clusters of each key component are clearly marked, and the global point cloud model is output. Identify fixed feature points of the workbench from the global point cloud model, determine their original three-dimensional coordinates in the global coordinate system through point cloud analysis, and output the original coordinate set of each workbench feature point; After the workbench position is changed, the image coordinates of the same feature point are captured by the camera (5), and combined with the global point cloud scale to convert to a new three-dimensional coordinate in the global coordinate system, and the new coordinate set of the feature point is output. The original coordinate set and the new coordinate set of feature points are input into the visual calibration algorithm to calculate the translation and rotation parameters after the change of the workbench position. These parameters are then integrated into a coordinate system transformation relationship, and the output is used for subsequent path planning and pose calculation.

6. The collaborative robot target recognition and grasping posture planning system according to claim 5, characterized in that, The optimization module also includes: The robot arm (4) motion, joint force, workbench switching, and gripping success rate are collected through a sensor network. After cleaning and labeling, the data are classified by type and time sequence, and the training set and validation set are divided. The structured system operation status training dataset is output. The training dataset is input into a virtual simulation environment built according to the entity scale. A reinforcement learning strategy is deployed. A reward function is designed with the goal of improving efficiency, reducing energy consumption, and reducing collisions. The simulation system is driven to iteratively adjust the control parameters. The optimal parameter combination is evaluated and retained in combination with the validation set, and the optimized control parameter set is output. The optimized parameters are imported into the physical control system to replace the original parameters. The system is then tested and the new running data is recorded. The results are compared with the simulation expectations. If there is a deviation, the data is supplemented and the system is returned to the first iteration. Once the target is met, the upgraded stable physical control system is output, and the closed-loop upgrade is completed.

7. The collaborative robot target recognition and grasping posture planning system according to claim 6, characterized in that, The perception module also includes: Images of the target material under different lighting, rotation, occlusion and workbench positions are collected by camera (5). The actual six-degree-of-freedom pose is recorded by high-precision equipment. The degree of occlusion is manually labeled to form a labeled dataset of original images, poses and occlusion degree association and output. Based on a convolutional neural network, an image enhancement module is added to the front end, a rotation-invariant feature extraction layer is designed, and the back end is divided into two branches to output six-degree-of-freedom pose parameters and occlusion evaluation results, and outputs the network structure containing the extraction layer. The dataset is input into the network, trained using a joint loss function, and the parameters are iteratively adjusted until the validation set metrics meet the requirements. Fine-tuning is performed on occluded samples to enhance feature extraction capabilities, and the preliminarily trained pose estimation model is output. The compressed model meets the real-time requirements. Through interface adaptation, the model receives image input from the camera (5), outputs standardized pose information and occlusion evaluation results, and outputs a deployable pose estimation model.

8. The collaborative robot target recognition and grasping posture planning system according to claim 7, characterized in that, The perception module also includes: Collect the physical parameters of the joint length and range of motion of the robotic arm (4), and the installation coordinate offset of the camera (5) relative to the end effector, form a set of basic kinematic parameters and output them; Based on the parameters, the mapping relationship between the end effector pose and the joint angle is established according to the screw theory, and the basic model of screw motion is output. Control the robotic arm (4) to move along a preset trajectory, record the actual end pose through the camera (5), compare it with the theoretical calculation value of the basic model and adjust the parameters to reduce the error, and output the calibrated spinor motion model; The logic is embedded in the model. After inputting the material pose and occlusion evaluation results, the difference between the current camera (5) pose and the unoccluded pose is analyzed, the optimal pose compensation amount to avoid occlusion is calculated, and the model that can be directly used for the posture adjustment of the robotic arm (4) is output.

9. A collaborative robot target recognition and grasping posture planning system according to claim 4, characterized in that, The control module also includes: Collect the structural parameters, force and position sensor data of the robotic arm (4) and the physical properties of the material. Record the associated data under different grasping scenarios through experiments to form a labeled force and position control dataset and output it. A dual-closed-loop control framework is constructed. The position loop calculates the position compensation amount using proportional, integral and derivative algorithms, while the force loop calculates the force compensation amount using an impedance control algorithm. A weight allocation module is designed to dynamically adjust the output ratio of the two loops according to the material characteristics, and the output is a control framework containing the dual closed loops and the weight module. Input the dataset into the model with the goal of minimizing position deviation, force fluctuation and maximizing grasping success rate. Iteratively optimize the position loop proportional coefficient, force loop impedance parameter and weight threshold. For materials that slip, increase the friction force and reduce the clamping force to prevent collision when they approach the worktable, extract the corresponding samples for parameter fine-tuning and output the optimized parameter set and preliminary model. A real-time feedback module is added. After detecting material pose deviation or force value exceeding the limit, the weights of the position loop and force loop are automatically adjusted. The input of pose deviation is received through interface adaptation, and the optimized grasping force parameters and position compensation instructions are output to form a model that can respond in real time. The model is embedded into the control system of the robotic arm (4). The position accuracy and force control stability are verified by physical grasping experiments. If there are problems, the optimization parameters are returned. Finally, the force-position hybrid control model that meets the requirements of accuracy and safety is output.

Images